[Q] Does the tvai_up filter modify or preserve the color primaries and trc?

The frame-level color characteristics of tvai_up filter output appears to be:

  • pixel_format == rgb48le
  • color_range == pc # Even if the source is limited
  • color_space == gbr
  • color_primaries ~= Unknown # (Unless the frame has been explicitly pre-tagged with setparams filter)
  • color_trc ~= Unknown. # (Unless the frame has been explicitly pre-tagged with setparams filter)

I’m specifically interested as to whether the color_primaries and color_trc are in fact modified to RGB/sRGB conventions by tvai_up or are just passed through.

Here’s an example where the source which is tagged at stream level, but is Unknown[2] at frame level. On the output of tvai_up, the frames have Unknown[2] primaries and transfer characteristics. showinfo demonstrates that the frame-level metadata is passed through.

$ ffmpeg-topaz -loglevel '+level' -hide_banner -colorspace 'smpte170m' -color_primaries 'smpte170m' -color_trc 'smpte170m' -color_range 'tv' -f 'lavfi'  -i smptebars=size='ntsc':rate='ntsc',tvai_up=model='iris-1':scale=1,showinfo -frames:v 1 -f 'null' -

[Parsed_showinfo] [info] color_range:pc color_space:gbr color_primaries:unknown color_trc:unknown

Here’s an example where the source which is tagged at both stream and frame level. On the output of tvai_up, the frames retain the frame primaries and transfer characteristics. showinfo demonstrates that the frame-level metadata is passed through.

$ ffmpeg-topaz -loglevel '+level' -hide_banner -colorspace 'smpte170m' -color_primaries 'smpte170m' -color_trc 'smpte170m' -color_range 'tv' -f 'lavfi'  -i smptebars=size='ntsc':rate='ntsc',setparams=colorspace='smpte170m':color_primaries='smpte170m':color_trc='smpte170m',tvai_up=model='iris-1':scale=1,showinfo -frames:v 1 -f 'null' -

[Parsed_showinfo] [info] color_range:pc color_space:gbr color_primaries:smpte170m color_trc:smpte170m

Since tvai_up is synthetically creating RGB frames within the rgb48le / gbr domain, are the real color_primaries and transfer characteristics of the output frames always absolutely equal to that of the source, or are the frames coming out of tvai_up actually RGB linear, or the sRGB convention of:

Stream #0:0: Video: wrapped_avframe, rgb48le(pc, gbr/bt709/iec61966-2-1)

or are they always preserved like the following?

Stream #0:0: Video: wrapped_avframe, rgb48le(pc, gbr/[AS_SOURCE]/[AS_SOURCE])

My testing suggests that the frame metadata is always preserved from source, but I question whether this is indeed the case since the frames are now in the rgb48le/gbr color model.

And why is this important? The challenge is that if a user wants to perform subsequent conversion to something like compliant yuv422p(tv, bt709/bt709/bt709), FFmpeg’s colorspace filter always requires that the frames are explicitly tagged or overridden with all of { color_space, color_primaries, color_trc} in order to perform a successful color conversion to bt709/bt709/bt709, otherwise FFmpeg’s colorspace filter will throw an error if any of the three values in the color matrix are Unknown[2]. This command fails, unless the user explicitly tags the frame metadata with setparams to avoid the input frames being Unknown[2].

$ ffmpeg-topaz -loglevel '+level' -hide_banner -colorspace 'smpte170m' -color_primaries 'smpte170m' -color_trc 'smpte170m' -color_range 'tv' -f 'lavfi' -i smptebars=size='ntsc':rate='ntsc',tvai_up=model='iris-1':scale=1,showinfo,colorspace=all='bt709':format='yuv422p':range='tv' -frames:v 1 -f 'null' -
[Parsed_colorspace] [error] Unsupported input primaries 2 (unknown)

FFmpeg’s colorspace filter has the strict requirement that the primaries and transfer characteristics are explicitly tagged at frame level (or overridden when initiating the colorspace filter). I would appreciate clarification whether the primaries and transfer characteristics coming out of tvai_up actually follow RGB/sRGB conventions or whether the primaries and transfer characteristics always representative of, say, the original YUV input. I want ensure that the frames are tagged correctly to meet the constraints of FFmpeg’s colorspace filter when subsequently converting to compliant yuv422p(tv, bt709/bt709/bt709).

In the above examples, I’m interested is what is happening within the filterchain, analyzed using the the showinfo filter, since it is these frame-level tags that FFmpeg’s colorspace filter needs. I fully appreciate that in a full color-correct workflow that as well as accurate frame-tagging that -colorspace, -color_primaries and -color_trc should also be passed as output options to the codec and that the [experimental-only] -movflags +write_colr can also be used at container level. The question is pertinent to what is happening within the filterchain - if the frames are tagged correctly, the rest is trivial.

Would someone from the development team be kind enough to confirm whether during tvai_up processing, the primaries and transfer characteristics are modified to meet RGB/sRGB conventions or always preserved from the source? Thanks.


System info for reference: Mac M1, TVAI v3.3.9 (N-111537-g11e71d5e7e). Using binary from 3.3.9 installer, aliased to ffmpeg-topaz to avoid conflict with brew’s FFmpeg…

$ ffmpeg-topaz -version

ffmpeg version N-111537-g11e71d5e7e Copyright (c) 2000-2023 the FFmpeg developers
built with Apple clang version 14.0.0 (clang-1400.0.29.202)
configuration: --prefix=./builds-arm --extra-cflags='-I./conan_arm/include/videoai -I./conan_arm/include ' --extra-ldflags='-L./conan_arm/lib -headerpad_max_install_names ' --enable-tvai --enable-libvpx --enable-libaom --enable-openssl --disable-ffplay --enable-shared --disable-static --disable-asm --enable-neon --disable-libxcb --disable-sdl2 --disable-xlib --extra-cflags=-I./conan_arm/include --extra-ldflags=-L./conan_arm/lib
3 Likes

tvai_up always converts the colorspace to rgb48le. I hope that’s what you’re asking… and it’s probably not.
From what I have been able to deduce, nothing is kept from the source. There have been color shift complaints since TVAI 3 came out (And I have issues with it in VEAI 2). The only way I can get colors to make it correctly from start to finish is to explicitly convert them each step of the way.

1 Like

Yeah, the rgb48le color_space isn’t what I’m asking, that’s set in stone - but thanks for replying. It is more about the tagging accuracy of the primaries and the transfer characteristics after the conversion to rgb48le/gbr colorspace within tvai_up.

  • (bgr/unknown/unknown) is fatal for FFmpeg’s colorspace filter
  • (bgr/smpte170m/smpte170m) is an unusual combo
  • (bgr/bt470bg/bt470bg) is an unusual combo
  • (bgr/bt709/bt709) is an unusual combo
  • (bgr/bt709/iec61966-2-1) is far more usual for sRGB
  • and old-skool RGB is sometimes linear

But in order to feed the beast that is FFmpeg’s colorspace filter, it is required that primaries and trc are (1) never Unknown[2], which can easily happen with primaries and trc passthrough AND (2) ideally accurate.

I’m hoping that a rep from TopazLabs will be able to confirm whether tvai_up is always (bgr/bt709/iec61966-2-1) or whether primaries and trc are always pssthrough, not just for me but for others who wish to use FFmpeg’s subsequent colorspace filter without [fatal] errors in their workflow.

Nevertheless, I do appreciate the confirmation that the colorspace is always rgb48le/bgr.

2 Likes