How does Topaz Upscale feature work?

Topaz Video General
1

Hey,

I’m developing a new software for editing videos (personal initiative, nothing profesionnal) and would like to integrate Topaz Video inside to ease my workflow. Basically, during export, the user has the possibility to choose adding Topaz features (enhancement, interpolation, …) to the pipeline if he has a valid license.

To do that, I saw Topaz Video is based on ffmpeg filters, tvai-up (upscale) and tvai-fi (frame interpolation).

tvai-up takes several parameters (model, model params and target resolution), but then the original ffmpeg scale filter is also used to “upscale” to the desired resolution. In other words, it looks like the enhancement feature is:

  • firstly improving the video stream, leaving the resolution as its original size
  • then upscale from original size to the new desired size.e

And I said it “looks like”. Because by testing this, I first get the original resolution size between tvai-up filter and scale filter, and the desired resolution after the scale filter, but I do have interpolated pixels in the final result, not “duplicated pixels” as we have using only the ffmpeg scale filter. Nevertheless, comparing the Topaz scale filter and original ffmpeg scale filter, I can’t see any difference between the two. And using tvai-up filter without the scale filter just doesn’t work. So… I don’t really understand how this is working, technically speaking, at ffmpeg level. I’m not interested in the tvai-up filter implementation, I totally understand that’s Topaz property. But I’m just wondering how we can have the same input/output size in tvai-up filter, then do an upscale using what looks like to be the original ffmpeg scale filter, and get interpolated pixels calculated in the previous filter :sweat_smile:

Thanks!

2

How are you defining scale within the tvai_up filter?
I do something like this for DVD to FHD:
scale=2.25
'tvai_up=model=' + model + ':scale=' + scale + ':w=1920:h=1080…
As far as I know, this activates the 2X upscale, so the output will be less than 1920 by 1080, but I account for that later with the ffmpeg scale filter.

3

Ooooh, you mean the tvai_up will apply a 2.25 scale to the input, targeting something close to your specified wxh you have to ensure with a ffmpeg scale filter to “downscale” to the desired value?! Mmm, ok, I’m letting this scale param to 1.

Actually, I launched Topaz upscaling from 7680x3840 to 8192x4096 to check the role of each params, and it gaves me a scale=1. but it should be 1.07, then ffmpeg scale to ensure 8192x4096, right ? Ok, I will tune that, thanks for this !

4

I’m pretty sure that the AI models have pixel scaling factors that they run on: 1X, 2X and 4X. If you say scale=1.0 it will always pick the 1X variant of the model—and scale=1.07 will also only pick the 1X variant. I only use 2.25 because that’s how they generated the command originally when the CLI first came out, but it picks the 2X variant.
I don’t know if it favors scale over w and h. I know it uses both to pick what variant of the model it runs.