Discussion about Frame Interpolation

I’m more concerned with frame interpolation than image enhancement because it’s still pretty immature. I hope to open a separate topic to follow up on the changes and problems of Video AI in frame interpolation, and communicate with you.
First, you should read Topaz Video AI v3.1.4 - #16 by q91632 , it’s the starting point of the topic.

I often need to use frame interpolation for rendering and production videos. Obviously, its requirements for frame interpolation are much higher than those of single-scene live video, and there is currently no ideal solution.

As mentioned above, Chronos Fast and Apollo also have many problems at present. Chronos Fast is completely powerless for slightly large changes, and the results for small changes have also dropped compared to the older version. Apollo hopes to do better than it, but it still has a lot of problems, such as reducing the quality of the input, blocks will appear in the transition, the original frame will not be preserved, and so on.

I understand these aren’t going to be fixed any time soon, and this thread was created to focus on that. I hope that improving the model results is one of the highest priorities for current development, while adding transition detection as soon as possible to improve quality and resolve issues. If you have any views, welcome to discuss.

what about Chronos (not fast)? how does that perform?

Interpolation in v2.6.3 doesn’t cope well with rotations and repeating patterns. Quite severe wobble is introduced under these conditions. Here is Topaz up against DAIN.


Chronos has not changed much from the old version, and if it is used to deal with small changes, it is no problem to use it.

So in V3, has this problem been solved?

In v3.1.5, Apollo will change the color tone of the picture.

On 3.1.9, apollo still has no progress on the above issues, it still casts colors and generates blocks, and even blocks appear more often and more frequently now.

Looks like they’ve given up though, from the new roadmap:

We’re now quite satisfied with the quality of Topaz Video AI’s frame interpolation, so will now refocus on improving our general enhancement models.

You may be whistling in the wind.

If you only talk about the quality of frame interpolation, there is indeed a lot of improvement. However, whether it is color shift or squares, it should be regarded as a bug. As long as it is not resolved, apollo will always be a toy rather than a tool.
Given that there are more blocks now, users won’t accept it, it needs to be fixed.

I have not checked to see if they’ve fixed the top-down spinning clip mentioned above, but all the other issues with Apollo mentioned in here are unique to 4k.
The color shifting is a problem on all models of TVAI, that can be bypassed through a lot of mind-numbing study and using the CLI. I’ve figured out how to do DVD to FHD correctly, but randomly I’ll find FHD movies that color shift if I try to use TVAI on them.

Since I don’t own anything that would benefit form 4k (House space, TV screen that’s close enough and big enough in relation to the seating), I find Apollo and Chronos interpolation models to be the best options I have tried. Very much not a toy.
Maybe my opinion comes from the other things I have tried: SVP and FlowFrames. SVP couldn’t handle things like panning side to side. The edges of the screen would jump around nastily. And that’s just one of the many artifacts that would pop-out at you as you used it. My TV screen has something similar built-in to it. Pretty distracting as well. FlowFrames, I remember liking it at first, but it didn’t interpolate a specific scene better than TVAI and was more of a hassle to work with. Maybe they’ve improved.

If I recall correctly, Apollo5 has no such color shift.

The blocks do appear at 4K, but it’s kind of funny to screw up at 4K considering Video AI claims to support 8K. Color shifting and squares will be the reason some people reject it, not many people would like to have “degradation” while enhancing the video, especially Apollo is a models that does not retain the original frame.

Scene detection should be part of Apollo, only in this way can it produce more perfect results.

Agreed. It would help a lot.

Another thing I want to bring up is that the Replace Duplicate Frames option does not work.
Most NTSC DVD sources originally from film, to accommodate the specification, add a duplicate frame every fifth frame. No matter what value ‘Replace Duplicate Frames’ is set to, the duplicate frames are counted as important motion—so much so that they make the motion regress a tiny bit.

I hope they keep improving all of their models. The pattern so far looks like they work really hard on one model, then move on and not look back.

Hello, I tried to smoothen camera pans with “Frame Interpolation”.
My tested footage has 25fps.
So far I was not successful in smoothen camera pans. I recognized, that the camera pans a more jerky than the original footage.
Do you have an idea, how the model settings should be? Or does Frame Interpolation don’t work like e.g. Optical Flow?

If there are no duplicate frames in the source, try unchecking ‘Replace Duplicate Frames’. If there are duplicate frames, you’re going to have to use something like the decimate filter in ffmpeg to correct that before passing it through TVAI.

From my testing, Apollo is the best model all around, but the motion dies with duplicate frames—no matter what you have the ‘Remove Duplicate Frames’ slider set to.

Maybe I just have jerky or poor quality video, but almost every time I’ve tried frame interpolation, the intermediate frames look bad.

I’ve only ever use it to increase the frame rate. Usually if the in-between frames don’t look amazing while frozen, they are unnoticeable when watching the movie at normal speed.

I tested now, to increase my old 25 fps footage to 50 fps and using frame interpolation Apollo (with the suggested value 10 under (replace duplicate frames).
The results are great and very impressive. Unbelievable.
The whole old videos looks now very smooth. All camera zooms and pans are now very smooth.
My conclusion in my case of old VHS videos is, that frame interpolation only works by increase/double the frame rate.
By using in addtion also the Artemis Strong Halo model the old (bad) VHS footage looks impressive good.
We saw yesterday evening some examples in my homecinema via beamer. We couldn’t believe how good now our old footage looks.

1 Like

What I noticed:
If I try to convert 25fps to 60fps then the motion is not really smooth…
The Interpolation is not „constant“, it’s more randomly add some Judder to it :rage: