Poor frame interpolation on scrolling text

Topaz Video AI General
1

The scene below was processed with Chronos, Apollo, and Aion. In all three cases, the text jumps and flickers. Later on, some of the words are also swapped and cut in half.

Chronos

Apollo

Aion

Chronos struggles the most. Apollo and Aion are about the same.

Here is the same scene with RIFE for reference.

The models have a difficult time with the amount of activity on the screen. Part of the problem seems to be their inability to identify text, which they can’t differentiate from the background sequence. I would more-or-less expect these issues with free software, but it’s disappointing to have the same problems on rather pricey paid software.
_
Posting some more keywords for people searching the forums or using search engines: cartoons, animation, television, closing credits

2

All of them look like you didn’t account for the DVD frame rate. What happens when you drop frames back down to the original before interpolating? Also ‘Duplicate Frames’ Replace must be unchecked. It doesn’t do what the name suggests. (At least on Apollo and Aion.)

Last time I tried FlowFrames, it was easily the worst of all the AI models I tried. But that was not with a cartoon. I wouldn’t be surprised if FlowFrames (RIFE) was made for cartoons.

3

are you sure that you deinterlaced the footage properly?

4

@DineshRepInt The rest of the video is fine, only the scrolling text in the credits has an issue. The problem is not interlacing.

@ForSerious I had never used FlowFrames before but I checked it out after reading your comment. It’s a nice GUI, looks like it has a lot of community support.

Two of the models in FlowFrames I was familiar with (RIFE and DAIN), but it has two others that I had never heard of – FLAVR and XVFI. Unfortunately, they both had issues with the scrolling text. XVFI was about as good as Apollo/Aion and FLAVR was on par with RIFE.

I got rid of FlowFrames and tried the CAIN algorithm, but it also struggled with the text (marginally better than Chronos). Then I tried IFRNet, which was again about as good as Apollo.

Finally, I decided to have a go with DAIN. I had always considered DAIN to be useless because it is so painstakingly slow – slower than Aion. In the past I had always canceled my DAIN transcodes before they finished because I did not think they were worth the extremely long wait. But, to my pleasant surprise, DAIN was the only model capable of successfully interpolating the scrolling credits with no jumping, flickering, ghosting, or other issues.

For anyone that would like to know how DAIN functions:

DAIN AI Frame Interpolation works with Stable Diffusion by analyzing video sequences’ complex motion trajectories and depth cues. Combining the depth-aware convolutional neural network with a spatial transformer, DAIN AI can generate accurate intermediate frames at a high-resolution level. This significantly reduces visual artifacts and ensures a smoother and more immersive viewing experience.

For the longest time I viewed frame interpolation as TVAI’s saving grace. It was the one thing I was confident TVAI could do better than anything else. If Topaz wants to TVAI to stay relevant, they absolutely need to focus on model development.

The worst-performing model in this test was TVAI’s Chronos, which I have never found to be useful. It produces poor visual results compared to Apollo and larger file sizes than Apollo. At this point, it’s essentially a legacy model, as are several other models in the program. I’m not sure why Topaz keeps them around.

5

Did you try my other suggestions?
I just want to know if that’s part of the issue or not.

About a week ago, I used Apollo to interpolate a TV episode. It seemed right for about 90% of the time. There were just enough things that seemed off, in some way or another, to leave me not wanting to interpolate the rest of the season.
Like, there wasn’t anything that I could solidly point out that looked wrong. I watch YouTube things that are recorded in 60 FPS all the time. They feel natural. My TV episode did not, in those 10% of random moments, and it made the whole watching experience feel uneasy.

From that, I agree that frame interpolation in TVAI is not what I want it to be. I also hope they can prefect it, but it is not there yet.

6

The frame rate of the source file was unchanged before interpolating. It was at the original DVD frame rate. As far as YouTube, if the video was recorded at 60+ fps, it’s generally always going to look better than an interpolated file. Interpolation isn’t quite at the point where it can make everything look 100% natural. But YouTube also encodes their best videos with VP9 and AV1, and that has a huge impact on quality as well.

One of the reasons I spend time restoring and enhancing videos on my own instead of relying on commercial restorations is that the work that can be done independently is usually a million times better than the commercial releases. Videos can be beautified to a level that is far beyond what commercial companies do. It feels like most companies just do the bare-minimum and most cost-effective level of restoration on older videos.

A few months ago, I learned that a show I had been working on had recently been re-released in 4k UHD. At first I was upset because it felt like I had wasted my time, but I decided to check it out. It turned out that the “remastered” re-release wasn’t even close to what I had been able to achieve on my own.

Along with video processing, I also restore the audio of the files. Between upscaling/enhancement, audio restoration, and interpolation (which TVAI usually does well, but, as I have just learned, there is still a lot of room for improvement), the videos typically turn out well. They’re not perfect, but it’s enough to make me want to keep going instead of relying on commercial restorations.

7

Good to hear. It’s rewarding to do better than the “professionals”.

The original DVD frame rate is a problem with TVAI. Apollo, especially, cannot calculate motion correctly with duplicate frames. Most commercial NTSC DVDs have videos that were filmed in ~24 fps but the DVD format requires ~30 FPS (Or PAL where it was still 24 FPS but is now 25 FPS—but that’s another issue). That means the DVD has a duplicate frame every 5 frames. I can see it clearly in your examples and so that’s why I asked if you had accounted for it.
I renecode DVDs before putting them through TVAI. I use the ffmpeg -r (rate) option to drop frames to the original frame rate, and that works perfectly for most DVDs.
Cartoons might be another issue for TVAI because they can have dual frame rates. The background panning can be in ~24 FPS while the animated characters can be in ~12 FPS.