SVP + Rife frame interpolation (updated to support tensor cores for 50-series)

Topaz Video AI General
1

Not sure how i have missed the software SVP. It’s $20 and use rife models which have had an update to support tensor cores for blackwell. They offer a 30-day free trial with access to all functions (no need to subscribe or enter card details - just download and start the trial).

Haven’t dabbled much in it but have watched a few videos with real time frame interpolation, and just finished interpolating a 1080p@23.97fps clip to 59.94fps at whopping speed of 280fps using “Automatic” mode, which didn’t seem to put barely any load at all on neither GPU, CPU or ram.

Result = OK:ish.

Then i ran interpolation with Rife v4.10 (still not entirely sure what the difference between the models are, but from the small bits I’ve read, the bigger and more recent models produce better results through better motion estimation, no artifacts etc. But they require more compute to render).
Tested the Rife model on another 1080p@23.97fps clip interpolated to 59.94.

Interpolation speed was around 140fps (around 4-5x faster than any of the TVAi models) with a GPU Power Draw of around 200W (~30-40% lower than TVAi) with the rtx 5080 which has a max power draw of 360W.

Result = really. damn. good.
Comparable, if not better than any of the interpolation models currently in TVAi.

SVP can also be used to interpolate in real time. The smaller models which are based on AviSynth+ don’t require much, and the result is really good considering the low hw usage. The larger models such as Rife using tensor cores require a bit more but produce results comparable with a high-quality video at target fps (can be set to monitor refresh rate or source x2, x3, pretty much however you like).

I’m literally blown away.

2

I’ve had SVP longer than TVAI. Last time I compared the two was about a year ago. SVP was worse than TVAI. I put in extra effort to test the RIFE model with it. It was about the same as Chronos.

1 Like
3

I had only tried flowfranes previously, a few months ago, also built on rife.

Render speeds were about the same. Didn’t check quality difference more than just a quick look at the clips to which I noticed no difference. So I stuck with TVAi so I’d have everything within the same app.

It’s also a bit meh regarding the transcosing bit now that I’ve ibterpolated a few videos. You have the possibility to output in mp4, but it renders to mkv first then remux it to mp4 which results in vfr. However, it’s possible to output to mkv, then remux it to mp4 using a set_time script which will output mp4 with cfr that is not forced cfr.

But the fact that it’s 4-5x as fast at interpolating with equal quality is a fat win in my book, since I’m interpolating a lot of <=30 fps videos to 60.

4

I tried to detail it all out here.

When I first got into the hobby of enhancing videos, I was very excited about interpolating. (Like I said, I got SVP long before TVAI.) Lately though, I’m finding the random motion inconsistencies way more distracting than anything else. Like my brain starts to hurt from trying to make sense of them. I’m actively redoing all the videos I interpolated to not be interpolated.
I find nothing distracting or abnormal about videos that are recorded in 60fps. I wish interpolation models could produce results on the same level as that.

1 Like
5

Interesting. I’m going to give your sample clip a to tomorrow. See what the different rife models are capable now, and perhaps figure out the difference between heavy an non-heavy.

From my understanding, if you know how to code, you can manually set parameters. But that’s far above my knowledge. I intercepted the process that interpolates in svp and the command “- of=matroska” among the fulcommand line uawd, but I can’t even find the file whare it’s specified or can be configured after looking through some 10 different settings files.

Would also be interesting to see how the other interpolare features measure up in svp.

6

Examples or it didn’t happen/its not worth anything.

7

Gotcha. Here you have a samples. One done with chronos, one done with apollo and one done with RIFE. Initial fps=25. Interpolated to 90fps 100fps with 4x slowmo back down to 25fps.

I used a clip @Mayday linked the other day that he had processed with Rxl1, ahq12, and thd3, since it’s got a bunch of scene changes some light blinking etc. I hope that’s ok Mayday! Cut out 6s in the middle of the clip.

Parameters:
TVAi v5.3.6
Slow motion x4 - duplicate frames unticked.
Models used: Aion, Apollo and Chronos.
With RIFE i used model v4.25 running on TensorRT (had to do x4 interpolation to 100fps and the slow it down to 25fps with ffmpeg).

Here’s a downloadable .zip file and here’s downloadable 7-zip file.

They contain 5 clips. Original 6s of video and 6s of each of the 4 different interpolations models.

Time required for each model to process 6 seconds:

Aion:    2min 28s
Apollo:  1min 10s
Chronos: 2min 16s
RIFE:    0min  14s

Yes, you read that right. It took 14 seconds with RIFE. Add 10sec for post processing to RIFE and it lands on 24sec. But that extra time spent to process will be pretty much the same no matter the length of the clip.

Now interpolated clips longer than 6s and the difference in processing speed makes a huge deal.

I’ve been comparing the clips in iCAT (free to download from NVidia and an amazing tool to compare videos with), and I honestly can’t tell much of a difference. Most notable however, is that RIFE handles scene changes better. No scene change has been interpolated. While Aion did a really good job, it still did blend scene changes, but much less noticeable compared to Apollo and Chronos.

What do you think yourself?

Edit* Changed time it took with RIFE as 2sec sounded a bit too fast, so I did a re-run which took 14s. I must’ve looked at the wrong log the first time around.

Edit2* After closer inspection. The scene changes using RIFE are that clean is because i had it set to duplicate frames on scene change. Which results in a ~3-4 duplicate frames on each scene change, resulting in a clean scene change, but about 120-160ms of no motion. Setting it to blend results in 4 frames which are just a mess. My final verdict: TVAi beats RIFE on scene changes if duplicate frames are a big nono. If 120-160ms of no motion is fine, which one won’t notice unless looking at it frame to frame, then RIFE beats the TVAi. At least in this clip. Would have to do more testing with different kinds of moving objects to better see the difference between them.

1 Like
8

I did another run of a different scene that looked challenging. This time with all 5 TVAi models, and it’s only 2*4 = 8 second clips.

Testsamples_2.zip
Testsamples_2.7z

9

There’s not enough motion in that scene to really evaluate anything. You need something like panning with people walking in it.
Also, it’s harder to evaluate in slow motion since some things can look fine, until you play them back at normal speed. And others look bad in slow motion, but are not noticed at full speed.

10

Send me a sample clip you think would be challenging and I’ll put together a comparison.

11

Here’s the one I’ve been using.
If this comes out good, I’m confident anything else will look good.
Original Clip.zip (19.1 MB)

12

What fps should I test it at? 59.94?

13

I was doing 59.94, but you can do 2X. Might work better with RIFE.

14

Haha damn. You weren’t lying. Could already tell from looking at the original clip that it would be challenging.

I’ve tried a few different rife models, both at x2, 59.94, x4 and x6. All are terrible at the scene where they are flying with a ship in the background. The wings are flapping like a bird at low fps interpolation.

Aion wiped the floor with all other models and actually produced a good job. If i would rank them from 1-10 where 10 would be perfect. Aion would get an 8, Apollo 7, Apollo fast 5, Rife and both chronos models a 2 at best.

Btw, your clip’s got variable frame rate. mkv container is faking it. Was noticing that the interpolated files was showing faulty fps so i remuxed to mp4 which show the clip as vfr. Fixed it with setpts=PTS/1,fps=“24000/1001” which duplicated 2 frames to achieve cfr before and ran the models again. Also encoded them to av1 and set stricter parameters regarding bit rate. I think the original h265 was at 10Mb/s. After remux and change of codec it has a bit rate of 2Mb/s, while the interpolated ones i restructed to ~5Mb/s.

StarWars_ForSerious_InterpolationTest 7-zip

Starwars_ForSerious_InterpolationTest zip

Don’t know what compression container you use so i compressed to both. They contain the same files.

Edit* Forgot to add the original file in mp4 container and av1-codec. So here it is: OriginalAV1.mp4

Edit2* Alright, should not have compressed it that much. Looking closer at your original and the encoded one, the encoded is missing a lot of details comparing frame to frame.

15

Nice. Thanks for taking the time to do all that.

I just cut the clip out with ffmpeg and H.265 set to CRF 20 or something. I see it’s set to variable frame rate now. Not sure how that happened, unless that’s what the original is set to.

16

Yeah i don’t know. Did some more encode testing on the original to see if i could retain details. Hw encoder/decoder aren’t able to no matter what i do. They all lose details. Even at >10Mb/s. So it’s not weird interpolation models struggle lol

Anyway, x265 at crf 23 seems to be the limit. Lower is a waste of bits and higher you start losing details. Re-encoding the clip instead of just muxing it will generate a clip with constant frame rate. Probably something to do with mkv containers in general that are iffy.

ffmpeg -i original.mkv -c:v libx265 -preset slow -crf 23 original.mp4

Will generate a clip with mp4 container and constant frame rate at ~4700Kb/s which maintains all fine grains and details.

17

I did a rerun with Aion and Apollo, using the re-encoded clip at 4700KB/s, and set bit rate to 12mb/s in TVAi to make up for the extra frames, and the result is pretty damn good. Aion has a slight advantage over Apollo. Considering Apollo is twice as fast, Apollo did an amazing job. It’s not perfect of course. But definitely watchable.

Here’s a zip with originial in mp4 at 4.7mb/s + aion and apollo interpolated to 59.94 with a bit rate of 12mb/s. Re-encoded origininal + Aion + Apollo

Funny how the explanation of the models would make one believe that Chronos would be the best model for this kind of interpolation, when in reality, it’s the complete opposite.

1 Like