Topaz Video AI Beta v3.3.2.0.b

[dunmer2007]

I thought I’d tell you that I was dead wrong about my method to fix the dupe bug, so instead of RIFE, I figured I’d run the video through TVAI, and clean up the frames after, as it’s still exponentially choppy as I usually do some jobs with 240 frames as I want accuracy for some of my own 3D renders as I do 3D art as a hobbyist.

Yet, the only thing TVAI’s interpolation accomplished was to actually multiply the duped frames, so instead of: 1-2-3-44-5-6-7-88-9, etc… I receive this: 1-2-3-44444-5-6-7-88888-9, etc… ⇐ (It scales exponentially the higher/lower the frames.) I’ll run a test and load my videos/logs once I’m done testing, of course…

EDIT: I finished testing and uploaded the results.

So yeah, RIFE is the only choice for interpolating videos that are not the standard, my video is 25 fps BTW, also of note, Virtuadub2’s remove dupes plugin is still pretty fast, relative to the frame count that is, so it’s a great resource if you’re pressed for time.

[Akila]

@dunmer2007

RIFE

what is rife?

[david.123]

Papers with Code - RIFE Explained

[joel]

when you trim videos, it just blacks the screen but leaves the audio and file stays original length. this program is always one step forward and 5 steps back.

[t9000]

For RIFE check out FlowFrames. That will get you started faster than looking at a paper and repo, and it’s free. Flowframes - Fast Video Interpolation for any GPU by N00MKRAD

[johnnystar]

Iris has a lot of dehalo built in that can’t be adjusted with the sliders. The ghosting you mention is a byproduct of that, as it is like a glow over the black parts. It is visible on some sources more than others. If the developer can reduce the dehalo effect, it would help a lot.

[jo.vo]

Apart from the slow performance I’m actually quite happy with the Iris model. It really helps in upscaling LQ/MQ SD content where Proteus earlier did a decent enough job except for faces which sometimes were really terrible. Iris keeps a more natural appearance here and also has less “oil-painting” (with the right settings).

Compared to what I’ve got before with Proteus that IS an advance.

[dunmer2007]

The only thing about flowframes is that in order to use the current version of RIFE 4.6, you will need the Patreon version, but that’s only around $6… (Donation at $6 per month) Also of note, the developer as well as the program takes a glacially long time to update the program/complete interpolating, but it’s still useful.

The current update was in February, and the one before that was in Sept 2022, and in all that time, mostly radio silence from the developer…

But to be fair, while it’s still only one developer, as a content creator/developer using that platform you still need to provide better support/progress in order for people to have a solid reason to pay money to support you financially every month.

EDIT: Another disadvantage with flowframes is that it cannot accept image sequences, unfortunately…

[dunmer2007]

It works very well with 240p videos, particularly with videos where the faces are of a few feet and mid-range distances but still has issues with faces any farther than say, 10 feet (3.05 metres), with 20 feet (6.1 metres) you’ve got mutant-generator on your hands!

This is from Cyrak’s “Beast Enders”:

It still has issues with facial approximation predictions, but that is mostly due to the model having to predict facial components from pixels affected by low rez artefacts, so it is constantly predicting over an ever-changing transition between pixels, as the artefacts disrupts accuracy.

The fact that it can accomplish this at all, which makes this tech pretty amazing in of itself, think of it as trying to approximate the correct face behind a pane of constantly changing patterned glass, so it’s quite remarkable, which should get better at predicting over the time of its development/learning.

The drawback is that it also ends up over-simplifying non-facial objects, so they end up looking like illustrations rather than realistic representations, but that too should get better over time, thanks to TPAI’s continued image interpolation improvements with it’s model!
NOTE: Cyrak, uses real-world images for his animations.

My guess is that the same tech used for TPAI will eventually be used with TVAI, which is different from a single image, as you have to account for an ever-changing progression of both the individual frames, and the artefacts across the video…

The good news is that TPAI has a new text interpolating model, so it will most likely make it over to TVAI, so hopefully, it will take care of the over-simplification of interpolating non-facial objects!

[sokr1114]

in a lot of low quality videos like this video, there are a few frames the face shown in good quality, this frames can be selected manually by user in VIA, and VIA enhance face in them and use them as reference to enhance the face throw the whole video, this is something like roop in stable diffusion, but instead of using it to make deep fake we use it to enhance the same person face, hope Topaz Team add this feature

[jo.vo]

That’s why I don’t overdo it with the “enhancement” and add quite some grain, that reduces that “painted” look quite a bit.

Even with larger crowds in the background (this is where earlier models often struggled and created “Zombie-faces”) Iris did fare quite well here:

But, yeah, there is always room for further improvement. Iris currently softens the output too much.

[dunmer2007]

You know, I avoided grain, due to it writing over the original frames as I’m a stickler for original preservation, which is why I use/encode files with Huffy-Lossless, in between programs, but I think I’ll give it a go at the end of my pipeline as most of my finished projects are not going to be encoded again!

[johnnystar]

Some questions for developers:

Will Iris v1 continue to be trained so it can pick up some of the fine details that it currently cannot?

Is it possible to tone down the built in dehalo function in Iris, so that the user has more control in manual mode? Currently, in some videos, the dehalo ‘glow’ or ‘shine’ makes blacks look gray or pale (even when the dehalo slider is at 0).

Can face recovery be a separate function from the model (so it can be used with any AI model), similar to the functionality in Topaz Photo AI?

Will Topaz Video AI ever be able to remember user settings (without having to make a preset), so opening a new video or a batch of videos can have the same settings applied immediately?

Will starting a new preview ever be as quick as in version 2.6.4 (which was pretty much instant)?

[johnnystar]

Sure, but Artemis doesn’t have adjustable sliders - big difference.

[dunmer2007]

I always found that bloody thing so condescending, much rather have the unpronounceable text instead!

Was it a GPU or CPU error?

[dunmer2007]

Better a software/driver error than a hardware one… as software can be uninstalled, so that’s good news, at least as far as your hardware’s concerned, those are pretty cool temps during a full-load by the way!

[dunmer2007]

Yeah, myself I keep my apt at a strict 62f/16c nearly all year 'round as I overheat incredibly easy!

My GPU temps are 23c idle/70+ under load and my CPU is 43 idle/75-80c with a full load running TVAI/VirtualDub2, and Avidemux2 doing full jobs while composing/rendering in Daz Studio! It’s why I bought my 3090/Ryzen9 3900x, as I am a huge multitasker!

[deepzone]

The temp file still is not removed when exporting out a file.
I usually export to a network drive, and all read/write rights are in place.

[martinsawicki]

Are there any plans to fix Chronos or Apollo to be able to interpolate low framrates? ie 15 to 60. Now only create dupes…

[dunmer2007]

A tip for getting the most out of iris, I use these settings to combat the iris Vaseline coating effect:

• 25-50% Revert Compression
• 50-75% recover details
• Sharpen can use the manual estimation setting.
• Reduce Noise: depending on the Vaseline level, you can actually turn it right off!
• Dehalo, same as reduce noise… yet helps with over-sharpening normally.
• To combat the Vaseline, turn up the deblur!
• Recover Original Detail, 50-75% works real well depending on how much was lost and replaced by the Vaseline effect!
• Last but not least, you can tidy up the overall effect by using noise/grain!

This vid was 300×270, upscaled to 1920×1728 I also HIGHLY suggest re-encoding any (1080p, ⇐ my mistake) 720p or above video down to at least 640p or lower with lossless containers to get the BEST out of iris, ⇐(That is IF you’re only upscaling up to 1440p, for some 4k is too demanding) as iris really shines with 3-4x upscale jobs, most resolutions lower than 2x is no different from Proteus 3!

EDIT: Feel free to bump up deblur, revert compression and recover details as needed.