Topaz Video AI Beta v3.2.9.0.b

Hello everyone,

We have another beta available for your testing.

Downloads:

Changelog from 3.2.8:

  • New enhancement model named Iris (previously known as Proteus v4) for enhancing face or low-medium quality videos
  • Updated Apollo Fast model for improved sharpness
  • Misc updates

Known Issues:

  • Not displaying the correct duration on some videos

Enhancement model - Iris

  • Primary aim of the model is to improve human faces
  • This model is also designed to enhance low- to medium-quality video, particularly, with noise and compression artifacts. The model tends to lose details on high-quality videos
  • This model is trained to work on all three categories - progressive, interlaced, interlaced progressive - but the main target is progressive videos
  • Since the model has significant differences from the Proteus model in terms of aim, scope, and performance, we have decided to give it a completely new name
  • Please test the model and give us your opinion. Thank you

Please upload problem videos and logs here: Submit Files .

5 Likes

Are there plans to improve this before the stable release? Or is this just a characteristic that Iris will have until maybe a v2 comes out?

2 Likes

Note Iris 4x upscale was used in my testing with full auto settings.


Iris (Proteus v4) continues to have the issue of over enhancing some details. For example the shirt and poster in this shot.

Or the text in this shot (it’s nice that Iris is trying to enhance the text, but there’s not enough information for it, so it produces some really wrong results. This is a case of Iris trying to “over enhance” or being “over confident”):

But then it’s inconsistent. Where it will enhance some text, but not others, giving a weird tilt shift like effect due to the blurriness of the non-enhanced text. This issue isn’t limited to just text, it impacts other things like faces (mentioned below).


Iris (Proteus v4) still enhances some out of focus faces.


Edit: The issue described below seems to be related to the video I’m using. Sadly I can’t share it due to copyright issues.

Along with that Topaz Video Enhance AI still isn’t aligning original and preview footage (as can be seen with some of the images in this comment).

6 Likes

If I recall correctly, in the road map announcements from the last few months (before they were edited), it was mentioned that Topaz Labs was working on a AI that handle faces better (Iris/Proteus v4) and an AI to denoise better.

Was the “denoise better” AI also Proteus v4/Iris? Because the lose of detail in high quality footage looks like an aggressive denoiser, and it would make sense.

On that topic, a simple idea for pursuing a Iris HQ (high quality) is to train an AI for facial improvements (the same way you did with Iris) along with the typical training you’d do for something like Proteus v3 or Gaia (rather than the denoising specific training I believe you did). That way you’d probably get a high quality Iris that can live along side the current Iris. Both of them having their own media types they work best in, but both enhancing faces better.

Of course, AIs are complex things and I’m making some assumptions here. I could be entirely wrong on my idea for why Iris has issues, and my idea on how to create Iris HQ may be ineffective.

3 Likes

I just ran a quick test that can help with some of the issues with Iris.

The footage I’ve found work best with Iris is DVD versions of movies from the 1980s. And is typical of DVDs of this type, they have noise, compression, and a relatively low resolution. So I’ve been using Iris 4x for most of my testing. But it seems Iris 2x with another pass of Gaia 2x or Proteus v3 2x to bring the total upscale to 4x can produce a more consistent upscale than a simple Iris 4x in some scenes. Maybe other people should test this out?

Hopefully we see the multi-pass upscale brought to the TVAI GUI soon so this is easier to do.

Just a quick one:

Two installed GPUs now show up as four GPUs in the selection (3.2.7 showed three…).
FP32 Models are loaded on GCN3 and GCN4 - speeds are slower again.

1 Like
Topaz Video AI Beta  v3.2.9.0.b
System Information
OS: Windows v11.2009
CPU: AMD Ryzen Threadripper 3960X 24-Core Processor   127.88 GB
GPU: AMD Radeon PRO W6800  29.956 GB
Processing Settings
device: -2 vram: 1 instances: 0
Input Resolution: 720x480
Benchmark Results
Artemis		1X: 	13.56 fps 	2X: 	07.32 fps 	4X: 	04.74 fps 	
Proteus		1X: 	13.59 fps 	2X: 	08.63 fps 	4X: 	05.36 fps 	
Gaia		1X: 	07.31 fps 	2X: 	04.50 fps 	4X: 	04.79 fps 	
4X Slowmo		Apollo: 	18.17 fps 	APFast: 	44.59 fps 	Chronos: 	04.23 fps 	CHFast: 	13.73 fps 	

These numbers doesnt really make sense to me, since Iris was running with a export in the background at the same time with a (720 x 576) 4x upscale + Stabilisation + Motion Deblur + Frame Interpolation @ 4.4 fps.

If i export the same video with another instance i get 2.8 fps in each instance that means 5.6 fps in total.



Maybe the Benchmark is just too small in load for the GPU and because of this it don’t realy see it.

1 Like

My first impressions with Iris V1 are pretty positive: the face enhancements are very good and effective without excessive increase of sharpness. The only cons is the speed in encoding (both ProRes and H265) slower by 3 fps respect of Proteus V3 and Artemis V13. My specs are Mac Studio M1Max with 32 Gbs memory and 24 GPU cores.

1 Like

This is likely related to the point I made last beta, which is that the 2x and 4x upscale models are different and don’t produce the same results. Given the 4x came first, it may be the 2x is headed in that direction - but I really hope not as the 2x is the best version of the model I have sen so far, with the 4x much worse.

To give you an example using this current Beta, this is a comparison slider between 4x images - The original non-upscaled input, Iris V1 Auto 2x, Iris V1 Relative to Auto 2x and Iris V1 4x Upscale Relative to Auto (same settings on relative).

If you compare the Iris Relative with Iris Relative 4x you will see what a whopping difference there is between the two - and not in a good way. Iris V1 Comparison - Imgsli

On a separate note, the text enhancement issue is one of those likely unsolvable issues. If you have text in the image that is so blurred that you cannot deduce what it says, the decision needs to be made on what to do with it in an upscaled image.

Usually if the text is that bad, then its either the worst footage imaginable, something the others screenshots don’t imply, or its so small that in the original you would not have been expected to actually be able to read it. That is, the director etc was providing one of those “filler” articles where the headline is all thats important etc.

These will never be able to be enhanced properly. Based on the output, i suspect whats happening is that because it cannot read it, its replacing it with a kind of pseudo text - something that looks like text but isn’t. This would be something you do to disguise the issue.

The best example is when restoring artwork and you have to paint in the missing sections. The restorer’s job is to make the image “whole” but not under scrutiny. If you stand really close, you can find the problems, but when standing back, the image needs to look natural.

I cannot see the frame as a whole, or how noticeable your specific example is when looking at the whole frame. When upscaling the 4k its probably unavoidable to be seen - and I am not saying its doing a good job - but I wonder if what happening is its attempt to disguise the readable text in a way that you don’t get drawn to noticing something is very wrong, by replacing it with something that with only brief exposure when watching as a video, you may miss noticing its gibberish as on quick glance, on background text, it “looks” like text.

This is a screenshot of your image when bundled into this post - IE without me clicking on it to magnify. It “looks” like text and if it was not showcased, or had its attention drawn to it and only appeared briefly, you might miss completely its gibberish - and I wonder if that was the intention to it.

The rest of this message is general feedback noting that I refer always to 2x upscale here as I treat the 4x as not really usable currently.

  • Comparisons between Iris Auto and Iris relative in the above show the incompleteness in detecting and sharpening the image properly. The Auto is not bad - but Auto on its own still remains the worst way to upscale anything I have tested. You must adjust those settings if you want a decent output.
  • The large checkerboard pattern appears to be gone so far - at least way less noticeable if it is as I haven’t seen it yet, so that is an improvement.
  • Exporting images still isn’t accurate. On this test I selected frames 58900 to 59000 and the frames export started at 58899 and finishes at 58998 meaning that of the exactly 101 frames I asked for, I got 100, with the wrong start and finish frame, so both dropped a frame, and the wrong frame sequence. I personally have no idea how exporting frames went from perfect with never an issue in V2 of Topaz, to months and months of endless problems with frames in V3, but honestly, I think its beyond a reasonable time frame to fix. I am at a loss as to how an entirely new model can be developed and tested, but if I ask for this specific extract, I cannot get it.
  • Export not Export As, allows for export of frame sequences with no name and thus auto defaults to the frame number. This is great, but Export As still requires something to be put into the field on export everytime. The old version used to allow this if required, but if left blank would just export the frame numbers as default. It is a small thing, but it bugs me no end, especially when testing frequent small samples.
  • I thought I would experiment with importing the original VOB to see how it would handle no pre-processing at all. When doing so, the Preview button is greyed out and won’t let me preview at all, but I can scroll the time bar and see the whole video and trim normally. Having said that:
    – It won’t trim
    – It jumps constantly on preview frame numbers and they are always incorrect.
    – Basically it seems to have no idea how to handle the file - but gives no errors as such, just a warped user experience.
    – It will not let me edit the input fps, which is incorrect for the file.
    I have not seen a video application have this much difficulty understanding a vob file, and all my other applications open it with no issue.
    – I attempted to export a clip of it saved directly to mkv using a different application, and this opens and seems fine with frame numbers, but it has lost its aspect ratio information and now displays wrong in Topaz.

Just a reminder that eventually I really suggest you decide at the back end to include the ability to import from Avisynth files again. I used to be able to import the vob files with little to no processing by using the d2v file assessment with it - but attempting to import the d2v file does nothing, so I assume you don’t support that anymore, and without a frame provider like Avisynth, I cannot translate that import either.

I bring this up because your goal is apparently to make it as user friendly as possible to use and upgrade your old media and home movies - but you made V3 of the program significantly harder to actually import those some media files. I have to jump through hoops that the average person doesn’t want to do just to import the same files and media V2 handled with no issue.

It is a bit of a paradox to me as to why you would intentionally make it worse without a solution to fix it in application. If I have to pre-process the files before I can feed them into Topaz, and V3 makes that harder than any previous version, then no lay person is going to be happy trying to use this application.

5 Likes

on the relative Auto, what values you changed/tweaked? and to what value(s)?

The new „Iris Model“ is definitely better than in the older Beta, but there are still some Issues

=> The Face is sometimes still looking strange. Specially with small faces in the Background. And it’s should be better interpolated over the Time to avoid changing eyes over time etc.

=> The Video is loosing still too much Details. This should be better adjusted.

3 Likes

Iris is reducing grain nicely. In some cases it even seems to restore the grain structure where the material is overly compressed. I’m still having the issue where the preview input and output are off by a frame. If this issue is a moving target and can’t be corrected in all cases, can we get preview frame offset + and - buttons in the meantime?

I’d know that news paper anywhere, taking Iris out of Full Auto can help get something closer to reality. It’s still not legible text, but it doesn’t look like a completely made up language and we probably wouldn’t even be able to read the text at a normal distance. Besides, that specific text is not legible on the official UHD remaster either. I usually stick with Artemis Dehalo Medium for these types of movies, leave it at 1x scale for Full HD material and slap a lighter layer of grain back on.

1 Like

these numbers definitely are too slow with this small input resolution. was it faster in previous versions ?

TVAI did export in the background, so GPU was under load.

My assumption is that because 2x upscale isn’t enhancing as much, it can more closely follow the original footage. While something like 4x has to make up more detail which can can result in a mix of “confident” 4x upscale and uncertain 4x upscale. If you do a 2x upscale then another 2x upscale, you get a total of a 4x upscale, but the results may be better because both steps of upscaling are more “consistent” since they’re not generating as many pixels with as widely varying levels of quality, confidence, and consistency.

And how text that can’t easily be identified should be enhanced is kind of down to personal preference.

For example, it seems you’re mostly okay with this “pseudo text” the AI is generating. It looks like text from a distance, and that’s good enough for most people (and good enough for me if I was watching this on a small screen).

But it’s personally not something I’m happy with. I would prefer if the AI kept the text kind of blobby. Because with the “pseudo text”, it’s not as temporally stable as the rest of the scene (which draws attention to the pseudo text) and the pseudo text generation is inconsistently applied across a body of text, such as in the image below, making the footage look inconsistently in and out of focus.

It’s probably an unsolvable issue in terms of video upscaling. In terms of video enhancement (E.G. Applying video upscaling for some parts, and AI video generation for other parts), it should be possible to solve at some point in the future.

I’ve noticed an interesting thing while seeking a variable-rate source video file with a :arrow_right: key in this beta - the seek bar skips a small amount of the “way” sometimes when I’m going to the next frame, and sometimes - a bigger amount. It’s totally fine, if the video has a variable frame rate.
I’d like to “convert” the video to the constant frame rate though, to avoid this choppy playback in the future - which options would be the best for VEAI to do so?

The Magic’s Gone!

For several / many releases I can’t match the quality of a version from about a year ago. The file date is 07/04/2022.

I wasn’t even planning on keeping it, but I’m certainly glad I did, because I can’t even come close anymore.

It’s an interlaced video from 3/4” to ProRes, 720x480, and I’ve enlarged it by 268% which was the even number that got me over 1080p, ie: 1930x1302. DDV3 was where the magic was. I’ve tried all I can think of now, including Iris now. In one shot of my mom, she looked like a different person!.

I’m giving up on it. I thought you should know. You perform magic at times. Same with the still photo apps, (and still are there, for my limited use). But I’ll have to move on to some movie transfers to progressive, they seem to work OK still.

Original and Iris

The previous trusty DDV3

And with the Magic…
GoodTopaz

Any suggestions…
Gene

These are the same image of Iris. We can’t see how modern DDV3 compares to the older DDV3.


Based on your comment, it seems DDV3 from roughly a year ago is significantly better for you than DDV3 in current versions of TVAI. But based on the input footage, I find it hard to believe that DDV3 could produce the magic results you have from roughly a year ago (It just seems too good to be true considering the colour shift, compression artifacts, and overall quality differences between the original and magic DDV3 from a year ago).

I think you should go back to an old version of TVAI from roughly a year ago and test it (This would probably be v2.6.X). This is just to make sure you can produce the magic results you’ve shown in your comment. If you can’t match the magic results, then it’s likely you were using a different video source roughly a year ago or you were using a different AI model to enhance it.
Older versions of TVAI can be found here: Releases - Topaz Community

Once you’ve confirmed that DDV3 is performing differently between the current and older versions, and the older one is significantly better than the newer one, you can provide the developers of TVAI with the exact version numbers you were using and they could take a closer look at what changed between then and now and possibly identify the cause and come up with a fix.

Along with that, providing the hardware you’re using might be helpful. For example, many of the AI models now use TensorRT on modern Nvidia hardware. TensorRT typically speeds up the processing, but it also has a impact on the quality of the output. The simple shift to TensorRT may be enough to explain the difference in quality. Macs also appear to use MLCore to speed up processing, but it also has an impact on quality compared to using other methods on other machines.

Also, providing the video to the developers could also help them with testing. There is a dropbox link in the first post that you can upload videos too.


Also, just confirming. DDV3 is Dione DV 2x FPS v3?