Iris v1: Face Enhancement for Low-Quality Videos (June 2023)

Topaz Video AI Roadmap
1

Enhance faces naturally

Today we’re thrilled to release Iris v1, the first face enhancement AI model designed specifically for video. Iris works best on interlaced, noisy, or compressed footage with degraded faces. After updating to Topaz Video AI v3.3, you can find Iris in the Enhancement filter under any Video Type:

CleanShot 2023-06-12 at 17.37.48@2x
CleanShot 2023-06-12 at 17.37.48@2x936×516 201 KB

As humans, we’re uniquely sensitive to seeing faces that look slightly off. The same methods that create believable texture on foliage, buildings, wildlife, and other subjects can often fail completely on human faces.

Face Recovery from Topaz Photo AI works well on still images but often appears unnatural when applied to video. The slight differences in orientation between each frame create compounding artifacts when stitched together. In order to solve this, Iris uses information from the same face across multiple video frames to achieve significantly better quality than was previously possible:

Iris minimizes flickering, artifacts, and motion inconsistency, and as a result offers more natural and detailed face enhancement than any other video tool currently available.

Best for low-quality videos

When upscaling low-quality videos with degraded faces, Iris offers significantly clearer, more detailed, and more natural results than Proteus or Artemis.

Iris was trained to aggressively remove extreme levels of noise and degradation from low-quality videos, so it has a tendency to remove detail from higher-quality footage. We still recommend Proteus or Artemis to enhance texture and detail in higher-quality videos with larger faces.

Improve interlaced or progressive footage

Iris is designed as a flexible model for all input videos, so it works well on both interlaced and progressive footage. In fact, we’ve found that it will often produce better results than the specialized Dione deinterlacing model, even for interlaced videos without faces.

Recover original detail

Use the new “Recover Original Detail” function to intelligently blend fine texture from the source footage into the output. Find the new slider in the Enhancement filter with Iris, Proteus, or Artemis selected:

CleanShot 2023-06-13 at 12.08.18@2x
CleanShot 2023-06-13 at 12.08.18@2x1278×719 152 KB

This pairs well with Iris in cases where it creates unnatural detail or overly smoothes the footage. Increasing this slider can reduce unnatural smoothness from AI enhancement, but can also bring back noise and compression artifacts. We also recommend caution when combining this with the Add Noise slider, as the noise might also show up in the output.

We hope you enjoy using Iris. Please let us know what you think and what you want to see next!

Other improvements

Since May’s update post we’ve improved on app input/output and stability:

  • Audio streams with metadata are now preserved in output
  • Default encoder for new installs set to H264 High
  • Added DPX support
  • Corrupted model files are now automatically re-downloaded
  • Fixed Estimate for some image sequences
  • Fixed copy ffmpeg command button
  • Fixed output video preset issue
  • Fixed occasional permissions issues when updating or uninstalling
  • Fixed occasional importing issues and processing failures on Mac
  • Max Processes now defaults to 1
  • Preview frame limit increased to 900
  • Failed processing now shows more specific errors
  • Various UI improvements for dialog boxes, buttons, and tooltips

Read the release threads for v3.3.0, v3.2.9, v3.2.8, and v3.2.7 to see the full list of changes.

Next

We’re working on a few concrete features that should wrap up within the next few months:

  • Detect scene changes internally to improve Stabilization and Frame Interpolation quality for videos with multiple scenes.
  • Allow pausing and resuming exports.
  • Faster playback and seeking in the in-app preview.
  • Allow applying a second Enhancement pass without exporting and re-importing.
  • Improve Output Settings interface and add audio transcoding options.

We’re also exploring a few directions that we could use your feedback on:

  • Explore NLE integration with popular editors. What are some other plugins that you use that you think we should look at?
  • Explore the next generation of video enhancement models for medium-quality and high-quality footage that will eventually replace Artemis/Gaia/Proteus. What kinds of use cases would you be most interested in, and could you provide sample videos?
  • Explore a specific noise reduction model for 1x enhancement for noisy or low-light videos. Do you have noisy or degraded footage that Proteus/Iris currently fails with at 1x?

Please comment and upload videos to Dropbox, and apply for the video beta program if you’d like pre-release access to new features. We’re excited to make some major improvements to Topaz Video AI in the next year, so we look forward to hearing your feedback.

23 Likes
3

The Iris Face Enhancement is an interesting inclusion to this app’. Proteus’ manual Sharpen can over sharpen, so this will assist in clips that require less pin sharp sharpening.

Question: How does Iris Face Enhancement decide what is a face when a figure moves quickly in any one direction?

4

What kind of settings are you guys using for IRIS? Auto? Or do you use manual and bump up some specific values?

1 Like
5

I am using relative to auto most of the time.

3 Likes
6

A quick follow up to say thank you to all the people who make the magic happen at Topaz Labs.

The inclusion of the iris face enhancement in Video AI is superb. It seems to function vaguely like Boris FX’ Silk smoothing filter, but permits sharpening of edges of video images.

3 Likes
7

The one thing that I feel like TVAI doesn’t do much for right now, but several users bought TVAI for such projects, is recovering family films.
For example: I have several digitized family videos that were recorded directly onto VHS by a camcorder. All the current models don’t really enhance those much. They do something, but not as much as what can be gotten from high quality sources.

I see a clear model training path from the amount of commercially released VHS and matching Blu-Rays. No idea how possible it would be to procure such a dataset, but it technically exists.

13 Likes
8

Great suggestion!

4 Likes
9

Iris doesn’t have any separate model to detect faces. It is one single model which implicitly detects and enhances faces (and other objects) in different situations.

2 Likes
10

It would be great to see how you could integrate with Davinci Resolve NLE. It’s the leading NLE when it comes to color (industry standard) and its neck and neck with Premiere Pro in terms of use in the professional market.

This would enable those who shoot RAW (RED, ARRI, BRAW) to keep the quality of the RAW file while utilizing the Video AI tech.

8 Likes
11

Yes, indeed. I imagine one problem that needs to be resolved with Topaz is color management, which as it is now, changes colors of the videos that it process. I would love to see Topaz finally solves that color management problem, so we can use the program more professionally. Especially in Resolve where color grading is important part of the app. Getting just right color or god forbid a color product shots and than Topaz changes the colors once processed. Not workable, until they resolve color management.

16 Likes
12

THIS!

2 Likes
13

Yes, you are absolutely correct. This is an area that we are working to improve immensely.

8 Likes
14

Glad to hear it.

2 Likes
15

It’s not all about the professional market though, and these days the line between that and ‘lesser’ NLEs can be blurry. So, as Topaz has said they are considering the development of a TVAI plugin for NLEs, the way forward should be I think to cover a wide range of them.

Davinci Resolve supports OFX plugins and here are some well known hosts that also support the OFX standard:

Autodesk Flame, Black Magic DaVinci Resolve Studio/Design Fusion Studio, Boris Silhouette, The Foundry Nuke, FXHOME HitFilm Pro, Magix Vegas Pro/Movie Edit Pro/Movie Studio, and others. Not to mention non-OFX software by Adobe and Avid!

There’s a big potential market there for just 2 or 3 versions of a TVAI plugin (OFX/Adobe/Avid), which could be offered at a discount from the full software pricing. I hope Topaz have the inclination and resources to do it.

4 Likes
16

I’m not much of a fan of this model to be honest. To me the end result looks cleaner, but also a bit blurry and less detailed which i find very important in a human face.

Its like the person in the video went to a professional makeup artist to ask them to hide any small flaws when it comes to their skin and its pigments.

Just my opinion though.

4 Likes
17

Would be good if Video AI could detect and train itself when a better quality version of the face /body exists in a shot and then apply that when the subject becomes smaller in frame, is concealed or in bad light. Another thing I’ve noticed is when someone blinks in lower quality video things can look a bit weird for a moment.

9 Likes
18

Yes, I have suggested this previously. If there is a closeup of a face somewhere in the video, the AI should be able to take that face and apply it to the scenes where the person is further back in the shot. It’s inevitable that this will be possible someday, but who knows how long it will take to implement this type of thing in this program.

It will especially help with eye detail.

6 Likes
19

I was going to say pretty much the same thing. It is a bit of a hassle to have to go outside DR just to do the AI enhancement. Plus, there is at least one other company who does AI enhancement that has a plugin for DR already. It would be a good idea for Topaz to jump on this opportunity before another company establishes dominance.

3 Likes
20

Upgrade Blu-ray (1080i, 1080p) to UHD Blu-ray (2160p).
Upgrade SD animation to FHD or UHD.

7 Likes
21

Have you increased the parameters in manual mode?