Face Reconstruction (Refinement) for small faces

It would be great VEAI included some kind of face reconstruction for face detail. :smiling_face_with_three_hearts:

When VEAI detect faces in scene, it should use a specific AI model to upscale the face.

Human is very sensitive to human face, currently VEAI upscale footage without restore face detail and it often create monster faces and ruin the footage completely.

Gigapixel (4x size very compressed):
629525ce0c9400757d562ed2891c86e870576349-very_compressed-scale-4_00x

Thank you for testing.

I also tried with Tencent ARC’s GFPGAN. :smiling_face_with_three_hearts:
Here is the result.

test image source_00

Compare Before & After:

4 Likes

Wow! Most impressive. Does Topaz have any plans to hire the folks who created GFPGAN’s algorithm?

If this ever comes to a commercial product, I’m going to have to redo a bunch of my videos. So be it.

1 Like

The images you posted are interesting as it contradicts the others I’ve found. The ones I found showed very little improvement in video quality and no improvement in detail. Did you use AVCLabs VEAI? I’m installing the trial version soon to test for myself.

Yes, I tested with AVCLabs Video Enhancer AI.
AVCLabs VEAI seem treat video as image, they upscale each frame separately using BSRGAN with face restoration. The Ultra Upscale takes very long time to process, the Fast Upscale produce significantly poorer result.

I think it highly depends on the content, BSRGAN is good at denoise very grainy old film photo or upscale animation. For normal video, it just remove too many detail and too much sharpening.

AVCLabs VEAI is not a replacement of Topaz VEAI.
Topaz VEAI is much better overall and it has improved a lots since it first release. AVCLabs is still very new and build in face restoration, may be they will improve or add more model in the later release.

I’m done testing it. Using the Ultra model, the program runs about 20 times slower than Topaz VEAI. There’s very little useful technical info on AVCLabs site or with the program. Like what is the bitrate of the output MP4? I found a webpage on their site that suggests it varies with the size of the output. I wasn’t able to gather any info from the partial MP4 because it deleted the file when I aborted the operation! x264 and x265 leaves the partial file if you abort. There appears to be no way (from the GUI) to change the output bitrate or CRF or to use a lossless option. I searched the web but it seems no one has posted anything about the program. They have no support forum.

It’s a moot issue as the quality is terrible. Looking at the preview output, it leaves jaggies everywhere and looks quite unnatural. With what they have now, it’s not at all competitive with Topaz VEAI. They need to go back to the drawing board.

1 Like

I didn’t tried the software but from what i see here,

  • everything is done to steal everything from topaz VEAI + their possible customers.
  • the demo seems to be acceptable but, remember that doing a test on a face and a video is something totally different ! an engine done to recreate face is not the same done for landscape or objets etc…
  • they took what i maybe understood the open source developpement from someone else as the base of their software
  • they took the UI from Topaz VEAI as well, the name too, etc…

all this not make me wanted to try it as it’s so looking like the so called “chinese” encoder based on ffmpeg that we see everywhere, where they sell a Gui at 50 or 70$… with “amazing BF discount at 5$ :D” or even “given for new year”.

+1 for GFPGAN

1 Like

GFPGAN can not for now be transposed to video, it only works on frontal faces and works more like deepfakes on low res faces creating totally random faces that could change on different frames of a video.
And creates teeths and sclera too white to be real.

2 Likes

I think the Theia refine detail mode might be able to partially do this job. Someone explained to me the Theia mode was made to recover details by looking into several near frames. In this case VE AI could improve even small faces. Just testing this right now. :slightly_smiling_face:

1 Like

GFPGAN tryouts. I don’t think you would ever want this in your videos :sweat_smile:

See how bad it is on profile faces and poping up those out of focus faces with very strange faces.

Also some “ghost” faces can randomly appear when there should be none :grin:

3 Likes

tell us when you’ll see Jesus ! :smiley: lol…

1 Like

Or Elvis :slight_smile:

1 Like

Jimmy Hoffa body as well, that would help :smiley:

Remini doesn’t really to anything but faces, but faces it does very well.

Remini202201132059078381

Nice, although I’m a big Topaz fan I have to admit that the Remini app I tried out on Android does a much better job at faces.

1 Like

I have seen Remini as well and the results on many old photos are nothing short of amazing. It doesn’t work on everything but, whatever they’re doing is worth analyzing to see if can be incorporated into VEAI.

3 Likes

Adding this feature would improve the subjective quality of the upscaling so much because human perception is especially sensitive when it comes to faces.

I don’t think it can be as straightforward as adapting a model made for still images to VEAI because we need temporal coherence. The output has to match the output of the previous frame/

3 Likes

Yes, I agree with you.
If we treat each frame of video as a separate photo, it looks unnatural.

Here is an example,
ezgif.com-gif-maker (1)

VEAI require a special trained face model designed for video.

At the moment, it seem that developer is planning to include the face enhancement model from GAI V5.8 into VEAI as a short term solution.

2 Likes