i have couple of old videos containing human faces. They are low quality ones so have lot of noise. I tried different models to see if they would upscale it to make more natural but none of them made any difference. They were mostly the same as source input with minimal smoothing. Are there any specific models that would work? Please suggest.
As far as I know, face detection tech is not incorporated, so you just have to play with it to find a compromise between alien faces and noisy video.
ok. Are there any known AI tools that do that already or is this an active area of research? Given the reason people try to upscale videos is mostly to upscale old videos containing their past self or family members it should have been at the top priority for topaz ai upscale algorithms.
There are a couple of cloud services that can do it. You pay a premium but the service is available.
If enough of the frames are in reasonable condition - and there are usually some - deep fake software should be able to do face replacement (of the same person) prior to the upscale. Train it on the good frames then deepfake them into the target video to replace the faces in all frames both good and bad. Tracking the face in question would obviously have to be part of it, especially if there are more than one.
Whether any of the current deepfake stuff lets you do that I have no idea but it should be possible. Maybe Topaz will get round to it one day but if not, it is surely just a matter of time before someone does. I canât see any of the current models being anywhere near good enough for any of this.
There are several AI tools have built in face enhancement AI model for video,
Such as, AVCLabs Video Enhancer AI , HitPaw Video Enhancer, CodeFormer Video Enhancement, etc.
However, (last time I tried) they only treat each frame as seperate image. Therefore the result can vary significantly depends on source. If the subject is looking straight toward the camera with little movement, the result is pretty good. But if lots of motion blur & subject movement, the result is horrible.
If you want VEAI / VAI to include face enhancement model, please âVoteâ here.
1 Like
Thanks for listing the applications that do that.
I used this for now codefarmer to upscale old videos containing faces. As you said above it only works correctly when the subject is facing directly at camera. The moment it rotates face or there is sudden jerky movement it makes it really messy. I had to install anaconda, cuda and build the codebase to run it but was able to do after few hours of grinding. So was worth it and definitely recommended if you guys donât want to put money on paid software.
To salvage the video to acceptable state i will merge source and upscaled videos in premier pro and use mask to only use the facial part of upscaled video in final result. Once they are merged i will then use topazAi to do overall upscaling of video.
The good thing about codeformer is, it export all the frame in png files during processing. I can keep monitoring the export folder during processing, sometimes it create monster face file due to motion blur. I can delete those frames and replace it with files from perious frame or after.
Camera shake probably yes, but blurry face turning, dancing etc. or a high percentage of blur? No because there will be too many consecutive blurry faces. I have many old videos like that.
That will need face tracking, identification of the best frames and using them for AI face replacement based on the best ones. I canât even begin to imagine how slow that process would be, it might still require some manual intervention, and so we are probably talking short clips only for a long time yet, even if such a model could be created.
2 Likes
Just extra 2 cents here. I tried GFPGAN and found it much better and accurate than codefarmer.
I think it really depends on the source image. 
For some image, I prefer GFPGAN over GPEN or CodeFormer.
But for very low resolution image, GFPGAN sometimes fail to recognize face.
CodeFormer tends to recover extreme low resolution face better.
Also CoderFormer allow fine tuning the face slightly by adjusting the ââfidelity_weightâ parameter.
1 Like
