Video Enhancement AI Important Upscaling improvement

WARNING
This is very long. Please take your time to read everything, to realy understand the message. Skipping lines will yield in bad understanding! With that said:

Hello everyone! I have to say. I tested Video enhance AI and Gigapixel AI. Both programs are absolutely promising. Nothing comes close to that, like already said.

But there is a lot of improofments to make to the neural network. I am also thinking, if it has enought neurons? I am no expert to that, but if I remember correctly, that is one way to improofe the ability to store much more information into the network. So if this is even based on neurons, could you somehow increase its count by a few percent? With that said, let’s get to the real deal.

Corridor Crew on YT (sadly I am not allowed to post the link) did something special: A deepfake to bring back 2 Pack to life. And I remember them having issues, because they only had samples of him from one side only. They mirrored it, to train it with that, which did not look great at all. The shadows where ugly.

How did they fix the shadow issue? And do not ask me at all, why that even works at all!
They changed the color hue hundrets of times, to increase the learning data. And strangly enough, this actually worked. This improofed it a lot. With that and the just basic know how, I think I know, what is mandatory, to increase the AI. The main problem becomes calculation power.

Why do I want this improofment? Well, I wanted to remove the artifacts of the 96fps>30fps slo-mo, that my Lumix GH4 did output. Of course, at such detail levels, the Bitrate becomes a bottleneck. The Video looks great, sure - but the compression in addition with the noise is a problem. It makes it look much less pro. So I did a test, if your Video Enhance AI could remove the noise.

I tried:

  • 100% (remove Blocks and artifacts)
  • 200%

The model was Gaia-HQ, since it works best on everything I tried so far - it does not even matter, if 2D animation or live action footage.

What is the result?

Nothing! The image was not improofed at all. I could not spot any noticeable difference - even frame by frame.

And I know why: I already figured out, that the AI is not good at enhancing zoomed in details. It works quite nice, if you scale down the video first (and for Gigapixel AI scale down the images). But loosing details to recover details makes no sense to me.

I beg you to do the following training to the network:

Let us pretend, all your training data is starting from one ultra high definition video with insane bitrate and basically RAW quality. (yeah, it does not work that way - who cares. Just an example).
Now follow me on that… How do we get out the low quality video, to get the input reference for training?

I think you did this:
(each line is another version of new training data)

  • Add some noise
  • scale down
  • decrease bitrate

I do not know, how many combination of that you realy did. But learning from Corridor Crew i beg you to train the model with every video you have with the following:
(#,##,###… means, there is a long footnote!)

  • Scale down in multiple different resolutions (obviously) - use “even” numbers (25%, 50% etc.) But please also use uneven numbers like 27.5% scale.# [apply the effect to: in+out]
  • Use as many possible combinations of bitrates and codecs as you can. For example MPEG2, h263, h264 and h265. ## [apply the effect to: in]
  • Add color filters (1.: change hue every frame; 2.: change hue for each color for full length ### [apply the effect to: in+out]
  • Add visual damage to the input. Try to create as many types of: Noise, Artifacts, Chromatic abarration as possible. Noise is not just Noise. [apply the effect to: in]
  • Mirror, Rotate [apply the effect to: in+out]
  • Distort (warp) the image [apply the effect to: in+out]
  • Try to get a lot of licence free Video material - it is free. If it is good, use it. ####
  • Can the training be spread over the www? Like folding at home, or however it is called.

I really hope, people take the time, to read this. I think in here lies basically the guidelines, to increase the quality in the long run by over an order of magnitude. AI is the future. But if you cannot keep up with Nvidia and AMD you will loose the advantage of market leader in the very long run. (Before they get anything usable it will take still around 2 years - but not any more!)

Suggestion is wanted. Please upvote, if you want to get an amazing product!

!Only explenations from here on!

#Just try for yourself. I bet 50 bucks this will improof already. My explaination to my theory: Not using 1/4 and 1/2 resolution will damage the quality of the “input” even more. The AI has new data it never has seen before. Also using all those scales will likely increase the quality of already high resolution video - which barely improofes in far to many cases. Also: Try to downscale the HQ reference footage to 1/2 and the input data to 1/4 - this also is new data

##Each codec (at all the different bitrates) compresses totally in a different way. Just render it back to a supported input format (as modern as possible) at a very high bitrate, so the 2nd compression does not “overwrite” or hide the compression of the other formats. If the input format is directly compatible with the training software, of course just use that. There can also be licencing issues with h264 and h265 (pro use costs money) - so already talking about codecs: what about an implementation of VP9 as output format? This is licence free and open by google.

###This is, what Corridor Crew used, to absolutely destroy my mind by improofing the video in such an insane way, that it is just not believable. Who thought, that a bit of crazy color and flickers could be used, to train an AI to increase fidelity by an order of magnitude. Will it be as good for Topaz? Only one way to find out.

####Ask the community, to gather sources of high quality video material. (of course only links, so you can find out, if the licenses are not mistaken by the community) Maybe even create a reddit like sorting system. People need to vote, if the video is high quality enougth, to be used for training the network. If the video has reached… say 100 Votes and shows 95% community approval, than use that. Just do it - I bet 100%, that there will be thousands of Videos available in a few months. Maybe you can use this in combination with some discount on your other products. Like: If you add at least 5 Videos with at least 60 Minutes of high quality video (community also approves it), you give a 10% discount for a month. Such simple things can hugely increase the manpower of your projects for almost free. Just the time, to set up the webpage for that. That would be like 5-10 hours of programming and done. Don’t get too fancy - just do, what really is required.