I was wondering if Video Enhancer AI could utilize a support to convert SDR videos into HDR videos with the correct colorspace and metadata?
An inspiration idea comes from the RED cameras utilization feature called “HDRX” Where the camera would shoot both: A normal exposed frame and an Underexposed frame simultaneously. Then the exposed and underexposed frames are edited together to make an HDR frame.
I was thinking that for making the Video Enhancer AI to add HDR to video, why not use two cameras? One that shoots the intended video setup at the normal exposure, while using a second camera to capture the underexposed video. Video enhancer AI would then take the underexposed video as a reference, train itself and then convert the normal exposed video into a High-Dynamic Ranged video with Metadata.
Being able to create an HDR video footage using Video Enhancer AI would be challenging but something to experiment or implement upon. Also to take into consideration, the color management to get the HDR AI enhancing would need to be worked upon to allow editing without any odd/weird effects such as artifacting or incorrect colorspace data.
5 Likes
I think this is a great idea and you should work for this. I think you need to spend a lot of time to make such an AI. Making High dynamic videos from a normal video with just a software is pretty cool. I think you can even sell it.
3 Likes
Microsoft did something like this for their HDR output on Gears of War and Forza Horizon. I’ve been playing around with something like this in the following three videos and the problem you run into is that the gamma curve for SDR content cannot be linearly adapted to an HLG or PQ gamma curve, as those usually transfer a lot more into the highlights. That means, that the majority of videos you would convert like this would end up being too dark. And just brightening midtowns results in weird looking shots, with some being too bright and some too dark. Unfortunately, that’s the reason why movies usually receive a separate SDR and an HDR grade. But again, it would be nice, if you guys could find a way to solve this somehow, maybe video game footage isn’t that bad of an idea, as you can create perfectly identical scenarios in SDR and HDR.
Very interesting! So it seems like the Gamma Curve is acting like a bottleneck of some sort. I’ve been playing around with DaVinci Resolve Pro recently and was experimenting some ways to adapt SDR videos to HDR. (Not that it would work), but I did find some interesting results. This is after “maybe video game footage isn’t that bad of an idea” quote:
I used OBS cause it had updates to allow capturing HDR footage and was using Destiny 2 as a test sample, I modified under the “Advanced” Settings:
That allows to capture Rec. 709 with 10-bit color encoding cause any game is already outputting raw colors and details for OBS to capture. Then I was able to tweak any game captures to output it to HDR. Its not perfect but…
Breaking The Ice [Destiny 2] - YouTube
Lost Sector: The Conflux [Master Difficulty] (Fallen Sunstar Exotic) - YouTube
Both are in HDR and you might need to watch them on an HDR monitor. The point however is that it is possible to edit SDR footage (or rather Rec. 709 SDR - with 10-Bit footage) into HDR. However, its more likely gameplay footage is better suited for the HDR enhancing workflow cause its already outputting RAW color data.
But then yet again after you pointed out the problem related to the “Gamma Curve”; it does sound more complicated than what any of us would have thought. For Live Action/Real Life footage (Compared to game footage), It will require some manual and or clever editing to make it look good in “HDR” if the Gamma Curve issue is addressed.
I wouldn’t call it a bottle neck, but it’s not an easy problem to solve because you’ don’t really have a good reference point to work from, as colour grading is an incredibly subjective thing. Maybe it could work somehow if you train different models for different criteria, like sport, cinema ect. to use different interpolations of the gamma curve. Davinci Resolve has a similar feature in their Colour Space and ACES Transform nodes. And you might also run into colour space transform issues as well, which may create colours that are off, too little or too saturated. The HDR topic isn’t really that simple to handle. In most cases I’d argue, you might honestly be better off just watching it in SDR.
To the topic of HDR transforms in Resolve:
Resolve Studio is actually rather good at transforming a Rec709 SDR video into an HDR one. Just set your colour management to Colour Managed, HDR, PQ or HLG (I would recommend HLG for playing around) and the peak brightness you’re aiming for. If you just drag your video into the timeline resolve will make an automatic colour space and gamma conversion that looks surprisingly good, but can end up being a bit too dark. But you can correct that in post if you like. That’s what I did for my videos on a shot by shot basis.
Just know, that HDR content transforms requires a very high quality and high bit-rate 10-bit source file to do a descent job, because if you use a low quality source file the lack in colour-depth and data-rate will create horrible artefacts that will make the video look like it was shot at 360p. In general, 10 bit is the absolute minimum for HDR, Dolby even goes as far - and I agree - that anything below 12-bit colour depth is probably not enough to create a proper HDR grade. At least not for anything higher quality than YouTube.
Ah, I understand.
I’ve been facinated with HDR ever since I’ve seen demo videos and was trying to find a way to make HDR footages. On the other hand, there is always a learning curve…
I guess this is where my handy Iphone 13 Pro Max comes in, but I was also hoping that Video Enhancer AI would someday be able to AI enhance SDR to HDR… Oh well!
1 Like
Honestly, from a pure processing perspective what the iPhones are doing with their Dolby Vision processing is incredible. I’d love to know what VEAI could do speed-wise if it would be optimized for Apple’s ML cores.
This is Davinci Resolve Studio. I use it to correct the videos I upscale with VEAI.
