Project Starlight - Video AI 6.1 Beta 3

Yes! This is exactly what my use case would be which is the reduction of aliasing/moire/chromatic abberation in super slowmotion video and other cameras that employ some sort of lineskipping in their recording modes (ie. Freefly Wave in 2K mode, Sony RX10 IV, Sony RX100 VII, Sony FS700, FS5 HFR modes). I believe taking that Starlight file and doing a 2X Proteus pass with some dehaloing and adding film grain would enhance it even further and make it look more natural. The other models (even Artemis AA) donā€™t seem to do well in terms of reducing this kind of aliasing/moire in severe line skipped video but Starlight is def. doing it much better! Artemis LQ and Artemis Strong Halo comes close, but it removes aliasing/moire in some areas while not doing it in other areas of the image making it look uneven. Looking forward to see how Starlight progresses and hoping for a local model or new model that focuses more on improving aliasing and moire reduction.

My current upload seems to be stuck on ā€œinitializing resources.ā€ Cancel or just let it run?

Should be good to just let it run, weā€™ve seen a spike in requests over the last 30mins so additional machines are spinning up now

Iā€™m not really a fan of cloud processing and paying for an intermediate coin currency that lowers the inhibition threshold to pay too much for a service. We went through this already a decade ago with Xbox and Nintendo points.
I rather invest in and own good hardware and have local processing.

Back to topic: The model works like magic for early CGI work: Topaz Compare
I look forward to get a smaller variant of it for local use.

2 Likes

Awesome sample!

I wonder if itā€™s possible to apply for Early Access without owning a X account.
Directly linking to that shady platform from inside the application is already weird.
Had to manually enable the domain in my filter settings to ā€œLearn Moreā€. :sweat_smile:

You are already in the beta group, you just need to download the installer or use the web app linked in the post

So this is a clip I recorded with one of my super slowmotion cameras. The resolution was 640 X 360 and at this resolution, I can shoot at 5899 fps which when played back on a 24 fps timeline, you get around 245 times slowmotion! I missed focused on the bee very badly and I have to say Starlight did a decent job with it despite me missing focus and the very low input resolution. Iā€™m sure with a few tweaks in post I could get the file to look a little better, but I think Starlight is headed in the right direction (at least for this scenario) as the older models I have tried with this clip havenā€™t come anywhere near this close! I tried Artemis LQ, Strong Halo and Rhea and the bee is still blurry. Even if I would not use this shot professionally, for experimentation with slowmotion wildlife, the Starlight file is def. the one I would choose for viewing.

3 Likes


Tried to upload video via web app and it failed

1 Like

Yes Topaz Compare

2 Likes

I just used 300 points to do a cloud processing of the whole early CGI video:

Original video: https://www.youtube.com/watch?v=JOVmmml7PCs

Enhancement I did with regular TVAI: https://www.youtube.com/watch?v=ry2xKqlqmT8

Project Starlight: https://www.youtube.com/watch?v=WH4XjO-jqBg

3 Likes

Did project Starlight use the original video or the one you enhanced with TVAI. Did you use Gaia in TVAI?

Project Starlight used the original video as an input.
The regular TVAI is chr2_nyx3_gcg5. A perfect solution would have been using Gaia, Proteus and Rhea for different scenes of the video. But I didnā€™t feel like editing it together afterwards.

One more example using CGI / Gaming Footage:

Original: https://youtu.be/Pgg3VG7Hug8
Project Starlight: https://youtu.be/xe3Kh9PmMXk

Still feels like magic what kind of fidelity it can deliver using heavily compressed source material.

1 Like

<Imagining additional 80GB H100s jumping into the fay> :slight_smile:

1 Like

One example of Sony RX10 III 960fps slow motion. Temporal coherence is very good, and very important for this sorrt of footage.

2 Likes

Nice to see another Sony RX10 user on here! I did more tests as well and itā€™s pretty consistent when it comes to reducing the aliasing/moire/noise artifacts in HFR mode, but it does tend to oversharpen shots too much where itā€™s akin to an unsharp mask or convolution sharpen plugin cranked way up. If they can find a way to make the restoration detail more subtle (ie. Artemis or Proteus) while keeping the aliasing/moire reduction properties of Starlight, it will look even better.

Below is an example of what Iā€™m talking about where it does decently reducing aliasing/moire but the restoration of detail is too aggressive and the banding in background is very posterized looking. Footage is from my Sony RX10 IV in 480 fps mode. I believe with more training Starlight will eventually get to a point where the detail restoration isnā€™t too bold or oversharpened. They can try to add some dehaloing elements from the Proteus slider to add more naturalness to it. The dehalo slider really helps aleviate that ā€œcrunchyā€ compressed look, and I hope we can get an improved dehalo element in all the models.

I agree with you, and itā€™s especially a shame about the poor quality of some backgrounds. If they can improve these aspects, we can breathe new life into these magnificent cameras that Sony has unfortunately forgotten to update.

1 Like

Yes, I hear you! Iā€™ve been waiting nearly 9 years for Sony to improve the high frame rate recording modes! And yet Sony started to remove the HFR feature out completely at the higher settings with their Sony ZV1 Mark II and also their successor Sony HXR NX 800 camcorder so itā€™s sad they didnā€™t care to improve the resolution or improve the fidelity in the 480 fps and 960 fps mode in order to give Topaz VEAI a higher input! But I believe Topaz, like you said will help breathe new life into our Sony cams that do these special slowmotion recording modes and make them much higher fidelity. Just needs tweaking which I believe they will be able to figure out.

Itā€™s true, in VHS quality terms, that video would be really good. Though, the blurriness and fringing indicates ā€˜analogue eraā€™ of video processing/recording. :slight_smile:
Iā€™m aware that VHS tape usually has way more ā€˜visual damageā€™, and Starlight has to rely on multiple frames of ā€˜thatā€™ to recreate a single frameā€¦ :slight_smile: