VHS - Additional Filter in TVAI as Preprocessing - Proposal

Topaz Video AI Ideas
1

Hello.
In the past two years ago, I worked on improving VHS video material using deep neural networks. For this purpose, relying on VIMEO90K data, I created high-quality material, then recorded it onto a VHS tape, and played it back from the VHS tape, thus obtaining “damage” to the VHS material. With the help of a UNet-type network and regular MSE prediction, but after necessary “calibration”, I was able to create a deep neural network specializing in VHS.

Of course, I realize that TVAI is a much more advanced tool and trained on hundreds of thousands of hours of various types of VIDEO material. Nevertheless, I wonder if I could potentially share my created materials so that TOPAZ could further improve its models in terms of typical VHS? If so, I will gladly provide appropriate videos related to this.

Please respond to this idea.

2

Im not from Topaz, but pls can you share before/after screenshots? Would be cool to see

1 Like
3

OK, the materials are being uploaded. As soon as they’re on OneDrive, I’ll share the link.

1 Like
4

And as for examples of improving VHS materials with the help of the created network, I will upload them later today.

5

6 files:
https://drive.google.com/drive/folders/1t9x1jvEQ9IkYmzKoVUij7-OyL3lB06hU?usp=sharing
demo - VHS quality before any improvements
demo_preprocess - VHS material (preprocessing) - enhanced through a deep neural network trained on data based on VIMEO90K with VHS degradation.
demo_thm2_iris1 - VHS material enhanced by iris1 and thm2
demo_preprocess_thm2_iris1 - VHS material: preprocessing + iris1 and thm2

and new, a little experiments with chroma:
demo_preprocess_chroma - demo_preprocess but only chroma color channels (luminance from original demo)
demo_preprocess_chroma_thm2_iris1 demo_proprocess_chroma + iris1 and thm2

2 Likes
6

And materials based on VIMEO90K in original (VIM) and with VHS quality. This materials are after needed calibration with number frames and space calibration:

7

Outside of any deinterlacing necessary, I too have found that applying an aggressive denoise on the chroma planes prior to TVAI made a huge difference on VHS sources - leaving the luma plane untouched.

Here’s the FFmpeg filter…

-filter:v hqdn3d=luma_spatial=0x00:chroma_spatial=0x0f:luma_tmp=0x00:chroma_tmp=0x0f

The chroma_spacial and chroma_tmp values of 0x0f = 15 are much more aggressive than many folks use out there, but it did help reduce the artifacts created by TVAI without being too destructive.

hqdn3d is one of the FFmpeg denoise filters that can denoise the luma/chroma planes with different settings. Unfortunately it is unsuitable to be included in Topaz, due to licensing. It would be cool if Topaz would allow separate control over luma & chroma denoise.

OP - neat job on training your private model. Very cool. Did you use a Time Base Corrector (TBC) in your tests, either standalone or integral in a semi-pro VHS player?

1 Like
8

Thank you.
I didn’t use any hardware TBC, unless you count the fact that each frame of the original image has a frame number encoded in it in quite a large font. This frame number served me for a certain simulation of TBC, which I called ‘frame calibration’. It is a fully software operation, which I implemented with the amateur and semi-professional equipment I had available.

1 Like