Hi Peter,
I assume the DVD in question is Progressive, if not, you will have to Deinterlace it first in Handbrake (or Shutter Encoder with BWDIF as Deinterlacer). To save the deinterlaced DVD video in lossless format in Handbrake you have to encode it with x264 codec with Ultrafast preset, Tune: None, Profile High444, Level: Auto with Constant quality RF0. (Only High444 profile in x264 is lossless for DVD and other 8-bit video formats, it is supported by Topaz Video, but many physical video players won’t be able to play it). In Shutter Encoder you can use FFV1 8-bit lossless codec, also supported by Topaz.
Now, a pleasant looking full HD upscale from a DVD is not always possible because some DVDs are released with extremely aggressive denoising and sharpening applied which destroy a lot of fine detail in the source that AI models are supposed to improve and in case of oversharpened video they often hallucinate and produce extremely sharp and distorted details.
Assuming the DVD doesn’t suffer from very agressive sharpening/edge enhancements you can try these combinations:
-
For a noisy DVD that looks sharp or a bit too sharp: two steps upscale using second enhancement. First model: Artemis Aliasing or Moire with Recover details 100%, Second enhancement: Proteus v3 with Recover details 0% (or Proteus v2 if Proteus v3 amplifies the noise too much). Sometimes the result looks best if you only upscale the DVD to 120% or 125% if the DVD source was sharpened too much and when any higher resolution only distorts and damages the original details.
-
If the settings above don’t look as any improvement, for example the image looks too blurry in 1080p, try the following settings: First model, Proteus v2, Recover details 100%, Second Enhancment Proteus v3 with Recover details 0%. You can also try Proteus v4 as a first model, but it is often too agressive on some DVDs and damages a lot of fine details because of its stronger denoise filter.
-
Iris Medium Quality with Recover details 60% or more, up to 100%, if it distorts fine details with Recover details 60%. It is a very slow model, sometimes the result looks better than Proteus if there are many scenes when you can see faces from a very close range (best model for a footage with a lot of large faces). It works very nice for such upclose scenes, not so much for usual scenes with small faces.
-
Proteus v4 one step upscale. Usually looks decent/pleasant for many DVDs, but only if you limit the upscale resolution to 720p. The setting for Proteus v4 720p: Recovery Details 0%, or you can try 10% or 20% if the result looks too clean with 0% Recover details to preserve some of the original noise. Sometimes you can do some edge enhancement/sharpening and futher denoise for the 720p Proteus v4 upscaled video by using Artemis High quality v10 model with Recover details 100% and keeping original resolution (no futher upscale, and in this step you don’t use Artemis model as second enhancment, but as a futher one step upscale). Usually though you wouldn’t need the extra sharpening with Artemis because allmost all TVs now have built-in sharp lanczos upscalers, (and on PC you have Kodi when you can choose the upscale mode for each video, DXVA usually is the sharpest, followed by Lanczos 3 and other Lanczos options.) The extra sharpening offered by Artemis may be usefull only for players/TVs that use Bicubic to upscale videos.
Now the important step: always save the upscaled video in lossless format - FFV1 8-bit, and later reencode it (if you want to save space) in Handbrake with x264 codec in Very Slow mode (Tune: None, Profile High, Level: Auto) with Constant quality RF15, frame rate same as source, and in the Additional Settings text box write “aq-mode=2“ (without quotes). For some reason the default x264 settings use aq-mode=1 and it looks worse to my eyes, especially for upscaled videos. In Handbrake’s filters settings make sure all the filters are off. For audio choose passtrough the original AC3 audio without converting, or convert it to AAC 320kbit to make it compatible with some devices/players that don’t support AC3.
If you save the upscaled video with GPU accelerated H264 or H265 encoders you would need an extremely high bitrate, probably 4 times more filesize to achieve x264 CRF15 quality.
By the way, I have never had any success with the latest Proteus Natural model, for some reason it produces a very distorted results on many DVDs, but you can try it anyway with Recovery details 98% or 100%. Maybe other people can say when they use Proteus Natural, because so far it doesn’t look natural at all on my RTX3070.
I hope it helps. Best of luck with your DVD.