Upscaling old DVDs to 4K (SciFi & Star Trek)

CodeGrey · June 27, 2021, 8:12am

Looks very impressive. Do you mind sharing your exact settings?

virtutis · June 27, 2021, 9:25am

First run through a two part TFM/TDEcimate. Mostly at the moment, just to confirm if the episode is almost entirely Film (which this one was). So this start bit may be convoluted slightly.

First Past, running through VDub:

SetFilterMTMode("DEFAULT_MT_MODE", 2)
D2VSource("D:\Video\Voyager\S04E20 Demuxed\VTS_03_1.d2v")

#Pass 01
    Checkmate()
    Bifrost(interlaced = True)

    tfm(d2v = "D:\Video\Voyager\S04E20 Demuxed\VTS_03_1.d2v", mode = 1, output="D:\Video\Voyager\S04E20 Demuxed\tfm_out.txt")
    tdecimate(mode=4, output="D:\Video\Voyager\S04E20 Demuxed\tdecimate_metrics.txt")
Prefetch()

Then I opened it for the second pass in AVSPMOD to let it generate the timecodes file, ie, using my VFR script below:

SetFilterMTMode("DEFAULT_MT_MODE", 2)
D2VSource("D:\Video\Voyager\S04E20 Demuxed\VTS_03_1.d2v")ConvertToYV12()
  
  Checkmate()
  Bifrost(interlaced = True)

  tfm(input="D:\Video\Voyager\S04E20 Demuxed\tfm_out.txt")
  tdecimate(mode=5, hybrid=2, cycleR=1, vfrDec=0, input="D:\Video\Voyager\S04E20 Demuxed\tdecimate_metrics.txt", tfmIn="D:\Video\Voyager\S04E20 Demuxed\tfm_out.txt", mkvOut="D:\Video\Voyager\S04E20 Demuxed\timestamps.txt", batch=true, tcfv1 = false)
 
Santiag()
MCDegrainSharp(csharp = 0.3)

Crop(6, 6, 708, 468)

Spline16Resize(720,540)
 
 GradFun3 ()

Prefetch()

Upon opening the timecodes for this one, it is almost entirely Film:

# vfr stats:  99.72% film  00.28% video
# vfr stats:  82473 - film  230 - video  82703 - total
# vfr stats:  longest vid section - 15 frames
# vfr stats:  # of detected vid sections - 30

So to save headache, I ignore the VFR encode (but entirely optional) and use an amended version of this script:

SetFilterMTMode("DEFAULT_MT_MODE", 2)
D2VSource("D:\Video\Voyager\S04E20 Demuxed\VTS_03_1.d2v")ConvertToYV12()
  
  Checkmate()
  Bifrost(interlaced = True)

  tfm(input="D:\Video\Voyager\S04E20 Demuxed\tfm_out.txt")
  tdecimate(mode=0, hybrid=1)

 Santiag()

Crop(6, 6, 708, 468)

Spline16Resize(720,540)

Prefetch()

Which I then use VDub again to export as an image sequence, now in CFR at 23.976. Given it was almost 100% 23.976 anyway, its easier in the long run - otherwise, if it is one of those episodes with a lot of 29.97 content, its a bit more complicated.

I take that, and feed into the new Proteus model in the latest Beta. 2.3.0.2b. This was the bit I wasn’t expecting. I did not have high hoped for Proteus but it seems to be based on the older Artemis so it can pull out a lot of details.

The settings I used were:

I am still playing a bit with it - it would not surprise me if I found some improvements, but it matched or exceeded my previous process whilst halving the time to complete it, so I am very happy for the moment playing with it and seeing how it goes.

After that, encode the images back and re-mux the audio. I am doing that in Premiere Pro atm, but whichever is easiest.

CodeGrey · June 27, 2021, 11:43am

Thanks for the detailed response. Very interesting. Actually I have been playing around with Proteus as well lately and must say, I do like the option to have a “custom made” template that I can adjust entirely to my own liking.
Personally, I don’t understand some people bitching about it, since it really provides you with a good alternative to “pre-determined” models - usually, I input my source, check certain still frames that are “typical” for the source (for example: when it is an episode that has a lot of exterior space shots or battle scenes, I take a look at two or three frames of those, hit “auto” and note down VEAIs suggestions and then start tweaking from there - same goes when the episode has a lot of actors shots or close ups).
Therefore you can take the programs recommendation but still adjust to your liking - it’s a nice compromise. There will never be a model that can “do it all”.

ppedro · July 4, 2021, 6:33pm

Hi,
thanks for your great work. But a question:
In your workflow at Step 5) MeGUI you wrote “Mode: slow down without pitch corection”, but in your example videos you are using pitch correction. I’m also using the PAL (German) DVDs of DS9 and without pitch corection the german voices seem to be too low.
I’m not sure about the original english audio but I think it also needs pitch correction when converting from 25->23,976 FPS.

CodeGrey · July 10, 2021, 1:20pm

Actually, due to the PAL speedup, voices have a “higher pitch” than the original. If you have never heard it otherwise, voices will seem “low pitched” after slowing the audio down, but some of the original VHS tapes I still own to this day had the “lower pitch version” on them which seems to suggest that this is the correct way to handle the audio track. When it comes to PAL DVDs, everything is adjusted to fit 25 FPS, which means speeding up the audio by about 4% I think. I simply “reversed” that.
Once you compare the NTSC to the PAL DVDs, the difference becomes (sometimes painfully) clear - with male voices it is not all that bad, but with womens voices or special effects like phasers and such it just sounds pitchy in a very annoying way, especially on Voyager.

Once I had heard the “NTSC-audio”, there was no going back for me.

CodeGrey · August 23, 2021, 1:35pm

Hello community.

I have something new for you, upscaling both Star Trek Deep Space Nine (Pilotepisode) and Babylon 5 (Episode 1 of Season 1).

First of, my samples on Star Trek Deep Space Nine (password is always “DS9Up”, without quotation marks):

Comparison Images (DVD on the left, upscale on the right):

Videosample (downsized from 4K to 1080p for size reasons):

Second, here are my samples from Babylon 5 (my approach here is to take the 16:9 live action shots from the DVDs and replace the CGI/VFX shots with the 4:3 HD remastered versions - those are stretched to 16:9 without cropping ; I know, not everyone will like that, just wanted to show an option here). The password here is always “B5Up” (without quotation marks):

Comparison Images (DVD on the left, upscale on the right):

Videosample (downsized from 4K to 1080p for size reasons):

Have fun and let me know what you think.

CodeGrey · August 31, 2021, 1:40pm

Hello there community.

I thought I would share something a little different with you today. It is something related to the SciFi series Babylon 5 again, but from a slightly more unique angle.

There were TV-movies produced for the series. One of those movies, called “The Gathering”, was intended as the original Pilot to the series and was produced in 4:3 aspect ratio. Furthermore, there was a special edition released on DVD of the same movie with altered scenes/cuts/music.
Since that release was (as far as I know) not available in PAL, I worked from the NTSC DVD.
I wanted to avoid the jerky motion that would have come with a self-created pulldown/deinterlace, so I actually deinterlaced with full field restoration, resulting in a video with 59,94 FPS.
Afterwards I ran it through my usual insane Upscale and Enhancement and came up with something that I thought worked well enough.

Here is the videosample of that (1080p downsized for size-reasons ; password: B5Up):

Comparison images between the NTSC-DVD (left side) and my upscale (right side; password: B5Up):

Let me kow what you think.

cypher · September 9, 2021, 1:39am

VideoEnhance let’s you export a preset for the model settings. Could anyone provide an export of the settings you guys are using for you upscales?

CodeGrey · September 11, 2021, 10:35pm

Hello community.

I have something new for you, upscaling Star Trek Voyager (Pilotepisode). Password always: VOYUp.

Comparison Images (DVD on the left, upscale on the right):

Videosample (downsized from 4K to 1080p for size reasons):

Let me know what you think!

CodeGrey · September 16, 2021, 4:18pm

Sometimes, I do change the models and settings slightly if the output doesn’t look right. So even if I were to provide the models, they would not apply 100% in each case. Actually, every episode requires a little tweaking.

CodeGrey · September 16, 2021, 4:25pm

Hello guys. New stuff is arriving.

Let’s start things with Babylon 5 - Season 1, episode 2 (Password: B5Up):

Comparison images:

Video Sample in 1080p:

Let’s continue with Star Trek DS9 - Season 1, Episode 3 (Password: DS9Up):

Comparison Images:

Video Sample (1080p):

And let’s finish things with Star Trek Voyager - Season 1, episode 3 (Password: VOYUp):

Comparison Images:

Video Sample (1080p):

Let me know what you think!

richardcrawford0 · October 9, 2021, 7:26pm

Thanks, going to keep this 10-bit tip in mind!

virtutis · October 13, 2021, 12:55am

I mentioned in the other post that also had Voyager I would upload a sample as well, but this thread is more current, so posting here.

https://drive.google.com/file/d/1_2KSEeyiIFz6uaddnxNGfk6Z2WZhsyLs/view?usp=sharing

This is essentially the same extract for Caretaker, just with the best results I have had for me personally, which is a bit different style. Note that this episode is handled unique by me and differs from all other episodes due to its more noticeable VFR and for some reason, entirely different scripts needed compared to other episodes.

I am still interested in the specific avisynth (or similar) filters you use for the CGI sequences. Occasionally I can get it alright for me, but I have seen time and again other people post CGI sections that look better and I can’t seem to re-create them.

I was at least happy with this episodes CGI in the end - enough to hopefully not have to revisit the episode as a whole again anytime soon… >.>

It drives me nuts that I can tweak the input to where the starfields look glorious - but the ships look crap, or make the ships look ok, but the star fields behind look like muddied mess, but can never get the majority of the ships to look very crisp. Anyway, looking at episode 3 for Voyager in the CGI sections you have a crispness on the ships that I sometimes have issues replicating, so if you are fine with sharing the pre-processing script used there, I would interested.

I will then just experiment inside TVEAI as I always do looking for the best output and see how it compares, and if works better, shamelessly steal it

CodeGrey · October 17, 2021, 4:19pm

Hello there community, I have been away for a bit, but I’ve got some quality updates headed your way.
I have recently adjusted my workflow to better handle the DVD-Source.

Here are new comparison images from the DS9 pilot:

CodeGrey · November 7, 2021, 5:27pm

Hello Topaz community. Have been tinkering with my upscales. New stuff on the way. Let me know what you think.
New comparison images below.

Show: DS9, episode 3x21 “The Die Is Cast (Part 2)”

Show: DS9, episode 5x26 “Call to Arms”

Show: DS9, episode 7x25;26 “What You Leave Behind (Parts 1&2)”

Show: VOY, episode 3x26 “Scorpion (Part 1)”

Show: VOY, episode 5x06 “Timeless”

Show: VOY, episode 7x25;26 “Endgame (Parts 1&2)” - Note: Please excuse the Borg Queen looking green and then blue in one image, I accidentally used the wrong image, sorry.

So, what’s your verdict? I have been doing filtering wirh StaxRip and Hybrid, followed by layer-based work in DaVinci in order to combine multiple AI models from VEAI (Gaia HQ, Artemis LQ and MQ as well as Gaia CG).

CodeGrey · November 7, 2021, 8:21pm

And here I am following up with the corresponding video samples (free to download, no password required):

Show: DS9, episode 3x21 “The Die Is Cast (Part 2)”
DVD: https://www.dropbox.com/s/1gf7zlyjkdaq1ex/DS9 - 3x21 - The Die Is Cast%2C Part 2 (SD-DVD).mkv?dl=0
Upscale: https://www.dropbox.com/s/s8v05sjkwrhy1js/DS9 - 3x21 - The Die Is Cast%2C Part 2 (4K-Test).mkv?dl=0

Show: DS9, episode 5x26 “Call to Arms”
DVD: https://www.dropbox.com/s/rr8wv0t1ujygrcm/DS9 - 5x26 - Call to Arms (SD-DVD).mkv?dl=0
Upscale: https://www.dropbox.com/s/erakyiabempe4u8/DS9 - 5x26 - Call to Arms (4K-Test).mkv?dl=0

Show: DS9, episode 7x25;26 “What You Leave Behind (Parts 1&2)”
DVD: https://www.dropbox.com/s/2r781sx8q7z1ae0/DS9 - 7x25%3B26 - What You Leave Behind (SD-DVD).mkv?dl=0
Upscale: https://www.dropbox.com/s/pr3gnnrq03l9rxe/DS9 - 7x25%3B26 - What You Leave Behind (4K Test).mkv?dl=0

Show: VOY, episode 3x26 “Scorpion (Part 1)”
DVD: https://www.dropbox.com/s/2eq7j3gk5djrsk7/VOY - 3x26 - Scorpion%2C Part 1 (SD-DVD).mkv?dl=0
Upscale: https://www.dropbox.com/s/ex11ftno848zsh3/VOY - 3x26 - Scorpion%2C Part 1 (4K-Test).mkv?dl=0

Show: VOY, episode 5x06 “Timeless”
DVD: https://www.dropbox.com/s/tt2nrzqfaouvfxn/VOY - 5x06 - Timeless (SD-DVD).mkv?dl=0
Upscale: https://www.dropbox.com/s/n7j4oc62m7k73ny/VOY - 5x06 - Timeless (4K-Test).mkv?dl=0

Show: VOY, episode 7x25;26 “Endgame (Parts 1&2)”
DVD: https://www.dropbox.com/s/ei5iphako0u9sje/VOY - 7x25%3B26 - Endgame (SD-DVD).mkv?dl=0
Upscale: https://www.dropbox.com/s/gnnprmm3mwgpfly/VOY - 7x25%3B26 - Endgame (4K-Test).mkv?dl=0

virtutis · November 10, 2021, 3:39am

Now I know who you are on Facebook - I didn’t recognise the name on Facebook, but I do recognise those examples ^^.

The Voyager videos are looking good, the only issue I have in comparing is that all three scenes chose are very dark scenes, which helps a lot in hiding any issues - I basically can’t really tell easily how the end result looks from the videos, but generally look good.

My only concerns remains the same as on Facebook - there is a strange resolution change in some of the comparison images that I cannot fully work out if its an artifact of the comparison, or some part of the process. I know that the images have to be adjusted from the source (and cropped in a lot of episodes to get rid of noise) so keeping the original resolution after upscaling can be painful, but some of these comparison images show stretching of the image distorting the original resolution.

Is that just in these comparisons?

CodeGrey · November 10, 2021, 7:10am

Actually, I did chose dark scenes on purpose to challenge the AI models - some times dark scenes or scenes with bad lighting cause the models to lose some fine detail, but I think they did rather well.

As for the resolution change: I did not always manage to capture the exact same frame, so perhaps that is the reason. I also did some slight cropping, since the borders are not always consistent (especially left and right side), which can be distracting. And since the original file resolution after crop (PAL-source) is 720x536, which would be an odd number for the upscaler and I want a “clean” 200% two step upscale, I adjust to bring it back to 720x540.

I know, we have a slight difference in resolution there, but I am careful to keep the pixel/aspect error ratio below 2% (otherwise my software would render anamorphic, which I don’t want, since I want the file to exist in a way that would allow for classic 4:3 viewing in 2880x2160 - of course one could use an aspect ratio of 16:9 or a stretch function on the playback device to fill the screen, but that’s up to the person who watches), so it’s visually not too noticeable. Of course you can spot the difference in direct comparison.

virtutis · November 10, 2021, 8:04am

What I was referring to wasn’t the usual adjustments to pixel aspect ratio, the image looks like it was resized into the wrong aspect ratio, which was my concern.

This image comparison, shows no change in image vertical movement, but from the middle to left and right images her head is stretched in both directions - basically a small but same result of trying to display a 4:3 image in 16:9 by stretching - just not to the extreme of this kind of stretching.

In my experience when cropping the edges to get rid of anomalies at edges then try to retain the correct aspect ratio, this occurs with me when I crop the edges and then resize the image back to the same width without applying the same adjustment vertically as horizontally, so in the end the entire image is stretched horizontally to fill in the gaps. I just cant tell from the images if the before image is the culprit and the after is the fixed one etc. Just wanted to mention in case it was the after as the whole episode will be essentially slightly stretched.

I just wasn’t sure if it was all images, as not all seem to display that and its hard to tell with the before and after when the before is taken prior to any processing - my before is usually the cleaned output from Avisynth original, so aspect ratio cropping etc are all done already meaning the upscale comparison should always look virtually identical, but yours don’t so I assume the before images are not entirely the same image input to the upscaler (or screenshotted, I am not sure which).

The dark sections comment is interesting. I have never seen the upscaler struggling much with darker areas - the darkness always seems to help hide anomalies for me. I find the biggest giveaways of processing always are in the lighter areas, especially faces. Faces moving to background that are lit well can be disastrous but the darker scenes hide that.

I also seem to recall, unless I am mistaken, you mentioning the process intentionally darkens the scenes further which can make them a bit dark in some place.

I can’t do a “great deal” of direct comparisons with your examples yet - as I order in the NTSC versions, I only have 1-4 seasons in NTSC and season 4 will be redone again soonish as its been so long since I did those, I need to re-do them with better methods. ATM finished season 1 and starting season 2.

Also I know you do 4K so mine will be different in any event. As my Scorpion episode was done long enough ago I am not happy with it, this extract was yesterday on my current best process as a comparison, though from episode 2, season 2.

https://drive.google.com/file/d/1LF39TkEyofWxjXCtSI032VlfEHxqwY4j/view?usp=sharing

I have a feeling I may still re-do the episode in another 12x months as something will happen and I will find a better solution…

I am never happy with the CGI. Occasionally it feels like I have a good result, then it feels like it goes backwards. As for the live action shots, it always seems to come down of the lesser of two evils. Still searching that mythical single process that works on most episodes and doesn’t need to be fiddled with every 3x days…

CodeGrey · December 6, 2021, 8:58pm

Hello community. Wanted to share something that I had “left over” from my upscaling trials (testing the workflow that has now become my standard), so you know I am still at it.
Please forgive me if I haven’t always hit the exact same frame or if the crop is a little off (I forgot to change the values), but since this was short ordered cooking, just focus on the visuals, ok?
New comparison images below.

Show: DS9, episode 1x06 “Captive Pursuit”

Show: DS9, episode 3x01 “The Search (Part 1)”

Show: DS9, episode 6x26 “Tears of the Prophets”

Show: VOY, episode 2x21 "Deadlock”

Show: VOY, episode 4x09 “Year of Hell (Part 2)”

Show: VOY, episode 6x26 "Unimatrix Zero (Part 1)”

And here you have the corresponding video samples (both SD-DVD from PAL-Source and 4K-Upscale - please remember that PAL has a speed-up to 25 FPS, which I corrected in the upscale towards 23,976 FPS, which is why voices sound different); free, no password:

Show: DS9, episode 1x06 “Captive Pursuit”
DVD: Dropbox - File Deleted
Upscale: Dropbox - File Deleted

Show: DS9, episode 3x01 “The Search (Part 1)”
DVD: Dropbox - File Deleted
Upscale: Dropbox - File Deleted

Show: DS9, episode 6x26 “Tears of the Prophets”
DVD: Dropbox - File Deleted
Upscale: Dropbox - File Deleted

Show: VOY, episode 2x21 “Deadlock”
DVD: Dropbox - File Deleted
Upscale: Dropbox - File Deleted

Show: VOY, episode 4x09 “Year of Hell (Part 2)”
DVD: Dropbox - File Deleted
Upscale: Dropbox - File Deleted

Show: VOY, episode 6x26 “Unimatrix Zero (Part 1)”
DVD: Dropbox - File Deleted
Upscale: Dropbox - File Deleted

Let me know what you think!