Workflow with a lot of footage

Does anyone have a suggested or preferred workflow for editing (relatively) large amounts of raw footage?

Here is my current project:
My dad is turning 75 in 2 months and I just got about 20 hours of his Video8 footage digitized into AVI (without him knowing it). The video input: 720x576 @ 25 FPS, 1.07:1 PAR. My plan is to use Final Cut Pro to edit the footage down to a manageable 10-15 minutes video that can be shown during his birthday celebration as a big surprise. I’ll be using an Apple M1 Max MacBook Pro to do the editing and work in TVAI. This will be the first project I use TVAI on.

  1. Would you recommend upscaling clip by clip after I have gone though all the footage and selected what I want to use or do it “tape by tape” (upscaling everything before selecting clips)? (What’s the trade-off between time and quality?)
  2. Do any of you have a suggested workflow for the upscaling?
  3. Can I get this up to 4K or will that be tough?

Any tips or advice will be greatly appreciated.

Highly recommend that you do a first pass cut in FCP to trim down the footage to determine what you need to upscale or enhance using Video AI. The processing times can take awhile when upscaling to 1080p and even longer when going to 4k.

Depending on the quality of the tapes and the digital files you get once you bring them in, try using the Dione or Iris models to start with enhancing the videos and upscaling. There will be some trial and error as each clip could need different models and settings.

Feel free to email the support team directly if you get stuck or need them to test a clip for you to help. help@topazlabs.com

1 Like

# 1

The former. You still need to go through the clips and select scenes and shots at one point or another anyway, so it makes no sense to waste time and electricity on upscaling stuff you’ll just be throwing away later on. Upscaling is very time consuming, so only do it on what you actually need.

#2

I tend to just load up the selected shots (scenes) in TVAI [*], scrub to around a bit trying to find some “representative” point in the clip such as the second to worst [**] looking point in terms of artifacts I want to get rid of (blocky, unsharp, haloed …) and use that as the baseline for manual setting of the entire clip, then set upscaling to 2X.

  1. I first check how Proteus-4 performs, and then Iris LQ. 95% of the time Proteus is the answer, so I often skip Iris all together.
  2. For each model I click the Estimate button and produce a 5 second preview.
  3. If it’s too soft I crank up recover details and a bit of sharpness.
  4. If that adjustment introduces halos, or there are already edge halos present in the source clip (common in old DVDs that were VHS captured), I increase Dehalo until I find a decent balance between getting rid of the halos and not softening the image too much. Typically if you increase Dehalo, you also have to increase Recover details and Sharpness as well, as Dehalo softens the entire image, so you need to compensate for that using the former two knobs. Rule of thumb; If you increase dehalo, increase the other two about 2-3x the dehalo increase.
  5. Once I’m happy with the result, I queue the entire clip (shot) with that setting, and move on to the next. Then repeat step 1 for each shot.
  6. I press Export [***], to export each clip to mkv (or mp4 if your video editor doesn’t handle mkv) with very high quality (low compression).
  7. I finally take the resulting upscales and throw them into the video editor for final project composition.

#3

I would say it will be tough. I’ve never seen a successful upscale of 480p to 4K. But “successful” is in the eye of the beholder.

If you look around the forum, some people are posting screenshots of babylon5 and startrek over and over, as they’re using that as their personal test baselines when fiddling with the application’s settings. As you see in all of them, they have a very unnatural AI-upscale type of look, and the reason for that is two-fold; 1) there simply isn’t enough information in 480p to faithfully reconstruct what a 4K camera would have captured. 2) The TVAI model’s haven’t yet reached a point where they’re able to do a 4x upscale leap. They’re about at the level where they can do 2x, but then things get wonky. This is core problem the Topaz company is trying to solve though, their Raison d’être if you will, and they are constantly iterating on trying to climb the scaling ladder ever higher, a tiny step at a time, but it is a very hard problem to solve.

Personally I find that 1080p is as far as one can get 480p (about 2x upscale) before my brain starts screaming “Annoyingly fake!”. But again, as I wrote, it’s all in the eyes of the beholder. For professionally trained eyes that know what natural looks like, how cameras, light, color and contrast work, it takes very few mistakes by the AI models to make the picture look unnatural and thus Not good/unnatural (and the Not Natural problem increases exponentially with each magnification / upscaling factor). Less trained eyes have a higher tolerance for the “Fakeness”, and for those eyes, TVAI 4K upscale “Looks Good”.

In short, try to do a 4K upscale and see if your own “eyes” deem simply a sharp image looking 4K-ish. If it does, go for it. Your dad probably is even less discerning than you. But if you find it just looks fake, then settle for something lower, like 2K or 1080p, and instead just let the player/TV upscale it on its own (typically they use non-“AI” bicubic interpolation which preserves the natural look, but doesn’t sharpen images). Often I find the latter to be much more pleasing, as from a distance, you can’t really see much difference between 2K and 4K. For me, even 1080P without the “AI fakeness” looks a lot better with bicubic upscale than a 4K picture where the TVAI models have been pushed beyond their capabilities. Let your own eyes decide how far you can push the upscaling before you think it looks bad, and settle for that as the upscaling amount.

[*] I actually use another tool for scrubbing, since it’s a LOT faster than TVAI. Then I press a custom developed keyboard shortcut that moves the TVAI time scrubber to the same position as the other tool is at. Avoids having to deal with the clunky TVAI UI more than I absolutely have to. Speeds up the selection process about 10-20x.

[**] I find the rule of thumb of going for “second to worst” statistically lands me at a representative segment in terms of what the overall TVAI processing result will look like. If picking “the worst” / most distorted section, it tends to over-correct the overall shot, leading to immediate “fakeness”. It’s just a rule of thumb though.

[***] Actually I don’t. I have another custom keybinding shortcut to export the FFMPEG settings to another custom program that queues and runs those commands, since TVAI is very unreliable in terms of exporting. Specifically, it sometimes crashes, or outright forgets / ignores your laboriously crafted settings for each clip and just uses some other random settings, leading to wasted hours of processed junk.

3 Likes

I agree with everything you say, I put my dv 720 x 576 videos in hd 1280 x 720, but I don’t go higher in definition because it looks so fake! just the fact of deinterlacing with qtgmc, even if it is the one that produces the least artifacts, and even if we deinterlace in “fast” or very fast" the details of the video will be less fine than the original video, and it is for this reason that now I deinterlace with vegas pro, even if it is not as good, proteus v4 is powerful enough to correct the artifacts that the deinterlacing of vegas pro produces. in short, in any case, the fact of upscaling in 1920 x 1080 or more, we lose more details and the video will necessarily look more “plastic”.

1 Like

This is a very important point you’re shining a light on. ML/AI models like Topaz’s are trained on a bunch of artificially created noisy video clips, and their reconstruction (fixing) is entirely based on the difference between their clean (ground truth) variants and the distorted ones.

When we run preprocessing through other tools before loading the clips into TVAI, we change the types of artifacts present in the source clips. What we’re essentially doing is introduce a new type of artifacting that Topaz doesn’t have in its training data, and consequently there’s no learned mapping from a specific type of artifact to a clean output in their models. This is a significant issue, as it leads to precisely what you point at; the TVAI models being unable to identify the very specific set of defects they’ve been trained on, and consequently are enable to find a mapping function that would remove (transform) the artifact. The observable result is precisely what you mention; lower fidelity output (less details as you put it).

For this very reason, I’m only using QTGMC for deinterlacing (with its image cleaning knobs turned way down), so that TVAI has as much of the source defects retained as possible. That way it’s able to produce more “detail” in the output, than if I was to also clean the sources while deinterlacing.

There are a few rather infrequent exceptions though. Sometimes the source footage is just too bad for TVAI to work with. Likely due to Topaz not having that specific set of distortions modeled for producing their “bad clips” training footage for their models. In those specific cases, doing a cleaning pass with QTGMC will produce clips with artifacts closer to what the TVAI models were trained on, and it will consequently let TVAI (e.g. Iris LQ) recover more features (details) from the original signal than if VAI had been fed the source clips with their prior unseen distortions directly.

But we’re talking horrendously noisy and compressed thumbnail sized source clips here. Definitely not the typical sort of clips most TVAI customers are operating on day-to-day.

1 Like

I appreciate the kind advice from all of you! This will save me a LOT of time.

I will make a rough cut first in FCP and then go to work with TVAI.

The workflow, thoughts and advice by @jojje is very appreciated, though I think QTGMC will be unavailable to me on a Mac. All of you have fine-tuned my expectations on how far I can expect to push the upscaling: it will save me from wasting precious time chasing a 4K upscale that most probably won’t pay off: I’ll limit myself to max 2X.

I’ve made some test runs in TVAI cleaning up detail and noise, without sharpening or upscaling and the results are formidable. I can now focus on the FCP edit knowing that I’ll be able to make the end product much more presentable in TVAI, when that time comes.

Again, thank you all for your help!

1 Like

I understand, I don’t use qtgmc anymore even if it remains the best for me in terms of cleaning noise and artifacts. Deinterlacing via gpu with vegas pro keeps finer details, with the same parameters as a deinterlaced video with qtgmc, it seems more real. and yes, I much prefer to upscale to 1280 x 720 even if it means letting my tv upscale the video to 4k on my screen, tvs upscale very well.

1 Like

One additional question:
I have now made a rough selection of about 40 minutes of clips, from what turned out to be 22 hours of footage. I will edit them down to a 15-20 minute long video.

Do you recommend finishing the edit, with effects and titles, etc. before doing the TVAI adjustments and upscaling the finished project, or to do the adjustments and upscaling on the 40 minutes of “rough” clips before starting the actual edit?

If your project consists of assembling a collection of many video clips shot over a long period of time, there’s a good chance that different clips will require different types of enhancement. Also upsampling and enhancement can sometimes do some odd things to effects and text.

So your best path is probably to enhance the rough clips, then assemble them and add effects and titles so any newly created content - which will be created at the final resolution and shouldn’t need any enhancement - doesn’t get run through VAI.

1 Like