Jerky Motion on One Video

Topaz Video AI General
1

Hi. I have been using VEAI for about a month now. I used it to upscale episodes of a TV show from DVD quality to 1080 (keeping 4:3 aspect ratio - 1416x1080) using GHQ-5. They all look great! Except one episode. When I watch the original it looks fine, but the upscaled version has choppy playback when there is a lot of motion (like a camera panning across the scene.)

I’ve tried going back to the original source, re-encoding the source into another format before upscaling and it always has choppy playback. I’ve done hundreds of episodes of this show and watched dozens of them. I saved my default settings so they have all been converted exactly the same way, yet so far, this episode is the only one that exhibits this behavior.

I am running on Windows 10 Pro 64-bit with 64 GB of RAM, 10th Gen i9 and GTX3080 graphics card and using PCIe NVMe M.2 SSD.

Any ideas on why this one episode would have the issue and how I might work around it?

Thank you.

2

Are you using a variable bitrate codec setting? Panning left or right puts a huge load on your CPU/GPU, and often the bitrate if fixed is not enough to deal with rhe huge amount of changing data.

3

if you upload it somewhere we could examine the video file. :slight_smile:

4

Hi. Thank you for the reply. I am using the mp4 output from VEAI. I’m not aware of any way to tweak the encode settings for mp4 output. None of the other episodes seem to have this issue. Although, your question does offer an idea. I can try outputting it in a different format.

EDIT: Just started it, it will be done in a little more than an hour (getting .10 - .11 seconds per frame!)

5

Alas, I do not think I want to get banned from the forums on my first day. :slight_smile:

What I can do is upload the ffprobe output from the input file and the output file. (Except I can’t because new users cannot upload attachments.)

Okay, here is the stream section of the input file, including an error message. This was created with handbrake - mostly I removed the chapters to make it smaller and the filename to remove the name of the show:

  Metadata:
    major_brand     : mp42
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    creation_time   : 2021-03-06T16:15:15.000000Z
    encoder         : HandBrake 1.3.3 2020061300
  Duration: 00:25:33.01, start: 0.000000, bitrate: 1714 kb/s
 Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p(tv, smpte170m), 708x480 [SAR 8:9 DAR 59:45], 1546 kb/s, 24.01 fps, 59.94 tbr, 90k tbn, 180k tbc (default)
    Metadata:
      creation_time   : 2021-03-06T16:15:15.000000Z
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 160 kb/s (default)
    Metadata:
      creation_time   : 2021-03-06T16:15:15.000000Z
      handler_name    : Stereo
      vendor_id       : [0][0][0][0]
  Stream #0:2(eng): Data: bin_data (text / 0x74786574), 0 kb/s
    Metadata:
      creation_time   : 2021-03-06T16:15:15.000000Z
      handler_name    : SubtitleHandler
Unsupported codec with id 100359 for input stream 2

And here is the stream section of the output file:

  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf58.20.100
  Duration: 00:25:32.99, start: 0.041000, bitrate: 13193 kb/s
  Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p(tv, smpte170m), 1416x1080, 13092 kb/s, 24.01 fps, 24.01 tbr, 570873562.00 tbn, 48.02 tbc (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 95 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]

Thank you.

6

I am not familar with reading Metadata I am afraid. :neutral_face:

7

.mov did not help, but it DID create a really large file.

I am doing .png now.

8

This is weird. It is rendering to png now. I just watched a section where I notice jerkiness. As it leads up to it, I can step through one png file at a time and each one is different than the one before, but when it gets to what I notice as the jerky part, there are 2 identical pngs before the picture changes. It’s like it’s duplicating a frame and dropping another.

I can actually take the original video and look at it side-by-side with the png files and see that frames are missing and duplicate frames are inserted for the missing frames.

What could possibly be causing this?

Thank you.

9

The Unsupported Codec is in all likelihood the cause.

10

But all of the episodes have that. It was encoded using handbrake. Does VEAI use ffmpeg? on the inbound side? I will try upscaling the original .mkv file and see if that helps.

11

either the output framerate does not match the input framerate exactly or the output is told to match the length of the audio which may stretch the video a bit by inserting unwanted frames. O.o

12

I cannot affect those settings. I can see that when encoding, it shows the frame rate at the bottom and they are the same. The ffprobe shows the frame rates are the same, and that the length is off by .02 seconds, which at the 24.01 fps works out to less than half a frame.

Seems like there is something about this ONE episode that is causing issues. Seems like Topaz might be interested in looking into what might be a bug in their software.

Thanks.

13

have you tried to play the video with different video players? I sometimes have videos that don t run perfect in one but smooth in another …

14

Is it possible that the original input, before handbrake, is interlaced? And do the jerky scenes play OK after handbrake but before Topaz (or is the problem there already)? I have been struggling with a DVD film rip, trying to clean speckles, comets, scratches etc and getting the same effect you describe. It turns out that the original source was reporting TFF whereas it was actually BFF and the jerkiness was only obvious in fast motion sequences

15

Should be the problem with compression. Increase the compression may help. I personally use CRF 16.

17

Eight months after this thread was posted, I’m running into similar problems. I’m running VEAI version 2.6.1 on Mac OS. The video I’m attempting to enhance is old “Hi8” analog (interlaced) video that had been converted to DV format. When feeding this into VEAI, pans show a “jerk” every few seconds in the output whereas the pan in the original video is smooth. Exporting to JPEGs (to eliminate the variable of video encoding, bitrate, etc.) I find that every so often the same frame is repeated 3 times and then the fourth frame jumps ahead to catch up.

Things I’ve tried that have made no difference:

  • Changing the output resolution (I’ve tried both upscaling to 4k and keeping the original SD resolution)
  • Switching AI models (tried both Dione and Chronos)

Any ideas here? This is happening with all the pans from this type of input source.

18

hola, I work on 8mm sd and hi8 videos also for a few years, the best solution for me is to deinterlace with the gpu with vegas pro and bring out the video in uncompressed avi by re-enterlacing with vegas, then with veai use dione tv or dione dv. but currently these models produce flicker on the movements that are unpleasant.

19

Thanks for the suggestion to get the source video into an uncompressed state, that appears to have done the trick! I’m working on a Mac, so here’s what I did:

  • Added the video clip in the Compressor application
  • In Compressor, applied the “Uncompressed 8-bit 422” setting
  • Ran the batch job in compressor

The above gave me an uncompressed (but otherwise unmodified, still interlaced, 29.97fps) version of the clip. I then imported the uncompressed clip into VEAI, applied the Dione Interlaced DV model, and rendered. The resulting upscaled clip is progressive at 59.96fps and the pan is now smooth (no jerks/repeated frames).

This adds some steps the the workflow, which I’ll see if I can streamline a bit, but solves my problem.

1 Like
20

Great that this is solved;) Do you also have this problem of flickering on the movements anyway? ps: we can also produce the video by re-enterlacing into mp4 or avi yuv, as long as in the project we deinterlace and re-interlace each image correctly

21

I haven’t noticed any flickering on movement, but then I’m still in the early stages of trying to upscale my old video. We’ll see how it goes as I dig into it deeper.