Quick algorithm to make the ETA more steady and accurate

Topaz Video AI General
1

After each frame is rendered, the time it took to render it is used to determine what the ETA is for the remaining frames. This unfortunately results in a very jumpy and inaccurate estimate, since each frame time differs slightly. This is a very common issue in pretty much all the software I use, and it’s honestly not that a huge deal. Still, it is a “quality of life” thing which I think your users would appreciate being addressed, and this solution really shouldn’t be too difficult to implement.

What I’m proposing is not to do an overall average of all the frames rendered, since that wouldn’t account for when render times change due to background processes. Instead, what I’m talking about doing is known as a “low pass filter”. It will give your ETA calculation a more stable and reliable average frame time to use. It will constantly adjust itself in case background processes impact render times, and it only takes one line to do all the work.

The variables:
CurrentFrameTime = The time it just took to render the currently displayed frame (this is the input)
FilteredFrameTime = Filtered render time which will be given to the ETA calculation (this is the output)

The algorithm:
if FilteredFrameTime == 0, then FilteredFrameTime = CurrentFrameTime
FIlteredFrameTime = (CurrentFrameTime * .1) + (FilteredFrameTime * .9)

Since ‘FilteredFrameTime’ uses itself in that second line, we make sure that the first time it runs (when it’s equal to zero), ‘FilteredFrameTime’ will initially be set to whatever the ‘CurrentFrameTime’ was. Without that first line, the ‘FilteredFrameTime’ would be slowly climbing up from zero. (Or it could also be a null or something else you wouldn’t want.)

In the main equation, the numbers .1 and .9 can be anything but they MUST both add up to 1. They are essentially the “strength” of the filter, and will affect how slowly the frame time adjusts itself. Having the smaller number applied to the input (CurrentFrameTime) instead of the output (FilteredFrameTime) will make the filter stronger and give much more useful results.

If the numbers were 1 and 0, then it would remove the filter altogether. If the numbers were .001 and .999 then it would still work, but it would be very slow to adjust itself, and ETAs for preview renders wouldn’t be that accurate. I think the first number should probably be somewhere between from .01 to .1 (and then the second would be 1 minus the first number).

Okay, thanks for reading all this.

2 Likes
2

I don’t see the point. Doing a Proteus job now. ETA looks to be dead accurate. Calculating ETA is a trivial operation, and really only inaccurate for, say, the first few hundred frames. After that – which is how statistics works – the mean time per frame becomes, per definition, more and more accurate.

3

I’ve never actually checked the initial estimate vs when it completed to see how accurate it is. I know with most encoders even though scene complexity can change the FPS tends to stabilize pretty quickly. so even if you have IPPBPBPPI or whatever there are so many frames that a pattern starts to emerge and the running time converges (is that the term?).

Have you waited to see the estimate after a few hundred frames have been processed? Doesn’t VEAI perform the same functions over the entire frame for all frames? Like if I do a regular denoise algorithm with some fixed settings on two files, same resolution, same format, same bit depth, etc. One is a noisy compressed jpg and the other is pure black, it takes the same time.

4

I’m working on a film that’s 3 hours long and the ETA is constantly jumping anywhere between 11 and 13 hours. And that’s after half an hour of processing. I wouldn’t be bringing this up if it was something that corrected itself after a minute. Besides, the fix is very easy.

1 Like
5

Using the Proteus model right now to tweak some HDV footage. I’m quite happy with the results and the processing time. However, I’m noticing a strange thing. I use multiple “desktops” or workspaces on this Mac. While Topaz is crunching on one desktop, I’m doing other work on another. Actually a couple of other desktops. When I switch to the dedicated Topaz screen, the ETA (at this point) shows 20 hours and change. However, as I stay on that screen the ETA drops quickly…down to 5 hours or so. This is very consistent, happening every time I switch. Obviously I would prefer 5 to 20, but more importantly why is that happening AND if it is a real datapoint, am I slowing down the process by having it work away on a different desktop while I work elsewhere? Seems that if the ETA data is accurate, this workflow must be slowing down the app. Your input is appreciated.

6

lol, so that rendering that I was doing earlier with the ETA between 11-13 hours ended up only taking a little over 5 hours. So yeah, the situation is actually a bit worse than I thought, similar to what @jeff.hanley is talking about above. While it was rendering, I was just watching YouTube and scrolling through social media.

In case Topaz needs specifics: Windows 10 Pro (64bit), running VEAI v2.3.0. I’m on a Ryzen 5 3600X with an EVGA RTX 2060 KO Ultra and 32GB of RAM. I was doing a Strong Dehalo render on a 3 hour long, 22GB h.264 file which was being cropped and resized with an AviSynth script.

7

Slightly off-topic, perhaps, but I am very curious as to how you accomplished this. I am familiar with avisynth, of course (I use vapoursynth myself, but same deal), and crop filters. But how did you supply the output of avisynth to VEA? (Other than having to resort to a 500G rendered, lossless pre-pass first).

8

You can drag an .avs file right into Topaz VEAI and it will treat it just like it’s a video file. It’s all very seamless.

I use StaxRip, myself. Just found it not too long ago. It’s pretty great, but a little clunky.

1 Like
9

Thank, bro; good stuff. :+1:

10

Funny, just started a 4K render from a 480p source. This time, the ETA fluctuations are much higher than with my 1080p job. My mind tells me this makes sense, kinda (as the upscale multiplication factor is much higher); but, from frame to frame, the ETA difference is as wide as ca. 2 hours (with 7,000 frames done now). And, worse, still set at ~16 hours, like it was at frame 1 (that’s no good, of course, ETA should start to decrease).

1 Like
11

for me it will say 10 hours and still say that after 30 minutes, and it ends up completing in 2. Makes it hard to plan PC availability.

12

Fingers crossed this is addressed in 2.5 which we should be getting… soon?

1 Like