Scene change detection progress

[jojje]

Continuing the discussion from Topaz Video AI v3.2.6:

@DaveL

TVAI would have to continuously monitor the rate of content change from frame to frame and be able to pick it up if that rate suddenly increased then immediately returned to its previous rate. So it might be a slow process and usable only for short clips but still, we’d have the option.

No, it’s super quick. One of the fastest video operations possible. I’m using a naive mean SAD on the next frame (one pixel diff operation per frame). This gives me SAD values for next frame and previous frames which I can compare against the current. With a very simple and cheap decision tree, it’s able to very accurately detect not only scene changes, but also duplicates. Took me 1h to implement the SAD computation. Training the model took a few more.

If topaz releases this in steps, they could spend 1h doing and testing the SAD calculation, 2h CUDA:fying while another person exposes a static threshold for their FFMPEG plugin. A third person could at the same time create a threshold input field and a toggle in the UI. Total estimated engineering time, including testing and code review: 2 days. Add QA and “mgmt meetings” to that, so multiply by PI. So one FTE week to get this out in a production ready state.

A subsequent release could train a model to (like I did) to detect the thresholds automatically, so zero user threshold configuration would be needed.

Point is, it’s a solved problem, trivial to implement and cheap. Costs almost nothing resource wise to run. It’s clearly just a matter of the Topaz backlog (other priorities) that prevents this from getting done.

[DaveL]

@jojje

Well it’s beyond what I can do, though missing frame detection and recreation by interpolation seems to be possible from the GIThub discussion I linked to and that’s my main concern. They suggested that would be slow but may have used a different method of course - and ‘slow’ is relative anyway. Would your method work for that too?*

If it would work for missing frames and you post this information in the Ideas thread, maybe tag one of the devs then hopefully it will get some votes and can be added to the development list, as I think plenty of videos would benefit, from missing frame detection/interpolation especially when producing slow motion video.