Mac Mini M4 "Distributed" Processing

Topaz Video AI General
1

For those who might be interested, I found a way to use 2 Mac Mini M4s together to upscale SD videos at realtime speed using Artemis, Proteus or Iris. So I can “stream” the result in VLC after just a few seconds delay. I purposely set the Max Memory at 100% to minimise use of the GPU cores - so the total power consumption of the 2 Minis is less than 30W while upscaling with the benefit of no audible fan noise. Although one of my Minis is an M4 Pro, the other is a base spec M4 which is plenty fast enough for half the load. So 2 base spec M4s would be fine.

Using ffmpeg in the CLI, there are a couple of ways to split a video across 2 or more machines for processing and then recombine afterwards, but I found the “select” and “interleave” filters to be the easiest. (Another way is to use segmentation and concatenation.)

Below are the 3 commands I used:

On Mini 1 - upscale odd numbered frames:

ffmpeg -hide_banner -y -i “/Users/Andy/Desktop/Original.mkv” -filter_complex “select=mod(n-1\,2),tvai_up=model=alq-13:scale=0:w=1440:h=1152:blend=0.5:device=0:vram=1:instances=1,scale=w=1440:h=1152:flags=lanczos:threads=0” -c:v prores_videotoolbox -profile:v xq -color_range tv -pix_fmt p416le -an -fps_mode:v passthrough -colorspace bt709 “/Users/Andy/Desktop/Odd Frames.mkv”

On Mini 2 - similarly upscale even numbered frames:

ffmpeg -hide_banner -y -i “/Volumes/Andy/Desktop/Original.mkv” -filter_complex “select=mod(n\,2),tvai_up=model=alq-13:scale=0:w=1440:h=1152:blend=0.5:device=0:vram=1:instances=1,scale=w=1440:h=1152:flags=lanczos:threads=0” -c:v prores_videotoolbox -profile:v xq -color_range tv -pix_fmt p416le -an -fps_mode:v passthrough -colorspace bt709 “/Users/Andy/Desktop/Even Frames.mkv”

On Mini 2 - interleave upscaled odd and even frames, adding original audio - concurrently with the upscaling, starting after a short delay:

ffmpeg -hide_banner -y -re -i “/Volumes/Andy/Desktop/Odd Frames.mkv” -i “/Users/Andy/Desktop/Even Frames.mkv” -i “/Volumes/Andy/Desktop/Original.mkv” -r 25 -fps_mode cfr -filter_complex “[0:v][1:v] interleave” -c:v hevc_videotoolbox -profile:v main -tag:v hvc1 -pix_fmt yuv420p -allow_sw 1 -g 30 -b:v 0 -q:v 82 -c:a copy “/Volumes/SSD/Final.mkv”

After another few seconds delay, Final.mkv can then be viewed by VLC in realtime locally or via network. VLC is the only app I know of that periodically re-reads the file header allowing for the file to be viewed while it’s still being written. Out of interest, does anyone know of other video player apps which do that? It would be nice to have alternatives to VLC.

Andy

1 Like
2

On a similar note, I found that using one vanilla FFmpeg 7.1 process to decode the stream and piping those decoded raw frames to the Topaz FFmpeg binary gave me a 20% increase in speed on my M1 Mac. Same machine running both processes. Here was the command that I used for testing purposes (no upscale, just very basic AMQ enhance). I removed audio in both cases so the pipeline was focused just on crunching video frames.

$ ffmpeg -an -i infile.mp4 -codec:v 'rawvideo' -t 60 -f 'nut' "pipe:1" \
  | ffmpeg-topaz -i "pipe:0" -filter:v tvai_up=model='amq-12' -codec:v 'ffv1' -level 3 "./test_speedup.nut"

I’m assuming it is to do with one of thread allocation, buffering, or perhaps some decoding optimization in FFmpeg 7.1, but it was an interesting finding nonetheless. There may also be some speedups in converting to RGB in the decode instance, because the TVAI filters think in RGB48le. Using FFmpeg hardware decode didn’t increase performance any further.

Thought it would be worth sharing in case anyone is interested in attempting to replicate. Your mileage may vary depending on hardware, environment and model selection.

The idea could be expanded by offloading decode to something like an RPi with sufficient interconnect bandwidth, but I did not bother investigating further. It made me wonder whether with a 3 machine setup, one could use one for decode-to-raw, one for raw-to-topaz-to-raw and one for encode you could let the middle machine focus on doing just TVAI filtering.

1 Like