Topaz Video AI v3.1.5

TicoRodriguez · February 20, 2023, 8:56am

I tried a few things.
“Process” - “Show Export Command” to see what options are set for FFMPEG.

In this test case, since the video is in 640x480 SD resolution, BT601 is generally applied.
If it is not to be enlarged, it must be output in BT601, and if it is to be enlarged to HD resolution, it must be set to BT709.

Also, according to FFMPEG specifications, if BT709 is to be used, “scale=out_color_matrix=bt709” must be set at the end of the filter.

Test 1 Noise reduction only SD(BT601) → SD(BT601)

ffmpeg -hide_banner -nostdin -y -nostats -i Z:/MVI_2232.AVI -sws_flags spline+accurate_rnd+full_chroma_int -color_trc 2 -colorspace 5 -color_primaries 2 -filter_complex tvai_up=model=prob-3:preblur=0:noise=0:details=0:halo=0:blur=0:compression=0:estimate=20:device=0:vram=1:instances=1,scale=out_color_matrix=bt709 -c:v prores_ks -profile:v 3 -vendor apl0 -bits_per_mb 8000 -pix_fmt yuv422p10le -map_metadata 0 -movflags frag_keyframe+empty_moov+delay_moov+use_metadata_tags+write_colr  -map_metadata:s:v 0:s:v -map_metadata:s:a 0:s:a -c:a aac -b:a 192k -ac 2 -metadata videoai=Enhanced using prob-3 auto with recover details at 0, dehalo at 0, reduce noise at 0, sharpen at 0, revert compression at 0, and anti-alias/deblur at 0 Z:/MVI_2232_prob3_temp.mov

The out_color_matrix=bt709, which should not be set, is set.

Test 2 Noise reduction & enlargement SD(BT601) → HD(BT709)

ffmpeg -hide_banner -nostdin -y -nostats -i Z:/MVI_2232.AVI -sws_flags spline+accurate_rnd+full_chroma_int -color_trc 2 -colorspace 5 -color_primaries 2 -filter_complex tvai_up=model=prob-3:preblur=0:noise=0:details=0:halo=0:blur=0:compression=0:estimate=20:device=0:vram=1:instances=1,scale=out_color_matrix=bt709 -c:v prores_ks -profile:v 1 -vendor apl0 -bits_per_mb 8000 -pix_fmt yuv422p10le -map_metadata 0 -movflags frag_keyframe+empty_moov+delay_moov+use_metadata_tags+write_colr  -map_metadata:s:v 0:s:v -map_metadata:s:a 0:s:a -c:a copy -metadata videoai=Enhanced using prob-3 auto with recover details at 0, dehalo at 0, reduce noise at 0, sharpen at 0, revert compression at 0, and anti-alias/deblur at 0 Z:/MVI_2232_prob3_temp.mov

out_color_matrix=bt709 is added.

Test 3 Noise Removal & Enlargement SD(BT601) → HD(BT709) Encoder is changed

ffmpeg -hide_banner -nostdin -y -nostats -i Z:/MVI_2232.AVI -sws_flags spline+accurate_rnd+full_chroma_int -color_trc 2 -colorspace 5 -color_primaries 2 -filter_complex tvai_up=model=prob-3:scale=0:w=3840:h=2160:preblur=0:noise=0:details=0:halo=0:blur=0:compression=0:estimate=20:device=0:vram=1:instances=1,scale=w=3840:h=2160:flags=lanczos:threads=0:force_original_aspect_ratio=decrease,pad=3840:2160:-1:-1:color=black -c:v hevc_amf -profile:v main -profile_tier high -pix_fmt yuv420p -b:v 0 -map_metadata 0 -movflags frag_keyframe+empty_moov+delay_moov+use_metadata_tags+write_colr  -map_metadata:s:v 0:s:v -map_metadata:s:a 0:s:a -c:a aac -b:a 192k -ac 2 -metadata videoai=Enhanced using prob-3 auto with recover details at 0, dehalo at 0, reduce noise at 0, sharpen at 0, revert compression at 0, and anti-alias/deblur at 0. Changed resolution to 3840x2160 Z:/MVI_2232_prob3_temp.mov

out_color_matrix=bt709 has disappeared.

Simply put, the GUI side of TVAI is not properly configured for input and output color space.

I have requested this many times before, but TVAI is too careless when it comes to color.
It should be possible to know the input color space and output color space on the GUI.

q91632 · February 20, 2023, 9:57am

@kit434 @TicoRodriguez Why don’t you guys take a look at the chat history, your problem is just that you are using a wrongly decoded player (or configuration).

TicoRodriguez · February 20, 2023, 10:15am

I think the first story was about the color change that occurs when using the trial version.
There was some erroneous information that it was affected by the watermark, but I have confirmed in testing that the color does not change when the watermark is added.

As per the FFMPEG options I pasted earlier, TVAI is not able to properly set color_trc, colorspace, and color_primaries, so this is a problem before the decoder.

For HD video, scale=out_color_matrix=bt709 is required for RGB->YUV conversion and should be set as -color_trc 1(bt709) -color_primaries 1(bt709) -colorspace 1(bt709).

An example of a strange video that was created.
Normally, it would be a total, BT709.

q91632 · February 20, 2023, 10:39am

I have observed a slight color shift in the results of Video AI, it may be something wrong, but this is not the cause of his problem.

icc.kl.chan · February 20, 2023, 11:41am

There is an extent of crf support in hardware encoders as well
though they usually go by the name of cqp

JakSpoon · February 20, 2023, 11:59am

Correct, but hardware CQP is not comparable in export efficiency to the software CRF of encoders such as x264 and x265.

Paul_Coddington · February 20, 2023, 12:10pm

Noticing with live action video, flesh tones which appear to lean towards having a slight pink tint in the original shift slightly toward leaning towards having a green tint. Might be related.

It is a small shift, but clearly visible on a calibrated monitor.

Paul_Coddington · February 20, 2023, 12:15pm

Although overall performance seems good, there is an interesting problem where split screen preview plays back at different speeds on each side of the split (NVIDIA).

Paul_Coddington · February 20, 2023, 12:36pm

With my preferred media player (madVR-based), I have to go full screen to have my monitor change refresh rate to match the video.

In Windowed mode, the player has to cludge the video frames to force them to match the 60Hz desktop, which causes visible problems with sources that are not 30 or 60fps (eg: 24fps movies, especially noticeable on scrolling credits and panning scenes).

Your viewing system may have a similar problem.

michaelezra · February 20, 2023, 3:45pm

1 second of 4K video stabilization speed difference on M1 Max 64 GB 32 GPU
3.0.12: 54 seconds
3.1.5: 83 seconds

That is 53% slower!

plasmavideo · February 20, 2023, 4:13pm

I second the request for the CRF/CQP Encoding capabilities. I have to export in Prores and use another app to encode to mp4. A one step process again would be helpful.

q91632 · February 20, 2023, 4:17pm

You just need to use the CLI and add the corresponding parameters.

johnnystar · February 20, 2023, 8:05pm

Can you check your logs? I suspect there is a bug where it is trying to download the same model with a different tile size. In your example the tile size is 1152x704, but each model has several other tile sizes. Usually, it will want to use a different tile size if the video resolution or upscale resolution changes, but in this case that may be the bug.

bjorn.granfoss · February 20, 2023, 9:43pm

A small thing but i would like the Video AI app to start with a big window filling the screen.
Like Photo AI do on Mac.
=O)

david-1264 · February 20, 2023, 9:53pm

It seems to try a few different tile sizes where it fails to download them then it uses the file that already existed and uses that and then previews it.

I’m currently testing with a 1920x1080 25 fps clip that’s 15 seconds long. I change the output to be 30 fps with Chronos Fast as the model. (edit: I’m a bit confused why it says “frames”:60 a bit further down - maybe for the log below it wasn’t done with the 15 sec clip as that shouldn’t have 60 frames - or maybe it’s because it only previews a short segment of it, not the whole clip)

There’s already a file called chf-v3-fp16-1152x704-ox.tz in the models folder (the only Chronos Fast one downloaded).

The first try of 2 times I tested very recently I pressed preview and it said “downloading mod…” then after 4 mins 33 secs it had previewed it. I think most of that trying to download.

The log file says:

2023-02-20 21-18-38 Thread: 27236 Debug Updating video info {"sar":1,"framerate":30,"startNumber":1,"frames":60,"progress":0,"status":2,"frame":0,"procStatus":{"status":2,"eta":0,"fps":0,"message":" Downloading Model chf-v3-fp16-1088x1056-rt806-8500.tz","pass":1,"error":"","progress":0,"frame":0,"priority":3,"requestPos":0,"processorIndex":0}}

2023-02-20 21-18-38 Thread: 27236 Info OUT: 2 :AIINFO: Downloading Model chf-v3-fp16-1088x1056-rt806-8500.tz

2023-02-20 21-18-58 Thread: 27236 Info OUT: 2 2023-02-20 21-18-58,114 Thread: 26548 Info RESULT for  https://veai-models.topazlabs.com /chf-v3-fp16-1088x1056-rt806-8500.tz  is  0

2023-02-20 21-18-58,114 Thread: 26548 Critical Attempt 0 failed for https://veai-models.topazlabs.com /chf-v3-fp16-1088x1056-rt806-8500.tz

it tries to download that 2 more times (critical attempt 1 failed then critical attempt 2 failed)…

More errors to do with that file.

then it tries another file:

2023-02-20 21-19-02 Thread: 27236 Debug Updating video info {"sar":1,"framerate":30,"startNumber":1,"frames":60,"progress":0,"status":2,"frame":0,"procStatus":{"status":2,"eta":0,"fps":0,"message":" Downloading Model chf-v3-fp16-1152x704-rt806-8500.tz","pass":1,"error":"","progress":0,"frame":0,"priority":3,"requestPos":0,"processorIndex":0}}
2023-02-20 21-19-02 Thread: 27236 Info OUT: 2 :AIINFO: Downloading Model chf-v3-fp16-1152x704-rt806-8500.tz

and that fails 3 times

then it fails to download https://veai-models.topazlabs.com /chf-v3-fp16-576x672-rt806-8500.tz
3 times

then it fails on https://veai-models.topazlabs.com /chf-v3-fp16-576x384-rt806-8500.tz
3 times

then fails on https://veai-models.topazlabs.com /chf-v3-fp16-384x672-rt806-8500.tz 3 times

then fails on https://veai-models.topazlabs.com /chf-v3-fp16-1152x1344-rt806-8500.tz 3 times

and it fails on some other resolution files…

then eventually it loads the file that already exists in the model folder and uses that:
2023-02-20 21-23-06 Thread: 27236 Debug Updating video info

{"sar":1,"framerate":30,"startNumber":1,"frames":60,"progress":0,"status":2,"frame":0,"procStatus":{"status":2,"eta":0,"fps":0,"message":" Loading Model","pass":1,"error":"","progress":0,"frame":0,"priority":3,"requestPos":0,"processorIndex":0}}
2023-02-20 21-23-06 Thread: 27236 Info OUT: 2 2023-02-20 21-23-06,059 Thread: 26548 Info Loading filename "chf-v3-fp16-1152x704-ox.tz" "fp16-[H]x[W]-ox.tz" "tfinetr2/m_train"

then seems to preview it okay.

johnnystar · February 21, 2023, 3:42am

I hope the devs pay attention to this.

aaron.lee · February 21, 2023, 4:15am

@yazi.saradest here are my logs from a similar crash after processing a DVD upscale, there were issues finding or resolving models, then it terminates and distroys the file. It appeared the video was converting fine, up until the finishing process. This was one of only a few applications running at the time, with 32GB ram, Mac Studio. It has been at least 4 or 5 beta versions from the last time I was getting these memory error crashes. Thanks.

2023-02-20 17-17-28 Thread: 0x103280580 Info OUT: 5 2023-02-20 17:17:28.569 ffmpeg[60796:1085879] Output Err: Error computing NN outputs.
2023-02-20 17:17:28.569 ffmpeg[60796:1085880] Output Err: Error computing NN outputs.
2023-02-20 17:17:28.569 ffmpeg[60796:1085878] Output Err: Error computing NN outputs.
2023-02-20 17:17:28.569 ffmpeg[60796:1085880] Deleting model cache directory: /Applications/Topaz Video AI BETA.app/Contents/Resources/models/coreMLCache/amqs-v2-fnet-fp16-576x672-1x-ml
2023-02-20 17:17:28.569 ffmpeg[60796:1085879] Deleting model cache directory: /Applications/Topaz Video AI BETA.app/Contents/Resources/models/coreMLCache/amqs-v2-gnet-fp16-576x672-1x-ml
2023-02-20 17:17:28.569 ffmpeg[60796:1085878] Deleting model cache directory: /Applications/Topaz Video AI BETA.app/Contents/Resources/models/coreMLCache/amqs-v2-fnet-fp16-576x672-1x-ml
2023-02-20 17:17:28.569 ffmpeg[60796:1085882] Output Err: Error computing NN outputs.
2023-02-20 17:17:28.569 ffmpeg[60796:1085882] Deleting model cache directory: /Applications/Topaz Video AI BETA.app/Contents/Resources/models/coreMLCache/amqs-v2-gnet-fp16-576x672-1x-ml

2023-02-20 17-17-28 Thread: 0x103280580 Info OUT: 5 2023-02-20 17:17:28.572 ffmpeg[60796:1085880] Output with name: fnet/output not found
2023-02-20 17-17-28,572 Thread: 0x2b48cb000 Critical Error debug information
2023-02-20 17-17-28,572 Thread: 0x2b48cb000 Critical Unable to run model with index  0  it had error:  
2023-02-20 17:17:28.572 ffmpeg[60796:1085879] Output with name: generator/output not found
2023-02-20 17:17:28.572 ffmpeg[60796:1085878] Output with name: fnet/output not found
2023-02-20 17-17-28,572 Thread: 0x2b483f000 Critical Error debug information
2023-02-20 17-17-28,572 Thread: 0x2b47b3000 Critical Error debug information
2023-02-20 17-17-28,572 Thread: 0x2b483f000 Critical 2023-02-20 17:17:28.572 ffmpeg[60796:1085882] Output with name: generator/output not found
Unable to run model with index  1  it had error:  
2023-02-20 17-17-28,572 Thread: 0x2b49e3000 Critical Error debug information
2023-02-20 17-17-28,572 Thread: 0x2b47b3000 Critical Unable to run model with index  0  it had error:  
2023-02-20 17-17-28,572 Thread: 0x2b49e3000 Critical Unable to run model with index  1  it had error:  
2023-02-20 17-17-28,573 Thread: 0x2b47b3000 Critical Caught exception in tile processing thread and stopped it msg: std::exception 1
2023-02-20 17-17-28,573 Thread: 0x2b49e3000 Critical Caught exception in tile processing thread and stopped it msg: std::exception 2
2023-02-20 17-17-28,573 Thread: 0x2b48cb000 Critical Caught exception in tile processing thread and stopped it msg: std::exception 2
2023-02-20 17-17-28,573 Thread: 0x2b483f000 Critical Caught exception in tile processing thread and stopped it msg: std::exception 2
libc++abi: terminating with uncaught exception of type std::__1::system_error: thread::join failed: Resource deadlock avoided

2023-02-20 17-17-28 Thread: 0x103280580 Debug passProcDone 0 5 5 6 1
2023-02-20 17-17-28 Thread: 0x103280580 Debug Updating video info {"sar":1,"framerate":23.976023976023978,"startNumber":1,"frames":127716,"progress":19,"status":0,"frame":24689,"procStatus":{"status":0,"eta":14097.587190782006,"fps":7.30713000875542,"message":"Out of memory","pass":1,"error":"- Process ran out of memory. Couldn't generate output from the model for: fnet/output.\n- Process ran out of memory.","progress":19,"frame":24689,"priority":3,"requestPos":3,"processorIndex":-1}}
2023-02-20 17-17-28 Thread: 0x103280580 Info Removing video "/Volumes/BlackWD2TB/Transmission 2TB/Weekend.At.Beernies.II.1993.DVD.BACKUP_amqs2_temp.mp4" 4
2023-02-20 17-17-28 Thread: 0x103280580 Info Close and delete path "/Volumes/BlackWD2TB/Transmission 2TB/Weekend.At.Beernies.II.1993.DVD.BACKUP_amqs2_temp.mp4"
2023-02-20 17-17-28 Thread: 0x103280580 Info EXITED: 5 6 1
2023-02-20 17-17-28 Thread: 0x103280580 Info ~TProcess(): destroyed

david-1264 · February 21, 2023, 6:59pm

Topaz support today said:

It says this because it is continuously attempting to reach the server.

The models are not being downloaded at all. You will need to determine what is blocking the connection in order to resolve this unfortunately.

I wonder if it’s anything to do with the spaces in the URLs that it shows in the logs.
eg. it says:

https://veai-models.topazlabs.com /apo-v6-fp32-1088x1056-ox.tz

That’s got a space after “.com” and before “/”. If I put that URL into the browser after removing the space that allows me to download it. If I don’t remove the space it doesn’t find it. So maybe it’s failing because of the spaces. It seems I should be able to download the models manually this way by checking the errors in the log files. I’ll let support know in case that’s the reason (or maybe it isn’t because for most other people it seems to be downloading the files okay in the app). But at least I have a way to get it working now (though it will be a bit time consuming to get them all that way).

aaron.lee · February 21, 2023, 8:35pm

Thank you for reposting this. I must have missed that repose in the long threads.

I wonder if managing the Models in a different location than inside the app would be easier. And to allow downloading of entire model segments like we did before. This would allow for troubleshooting specific sets of models, and finding if a typo like a space in a file name easier to fix.

I’m loosing 6hr conversions because the says it “runs out of memory” even though I have 28GB available, and a fiber internet connection. Let’s fix this asap as a typo is a pain.

gregory.maddra · February 21, 2023, 9:13pm

A new version is now available.