I am having a strange issue where if I try to convert more than one 1080p video to 8K and 60fps at a time using Artemis HQ, the first video in the queue will process just fine, but the second conversion will slow down to 0.0 seconds per frame with an ETA of several days (after about ten minutes of processing at normal speed). Then it will just hang at the 0.0 seconds/multiple days ETA forever.
I’ve checked the system performance and everything is normal - GPU, CPU, RAM all fine.
The problem happens for any 1080p video used; videos under 1080p process as expected. I’m just trying to figure out what may cause this to happen. If I close and reopen Topaz, then process the “stuck” video by itself, it finishes just fine in a few hours. The problem only occurs when I process two or more 1080p videos in a single session; the first video will process normally, then everything afterwards slows down and gets stuck.
Again, I think this is a Topaz issue rather than a system issue because my performance is fine and I know that the system is capable of performing this task. Could this be some kind of memory allocation error inside of Topaz?
Processing multiple videos to 8K will likely introduce memory allocation issues with most hardware.
I’d recommend trying the same clips again after changing the “Max Memory Usage” setting to 50% or even 20% to see if there’s an improvement in stability
I’m having a particularly rough time with the Iris model.
The initial post was about Artemis HQ. Proteus has no observed problems. Artemis has an issue with consecutive 1080-to-8K conversions. Iris cannot perform any 8K conversions.
All of the Iris upscales have failed after 1-2%. Even the previews fail. I followed the suggestion in this post and lowered the max allowable memory usage. The processing lasted for slightly longer but still failed. The input video is only 640x360.
Here is what the log file says:
2023-09-20 12-34-36.247 Thread: 3988 Info OUT: 2 2023-09-20 12:34:36.2477658 [E:onnxruntime:, sequential_executor.cc:494 onnxruntime::ExecuteKernel] Non-zero status code returned while running DmlFusedNode_0_3 node. Name:‘DmlFusedNode_0_3’ Status Message: D:
2023-09-20 12-34-36.248 Thread: 3988 Info OUT: 2 a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\FusedGraphKernel.cpp(397)\onnxruntime
2023-09-20 12-34-36.248 Thread: 3988 Info OUT: 2 .dll!00007FF98AEF92B0: (caller: 00007FF98AEF7575) Exception(2) tid(1160) 887A0006 The GPU will not
2023-09-20 12-34-36.249 Thread: 3988 Info OUT: 2 respond to more commands, most likely because of an invalid command passed by the calling application.
2023-09-20 12:34:36 4448 CRITICAL: ONNX problem: Run: Non-zero status code returned while running DmlFusedNode_0_3 node. Name:‘DmlFusedNode_0_3’ Status Message: D:\a_work\1\s\onnxruntime\core\providers\dml\DmlExecutionProvider\src\FusedGraphKernel.cpp(397)\onnxruntime.dll!00007FF98AEF92B0: (caller: 00007FF98AEF7575) Exception(2) tid(1160) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.
2023-09-20 12:34:36 4448 CRITICAL: Model couldn’t be run for outputName [“out”]
2023-09-20 12:34:36 4448 CRITICAL: Unable to run model with index 0 it had error: no backend available
2023-09-20 12-34-36.249 Thread: 3988 Info OUT: 2
2023-09-20 12:34:36 4448 CRITICAL: Caught exception in tile processing thread and stopped it msg: unable to run model with index 01
2023-09-20 12-35-07.786 Thread: 3988 Info EXITED: 2 -1073740791 1
The error doesn’t say anything about memory. There was some type of runtime error. Looks like this might be a bug.
4GB of dedicated VRAM is most likely going to have issues with 8K processing, I’d recommend setting the “Max Processes” setting in the app preferences to 1 and letting items process one after another:
It is already set to 1. I don’t have any issues processing individual items to 8K+60 fps, only have this strange problem with multiple items. But it’s not the end of the world.
Based on the preceding comment, I’m not sure Topaz understood the problem being described here.
If multiple 1080p input videos are selected on Artemis HQ for 8K+fps conversion, with max processes set to 1, the first video in the queue will always process normally, but the second video will freeze around 12% or so with a very high ETA, after processing just fine prior to this point. This seems very strongly to be tied to improper memory allocation within Topaz, as I don’t see why the program would still be allocating memory to the finished video. Using larger input videos just made this issue more difficult to hide. (And, again, if Topaz is closed and the stuck video is reprocessed by itself, it will always work.)
There are other areas where memory allocation seems to be a problem. For example, if enhancement parameters are estimated and then a particular parameter is set manually (e.g. ‘Sharpen’) on Proteus, Iris, or Theia, when a new input video is estimated, Topaz never forgets the manually-selected value. If I manually set ‘Sharpen’ to 33% for one video and then run estimations again on completely different videos, ‘Sharpen’ will always be set to ‘33%’ and will not be affected by the new estimations. Even closing all of the inputs or selecting a different AI model or video type does not resolve the problem. The only way to force Topaz to forget the manual value and begin estimating again is to close the program and relaunch. It doesn’t know how to deallocate memory from the manually-set value.
You’re probably right, a memory allocation issue is causing the next items to fail during processing even though the previous process is completed. I’ll check with our team to see what can be done to improve stability during long export queues.
Following up on the Iris v1 problem, I discovered that if I was using a constant video bitrate, Iris would always fail if the bitrate was set too low. When I increased the bitrate, Iris v1 ran successfully.
I found this out while running two conversions on the same video, one to 4K and one to 8K. The 4K conversion failed several times, but the 8K finished successfully. When troubleshooting what went wrong, I set the 4K conversion to the same bitrate that I gave the 8K conversion, and it processed successfully.
So the original 8K conversions may have been failing because the chosen video bitrate was too low.
