Problems when using ProRes or DNxHRX for export due to exorbitant continuous read terror with larger files

[seifenchef]

TOPAZ: Model IRIS; Input: 4K - H264, Output: 4K - ProRes 422 or LT (both tested)

Computer: i9 11900 (OC 4.5 GHz), RTX 5080, 32GB RAM 4400MT/s, PCIe4,

HD source is a Samsung SSD EVO Plus 4TB on PCIe4,
HD destination is a Samsung SSD EVO Plus 2TB on PCIe3,

Computer utilization at start: CPU 75 - 95%; GPU 55 - 75%

Problem description:

The computer starts up with normal/good parameters, but after a few minutes it gradually slows down and after 20, 40, or 60 minutes, a reading frenzy occurs on the disk, which is Topaz’s output target.

The file is read billions of times, with over 10 Gbit/sec as a permanent state.
See screenshots.

As a result, the computer has now slowed down by 15 to 22% on average!
This is unacceptable given the computing times that are to be expected with video AI today.
For a job that takes many hours, the computer is only working at about 83% of what it could actually do without this bug.

The ‘read terror’ occurs on the disk where the file to be written by Topaz is located.
It is read billions of times at 1 to 1.7 GB/s.

Here is a description of the process in terms of time:

1st test: ProResLT output

Start speed: 5.9 fps – cyclic read approx. 100–300 Kbytes/s (H264) – cyclic write: 8 Mbytes/s (ProResLT) – everything great
After 10 min: 5.8 fps
After 15 min: 5.7 fps
After 25 min: 5.6 fps
After 40 min: 5.5 fps - Cyclical reading - in blocks lasting approx. 20% of the time - from 1.2 to 1.4 GB/s begins! (visible in Task Manager)
After 50 min: 5.4 fps - during the ‘meaningless’ read operations, GPU utilization drops below 45%
After 65 min: 5.3 fps
After 75 min: 5.2 fps
After 90 min: 5.1 fps – frequency of ‘pointless’ read operations increases
After 100 min: 4.9 fps – frequency of read operations continues to increase
After 130 min: 4.8 fps – pointless read operations now occur continuously
→ CPU utilization as at the beginning, BUT: GPU only 40 - 55 % !
→ Note: sometimes the ‘read terror’ only starts after 50 or 60 minutes (with Apollo), but usually after about 40 minutes …

2nd test: ProResStd output

Basically everything as above, except that now the ‘read mania’ / ‘read terror’ starts after only 18 or 22 minutes around !

3rd test: JPEG output

Everything runs smoothly and without errors or problems!
→ I no longer have 10 bits, and 16-bit formats create monster files that nobody needs!

4rd test: DNxHR - Output

To get 10 bits, you have to select the HQX quality level, which I did.
The file runs through the task without any problems.
→ But I don’t need a format that generates an output of 60 gigabytes for a 5.5-minute 4K video file!
Even though it ran smoothly, I can’t afford a million expensive NVMEs…
… AVID is just crazy that only the two highest standards support 10 bits … SCNR

Summary:

I hope that Topaz can fix this bug soon so that our computers can work at a good capacity and I can work semi-professionally with ProRes in 10 bit.
Furthermore, SSDs are certainly less susceptible than mechanical hard drives, but they also have a statistical failure rate, and that becomes uselessly close in such a devilish scenario.

kind regards

[seifenchef]

A few words about the two images.

In the first image, the source and target files are on separate NVMe drives.
( 3 is source, 4 is target - booth are the same type, sorce on PCIe4, Target on PCIe3)

The system runs much more smoothly, and the GPU is better utilized.

The second image shows the state before I made this separation.
The computer is virtually at its limit when it has to read and write these exorbitant data rates (ProResLT or Std) simultaneously from one and the same NVMe…
Although, in my opinion, these data rates should not be too much for these NVMe’s, but well, that’s theory, and this is practice.

But you can probably see in both pictures that Topaz is literally to go on a rampage with the discs.

So if high data rates such as ProRes or similar occur in 4K, we definitely recommend having separate NVMe drives for the source and destination!

[seifenchef]

I spent several days here testing with the AI to narrow down the cause: stress, stress, and lots of energy, because the error only occurs with longer files, and here in Germany we have the most expensive electricity in the world. …

[kyle.topazlabs]

Can you share the logs from the app as well for review? You can attach them here, or send directly to the support team at help@topazlabs.com

To gather logs, please select Help > Logging > Get Logs for Support and attach the zip file to your reply.

Here is a video to help with the steps of how to collect the logs.

[seifenchef]

Hi,

logs sent from Topaz app

Here are some screenshots of the timeline again. The clip was 4:40 min long and took 28 min to render.

It’s easy to see how the GPU gets less and less “work” due to the “read terror.”
GPU utilization drops from over 60% at the beginning to below 40% at the end.

And as I said, this only applies to ProRes files.

kind regards

[seifenchef]

Yesterday, another run with 4K:
Iris and Apollo simultaneously, output 4K ProResLT.
Started at 3,1 fps. OK. (GPU 90%)

After about 45 minutes, the reading terror began again.
The speed dropped to 2.7 fps.
The graphics card utilization decreases.

Then I had an idea !

Just quit Topaz in Task Manager.
And lo and behold, the useless read load disappeared immediately, and ffmpeg ran smoothly.

The graphics card utilization rises above 92% again.

The job took 6 hours; with the read terror brake, it would certainly have taken 7 or 7.5 hours.

Now there is no progress bar, and the final file has no file header.

(This means that most players cannot read it (crash), especially if it is large/long).

Bandicut takes care of this: file in, and out again as SmartRenderer.
Duration for 125 gigabytes approx. 3 min.
(Unfortunately not measured, only estimated).

PS: I get the best graphics card utilization when the source and destination files are located on different system buses and Topaz does not interfere.

That’s how you want your graphics card to be utilized!

Is there actually a Topaz wiki where users can look up how a Topaz computer should be structured?

Best regards

[seifenchef]

Hi,

Does anyone here work with ProRes and can comment on my problem?
If so, please specify which version of ProRes you are using and whether the files are small or large.
(Topaz processing takes longer than 30 or 60 minutes.)

Best regards

[david.clarke-6312]

I make prores files and have problems making large files. I have not done anything like the testing above but when have noticed when I have done a long render, making ProRes HQ, and it fails the “unfinished” prores file causes problems - ages to just open the folder in which it sits, very hard to delete etc.
As Topaz is adding to an “unfinished” file as it renders I am guessing this is why it slows the render as the file gets bigger.
My guess is that the problem is QuickTime rather than Topaz. If I try to access an unfinished file on Linux it causes not disk problems and I can delete easily (an unfinished file is never playable).

As a results if I have to do a long clip I will chop it into 10 minute segments and stitch them back together afterwards. The alternative is making a different format like MP4 - which I do not want to do as it will not be as good quality - or making a series of stills. The latter will use far more disk space than Avid files.
I don’t render to NVMEs either - I just use normal drives which are not too expensive for the very large files that get made if rendering as stills.
All of this is a PITA but it has been like this for ages and I really have no idea if it is a Topaz or QuickTime problem. All I do know is I can’t fix it so I look for workarounds.

[seifenchef]

Hi,

Yes, I also think ProRes is the best solution if the files are going to be edited further afterwards.
(10 bit, no GOPs, works great in DaVinci)

Yes, Topaz logically writes a file successively during rendering.

(This often saves the day (if Topaz doesn’t delete it, which they have fortunately changed now) if Topaz crashes during the final stage, which often happened to me with Apollo).

And it reads and reads this file over and over again, eventually permanently during long rendering processes, which causes total data stress that slows down the entire system.

The funny thing is that when I close the Topaz app in Task Manager, the ffmpeg rendering process continues without any stress!

Of course, you don’t end up with a finalized file, but BandiCut takes care of that for me (SmartRenderer – just pop it in and pop it out again).

But it’s good to hear that I’m not the only one. :-)…

and … so far, I’ve only encountered this bug with ProRes …

PS: … of course it’s possible to share files … but it’s not exactly ideal …
I think I’ll just close the app in future …

Best regards

[seifenchef]

Hi,
By the way, I have to correct myself.

I’m currently running a project, 4K - IRIS - 5.2 fps in DNxHRX (10 bit).
The read terror is also occurring here. (screenshot after 50 min)

This causes the GPU utilization to drop from approx. 64% to below 50%…
I then closed the Topaz app in the Task Manager again … and everything is running smoothly …

Best regards

[seifenchef]

Hi,

Are there so few amateurs/professionals here who work with ProRes or DNxHRX?
(intraframe-based 10-bit codecs)

Best regards