It is fairly normal for me to run something in ProRes422HQ that is 600,000+ frames in length.
That said, the issue i’ve been running into is that these files get in the 300-500GB range, and when topaz is finished, the temp file seems to copy itself into the “final file.”
I do have a NAS that can saturate 10gigabit lines pretty well, and I do have pci gen4 nvme drives, which help a little, and I do realize the “fix” here is to make the temp file go to one directory and the final file be put into another directory (SSD>NAS would be the likely candidate)
However, i am used to keeping workflows on a single drive until finished. The problem i’m running into is that topaz finishes the render in prores, and then runs out of swap space when making the final file from the _temp file. This throws the “out of space” error, and there is no recovering from this. I have to run the entire thing again.
Like i said, i know a solution to this, but it’s going to affect people with slower drives/less capacity far more than it is going to affect me, and it seems to trigger the error before the drive is even completely out of space.
Is the solution moving forward going to be “use two drives” or is this an issue that can be addressed in a future version?
This isn’t really going to be a solution for some people, but if you run the Topaz Video AI ffmpeg process from the command line directly, there is no “copy from temp file to final file” process meaning you won’t run into this issue.
The temp file is in the same codec and format as the final export only difference is a few extra headers.
The only reason for the last pass is some video editors and players do not handle moovflags correctly. The temp file is generated with extra headers to allow for export playback while exporting, to keep the video file readable in case of crashes or other failure.
We will add an option to preferences to disable the last pass and keep it off by default.
Sadly, I’ve found this to be true. My (post-pass) VapourSynth LibavSMASHSource filter cannot read the first-pass .mov directly.
Question, of course, is, Can those compatibility headers not simply be written along with the stream of the first pass? I know they apparently can, as that is what 2.6.4 did.
N.B. FYI, yesterday, a 2nd pass of 900G took 2.5 hours to write (normal SATA3 disk). That is almost half the time of the entire rendering process.
I thought it was bad for me using PCI gen 4 NVME drives and a NAS that is capable of saturating 10-gigabits… would be absolutely unbearable on SATA – especially using the same drive for the final output!!!
I assume this part of topaz is related to the effort to be able to pause and resume in-progress projects, which would be a great feature to have, especially now that v3 is pushing my 30-series cards much harder than anything has utilized my nvidia gpus since 1.5.3 (which was CUDA dependant if i remember correctly)
Double edged sword, but I do think this ability to use one drive for temp file and another for output already exists in the options. might need some input here.
I have 3x PCI gen 4 NVME drives as well, but not going to use those for TVAI. A modern SSD can be written for, what, 256 Terabytes? At one Terabyte per file (and then times 2, for the temp file), that means I can throw away my expensive SSD within a few months. Ain’t gonna happen. Hence why TVAI output drive remains a regular SATA3, mechanical disk.
Well, the latter isn’t exactly my choice.
No, it doesn’t. You can define a temp folder, but it’s not being used for writing any of the output files to.
Wait for the next version, will add a preference option for avoiding the second pass.
Preference option will have: (Open to suggestion on option names)
Use Export Location (Temp File will be stored in export location - default)
Export Without Preview (Single Pass) (Files should work everywhere, no recovery on failure)
No Cleanup (Single Pass) (Files may have issues with some editors/players)
Use Temp Directory (Two Pass) (Temp folder will be used to store the intermediate file)
The method we used a very long time ago, to keep files from being ‘interfered’ with by other apps and eliminate the need to do a long copy was to simply do a quick rename when the job was done.
On that temp folder, I find the currently defined AppData\Local\Temp folder name confusing, as it’s clearly not referring to the temporary output file as discussed here.
Please, clarify. Are you saying this will do a single pass, but you can’t use Preview in the UI when having selected this? That would frankly suck, as I need Preview for determining best settings.
Is this the state of the current 1st pass file? (Playable only in like VLC, but fails for many other readers (including VapourSynth filters).
Originally the temp file was stored in the temp location and this was causing issues for people doing large exports who didn’t move the temp directory.
There will not be any preview of the result while exporting a file. If you use preview it should work fine. This will also lead to no possibility of recovering the output if the export fails for some reason.
In this state there will be intermediate headers which will make certain programs read incorrect duration.
Since there is no recovery/pause/resume option yet. I would recommend using the No Preview option. Exports when not played/previewed are always faster.
The reason for the temporary file is not only interference from the other programs which we handle by locking the file, but the fact that video headers need to be inserted from time to time for the file to be readable while being written to, and also be readable if the export fails. So essentially the file is written as a video stream and then the 2nd pass makes it a file with a single header at the front like all video files.
Thanks suraj, your initial reply explained everything very well, and I was pretty sure what you described was the “issue” (not really an issue, more of a feature)
The solution you explained should be perfect, I don’t typically ever need to pause but it would be nice to have the option, and to be able to use separate drives for temp/final output. (which i think you said was going to be an upcoming option.)
I don’t think there’s need to explain further, this was addressed perfectly, people just need to read more carefully
Thanks for the quick reply with a solution that should make everyone happy when implimented !
Suraj,
That part of the process is understood. But considering the different hardware deployments that TVAI is being run on, perhaps there should be a few controls the end-user can adjust to make better use of their storage scenario and also allows TVAI to have optimal temp and storage space given that machine (& network’s) topology.
And, BTW, I understand the need for temporary files, multiple passes, and temporary names until the end product is ready. - I’ve been using many of these techniques to handle/update data files and relational database structures for decades.
Here is an alpha version with the feature I described implemented. Let me know if there are still questions or more options are needed.
Preferences → Output → Video Export
I found a new option, Keep Extra Headers. From what I can tell, that is actually what I was looking for: a single-pass file, with Preview while exporting (!), and still readable by my VapourSynth input filter when written as single prob3 file.
Sad to report to use Keep Extra Headers. was a complete bust. Worked fine on a small test file; but on the main file, TVAI crashed at 99.99%. Result? Unplayable mov file. And since single-pass ‘Without preview’ also yields an unreadable file, I’m back to square one, and loss of a day.
The document linked to [?] in preferences would be updated before release.
With Preview (Temp File will be stored in export location - default)
Without Preview (Single Pass) (Files should work everywhere, no recovery on failure)
Keep Headers (Single Pass) (Files may have issues with some editors/players)
Use Temp Directory (Two Pass) (Temp folder will be used to store the intermediate file)
My (post-pass) VapourSynth LibavSMASHSource filter cannot read the first-pass .mov directly.
