During multiple attempts to upscale my existing movie library to 4K 120fps using Topaz AI on my RTX 4080, processing jobs lasting several days have been unexpectedly cancelled due to overnight system reboots. The current functionality of Topaz AI lacks the ability to save intermediate processing data, leading to complete loss of progress in these scenarios. To address this issue, I propose the implementation of a checkpointing system that automatically saves the state of ongoing tasks at regular intervals, allowing users to resume processing from the last saved point in case of interruptions.
8 Likes
If my system was rebooting like that no matter the software I’d be looking into a hardware problem.
1 Like
I am sure its windows 11 the culprit here. it mostly happens on friday night which i am sure is sneaky windows update. However, it would be nice for topazAI to either block that restart or atleast save its current state so new i start topazAi again it could let me continue the processing again.
1 Like
I am going to upvote this. There really is no other feeling of disappointment than having your 1+ day upscale be interrupted for any reason. These video upscales are incredibly computationally expensive, and it is also a massive waste of power when your computer draws 500+, 600+ watts for 12+ hours just to end up with nothing.
I have also been having jobs failing with no explanation, despite the fact they said they’d add a more verbose export debugger, and having the temp file get deleted for no reason. Topaz really needs to figure this issue out otherwise I will not be renewing my license next year. It is entirely system agnostic too, as I have used four separate computers now (AMD 5900X+Radeon 7900 XTX, Intel 10900KF + 3070 Ti, Intel 12700KF + 4070 Ti, and Intel 11700KF+RTX 2060) and they all have the same issues.
1 Like
The issue isn’t with the hardware; it’s with Topaz Labs’ software, a high-processing tool that consumes significant resources and time. In modern times, a tool susceptible to abrupt failures without recovery options is simply unacceptable. I sincerely hope the development team acknowledges this and prioritizes improvements in their upcoming development cycle.
Have you tried turning off Hardware-accelerated GPU scheduling ?
To do this go to Settings → System → Display → Graphics settings (down the bottom of page) → Hardware-accelerated GPU scheduling → Switch to “Off”.
This is an important need that goes beyond just individual systems. With increasingly frequent climate incidents (240 major incidents worldwide in 2023 plus uncounted local storm-downed power and internet lines), the chances that a many hours long process may be interrupted by a loss of power, internet connectivity or both is a real concern. Any application that runs long processes and depends on a net connection absolutely NEEDS to have the ability to set autosave intervals AND to resume interrupted processes.
1 Like
I 100% agree to your sentiment. The recent surge in climate-related power outages throughout the world underscores the critical need for automatic saving functionalities, particularly in projects like long-term video upscaling. With substantial effort already invested, imagine losing 98% of progress due to a sudden blackout. Integrating auto-save should be a top priority for the development team.
The problem is not with Topaz Labs AI shutting down; it’s when the entire PC shuts down due to various reasons. In that case, I don’t want Topaz Labs AI to lose existing upscaling jobs. It should save them so that when I start it again, it should ask me to continue with the processing.
The headscratcher here is why this hasn’t been done already. TVAI already saves exports in progress in dribs and drabs in the preview folder (you can view them while export is going on). So the app is already halfway to save/resume as long as that partially done export doesn’t get deleted when the system or app experiences an unplanned stop.
1 Like
Try splitting the render up into smaller “clips” then stitch it together using ffmpeg command line.
as a workaround you could export the video to frames so that in theory you could continue with the next frame if the processing ended unexpected. unfortunately this might not work well since version 3.x because of a missmatch between the saved frames and the frame numbers shown in the application. 
1 Like
Well there are workaround. But, when you pay 150 for a product, that should be provided by default. This kind of workaround sounds more like a college guy hack that you get from GitHub. This post is more of a feedback for future versions of topazAI.
