Upscaling 4k videos to 6k+ (possibly any resolution to any resolution) crashes with 100% frequency on latest TVAI version
i7-13620H, 32gb DDR4, Geforce 4060 mobile
Here is the relevant log entry: OpenCV(4.5.6) Error: Insufficient memory (Failed to allocate 402653184 bytes) in cv::OutOfMemoryError, file C:\Users\Gregory.conan2\p\b\opencb7b82bbe96ee7\b\src\modules\core\src\alloc.cpp, line 73 - I see only regular frame output in the logs leading up to this.
Windows event viewer correlates the low memory condition (resource exhaustion errors).
I have previously been processing for months without error, and stress tested my system to be sure it is stable since encountering errors with TVAI. I have now lost 5 upscales (and many weeks worth of time). So would love to have a fix/workaround ASAP.
I can post full logs if necessary but would like steps to sanitize them if so. The only relevant log entry I can find is the out of memory error which seems to occur every failure.
Can you check in task manager what your GPU memory usage is? Performance > GPU 0 or 1 (NVIDIA RTX 4060 Mobile) > Dedicated GPU Memory / Dedicated Memory Usage.
Sure, do you just want it immediately when I start a render, or after a certain amount of time? I regrettably cant get the info when it crashes because the program closes and GPU memory drops back to idle levels. I could run a utility like HWmonitor to try and get more info as well but at this point im hesitant to spend another 2 days of electricity and heat on a render that I know will fail.
Edit: Another PC (with a beefier, higher-VRAM GPU) I’m running a render on currently is at 6.3GB GPU memory usage. Same Model (Artemis), same dimension upscale (4k to 6k). The 4060 only has 6gb total so maybe the model has grown beyond 6gb of VRAM as a requirement?
Edit: Another PC (with a beefier, higher-VRAM GPU) I’m running a render on currently is at 6.3GB GPU memory usage. Same Model (Artemis), same dimension upscale (4k to 6k). The 4060 only has 6gb total so maybe the model has grown beyond 6gb of VRAM as a requirement?
Yes - this is what I am inferring, that you are running out of VRAM - based on the snippet of the logs and your situation (4K → 6K) on your 4060 Mobile.
You may want to look into an eGPU if your laptop can support it (Thunderbolt, USB 4, etc) or otherwise build a dedicated desktop machine or purchase another type of system (Apple products) to meet your needs/budget.
Although it sounds as if you may already have another machine you can use.
I dont think it’s reasonable for Topaz to just cease working on 6gb GPUs, especially when I’ve been using it successfully for the last year to process videos, and the system requirements state a 4gb requirement.
Hoping support (or an update) will be able to solve this, or provide a workaround and/or license extension.
Also to be clear, the windows event viewer logs state resource exhaustion of system memory, not GPU memory, though that’s not to say GPU isn’t overrunning as well.
Well if you can check what I said before, that would be useful to help diagnose. Depending on what driver version you are at, I would have thought the GPU would overflow to system RAM if it ran out.
Also you can blame NVIDIA and the like rather than Topaz as such. 6GB is very small amount of VRAM in 2023. NVIDIA GeForce 1080 Ti in March 2017 (almost 7 years ago) had 11GB VRAM.
I personally moved from an 8GB 3060 Ti to a 16GB RTX A4000 and now 48GB RTX A6000 to make the most of AI technology. If you want to stay on a laptop you can consider a workstation laptop with RTX Axxxx or RTX xxxx Ada GPU.
For example you could get (on mobile/laptop) - RTX A3000 12GB, RTX A4500 16GB, RTX A5500 16GB, RTX 3500 Ada 12GB, RTX 4000 Ada 12GB, RTX 5000 Ada 16GB, etc.
Topaz has actually done a lot of work to reduce VRAM usage and nowadays I am seeing a lot lower VRAM use in TVAI compared to 2.x and 3.x days - though I am mostly doing 1080p/1440p/4K (max). If you have the requirement to deal with 4K/6K footage then you are going to need to get the hardware to match.
When I am running LLMs/voice synthesis/diffusion I am seeing 32-48GB VRAM use - a consumer GPU has no chance to do such things (without overflowing to system RAM and becoming sloooowwwwww).
See if you can post up more information to help diagnose the issue - but I would say it is coming down to the resolutions you are dealing with and your hardware capabilities/specifications.
I agree 6gb is anemic in 2023 but the fact of matter is I’ve upscaled files in the past to 8k on this GPU (dozens of them, in fact) and now crash doing 6k after a Topaz update. The crash does not occur generally until 40-60% in to the encode which, to me indicates some sort of flaw in memory handling, or bloating of the model in this version not present in previous versions. This is not a LLM and I’m using one of the simpler models.
If 6gb is no longer sufficient to run the program then the minimum requirements should be updated to reflect this, as they currently list 4gb.
At the end of the day, the update broke functionality previously present, forcing me to sacrifice my desktop computer (which is DRAMATICALLY less power efficient than my laptop, as well as much less convenient) to processing for the time being. I’d like a fix from Topaz, or at least official acknowledgment that 6gb is no longer sufficient to run AI models at anything above consumer TV resolutions.
I am happy to give any additional information you want. To be clear are you asking me to give you DGPU memory usage off the laptop still? And if so, immediately after starting a render, or should I run a long-term monitoring utility like hwinfo?
I missed that you were processing the same/larger files for a while previously - that part could be important, sure.
I have no association with Topaz, I am just trying to help you out, the more information you can give us (the community) or Topaz - then the better you can be helped.
You can download a program like librehardwaremonitor which can log VRAM (as well as system RAM) usage to a file and this way even if it crashes you should have a record of what the values were before crashing.
Here for example I used TVAI 4.0.8 and took a 4K H.264 file and upscaled it (preview only, 899 frames) with Artemis HQ V12 (0/20) to 8K (Prores 422 Proxy, Audio Copy, MOV container) and my system RAM went from ~27GB to ~43GB and VRAM to about 3GB.
After a few minutes my system RAM is bouncing around between 43 and 45GB usage - so that is about 18GB of usage from my starting point. My A6000 which is basically a 3090 was ticking along at 1.5fps - so I am kind of concerned at the thought of how long your encodes are taking on a 4060 mobile.
Anyway - maybe you can post some screenshots/diagnostics so we can actually help you solve the problem???
Absolutely, I just wanted clarity for what you wanted in terms of data - since it takes a couple days for the crash to occur (and power is VERY expensive in my area) I didnt want to burn 2 days doing the encode without getting the data you wanted. Ill run another encode and hwmonitor to log some data on GPU memory usage and other system info. Talk to you in a couple days once it’s crashed!
I get about 1-1.4 fps depending on the video. The GPU is actually surprisingly potent, despite the lack of VRAM I did the math and compared to my desktop its roughly 50% less energy consumed per frame so I dont mind that my upscales take 2-2.5x as long in exchange for the energy savings.
Ill get back to you with that data once I experience the next crash
If it is happening after longer runs then you may want to do something quicker like a memtest (just get the free version of memtest86) which should take about 1hr or so for 32GB - though I would do 2-3 runs to make sure.
You realise also that if your laptop uses 50% less power but takes more than 2x the time you are not saving anything though, right?
0.5 (power) * 2.0 (time) = 1.0 (energy)
If it is anything more than 2x as long then you are actually better off using your desktop. But you said you did the math so hopefully you have come to the right conclusion
Good luck and hope you can solve the issue or get the requisite reports for Topaz team.
I do, that is why I quoted energy consumption per frame, not energy consumption of the machine while running encodes. The machine consumes roughly 1/3 the power of my desktop, while rendering frames at 50% of the speed, these are the raw numbers:
Desktop: 600w @ 2.4 fps = 250 watt-seconds per frame
Laptop: 200w @ 1.2 fps = 166 watt-seconds per frame
I have run stability tests - Testmem5 + and OCCT. Memtest86+ has become largely useless in modern times at identifying transient errors, especially ones introduced during high load scenarios (thermal constraints/system load fluctuations/etc.) - Testmem5 is a much better pure memory test and OCCT can stress multiple systems at once (ram, cpu, gpu) making it a better overall system stability benchmark. This was the first thing I did after retrying the initial encodes + running system updates (and ultimately fully reinstalling windows to be sure) to try and fix the issue.
I completed an upscale the night before, Updated Topaz, then began getting crashes every upscale, so I do believe it is a program fault, but I have a render running right now with hwinfo64 logging video memory usage and a few other system statistics and will post back when I get the results.
You sound like you know what you’re doing. If it’s that clear cut in going Version A (fine) > Version B (crashing), have you tried just rolling back the version?
All of the previous versions should be posted on here somewhere. What are you on now and what were you on before?
TopazAI support - I am still experiencing this issue. I have been able to consume it is system RAM exhaustion causing the issue, but I have 32gb of RAM and windows has dynamically allocated a 60GB page file (normally its around 8gb at boot time, so it clearly increased a LOT to try and accommodate topaz)
I would like this high memory run-away condition resolved and a license extension.
I reached out to the support team a while back and they sort of stopped responding after the last update.
With the recent updates to Topaz I am no longer crashing, however I lost a month of processing time to this issue and would still like a license extension, which I’ve mentioned multiple times in the ticket now. If there is any way you can help me, I’d really appreciate it!
I did the math and compared to my desktop its roughly 50% less energy consumed per frame so I dont mind that my upscales take 2-2.5x as long in exchange for the energy savings.
