Kind of a weird one, unfortunately I have not had a chance to attempt a repro.
Both Photo AI and Video Enhance AI were VERY slow even with GPU selected. I was getting 0.1 fps with progressive video enhancement at 1080p and several seconds for denoise on a 3mp picture. GPU load on gpu-z showed about 5% utilization. I tried reinstalling which made no difference. I’ve also tried switching between CPU and GPU several times. Other applications (handbrake, games, etc) had no issues utilizing the GPU.
Then restarted the Topaz products. Everything worked after that and I was getting 11+fps with much better GPU utilization. I got the hint to delete the registry keys from the Topaz CLI documentation.
My specs:
Intel i7 7700k with Resize BAR patch
48gb ram
Intel Arc A770 GPU 16GB
Hopefully this will help anyone in a similar situation!
========= Dev repro hints underneath ============
From my memory, I did a few things which might have caused this:
I enabled the iGPU.
I switched from hdmi output on my GPU to displayport.
**** Problem already manifested after these two steps ****
I disabled the iGPU. No changes.
Unfortunately I did the first two steps concurrently so I have no idea which one might have caused this issue.
Ah! I actually found a bug I think. From the logs of PhotoAI:
[2023-03-02 22:07:24.948, 13.30 μs] [44f4] Info | Device count: 1
[2023-03-02 22:07:24.948, 12.00 μs] [44f4] Info | - Index 0 Name Intel(R) Arc(TM) A770 Graphics Cores 0
[2023-03-02 22:07:24.948, 31.00 μs] [44f4] Info | VRAM 15.875000 GB Total / 7.687012 GB Used DataType 2
[2023-03-02 22:07:24.948, 24.40 μs] [44f4] Info | Serial 0000000000010f4c Legacy 0 CudaId -1 Capability 0
And later…
[2023-03-02 22:08:07.546, 1.14 s] [55ac] Info | check CPU: intel(r) core(tm) i7-7700k cpu @ 4.20ghz VS intel(r) arc(tm) a770 graphics 0
[2023-03-02 22:08:07.557, 11.36 ms] [55ac] Info | check GNA: gna_sw VS intel(r) arc(tm) a770 graphics 0
[2023-03-02 22:08:07.557, 113.70 μs] [55ac] Info | check GPU: intel(r) arc(tm) a770 graphics (dgpu) VS intel(r) arc(tm) a770 graphics 1
[2023-03-02 22:08:07.558, 35.00 μs] [55ac] Info | OpenVINO device string for 0 is GPU
[2023-03-02 22:08:09.252, 1.69 s] [55ac] Info | - Running CPU fallback due to error [ cldnn program build failed! Incompatible OpenCL runtime: no binary section found. ]
[2023-03-02 22:08:09.276, 24.31 ms] [55ac] Info | - Loaded with CPU
Note that I have already detached the the integrated graphics at this point and disabled it in BIOS. At this point, the igpu is disabled and unavailable. It looks like OpenCL maintains a GPU even after it has been disconnected and will reorder devices on connect (why Intel why??)
It looks like there is a disagreement for the indices between Topaz and OpenCL. My guess is that Topaz lists the list of available devices and assumes that whatever index they get from the the list will match OpenCL’s device enumeration. I confirmed this is the case by manually setting the deviceId key in the registry to 0x01 and PhotoAI was able to use my GPU. I’m guessing the fix on Topaz’ end would be to grab the device index from OpenCL instead of implicitly relying on the ordering for id.
On a side note, I’m wondering what would happen if the user selected GPU, then enables the iGPU. Would Topaz correctly update its device selection in the registry? Or would it end up defaulting to the iGPU because of the reordering quirk?
With this I think I have the full repro:
Have a machine with Intel CPU and Intel GPU. Make sure Intel drivers are not installed.
Make sure the iGPU is disabled then boot the machine.
Install all necessary drivers.
Enable the iGPU.
Disable the iGPU.
Topaz should now be stuck in CPU only mode now.
Workarounds:
Use registry hack and manually set to the correct device id.
Re-enable the iGPU and select the correct GPU in the menu.
Ok more updates lmao. So the second workaround doesn’t work. Topaz is just not using the correct indices:
[2023-03-02 22:54:00.586, 23.90 μs] [5674] Info | Device count: 2
[2023-03-02 22:54:00.586, 23.90 μs] [5674] Info | - Index 0 Name Intel(R) Arc(TM) A770 Graphics Cores 0
[2023-03-02 22:54:00.586, 26.70 μs] [5674] Info | VRAM 15.875000 GB Total / 7.687012 GB Used DataType 2
[2023-03-02 22:54:00.586, 24.30 μs] [5674] Info | Serial 000000000001101b Legacy 0 CudaId -1 Capability 0
[2023-03-02 22:54:00.586, 23.50 μs] [5674] Info | - Index 1 Name Intel(R) HD Graphics 630 Cores 0
[2023-03-02 22:54:00.586, 27.10 μs] [5674] Info | VRAM 0.125000 GB Total / 11.719750 GB Used DataType 2
[2023-03-02 22:54:00.586, 25.20 μs] [5674] Info | Serial 000000000001144d Legacy 0 CudaId -1 Capability 0
[2023-03-02 22:54:36.730, 0.00 ns] [42f4] Info | check CPU: intel(r) core(tm) i7-7700k cpu @ 4.20ghz VS intel(r) arc(tm) a770 graphics 0
[2023-03-02 22:54:36.731, 708.30 μs] [42f4] Info | check GNA: gna_sw VS intel(r) arc(tm) a770 graphics 0
[2023-03-02 22:54:36.731, 94.40 μs] [42f4] Info | check GPU.0: intel(r) hd graphics 630 (igpu) VS intel(r) arc(tm) a770 graphics 0
[2023-03-02 22:54:36.731, 41.90 μs] [42f4] Info | check GPU.1: intel(r) arc(tm) a770 graphics (dgpu) VS intel(r) arc(tm) a770 graphics 1
[2023-03-02 22:54:36.731, 25.50 μs] [42f4] Info | OpenVINO device string for 0 is GPU.1
[2023-03-02 22:54:48.626, 11.90 s] [42f4] Info | - Running CPU fallback due to error [ cldnn program build failed! Incompatible OpenCL runtime: no binary section found. ]
[2023-03-02 22:54:48.741, 114.71 ms] [42f4] Info | - Loaded with CPU
Notice that A770 is listed at index 0 in the first log snippet, but is actually at index 1 in the second log snippet. :((( Clearing the registry does not help, so I’m guessing a bad assumption was made somewhere. Guess there’s just the registry hack left.
I have seen in other posts that for the Intel ARC770 you need to disable the REBAR parameter from the BIOS. Try that to see if it improves, note that if you are gaming as well, you will need to enable it for gaming.
I am not sure what the effect is if you are using a iGPU though.
Thanks, unfortunately not easily done because of my kludge set up. From the logs it does look like an actual defect in the code. The registry hack does gets me back to the usual speed again
It seems a strange message about OpenCL “Incompatible OpenCL runtime” for the Open VINO as this should only happen on a CPU that isn’t 6th gen or better. Almost as if it only sees OpenCL 1.2.
Copy the files from runtime/bin/intel64/Release into the root directory of Topaz. (e.g.: C:\Program Files\Topaz Photo AI)
It should ask you to confirm whether or not you’d like to replace the files. After that everything should work as expected. I still find it strange that benchmark software passmark still works. Maybe they don’t use OpenVino ¯\ (ツ) /¯
Confirmed, this fixed VEAI for me. Performance is even faster on this updated driver (4146) than it was before this driver broke Topaz. Thank you for for the solution!
I was getting the same issues using my A770 (export stopping after less than a minute, unknown error). I tried replacing the OpenVINO files and that appears to work.
Hello, back again! I missed one thing which causes issues with Photo’s sharpening and probably causes the unknown error in VEAI. Unfortunately I can’t edit the post anymore…
Rename openvino_tensorflow_frontend.dll from the downloaded Vino toolkit to openvino_tensorflow_fe.dll This should replace the existing openvino_tensorflow_fe.dll.
As for the video, I’ve noticed some failures too… I get a blackscreen then failure occasionally when I’m using the GPU for something else (e.g.: watching a VLC video). Unfortunately I don’t use VEAI as much as I do the Photo AI so there might be some combination that I don’t know about. I’ll try running a longer clip through VEAI tonight and report back if it succeeds
Edit: things like youtube also use the GPU. One option is to disable hardware acceleration in Chrome. Kinda sucks that we have to jump through these hoops but that’s what we get for trying something new
Edit 2: Woohooo! I got a video to finish exporting. Set the max processes to 1. It looks like there’s some funniness going on with having multiple GPU processes running at the same time. Maybe some context switching bug?
Sorta success… I get unknown error with motion deblur and stabilization turned on. Normal video enhance seems to work ok on the two videos I’ve tested it on. I guess we’ll just have to wait for Topaz to fix the issues.
Please let me know if we need to spin off a different thread for my issue due to the version.
As of Intel ARC drivers past 4125, VEAI 2.6.4 has the same issues noted above. Regardless of selection, rendering falls back to the CPU and doesn’t appear to utilize the GPU whatsoever.
If I apply the openvino patching noted above, VEAI 2.6.4 will close without error when attempting to preview or run any model. 2.6.4 appears to use openvino 2022.1.0.0 and I’m trying to use it with 2022.3.0.9052. Topaz Video Enhance AI.exe.2452.zip (4.3 MB)
It looks like it’s closing due to C0000419 FAST_FAIL_INVALID_ARG, but I can’t diagnose much further past this.
IMMEDIATE EDIT
For those of you with older VEAI versions, it seems to be fixed by only updating the following files:
openvino.dll
openvino_c.dll
openvino_intel_gpu_plugin.dll
You can leave the other openvino files as-is. This increased my Proteus rate from 0.96/s to 0.15/s on the A770 on driver 4255.
2nd Edit
There is an incompatibility with the calls to openvino_ir_frontend.dll in the latest openvino release and VEAI 2.6.4 that causes the crash. If you copy all the files EXCEPT the ir_frontend DLL, the optimizations will return.
