First off, have you opened a Ticket with Topaz yet?
When you pay for a subscription service it better work and if not they should either help you figure it out or refund your money.
Your card is not that old, but you didn’t list Operating System. Besides you might have corrupt software or a driver issue. AMD graphics cards have not been embraced by a lot of companies yet, so it could just be a compatibility issue. AMD graphic cards have to be program differently than NVIDIA or even Intel built into the CPU graphics.
Also, there is the fact this company doesn’t guarantee that their software will work well on a retail graphics card. They just say it should be doable.
I haven’t been able to figure out if a Maxwell graphic card is still supported with their software, but Pascal runs like a champ. If you’re not doing this professionally and got time to spare. Try getting your hands on an a refurbished small form factor business computer off eBay and slap an NVIDIA Quadro P2000 in it. As long as it has at least 16GBs of ram in dual channel and a couple of SSD drives in it. It should upscale 2 or 3 TV shows a day with no problems.
My dumb butt wanted it all in one machine and eventually figured out that an NVIDIA Quadro P4000 could do everything I could want for now without having to use any system memory. Got lucky and only spent $500 bucks on it and it was a spare card for some business too, so the packaging wasn’t even opened.
I’m a crazy man though, so I don’t recommend my route. Plus running two graphic cards that don’t act as one graphics card somehow, does get complicated. Like having to tell each program your using which card to use. Couldn’t even watch YouTube without doing that step.
Best of luck and remember to put in the solution into this threat if you figure it out.