Proteus is generating artifacts no matter what i do!

Hello everyone, i hope someone can help me, because i tried around for hours with no solution.
It seems like Proteus is always generating compression artifacts/ color banding on its own, no matter which settings i choose. It happens in every video (h.264 export is even worse)
I tried to upscale an 720p music video from youtube which was very compressed. At first i cleaned it up with Nyx and it looked a lot better. All the color banding and artifacts were gone. I exported that file with h.265 4mbps ( before it was 1,5mbps). Then i tried to upscale it with proteus to 1440p h.265 12mbps. I tried everything. Sharpening 0, remove compression 0/50/100, increasing bitrate to 40mbps and above, nothing helped! The strange thing is, that the Preview always looks fine, but the end product looks like compressed garbage. Could someone please tell me what im doing wrong?

Im using AMD 6600XT - Ryzen 5 5600 - 16GB Ram - newest drivers for everything

picture 1 - the preview from Topaz - Nyx 720p left - proteus 1440p right -looks fine!
picture 2 - the end result - dark background compressed to death with color banding
picture 3 - exported nyx 720p - (looked much cleaner before uploading it here…)
picture 4 - exported proteus 1440p




Did you choose Quality level high in the output panel?

Yes, i choose profile “high”.
For Bitrate i set a constant amount, since it seems to give better results than dynamic most of the time. But this problem persists on dynamic bitrate too.

Not Profile, Quality level.

Yes, with Dynamic Bitrate high, the problem is still there. I once had a video with a bitrate of 220mbps and this problem still happened.

Is Proteus producing different color banding and artifacts, or are they the same ones Nyx removed? Perhaps Nyx isn’t really removing them, just reducing them to near-nothing and then Proteus is enhancing them back up again?

If you haven’t already done this, try setting Proteus to Manual/Estimate and turning Improve Detail and Recover Detail to zero.

Please look, this is the png screenshot i made from the 720p nyx video

I always set proteus to manual and let it estimate the settings. Then i change it how i like it, but i use that as a guideline. In Proteus i added 60 detail, sharpening 0, reducing artifacts to 0 (because i already did that with Nyx!), reduce noise 0 (already removed it with nyx), dehalo 0 (already done with Nyx), Deblur 18. Recover original detail is set to 0 too. Then i added grain, but that shouldnt be the problem.

I already tried to reduce artifacts AGAIN with values like 35 (proteus estimated it), but the end result was the same. i also tried to reduce ,increase detail’’ to values like 15. Nothing changed. The end result is always crap. Increased detail is useless if i get color banding and artifacts again. But why the hell is the preview looking alright?

Does the preview still look ok if you zoom it up to 200% or so?

yes, of course i zoomed in :slight_smile: I even looked at it from different angles and brightness settings on my screen.

What happens if you use Iris or Artemis on this video instead of Proteus?

I tried Artemis medium and its the same bad result as with proteus.
When you compare picture 3 from my first post to the picture i sended later (the png) it is basically the same effect as with the video. Its like the video is compressed to jpeg and the preview was the png. :thinking:

Do you have this issue with any other videos? If all the models are doing it, and it’s only with this video, then it’s got to be something in that video. Could be that your optimum enhancement for that one is to upsample with Nyx only.

I also have this problem with other videos, but its more visible in darker scenes. I mean the image quality looks fantastic, but it gets ruined by those artifacts and banding. Its frustrating. It cant be the bitrate because even if its 100x higher than the source, the problem still occurs. It cant ve the “fix compression” setting either, because different values wont change anything. So what is the reason? Maybe its different when i choose “proRes” format, but i cant play those files (the VLC player is freezing and wont react anymore). Could it be a problem with AMD GPU’s? (But the preview looks fine!!)

I have AMD GPUs. If you want to post a short clip, I can try it here and see what happens.

Can you please tell me how to upload in original quality? When i upload it to youtube, its gets compressed again.
Maybe tommorow i can install Dropbox and try to upload the nyx file there. My Upload is very slow and its already late.
Thank you very much for your help.

Dropbox is probably the best way. You can also atttach uploads to direct messages here, but I’ve never tried that so no idea if they compress as well. Maybe if you try uploading them as zipped files.

Please upload a clip of the original source as well as your Nyx export. A minute or so should be more than enough.

1 Like

I would check if your monitor black levels are too high.

For ProRes playback I recommend Media Player Classic - Home Cinema (MPC-HC). I would actually recommend it for all playback.

When i watch the video on my phone (OLED) i still see the artifacts, so it cant be an issue on my Screen. And when you look at the pictures, you see the artifacts. If it would be a problem with my screen, it would look ok for you, but it doesnt, right? :saluting_face:

Here on my work monitor, it does look mostly OK :man_shrugging:

This is very weird. My phone screen is perfect and i still see those artifacts.