Took a while to find the post Richard mentioned. Here’s a copy of the post, as I could not generate a direct link to the great post by Joel Hruska. No intention to plagiarize; just to facilitate the discussion here.
General advice to the group: Use Proteus, but ignore the auto tuner. The auto tuner can be useful, but it does not give the kind of achievable quality boost that Proteus can.
How to Configure Proteus:
When testing Proteus against content, you should use the following settings:
Revert Compression: Between 20 - 75.
Enhance Detail: Test a range between 20 - 75. Above 75 is almost always too much. Below 20, and the improvement is quite small.
Sharpen: I recommend keeping this very low to off.
Reduce Noise: I denoise with other applications and only use this option a little. Recommend settings of 10 - 25, if used. If you perform noise processing elsewhere, you may be able to leave this off.
DeHalo: If you have haloing problems you can turn this up. If you don’t have haloing problems, don’t. Leave it low in any case, it’s damaging to detail. I recommend testing settings of 0 - 25.
Antialias / Deblur: This is one of the most important settings, and how you use it varies by source. You need to test both negative and positive values on your content while holding other settings constant.
For example: Here is a frame comparison between two different Proteus settings. They use the same value for five settings: 60-60-0-0-0-30, and 60-60-0-0-0-(-30).
I’m holding most values at 0 so you can see the impact of the two antialiasing values. This particular frame has a lot of fine detail in it, so it’s a good test for the idea.
So what happens when we vary the first two options? I recommend keeping them similar, though I sometimes let one slide 10-15 points against the other (60 / 75 or 50/65, etc).
This comparison shows 60-60-0-0-0-30 against 45-45-0-0-0-30:
All of the versions of this frame that I’ve showed are promising, but none of them are exactly what we want. One set is too blurry. The other set is a bit too sharp. What do?
Let’s add some secondary settings. Here’s 45-45-20-10-30 compared against 45-45-0-0-0-30.
We get some improvement. It’s not enough. In a few cases, the new settings actually make the problem worse.
This might seem to be a showstopper, but it shouldn’t be. There are two ways to solve this problem:
1). You can spend the time to test Proteus settings carefully, making very fine adjustments. You may or may not get exactly what you are looking for.
2). Start blending in output from other models using an application like DaVinci Resolve. You do not need the paid version to do this.
The comparison below shows 30-30-0-10-5-20 against 60-60-0-0-50 – but the second encode’s output has been blended with the result from a standard ALQ run.
Blending back the data from a different encode makes a difference in the final quality. Here’s a comparison of ALQS blending against Dione:
All of these outputs can also be adjusted depending on your AviSynth encode settings. Sometimes you may want to perform less sharpening or other image processing in AviSynth in order to push more work over to Proteus.
Hopefully these comparisons are helpful as far as output tuning and what the various possibilities can be. Do not be afraid to screw around with Proteus tuning. Try both positive and negative Antialias / Deblur values, and if one doesn’t work, test its opposite (-30 / 30, -50 / 30, etc).
