Nvenc AV1 output quality

Topaz Video AI General
1

Has anyone experimented with output directly using nvenc AV1 encoder that’s available to RTX 4000 cards? I know hardware encoder suffers in general with compression efficiency/quality to achieve speed. But if I allow a reasonable bitrate and select a slower preset for the AV1 encoder, can I achieve output that’s similar in term of size and quality? If so, I can replace ProRes intermediate step and subsequent software x265 encoding with direct output to AVI in Topaz during upscale step. This way I can offload some of the CPU load during upscale to my GPU.

All the testing that I can find are aim at streaming where they are focused on quality comparison at lower bitrate. Those comparisons are not helpful for me since I won’t need to limit my bitrate to some lower value.

I’m mainly doing 1080p to 4K upscale to view my favorite films on 4K TV more enjoyable. So I am aiming toward preserving as much of the detail as possible with reasonable file size and encoding time. I have been using software h265 encoding (CRF = 18 and medium preset) to convert the intermediate ProRes output to final output that I am keeping.

3

Hello,

AV1 is generally more efficient than HEVC encoding, but for high quality output with less priority on file size reduction I would recommend trying -c:v av1_nvenc -preset p7 -tune hq -cq:v 20 -rc-lookahead 48

-rc-lookahead can be adjusted based on the framerate of the input file, but usually it’s recommended to set this to double the framerate.

Here’s a good resource for comparing different options in the CPU and GPU encoders.

5

Thank you for the tip, -cq:v 20 option is similar to crf? Does using -crf 20 do the same thing?

Also if I want 10 bit output, -pix_fmt yuv420p10le this is correct?

6

-cq:v does work similarly to -crf, but the values are not exactly equal. 20 should be relatively high fidelity but the best way to check would be to compare the AV1 output with the input file using either PSNR or VMAF.

yuv420p10le will produce a 10-bit video with 4:2:0 chroma subsampling.

7

One option you have to off load CPU load during upscaling is to use the GPU encoder in a really high quality mode, or even a lossless mode, then transcode with x265 later. That way you get no CPU impact during upscale, since the GPU is encoding. But you still need to use the CPU later on during for the x265 transcode.

For example you can use the hevc_nvnec (h265 nvenc) encoder in lossless mode using these setting with ffmpeg (what Topaz Video Enhance AI uses for encoding): -c:v hevc_nvenc -tune lossless

But you may want to remove the CPU entirely (E.G. No CPU transcode later on) and switch to AV1, in which case I have included some extra information below related to the AV1 stuff.

Please make sure your TV, or the device you’re using to play the video, supports AV1 playback at reasonable performance levels before making the switch to AV1. Most TVs don’t support AV1 playback, and only modern GPUs support hardware AV1 decoding, and not all major video players have support for hardware decoding (VLC 3.X doesn’t support hardware AV1 decoding).

Software decoding is a thing, but if your CPU isn’t fast enough, you may not be able to play back 4k AV1 video reliably.

I will be running some tests tonight comparing av1_nvenc to libx265 and post the results when I get them.

As you pointed out, hardware encoding typically suffers in compression efficiency in exchange for speed compared to software encoders. But when it comes to AV1, the trade off may not be that large.

Yes, hardware encoding will be worse in terms of quality compared to software encoding. But AV1 software encoding is SO SLOW for high resolution content on average consumer CPUs that the trade off is probably worth it.

As for how av1_nvenc compares to libx265 that’s a more interesting question that you probably can’t find an answer for. When I get my tests done, I could potentially answer that.

av1_nvenc, the Nvidia AV1 encoder doesn’t support CRF, so I believe -crf 20 will not work. CQ is slightly different from CRF, but apparently it behaves similarly to CRF. For reference, CRF is “Constant Rate Factor”, CQ is “Constant Quality”.

av1_nvenc does support some other modes, Varibale Bitrate, Constant Bitrate, Constant Quantization Parameter, but you probably don’t want to use any of those if you want to output directly from TVAI to a video file you plan to keep around long term and watch.

Based on testing done by myself. H264 is better at preserving detail than AV1. Although I am comparing libx264 to svt-av1 at high bit rates (E.G. 50mbps). Low bit rates are a different matter.

This seems to be because AV1 priorities providing a “clean” image rather than a detailed one. A result of this is that svt-av1 looks better when you’re watching a video, but it has less detail and thus looks worse when you pause it and compare the compressed videos to the original source material (an admittedly uncommon use case).

So if your goal is to maintain as much detail as possible, AV1 (at least svt-av1) isn’t for you. But if you want a high quality compressed video, AV1 can do that.

An extra thing that might help. You can find a full list of av1_nvenc settings by running this in a terminal: ffmpeg --help encoder=av1_nvenc

8

Actually, it seems the link provided by @tony.topazlabs already answers that. 画質比較 (2022.12)

At the same bit rate, av1_nvenc has a higher VMAF score (higher is better) than libx265 with the medium preset.

This means av1_nvenc provides “better quality” than libx265 at the medium preset.

libx265 can still “beat” av1_nvenc if you change libx265 to a slower preset. But since speed seems to be a concern for you. I feel like you may not be comparing libx265 veryslow to av1_nvenc.

9

Wow, thank you very much for the detail explanation.

So if I want to go with the route of generating a high quality intermediate using GPU encoder (to reduce CPU load), then followed up a second encoding step, which (h265 or h264 nvenc) is better suited for this? What kind of file size can I expect for using lossless (similar to ProRes size)?

Not familiar with ffmpeg command, but does the following make sense? Anything else I should add?

-c:v hevc_nvenc -profile:v main10 -preset lossless -tune lossless -pix_fmt p010le

do I need both preset lossless and tune lossless?

10

What I assume you’re doing is you’re using nevnc in lossless mode to export from TVAI, then using libx265 to make the final video you’ll watch on your TV. In which case I would recommend using hevc_nvenc for the lossless step. h264_nvenc has more limitations than hevc_nvenc.

Running some quick tests, comparing ProRes 422HQ (pix_fmt yuv422p10le) to hevc_nvenc lossless (pix_fmt p010le), hevc_nvenc is smaller (3/4 the size of the ProRes video). But that’s probably because of the different pixel formats.

No, you do not need the preset lossless, it’s mostly a left over thing from older version of NVENC. It should just be:
-c:v hevc_nvenc -profile:v main10 -tune lossless -pix_fmt p010le

I just wanted to add another note. I was like you. I wanted to export my videos with TVAI in the highest quality possibly then transcode it to something else later with a software encoder. I used to use HEVC_NVENC lossless for my TVAI output. But it took up a bit too much space for me. So I switched to just “high quality HEVC_NVENC”.

My personal settings are:
-c:v hevc_nvenc -profile:v main -preset p4 -rc constqp -qp 0 (I use profile main since I work with 8 bit content)

For you this might look like
-c:v hevc_nvenc -profile:v main10 -preset p4 -rc constqp -qp 0 -pix_fmt p010le

To the average human, it’s identical to lossless. To a computer it is different, but the differences are so tiny it doesn’t matter. And it can offer some space savings for your intermediate step. Anywhere from 10% to 50% space savings.

1 Like
11

I am really interested to know your Finding. since AV1 is much slower then x265 both in encoding and decoding, It interesting to know if the file size gap between the two (same quality give or take) is worth the hardware power and time or its already a diminishing return.

maybe will have an effect and noticeable if you planing slow motion your video… WDYT?

12

I probably won’t do this test, you can already find the answer to this investigation from a different source, see this comment: Nvenc AV1 output quality - #8 by 20rushtonj

1 Like
13

Thank you very much for the helpful tips. Really appreciate it.

14

This is getting a bit off topic. Maybe we should open up private messaging to each other or start a new topic.

Just for reference, we are talking about the issue where svt-av1 is unable to retain as much detail as libx264 at high bit rates. As such when you pause the two compressed videos and compared them to the source material, libx264 typically looks better due to the increased detail. However if you were to just watch the two videos, libx264 and svt-av1 both look fine.

It depends.

At low bitrates, svt-av1 looks, and is, higher quality than libx264 in most videos.
At high bitrates, svt-avt and libx264 will both look good, especially when the video is playing. But libx264 will have more detail most noticeable when the video is paused.
At a medium bitrate, things get much more complex.

So let’s run over some different scenarios with “slow motion” to see when each encoder is better.

Let’s say you have a video, you run it through TVAI with a slo-mo filter, save it in a lossless format, and you want to know if you should transcode that video with libx264 or svt-av1 to save space. svt-av1 would almost certainly be the best bet simply because it can get to a “high quality” with fewer bits than libx264.

Lossless video → Slow-mo → compress with svt-av1 or libx264?
Use svt-av1

But let’s say you have a lossless video. You transcoded it with libx264 AND svt-av1 with a high bitrate to try and keep as much detail as possible. And you now open it in a video editor and slow it down (so now a single frame from the original video displays for 20 frames in the video editor), or you want to zoom in and slow down the video with a video editor. Then the libx264 video will look better due to it’s ability to retain higher detail than svt-av1 at high bitrates. If you were to use a lower bit rate, then svt-av1 would look better simply because it looks better than libx264 at low bitrates.

Lossless video → transcode and compress with svt-av1 or libx264? → slo-mo in video editor
Use a high bit rate libx264.
If you are limited to a low bitrate, use svt-av1.

If you were to take a lossless video, and you wanted to compress it while retaining as much detail as possible, and you had plans to run the compressed video through the TVAI slo-mo filter some point in the future, and you had to pick between libx264 or svt-av1. Then a high bit rate libx264 would be a better pick than a high bitrate svt-av1 due to the extra detail it keeps around that TVAI could use in it’s processing. But if you were to use a low bitrate instead, a low bitrate svt-av1 encode would probably be better than a low bitrate libx264 because svt-av1 will have fewer artifacts which the models in TVAI would probably have an easier time working with.

Lossless video → transcode and compress with svt-av1 or libx264? → slo-mo in TVAI
Use a high bit rate libx264.
If you are limited to a low bitrate, use svt-av1.

Note: This is all based on my testing, which could of been flawed.

How do other codecs (libx265, libvpx-vp9, libaom-av1, nvenc variants of h264, h265, av1, QSV variants, etc) compare in all this? I don’t know, I didn’t test them, and I don’t plan to test them.

1 Like
15

I know we are digressing , but question from your last long response. I saw you provided use cases between AV1 and h.264. are the same uses cases valid between AV1 and h.265? or with x265 it is totally different (AV1 vs. x265)? forget a second the file size. let’s focus on the use cases , the details retaining , better looking, high bit rate, low bit rate, etc. the same uses cases.
if i use x265 and not x264, can i take the same examples and apply it as is for AV1 vs x265 or it’s only valid for AV1 vs. x264?
I know you did not test them all (you mentioned it at your post), but based on your cumulative experience (even though it might be more of a “guess” / “assume”).

16

I really haven’t played around with h265 much outside of using libx265 in lossless mode and hevc_nvenc in close to lossless. As such I have no idea how h265 behaves in real world use cases and thus can’t even guess how h265 encoders compare to AV1 encoders or how they compare to h264 encoders when it comes to detail retention and other factors at different bit rates.

I should also note that AV1 encoders typically have a lot of settings you can tune. In my tests I was doing a simple -c:v libsvtav1 -preset 6 then setting the desired bitrate and stuff.

1 Like