Hi - do you get this error in our app when the preview or export fails? Can you please send me the video? You can send it to Topaz Labs Dropbox Folder and let me know once you’ve sent something. Thank you!
Experimenting with the GUI and FFMPEG, I see that for my 3840x2560 video, setting alqs-2 and resizing with scale 188%, ffmpeg gets the following parameters:
-vf veai_up=model=alqs-2:scale=0:w=7220:h=4812,scale=w=7220:h=4812:flags=lanczos
But here’s the problem: the scale filter only gets a stream with a resolution of 3840x2560, so the AI really just upscales within the limits of the original resolution, and the lanczos scaling algorithm takes it to 7220x4812, which doesn’t improve the quality as much as it could… Setting the scale of veai_up to 2, I get 7680x5120, which would be fine, but the Meta Quest 2 doesn’t support more than 35 million pixels with H265, so I have to keep it just below that. What’s the solution here? Does the AI even support custom scales like 188%? Does it have to be scaled to 200% and be downscaled by ffmpeg afterwards?
Also, the preset p7 with nvenc should have much better quality than medium. NVenc only makes up <1% of the utilization anyway, so that added quality would be great for a few additional seconds of processing time!
@suraj - On Mac M1 MAX, 32GB RAM. First, let me say I appreciate all the work here and the problems you are trying to solve. Couple initial thoughts / Issues…I’m sure you have a lot of this in progress already, so take this as my first-blush reaction to trying the alpha.
-.Mp4 XAVC-S Files, and MXF XAVC-L files preview at just a few frames per second, and the GUI can get very slow. Pinwheels all over the place! 
The app has an overall slow and unresponsive feel that makes It clear that it is simply a shell to the ffmpeg backend, and this makes it feel like a clunky Linux style app like GIMP
-Lack of live preview slows down my workflow tremendously. For example, I use Proteus all the time. It is not uncommon for me to tweak the settings 15-20 times before settling on the best solution. Often, I just need to see a few frames before I know that I need to change setting. I understand that using ffmpeg as a backend might hamper this, but from a usability standpoint, lack of live-preview is huge for me. It is a big enough problem that I would continue to use 2.6.4 if this were not changed.
-The scale of the UI is huge. The sliders are quite large, and spaced far apart. Lots of screen real estate is taken up.
-It isn’t clear to the user how the choices on the right side connect to one another…I assume that ‘resize’ feeds into the ‘Enhance’ model selections…but ‘resize’ it is lower on the list. It looks like an entirely separate operation. It would be logical to think that ‘resize’ is simply a bicubic or Lanczos resize and not a parameter of the AI models. A node-based solution might be clearer (though more complex to program)…Or some other way to make clear how the various functions tie together and feed into one-another.
-On shots with a lot of camera movement, the stabilize function does not ‘catch-up.’ If a shot begins with a big move, and then settles on a largely static composition, very large AI-created borders remain on the image permanently - the algo doesn’t slowly catch up with the new composition, but tries to maintain some earlier framing…resulting in huge AI-borders.
-The AI-borders in the stabilization filter are great in principal…but I would never be able to use them in my film work…they aren’t useable in any scenario other than when they are barely seen. Will these see dramatic improvement? The Warp-Stabilizer in AfterEffects is the gold standard for post-stabilization (better than all third-party if you know how to deal with its quirks, and even Resolve). Unless VEAI gets as good or better, I would not include it in the product. I think it lowers the overall product standard to offer a feature that is not best-in-class. Your uprez AI is best-in-class…I’d hold back the stabilization until it is clearly head-and-shoulders better than other offerings out there - I’m sure you will get it there!!
Please take these thoughts constructively, and thanks for your work!
4 Likes
Thanks for sharing it.
Hopefully we’ ll get an update soon.
Regards
Feature Request:
Occasionally, I come across a video clip stretched wide or tall. I want a way to restore its aspect ratio to normal.
I am not using the VEAI3.0 GUI and am using ffmpeg with CUI.
When I use the VEAI3.0 GUI, the colors are altered because swscale is not set.
https://trac.ffmpeg.org/wiki/colorspace
Send a video comparing the video with and without swscale settings.
You probably should have tagged at least one of the devs in your post.
Fixed!
1 Like
Detailed logging both generates large files and also slows down processing. So keeping it on always is not the best case. Ideally, most issues should be detectible from normal logs needing detailed logs in rare cases. Most issues are reproducible for now so turning it on when required is fine for now.
Hopefully, we will need fewer and fewer detailed logs in the near future.
1 Like
Thank you for the detailed command options and files. Our focus currently is to get rid of the GUI sluggishness. Will fix the color related issues after that.
Thank you for your feedback, tag me if you have more questions.
-.Mp4 XAVC-S Files, and MXF XAVC-L files preview at just a few frames per second, and the GUI can get very slow. Pinwheels all over the place!
- Yes, the GUI is very sluggish. The GUI should be faster in the near future, pinwheel should disappear soon as well.
Lack of live preview slows down my workflow tremendously. For example, I use Proteus all the time. It is not uncommon for me to tweak the settings 15-20 times before settling on the best solution. Often, I just need to see a few frames before I know that I need to change setting. I understand that using ffmpeg as a backend might hamper this, but from a usability standpoint, lack of live-preview is huge for me. It is a big enough problem that I would continue to use 2.6.4 if this were not changed.
- Lack of live previews is due to the use of video codecs for encoding output, video encoders like to queue frames and write together leading to longer wait times. In the future we will allow instant previews by writing them as tiffs.
The scale of the UI is huge. The sliders are quite large, and spaced far apart. Lots of screen real estate is taken up.
- Can you post a screen shot and screen resolution.
It isn’t clear to the user how the choices on the right side connect to one another…I assume that ‘resize’ feeds into the ‘Enhance’ model selections…but ‘resize’ it is lower on the list. It looks like an entirely separate operation. It would be logical to think that ‘resize’ is simply a bicubic or Lanczos resize and not a parameter of the AI models. A node-based solution might be clearer (though more complex to program)…Or some other way to make clear how the various functions tie together and feed into one-another.
- The idea is to eventually get to a node based structure. For now they are just linear filters with parameters/options moving up and down the filter list. For example if no enhance model is chosen with resize the output is scaled using lancoz with no AI models running. The grouping of them is still work in progress and will be adjusted based on user feedback. The goal is to provide good enough presets so users don’t have to configure filters directly as much.
On shots with a lot of camera movement, the stabilize function does not ‘catch-up.’ If a shot begins with a big move, and then settles on a largely static composition, very large AI-created borders remain on the image permanently - the algo doesn’t slowly catch up with the new composition, but tries to maintain some earlier framing…resulting in huge AI-borders.
- Stabilization with content addition using AI is a very complex problem. This is one of the reason why the crop option was added to stabilization. The AI borders/region generation is still in training and hopefully will improve further. There is a lot of potential in generating regions using AI beyond stabilization for things like reframing, motion blur removal etc.
The AI-borders in the stabilization filter are great in principal…but I would never be able to use them in my film work…they aren’t useable in any scenario other than when they are barely seen. Will these see dramatic improvement? The Warp-Stabilizer in AfterEffects is the gold standard for post-stabilization (better than all third-party if you know how to deal with its quirks, and even Resolve). Unless VEAI gets as good or better, I would not include it in the product. I think it lowers the overall product standard to offer a feature that is not best-in-class. Your uprez AI is best-in-class…I’d hold back the stabilization until it is clearly head-and-shoulders better than other offerings out there - I’m sure you will get it there!!
- Model development is an iterative process. All our models started at various levels compared to state of art and improved from there. I’m hoping stabilization would be similar, I do agree that AfterEffects Warp-stabilizer is very good. We should hopefully be able to surpass it in the future.
2 Likes
We are switching over to a Tuesday weekly release schedule. Thanks for all the feedback, please be loud if the issue you reported is not fixed or even acknowledged.
The new alpha is posted here: Video AI 3.0.0.40a
3 Likes
