Hi All,
After using VEAI for a few months, going through its code base, the list of issues, the feature requests and many frustrated customers, it has become clear that VEAI has to change significantly to evolve into a product loved by all users, beginners and experts.
All the issues and feature requests can be categorized broadly into 3 types:
-
Video read or write:
No support for various output codecs or even codec profiles
No pass through streams
Issues handling various color spaces
Inability to handle variable frame rate
Meta data and other streams of information are lost
Audio channels and bit rate changed to low quality
Frequent crashes on native M1 builds while opening certain mp4 files -
Workflow:
Periodic re-login is required
Ability to work offline has become hit or miss
Missing command line support
Inability to play around with model versions or tweak model parameters in the json files
No ability to pause or recover after processing for days
The current workflow of using VEAI is very time intensive with tons of waiting for processing or preview to finish, preview lengths are fixed and huge memory hogs
The UI needing to process video again and again even though it was processed before
Need to use external programs to preprocess videos before using VEAI
No availability of plugins to make VEAI part of a video editing pipeline
Slow performance or not using all of the GPU resources -
Models:
Artifacts or no significant quality improvements
Inability to switch between model versions except the ones visible in model manager
Manually having to fix frames and color after using VEAI
Request for new models to do solves different problems
Most of the problems in VEAI stem from the fact that it treats video as a series of constant rate images. All the current models run on these assumptions and as a result have issues processing variable frame rate videos, frame repeats, scene changes etc. This also hampers development of future models like stabilization, scene detection etc. that require multiple passes over the video.
To address these concerns and allow seamless additions of new features, we have decided to rewrite VEAI as a new application with a whole different architecture and way of doing things. Starting today and over the next few weeks we will be sharing various alpha and beta releases of VEAI 3.0 in this category.
VEAI will consist of two independent parts: the GUI and the backend FFmpeg filters. The GUI will merely call the FFmpeg backend and spawn separate processes. This will finally allow VEAI to utilize the newer RTX 30 series GPUs to their full extent. Currently users have to use multiple instances of VEAI to optimally use the GPUs.
Video IO
The use of FFmpeg directly will also resolve all of the video read/write issues. These filters will be open source and will allow interested users to use our C API to write plugins, filters etc. for other applications. We would also be adding more plugins/filters for other applications in the future.
Our public FFmpeg repo with filters is hosted here
GUI
Create and store projects, so batches, previews and settings can be retained
Ability to create/modify and share presets
Customize the FFmpeg command being run to allow use of other filters from FFmpeg
Users will now be able to process a video in the background and continue using the app
Complete offline support with an option to download all models (that actually works)
New flow to allow login for machines that are completely offline
User editable model and preset files for easy sharing
Retain previews on disk, eliminating need to process already previewed video
Process video using multiple models at the same time
Generate previews of whatever length you might want
Recover from crashes and continue processing
Ability to pause and continue on another machine
GUI alpha/beta will start next month
Models
VEAI 3.0 will also introduce a stabilization model that will allow users to stabilize their videos and keep the original resolution. The model will not only smooth out the camera motion and reduce/remove camera jitters it will also predict the missing frame data when the correction is applied. Expect the FFmpeg filter for the model in the next couple weeks.
Ideally this approach would allow us to focus on our core expertise to create and improve our models to perform better on all user videos.
Please let me know and start a discussion below if you have any suggestion, concerns, etc. about the current roadmap for VEAI.
Thanks
Suraj
!
)