Linux support

Video AI Ideas
125

One of the big things driving the need for a Linux version is long-term stability for the processing of videos. I have a current project which is requiring just under TWO MONTHS of processing time using my currently available computer for a video which is just under 1.5 hours long. I am upscaling a 720p video to 1080p, using the AI features with motion deblurring.

To make use of Video AI, and the features above and beyond normal upscaling would be a “normal use” situation (the main reason to use this program versus ffmpeg or HandBrake), in my opinion. But when it takes a dedicated desktop machine almost two months to process the file, things like power quality issues, overall OS stability due to other programs being buggy, etc., and loss of that machine for other purposes start coming into play.

If a version were compiled to be used in a command line environment such as Linux, power quality might still be an issue, though more server-level machines are typically put on battery backup, but long-term stability and lower competition for machine resources due to the Linux machine not being the primary workstation the user would interact with would benefit the projects put on it.

One feature that could help mitigate problems with longterm machine usage, and potential interruptions to the workflow would be the ability to pick up the video processing at the last successful position. The processing could be started and stopped as needed without losing days (months?) of time when something went wrong.

I thought I remembered the previous version, Topaz Video Enhance AI, having this ability, but maybe not.

I would think it was absolutely essential for a main feature.

126

Linux support would be greatly appreciated. Some stuff in some versions work with Wine (with concrt140.dll native), but it’s a pain, and also doesn’t have gpu acceleration.

127

I for one would be very happy if there was a way to run the models on linux.

I already run most of my ML training and inference on my headless linux box. In fact, just like Topaz, I convert my pytorch and tensorflow models to ONNX for the exact same reason; portability and ease of inferencing with ONNX Runtime. No dependency on any other microsoft cruft. Definitely not DirectX as someone else speculated about in this thread. Native Linux runtime.

My workflow is as follows:

  1. Prepare source clips for a project in some GUI NLE on Mac or Windows.
  2. The clips that need enhancement, copy them over to a SAMBA share on the linux box and launch a CLI over there to run the number crunching (custom filters or ONNX ML models), over night.
  3. Use the processed clips in the NLE editor.

If the work is really heavy I spin up a small farm on AWS, and have the CLIs wrapping my models kick off the inferencing in response to file uploads to a certain S3 bucket. Since that uses segmented processing, what would take weeks on my local 32 core linux machine with one RTX 3090 is done in minutes to hours on a farm of cloud machines equipped with GPUs.

I for one would be content if Topaz just provided the passwords for the models tz zip files, and a document which describes what the model input parameters are, and the image (frame) preprocessing required to make the frames model-compatible + post-processing to turn the prediction results back into image data that can be serialized to TIFF/BMP/PNG.

Now I realize that the model files are half of the Topaz R&D investment, so I don’t expect you to do that. Instead, a compromise would be to offer a CLI with flags for configuring the model(s) to be executed. I expect this will be really easy for you guys, since you’ve already structured the code along those lines. You provide meta-data for the models (json files) that define input parameters already, so the code that pre-processes and hands the tensors over to ONNXRT I’d bet good money you would be able to just wrap a simple CLI around.

Having 1 instance of the CLI included in the license would be awesome. You can verify this the same way you verify the license in the GUI, just make a web call and write the license info to the machine the CLI runs on. For “Farm use”, there’s new revenue potential. A discounted “CLI” license for those of us who want to speed the processing up by using a set of beefier cloud machines.

For licensing inspiration, look at Duplicacy’s licensing. You’d need to discount the CLI significantly for farm use though, else you’d get zero sales. Have a conversation with us the community to help calibrate a suitable price point.

128

Can only speak for myself, and those I know who do anything ML related. Yes, CUDA 100%. AMD doesn’t exist in this space because they’ve never invested one iota in the ML landscape. AWS, GCP and other providers are all 100% CUDA, so start with that if you want to make money and minimize development investment :wink:

None. I prefer curl <site>/filename.tar.gz | tar xzf -
But really, anything would go, since I wrap all my production pipeline software in container images anyway. If you pick debian, ubuntu, alpine or something else common, I’ll just use a base distro that works with whatever package manager you go for.

I have zero need for this. Do the least amount of work. The FFMPEG dependency that you mentioned (even that is a stretch-goal, since I’d be happy with just having image frames written to a folder) and a CLI to kick the model inferencing off.

Thanks for the link. I definitely will !

129

Pro-tip. Extract the clip to individual frames and process those images. Solves both the “restart” problem, as well as making distributing the processing over infinite many machines trivial (well, OK feasible).

I’ve had similar use cases for a decade, and been doing exactly what I mention.

  1. Dump the meta-data about the clip (time codes, track info etc) and split the clip into its individual tracks (demuxing).
  2. Dump the video track to frames (a gazillion of them) on shared storage (NAS “at home” or buckets in the cloud).
  3. Process the respective streams with whatever filters you need in parallel (e.g. audio enhancement, noise removal, optical flow analysis, compensation, upscaling …).
  4. Stitch the images back into a video file using the video stream info dumped in #1.
  5. Update the time codes if any frame rate manipulation was done (e.g. decimation, interpolation, conversion …).
  6. Mux all the streams back together into a mezzanine clip (“master”).
  7. Perform ladder encodes on the master clip.
  8. Archive or stream away :slight_smile:

When I now need to process a clip, instead of waiting days for it, it takes minutes. Just spinning up 1K cores in the cloud makes short work of most of what I have to wait around for. Many small machines means linear scalability, faster results and lower $$$. You would be amazed at how inefficient most any software is at using multiple cores on a single machine due to Amdahl’s law and I/O bottlenecks.

130

Since the devs asked for proof that there’s demand, I’m just adding my comment here. I only use Linux so can’t even use the free trial at the moment.

I’ve tried installing with Linux/Wine but not even that seems to work any more, as per another thread in this forum

131

Thank you, suraj! I will be joining the linux beta group.

Even though I mentioned several things in my earlier post, my most basic reason for not just wanting, but needing a Linux/Unix version, is that even if I had a desktop that was tricked out, and cost thousands of dollars, there are just too many things I have to use it for which could potentially compromise the processing of a video file.

Stability and longevity of processing are what Linux brings to the table. I don’t even care about a GUI version on linux if I have some way of passing the processing off from my desktop to the Linux box, or cluster, if I can set that up, and just let it run in the background for however long it takes, freeing up my Desktop for the day-to-day stuff.

On my desktop, Video AI, for a “simple” upscale with motion deblur on a 2 hour movie was going to take close to 3 to 4 months to process. I wasn’t even trying to take it up to 4K.

I just don’t have that kind of stability built in to my desktop, and can’t protect the investment in processing from all of the instability-producing, multiple use, and multi-location things I have to have my desktop available to do.

I’m sorry, but I’m not rich, and try to make the most use of existing, and older computers I have available. Therefore, I don’t have a tricked-out computer which can do that much processing in short enough time to process a video file so that it wouldn’t be jeopardized by something changing due to my desktop being needed for something different.

Stability and long-term processing are my most important reasons for needing a Linux version.

I need to be able to submit the processing job, or directory of files to be processed, and check back every so often to see what has been completed.

132

What kind of machine are you using that takes 24 mins to process 1 second of video ?

133

Of course your calculation would depend more on how long the video was, but you’re not far off.

It is the fastest machine I have available right now, but I can’t devote it to the task of processing since I have to use it for various other things throughout the day. However, it is a 2.3 GHz Dual-Core Intel Core i5, with 8GB memory, and an Intel 1.5GB graphics card.

Which is why I am hoping for a more stable Linux version to let me process videos on another machine in the background, and then come back and check on progress every so often.

I would love to be able to throw money at the problem and get a fast machine to reduce the processing time, but that isn’t my reality right now.

I believe I was trying to upscale an older movie to 1080p, and was trying out the newer motion deblur option. I didn’t think I was going crazy with the options, but apparently it takes quite a bit of processing power to do it quickly, which I don’t have available right now. Hence, the need to offload the processing from my main computer to a Linux box that would be stable, and that I could just leave running with the job.

134

Flatpak makes the most sense. It is a distro agnostic distribution format/platform that “just works”. Containerization is the future and many Linux apps are heading that direction.

135

Great to hear that Topaz apps will be on Linux! Very exciting news, sign me in

136

so the video ai for linux is working ok? would love to test it out myself.

137

Definitely and please bring the Linux support. I don’t know a single engineer and I know quiet a few, who works with Machine Learning on Windows or Mac. All of us are on Ubuntu. My Ubuntu desktop has multiple GPUs and I have no choice, but to run on my Windows laptop, which has one GPU that is nearly not as fast as the desktop GPUs. I’m all in for Linux. Thank you.

138

Yes, all AI is on Linux. And many software developers are as well.

So please give us a native Linux version!

139

Had it for a while

https://community.topazlabs.com/t/topaz-video-ai-linux-beta-v4-0-7-0-b/58180?u=menditsa

140

Topaz Video AI is a wonderful tool already, but many users do not have access to either powerful or multiple computers. The temptation is to use our main or most powerful computer to do the video rendering, but the downside is that this computer is now tied up, many times for days or weeks (months?) trying to clean up, or upscale, and render one, or more, video(s).

I know many have already requested this, but I’m also requesting a Linux/Unix version which supports both GUI and CLI, much like other video processing projects do, such as HandBrake, and of course ffmpeg (mostly CLI).

Many users have one or more Linux/Unix boxes because it is less expensive, and allows reuse of older computers which have outlived their usefulness as desktops.

If Topaz Video AI could offer a Linux-based version which opened up CLI access, it would allow background, and possibly batch, processing of video files using our older, non-primary computers, so we could continue using our main/higher powered computers for day-to-day projects.

Network Rendering would be another request. If the program has already been optimized for multithreading, it should be well on the way to take advantage of implementing network rendering, so we could use multiple, older computers to help with reducing the rendering time.

The master computer could either be the desktop/workstation (Windows or Mac based… possibly Linux), or could have a Unix box designated as the master for the render queue, to render by itself, or with the help of slave computers to network render the queue jobs. The job or queue itself could be moved as needed back and forth between the desktop/workstation or the Unix master box on the network. To simplify interaction with Topaz on model updates, the designated master would be the computer to pull updated models from Topaz, and could share or handoff those updates with the desktop/workstation.

Of course, any feature/version request is normally very simplistic compared to all of the work involved to get the program working, but the additional features could be monetized to offset the expense.

It would be nice to have all of the features included in a one-program price, of course, but additional levels of enhancement could be offered as an add-on, or a more complete version for additional cost.

This would also position Topaz Video AI to not just be a workstation-only program, but also have the option of being adopted as a workflow component for larger content companies looking to batch process multiple videos.

Thank you for considering these suggestions!

141

I won’t be responding to the entire message.

The team at Topaz Labs has released a beta version of a CLI only Linux version of TVAI. It is kind of limited (Nvidia is the only supported GPU vendor to my knowledge) and when I tested it, all my exports on the GPU had large artifacts. The CPU didn’t have any artifacts, but it’s slow.

Another issue is that the Linux version is low priority so it hasn’t really recieved any updates and as a result it’s out dated compared to the latest public release (It’s missing the performance increasing enhancements and the new motion deblur AI filter)

For reference there has been a request for a Linux version in the past (see the thread linked below).

You might be wondering, how can you access TVAI for Linux? You have to be part of the closed group of beta testers. I believe the process to join is to go here Topaz labs video beta testing group - Topaz Community and click Request Access, write up why you want to join the beta team, submit it, then wait for a response to see if you get accepted into the beta or not.

Here is a summary of the release notes for the Linux Beta from November 2022

We have been working on a Linux version of Video AI for use on some of our training machines. While this is mainly an internal tool at the moment, we’ve decided to release a build for any interested alpha/beta testers.

A few important notes before starting:

  • This is the first time we’ve tried packaging a Linux version of our software for external use, there may be dependency problems and other issues.
  • This version is not officially supported at this time.
  • This build is CLI only
  • The Linux version will be updated much less frequenctly than the regular Mac & Windows releases.
  • We have very few encoders enabled in the distributed FFmpeg build (even fewer than Win/Mac). Headers and libraries needed for a custom build with your preferred encoders are included.
  • TensorRT models are currently unavailable, as they require reconversion on our end.

System Requirements

  • Ubuntu 22.04 or newer
    • or a distro with GLIBC 2.35 at minimum
  • libvulkan1 (maybe)
  • Proprietary NVIDIA drivers
  • For ONNX (GPU) models, a Vulkan-compatible NVIDIA GPU meeting Video AI’s minimum requirements
  • For CPU models, a CPU supporting AVX2 instructions
142

This idea has been marked as “processing” for almost a year. I wonder how far it’s come.

Releasing the Video AI for Linux would definitely give them an edge over competitors, as the only professional video editing program with AI capabilities that runs on Linux atm is DaVinci Resolve. So, I support this.

143

It is even a little puzzling why they don’t at least release the engine itself on Linux. All the main components used in the tool chain are Linux software in the first place. Both ffmpeg, the ML engines and even the Qt framework used for the GUI.

144

As a beta tester of the Linux variant I think I can answer that. It still has a long way to go. Not all models work yet, performance is significantly lower than on Windows, and it currently has limited hardware compatibility preventing it from working on a sufficiently large set of the most common hardware. It needs more “love” before it’s in a shape where it’s acceptable for actual use. Progress is slow, but every other month or so, Topaz has been making small incremental progress. Why it isn’t progressing faster, I suspect, is because not many customers have loudly and visibly been drumming impatiently for this version.