First off, I appreciate that Suraj provided the shell scripts for batch-downloading of models.
However I found a number of issues with them:
- Slow. Each network (model file) requires a separate launch of ffmpeg, with all the latency that spinning up the VAI engine requires. Took me hours and hundreds of VAI launches to get a few models downloaded .
- Unreliable. Sometimes VAI crashed during the matrix downloading, and I had no idea if the model files that were intended for downloading had been downloaded or not. So I had to run the script several times, and pray that I got all the files. I still didn’t know.
- Not comprehensive. Even after running the scripts, I noticed that VAI still wanted to download even more files. I attribute this to the specific set of resolutions requested in the script, which do not seem comprehensive.
To address all these issues, I created my own improved version, that is about two orders of magnitude faster (provided you have the bandwidth), downloads all model files required (to my knowledge), and is meticulous about handling failure. Also allows abort and restart from where you left off, without having to worry about you corrupting any model files. It will only download model files you don’t have, so you don’t have to figure that bit out yourself either.
It has a set of additional features as well:
- Ability to choose only the specific subset of model(s) and versions you are interested in. E.g. prob-3 and amq-13 for onnx16 or any other combination you like.
- Lists all the models and all versions of them with information about what they do, in an easy to view list.
For more details, view the comprehensive documentation on the PyPi repo, or the github project that’s backing the release. Give it a shot if you like. Testing, feedback and issue reporting are all welcome.
To install:
pip install vaimm
To use, for example:
vaimm download --backend onnx --include prob-3,amq-13 --dir . --cookie "$COOKIE"
Downloading: 100%|████████████████| 54/54 [00:18<00:00, 3.01 files /s, speed=93.12 MiB/s]
Download completed successfully with 1.63 GiB of data fetched.
Links:
- PyPI page
-
Github repo for issues and PRs
Note: the package is automatically built from source in github, and pushed from there to PyPi. So, you can validate the code and the build steps before you install it, if you happen to be as paranoid about third party software as I am.