This explains how zmon works…
zmon design
The script is written with bash included with macOS. I know there is a new version of bash on brew, but I didn’t lookinto it.
in this post, I will explain critical elements of the design so you can adapt to your needs. Sometimes you will see snippets of code that are non-trivial (at least to me). I am not a scripting expert (I am a more 'traditional" programmer), and I will be happy if you can show me how I could improve!
Working folders
I have created a folder called “Video AI” on my home location : ~/Video AI. Below I have 3 folders I call src, work and backup
- src will contain the video source files
- work will be the output folder. You have to select it everytime you start Video AI, even if you set it up as default, it will not automatically be selected when you export.
- backup is where the frames will be backed up.
You could have backup on the same physical drive if you have enough space or use a shared folder if you have another machine or a NAS.
ln -s /Volumes/NAS/Video\ AI ~/Video\ AI/backup
This will create a soft link in your ~/Video AI folder called backup and will point to your NAS (you need to have it mounted of course)
Parameters
If no parameter is provided, zmon will monitor Video AI on the current machine. If parameters are provided, zmon will monitor local or remote machines.
- local machine: parameter needs to be the name of the host (obtained via hostname -s)
- remove machines: zmon will use ssh to run a portion of the script on the remote machine. ssh key pairs must be setup to enable pre-anthentication from the monitoring machine to the remote machines
list of parameters will be placed in an array called files
read -ra files <<< "$@"
Note:
- <<< passes a single string directly to the standard input (stdin) of a command (Here string)
- $@ contains the list of parameters
The script will loop indefinitely and check every machine each minute until stopped (Ctrl-C).
Definition and usage of variables passed in the loop
zmon will communicate between local and remote machines by passing one string containing multiple elements separated by pipes “|”. The structure of the string that is passed as arguments back and forth is as follows:
| Variable |
Utilization |
| MAC |
machine name |
| VIDEO |
video name |
| OUTPUT_ROOT |
root of output folder |
| STATUS |
status |
| CURRENT_OUTPUT |
current frame # |
| GOAL |
last frame # |
| SPF |
Seconds per frame |
| FINISH |
estimated finish daye/time |
| AGE |
how long ago was the last change |
| EARLIER |
time of previous frame |
| NEED |
space required to complete processing |
| AVAIL |
space available in GB |
| MSG |
optional message, e.g. used when new frames are added |
This will retrieve all “variables” passed as one string argument
IFS='|' read -r MAC VIDEO OUTPUT_ROOT STATUS CURRENT_OUTPUT CURRENT_BACKUP GOAL SPF FINISH AGE EARLIER NEED AVAIL <<< "$1"
this will “return” all “variables” for the next loop
echo "$MAC|$VIDEO|$OUTPUT_ROOT|$STATUS|$CURRENT_OUTPUT|$CURRENT_BACKUP|$GOAL|$SPF|$FINISH|$AGE|$EARLIER|$NEED|$AVAIL|$MSG"
I use pipes “|” as separators so I can have spaces in variables. MSG is temporary and never passed through the loops.
I’ve read there might be better options with newer versions of bash, but I’ve decided to stick to whatever is provided in macOS for now
zcheck
This is the main function that will be executed locally or remotely via ssh. zcheck will return state variables that will be displayed and used for the next loop
Local processing
files[$i]="$(zcheck "${files[$i]}")"
Remote processing
SSH=$(printf '%s\n' "$(typeset -f zcheck)" "zcheck \"\$1\"" | ssh "$USER"@$MAC.local "bash -s -- \""${files[$i]}"\"")
When successful:
files[$i]="$SSH"
Calling ssh with passing parameters and returning something is a little tricky, I did consult AI to figure this one out! Note $USER is used to autherticate (name of the current user). .local is added to the name of each machine (should be on the same LAN)
subfunctions of zcheck will be explained later
displayStatus
After completion of zcheck, displayStatus will be run (locally) to display the information returned by zcheck. it will take care of formatting and some color coding
The following information will be displayed
| Header |
Content |
| Machine |
Name of machine |
| Video |
Name of video being processed |
| Status |
State of machine (see below) |
| Estimated |
Estimated day/time of completion |
| Sec/fr |
Seconds per frame |
| Done (%) / Total |
# frames in backup, % and # expected frames |
| Last update |
How long ago a batch of frames was generated |
| Required/avail |
Space required / available in GB |
State machine (STATUS variable)
The STATUS variables will contain different states that will change as more information become available from the log files.
| STATUS |
Meaning |
| - |
monitor just started, no information available from log files |
| STARTED |
video processing started, no frame generated yet |
| SYNC |
frames available, in sync with backup folder |
| >xxx |
xxx more frames are available in the backup folder. This can happen on rare occasions during transition to the next folder. This is normally resolved at the next loop. |
| <xxx |
xxx more frames available in output folder. This can happen when rsync is “behind schedule”. This is normally resolved at the next loop. |
| INACTIVE |
no new frames have been seen in a while. As the scripts calculates the amount of time it usually takes to generate one frame, knowing they come in batches of ~100, if we haven’t seen anything in that time + 50%, there is a problem |
| DEAD |
script cannot find the Topaz Video AI or Neuroserver running |
| PAUSED |
One process is still alive but not using a lot of memory. This could be an indication of a “Recoverable error” (which is usually not recoverable!) |
| NOFOLDER |
output folder or backup folder removed!!! I have seen this happening after resuming from crashes!!! |
| LOWSPACE |
there will be <10G of disk space on the main machine after estimated completion of processing. This could be an issue where Video AI decides to stop. |
| NOSPACE |
there is not enough space available in backup to store all frames. Script will calculate the average size of one frame and multiply by the remaining number of frames to be processed |
| DONE |
processing is complete, script will reinitialize itself and wait for the next video in queue |
zcheck subfunctions
- readlog
This portion will look for location of log files (release or beta version) and take the most recent one. the log file will be parsed to collect information to monitor the video processing
- location and name of video source file
- location of output folder
-
of frames to be processed
- time when processing started
-
of frames already generated and timestamps
zmon can be stopped and restarted at anytime, and will attempt to resume based on what’s in the log file
Folders scanned are:
~/Library/Application Support/Topaz Labs LLC/Topaz Video/logs
~/Library/Application Support/Topaz Labs LLC/Topaz Video BETA/logs
zmon will only look at lines that contain **Thread: **and program or Processing. if Topaz changes the content of their log files (which they can without notice!), zmon will have to be updated
while IFS= read -r line; do
... analyze log file
done < <(grep -E 'Thread:.*(program|Processing)' "$LOGFILE")
Note (with a little help from AI on this one):
- The syntax < <(process) combines Input Redirection (<) with Process Substitution (<(…)).
- It tells Bash to run the process in the background, treat its output as a temporary file, and feed that file into the command’s standard input (stdin).
- checkprocesses
This calls ps to check for Video AI and Neuroserver processes
PS=$(ps -ax -o %mem,command | awk -F- '/Topaz/ && !/tail|grep/ {print $1}')
if %memory utilization is <0.3, zmon considers this is a warning and one process may be stalled. It might happen from time to time and will most likely come back to normal.
- checkspace
Checks available space for output and backup. Folders can be on the same machine or backup can be a link to a file share (NAS).
For working folder, zmon assumes there could be ~100-200 frames saved and would require an extra 5GB for Video AI to not stop processing. Thus the check for 10GB
For backup folder, zmon will do the math based on the average size required for each frame, space available and # of frames that still need.
- checknewframes
Will check for folders and see if the output folder has been updated since last time (EARLIER). If the update happened less than 20s ago, it’s possible Video AI is still generating some frames, so we will wait for next loop. Otherwise, will call backupframes and estimate finish
** 4.1 backupframes**
Uses rsync to automatilly copy any new frame into the backup folder. Backup is organized in subfolders of thousands of frames
video_000 will contain frames 000000-000999
video_001 will contain frames 001000-001999
video_XXX will contain frames XXX000-XXX999
I implemented this as it was getting painful to browse a folder of 100,000 frames on my NAS (I use Synology, which in theory doesn’t have a limit with ZFS, but they still recommend to stay below 100,000 files per folder).
When a folder contains 1000 frames, it’s considered “complete” and matching frames will be removed from the output folder. This avoids duplicating every frame!
note: I updated rsync with brew to increase performance (macos rsync is quite old). This should work OK without the faster version though.
** 4.2 estimatefinish**
This calculate estimated time of completion with the knowledge of time it took to generated previous frames. Calculation varies based on information available, first I use the time the video file was queued, then I refine the time calculation with the history of frames generated. So, you will see this time fluctuate.
SPF will be recalculated when we get a batch of new frames.
Time will also change if you run other processes on your machine, especially when using the GPU. Switching to battery on a laptop will also increase your “SPF” (Second Per Frame) and estimated time will change at the next loop.
CURRENT_OUTPUT, CURRENT_BACKUP are critical variables that track the # of frames produced in the output and backup folders
Finally, when you queue a 2nd video, it will only start when the first video is complete. SPF and time to completion will be bad as it relies on the time the file was originally queued. When new frames are generated, you will get a closer approximation of time to completion after the 2nd batch of frames.
Final words
There are many more comments in the script itself if you interested in understanding how things work or want to make it better.
The script is provided “as is” and I will not be responsible for anything bad that happens! Hope you find it useful. If you have suggestions or made some modifications and are willing to share, please feel free to do so.
Thank you for reading and best regards,
Dom
