[1.6.1] Every new export worker: ~3 min model prep before cached TensorRT load (Apollo .tz3)

[coldbricks]

Summary

Every new export/preview ffmpeg worker spends ~2.5–3 minutes on a single CPU thread after locating the .tz3 model and before Loading filename / TensorRT engine load. During that time the GPU is idle. After that silence, the already-cached TensorRT engine loads in well under a second and rendering proceeds at normal speed.

This is not fixed by moving the model folder, running as Administrator, changing VPN/network, or reinstalling the same models. The cost repeats for every queued job with the same model and resolution, so a queue of many short clips is dominated by model prep, not by GPU work.

---

Environment

• App: Topaz Video 1.6.1
• Video AI runtime: 2.0.32
• AIEngine: 3.8.27
• OS: Windows 11
• GPU: NVIDIA (TensorRT path working)
• Example model: apo-v8-fp16-576x416-ox.tz3 (~239.95 MiB package; inner ONNX ~270 MB)
• Persistent TRT cache: Present (vtrt engine ~274 MB); engines write and load successfully
• Post-stall TRT load: Sub-second
• Actual short-clip render after load: Normal GPU use (e.g. ~10–15 s for a short test)

---

Exact log sequence (representative)

Control run with Topaz’s default model directory on C:\ProgramData\... (same behavior on a custom local U:\ model path):

15:52:31.744  Locate apo-v8-fp16-576x416-ox.tz: status 0
15:52:31.744  Cannot open zip ...apo-v8-fp16-576x416-ox.tz
15:52:31.744  Locate apo-v8-fp16-576x416-ox.tz3: status 1
15:52:31.744  CACHING MAY NOT WORK
              [~176 seconds of worker log silence]
15:55:27.875  Loading filename apo-v8-fp16-576x416-ox.tz3
15:55:27.876  Unable to create or use .../jitRTCache/
              (sometimes seen; does not explain the long gap)
15:55:28.275  Cached TensorRT engine loaded
              (~0.4 s after "Loading filename")
              then normal GPU render

CACHING MAY NOT WORK appears at the start of the silence. Prior staff comments indicate this is printed while checking an in-memory cache and is safe to ignore as an error string. The bug is not the message itself — it is the multi-minute work that follows in every fresh worker process.

Staff note (historical):

Topaz Video AI Linux Beta v4.0.7.0.b

1. This is safe to ignore, it’s printed when we check if a model is already cached in memory. 2. I’m assuming you’re referring to the multiple CACHING MAY NOT WORK messages? This is just an artifact of how we’re checking for the model file’s existence, it should still only be being loaded once. 3. This is the parameter estimation model for Proteus. 5. No, the model is for 30X0 devices. You’re just using a different card in that series than we used to compile the models on our end.

───

What the worker is doing during the silence

Measured on a fresh worker during the gap:

• CPU: One thread saturated
• GPU (nvidia-smi): ~1% util / idle power
• Network: No transfer (VPN on/off unchanged)
• Physical disk I/O on model file: Not required (often fully cache-resident)
• Large temp unpack file: Not observed
• Logical reads (process): Increase roughly linearly over the gap
• Working set: Bounded (hundreds of MiB order; not unbounded growth)
• Worker log: Silent until Loading filename

After the gap: TRT cache hit → normal GPU load → normal encode/render.

───

Controlled experiments (this machine)

• Normal restart — Custom local NTFS NVMe (U:...\models) — ~171.9 s — No improvement
• Run as Administrator — Same U:\ path — ~168 s — No improvement
• Default model directory — C:\ProgramData\Topaz Labs LLC\Topaz Video\models — 176.131 s — No improvement
• VPN on / off — same behavior — Not network/CDN
• Back-to-back same model + resolution — repeats every job — Not a one-time first-run cost

Notes:

• C: and U: model packages were byte-identical
• TRT engines write and read successfully
• The long delay is before the fast engine load
• Across packages on this machine, pre-load delay scales with package size at roughly 1.4–1.5 MiB/s (wall time vs package size)

───

Independent corroboration (public support logs)

Related thread:

Critical Initialization Stall (moov atom not found / Loading loop) with RTX 5070 Ti

Hello everyone, I am experiencing a severe and persistent issue with Topaz Video AI (v5.x/v4.x) on a clean installation of Windows 11. The software freezes on “Estimating” for over a minute before starting any preview or export, and the logs are flooded with cyclical errors every second. My Hardware Configuration: CPU: Intel Core i5-14600K (14th Gen) GPU: NVIDIA GeForce RTX 5070 Ti 16GB (Blackwell Architecture) OS: Windows 11 Pro (Clean install, running a Local/Anonymous User Account,…

Logs in that bundle (Topaz Video 1.6.0 / runtime 2.0.27 / AIEngine 3.8.20) show the same pattern on separate FFmpeg preview workers:

Locate .tz3 → long gap → Loading filename

Approximate pairing by package size:

• Iris v2 (~56.6 MiB) — ~23 s — ~2.4 MiB/s
• Apollo Fast v2 (~76.6 MiB) — ~30 s — ~2.5 MiB/s
• Apollo v8 (~240 MiB) — ~95 s — ~2.5 MiB/s

Near-linear in package size (near-zero fixed intercept). Iris runs often report loading from jitRTCache after the package gap.

During the wait, UI polling of unfinished MP4 previews produces moov atom not found / QML errors about once per second. That matches preview file not ready yet, not a separate root cause: no frames exist until model prep finishes.

───

What this is not

Ruled out by controls and measurements:

1. Model drive / folder ACLs — same stall elevated and on default C:\ProgramData models
2. Network / VPN / CDN — no transfer; VPN state irrelevant
3. TensorRT engine rebuild — sub-second cached engine load right after the silence
4. NVENC -rc-lookahead / encoder GOP — would scale with frames/encode path, not .tz3 size; GPU idle during gap
5. moov atom not found as the cause — symptom of unfinished preview output while the worker is still in model prep

───

Working diagnosis (for engineering)

1. Queued jobs spawn short-lived ffmpeg workers.
2. Decoded / materialised model state appears process-local (consistent with “load once per process” and with CACHING MAY NOT WORK as an in-memory check).
3. On a cache miss (every new worker), the engine performs a size-proportional, single-threaded pass over the encrypted .tz3 (read / decrypt / inflate / authenticate / validate — exact routine proprietary) before consulting the persistent TRT engine cache.
4. Persistent TRT caching works; it does not skip that pre-step in the current loader order.
5. User-facing impact is O(number of clips) prep time even when model, resolution, and TRT engine are unchanged.

───

User impact

Example: many short clips (e.g. ~10 s each), all Apollo:

• Prep ≈ ~3 minutes × N jobs with GPU idle
• Actual render per short clip may only be tens of seconds

A large queue becomes hours of single-core model prep wrapped around a modest amount of GPU work.

Workarounds that help users but should not be required:

• Concatenate many same-settings clips into one export (pay prep once)
• Use a smaller model when quality allows

───

Request to Topaz

Please treat this as a loader / worker-lifetime issue, not a disk or NVENC ticket.

Questions for engineering:

1. Why is the .tz3 fully walked/materialized/validated for every new ffmpeg worker when a matching TensorRT engine is already on disk?
2. Is the decoded model / package cache intentionally process-local only in runtime 2.0.32 / AIEngine 3.8.x?
3. Is there a supported long-lived worker, shared model cache, or path that skips package materialization when TRT cache hits?
4. What condition prints CACHING MAY NOT WORK, and is the in-memory model cache meant to be shared across queued export workers?

Ask: a build with model-loader timing, or a fix that reuses materialised model state across queued jobs (or avoids re-walking .tz3 when the TRT engine is valid).

───

Log sharing note

Raw *-Main.tzlog files can contain authentication tokens in auth-manager lines. For support, please use redacted extracts only:

• Timestamped Locate / CACHING MAY NOT WORK / Loading filename / TRT cache lines

Do not upload full unredacted main logs.

───

Reproduction (minimal)

1. Install models locally (Apollo v8 package is a clear amplifier).
2. Ensure a TRT engine for that model/resolution already exists (after one successful job).
3. Queue two separate exports: same model, same resolution, short clips.
4. In each worker log, compare timestamps:
   Locate …tz3 / CACHING MAY NOT WORK → Loading filename
5. Expect ~minutes of CPU-bound silence and idle GPU per job, then sub-second TRT load and normal render.

Happy to provide redacted timestamp extracts and the experiment list above for support.

[coldbricks]

Update — diagnosis confirmed; cross-process package cache closes the stall (measured)

Thread: [1.6.1] Every new export worker: ~3 min model prep before cached TensorRT load (Apollo .tz3)

Following up on the OP with tighter measurements and a working demonstration of the missing product piece.

---

Confirmed

The stall is not TensorRT rebuild, disk path, VPN, or admin rights. Each new ffmpeg worker pays a full package materialize of the encrypted .tz3 before the already-cached TRT engine loads. CACHING MAY NOT WORK remains a red herring (in-memory check; cache is empty in every new process).

---

Measured (1.6.1 / AIEngine 3.8.27, real UI export workers)

Apollo-class packages (~270 MiB materialized map), same machine as the OP:

Event	Wall time
Cold readCompressedFiles on Apollo .tz3 (new worker)	~161–162 s (matches OP ~1.5 MiB/s)
Warm path after persisting the materialised package map to disk	~250–280 ms on the next worker process
Same Apollo tile, subsequent queue jobs	repeated ~250–280 ms hits (not multi-minute)
Iris-class earlier control	~39 s cold → ~61 ms warm

Example pattern (redacted): same Apollo tile file

• Worker A: cold ~162 s → write disk cache
• Workers B / C / D …: DISK HIT ~0.25–0.28 s, then normal TRT + GPU render

Important: cache is per model package file (e.g. apo-v8-fp16-1088x1056-ox.tz3 vs …576x416…). Different input sizes that select different tiles still pay one cold per unique tile, then stay warm. Same-size stacks in the queue are where this wins hardest.

---

Screenshot — warm path = GPU working, not 3 min idle

What the capture shows on an RTX 5090 Laptop after package map is warm:

Signal	Before fix (every new worker)	After durable package map (same tile)
GPU 3D	~0–1% for ~3 minutes	~63% active (Apollo inference)
GPU temp / load	Idle while CPU one-thread spins	58 °C, real work
Export UI	Multi-minute “Estimating” / silence before frames	Seconds remaining @ ~31 fps, jobs completing in ~20s–3 min of actual render
Queue	N × ~3 min prep + short GPU	Prep amortized; queue is GPU-bound

Before: one CPU thread hot, GPU parked, log silence after CACHING MAY NOT WORK.
After: same Apollo tile → map load ~0.25 s → TRT hit → GPU graph actually moves.

This is the user-visible proof the stall was package materialize, not “the 50-series is broken.”

---

What this proves for engineering

1. The expensive work is package materialize, not TRT.
2. That state is process-local today.
3. If the same materialised map is reused across workers, the multi-minute gap goes away with no change to model quality, TRT engines, or encode settings.
4. Product-side options that would make this unnecessary for users:

A. Persist / share the package entry map (behavior-preserving)

After first successful materialize for a given package (path + size + mtime), reuse it for later workers. DPAPI-sealed or equivalent is fine — point is durability across process death.

B. Make cold decrypt fast (helps every first load, every backend)

Independently of caching: wall time is still ~1.4–1.5 MiB/s on a single thread with GPU idle. That is far below bulk AES-NI. Analysis of the stack points at libzip’s Windows CNG path generating AES-CTR keystream in tiny blocks (per-call overhead dominated). aiengine already ships OpenSSL / libcrypto for other uses. Building libzip against a bulk OpenSSL crypto backend, or batching CTR in the Windows crypto backend, should collapse cold package open from ~minutes → ~seconds for Apollo without any new cache surface. Happy to share methodology / redacted timings if useful.

C. Loader short-circuit when TRT is valid

Still valuable, but narrower than A/B: only helps TRT-warm path; A/B help cold first-use and non-TRT backends too.

D. Long-lived worker for same model/settings

Also valid; larger lifecycle change.

---

User workarounds (still valid)

• Batch / concat same model + tile into one worker (pay cold once) — zero risk, no tools.
• Local experiment: persist materialised package maps across workers (what I measured above). This is unsupported, can break on updates, and is not a substitute for a product fix. I’m not shipping a public injector here; the measurements are the point for staff.

---

Ask

Please treat this as confirmed loader / package-lifecycle + (likely) decrypt-throughput, not “user disk / GPU.”

Ideal ship:

1. Shared / durable package map (or fix B so cold is cheap), and
2. Optional: skip full package walk when TRT engine already matches.

I can provide additional redacted timestamps (Locate → silence → Loading / TRT) and cold/warm pairs per tile if that helps triage.

Thanks — multi-short-clip Apollo queues are usable again once the package map isn’t recomputed from scratch every worker.

[kyle.topazlabs]

Very much appreciated for putting this report together and posting. The team is reviewing everything you shared about this situation.

[coldbricks]

Thank you very much, Kyle. Looking forward to seeing what the team has to say.

David