Use available system resources when processing batches

Batches processing doesn’t use available system resources. My relatively powerful PC running windows 10 has 128 GB of RAM, a 32-core (64 hyperthread) processor, and an RTX 3080 Ti graphics card.

TPAI uses these resources when processing a batch:

  • CPU utilization: 10%
  • GPU (3D) utilization: 40%
  • free RAM: 70% of 128GB
  • 2.2/12.0 GB of dedicated GPU memory
  • 0.2/63.0 GB of shared GPU memory

Free RAM and GPU memory utilization stay the same after a batch is done. CPU and GPU utilization drop to around 1%, normal idle for this machine.

How about processing a few images at a time in parallel, reducing wall time by 2x or 3x?

Thanks,
- Eric