I had Gemini explain to me how training with reduced precision is affected by training with compressed (lossy) data.
Because i’ve seen from the generative models some broken gardiens and different kinds artifacts
How Does Training AI Models with Reduced Precision Images Affect Them?
The impact of training AI models, especially in image processing, with images of reduced precision is a crucial area of research. Here are the main aspects of how this can affect the models:
- Quality Loss in Training Data (e.g., due to JPEG Compression)
When AI models are trained with images that have already lost quality due to lossy compression (like multiple JPEG saves), it leads to several problems:
- Learning Artifacts: The model learns not only the actual features in the image (e.g., edges, textures, shapes) but also the artifacts created by compression (blockiness, ringing, color noise). If the model is then applied to uncompressed or differently compressed images, it might struggle to interpret them correctly. It generalizes poorly to “real” data.
- Loss of Fine Details: High-frequency information (fine details, sharp edges) is often the first to be lost with heavy JPEG compression. If the model is trained on such images, it won’t learn to recognize or utilize these details. This can impair the model’s performance in tasks requiring high detail accuracy (e.g., medical image analysis, quality control).
- Reduced Robustness: The model becomes less robust to variations in image quality. If it has only seen “noisy” images, it might have difficulty processing clear images, or vice versa.
- Poorer Overall Performance: The model’s accuracy can suffer. In classification tasks, the model might misidentify objects; in segmentation tasks, boundaries might be imprecise; in generative tasks, the produced images might themselves exhibit artifacts.
- “Model Collapse”: Especially in generative AI models (e.g., Stable Diffusion, Midjourney), there’s a phenomenon called “model collapse.” If models are repeatedly trained on synthetic data generated by earlier versions of the model, errors and artifacts can accumulate in the training data. This leads to models producing progressively worse and more unusable results over generations, as they learn from increasingly degraded data. Quality loss due to compression can be a contributing factor to this.
Ideally, AI models should be trained with the highest possible quality of images relevant to the specific task. This means using raw data or lossless formats to minimize information loss.
- Training with Reduced Numerical Precision (e.g., FP16, INT8 instead of FP32)
This is a slightly different aspect of “reduced precision,” referring to the computational accuracy during training, not the quality of the input images themselves.
- What is it? By default, many neural networks operate with 32-bit floating-point numbers (FP32, “Full Precision”). To reduce memory consumption and computation time, techniques like training with 16-bit floating-point numbers (FP16, “Half Precision”) or even 8-bit integers (INT8, “Quantization”) are often used.
- Advantages:
- Faster Training: Fewer bits mean faster computations on specialized hardware (e.g., Tensor Cores on NVIDIA GPUs).
- Lower Memory Consumption: The model requires less RAM, enabling the training of larger models or batch sizes.
- Energy Efficiency: Fewer computations and memory accesses consume less energy.
- Disadvantages/Risks:
- Potential Accuracy Loss: Due to the lower precision of number representation, rounding errors can occur. This can cause the model to converge less effectively during training or slightly decrease the model’s final accuracy.
- Stability Issues: Especially at extremely low precisions (e.g., 8-bit), the gradients (which are crucial for learning) can become so imprecise that training becomes unstable or completely collapses.
- Complexity: To minimize the disadvantages, special techniques like “Mixed Precision Training” (where critical parts of the model continue to be computed in higher precision) or “Quantization-Aware Training” (where the model is prepared for subsequent quantization during training) are often employed.
In summary, while using images with reduced precision (in terms of quality loss) directly degrades the model and makes it learn artifacts, reducing numerical precision (for the model’s internal computations) aims to increase efficiency, often with an acceptable compromise in final accuracy, but requires careful implementation.
And because Gemini did get me wrong and i did copy the text without reading it full. 
Here the right version:
You’re absolutely right, my apologies! I treated the two aspects—quality loss in training images and reduced numerical precision of the model—separately, instead of addressing their combination. That’s a critical point, and yes, it can indeed lead to significant problems for an AI model.
The Double Whammy: Compressed Images and Reduced Numerical Precision in AI Training
When an AI model is trained with images that have already been degraded in quality by lossy compression (like JPEG), AND the model itself operates with reduced numerical precision during training (e.g., FP16 or INT8), the challenges are compounded. Here are the main problems that can arise:
- Amplified Information Loss and Artifacts
- The “Snowball Effect”: Image compression already removes fine details and introduces artifacts. If the model is then trained with lower numerical precision, which itself can lead to rounding errors and a certain amount of “information noise” in calculations, the effect is amplified. The model receives “noisy” inputs from the start, and its internal calculations add further “noise.” It’s like trying to understand a very quiet conversation (compressed image) in a noisy room (reduced computational precision)—information is lost even faster.
- Learning Errors: The model doesn’t just learn artifacts from the compressed images; it also learns the errors that arise from numerical imprecision in the model’s weights and activations. This leads to a model that potentially deviates even further from the desired “true” features of the image.
- More Difficult Convergence and Instability
- Distorted Gradients: The learning process of a neural network relies on calculating gradients, which indicate the direction in which model weights need to change. If the input data (compressed images) are already imprecise and the model’s internal calculations (reduced precision) introduce additional rounding errors, the computed gradients can become very unreliable and inaccurate.
- Challenging Optimization: The optimization algorithm (e.g., Adam, SGD) struggles significantly to find a stable path to the optimal solution space. Training can:
- Converge slower: The model takes much longer to achieve even acceptable performance.
- Fail to converge: The model learns nothing meaningful, and performance stagnates at a low level.
- Become unstable: Training curves can fluctuate wildly, or training might crash completely due to values exploding (NaNs) or vanishing (zeros).
- Significant Performance Drop
- Reduced Accuracy: The overall accuracy of the model will likely decrease significantly. For classification tasks, this leads to more misclassifications; for segmentation tasks, to imprecise masks; and for generative models, to images with even more artifacts and lower photorealistic quality.
- Poor Generalization: The model will be less capable of applying the learned patterns to new, unseen data, even if it’s of similarly (low) quality. It becomes overly tailored to the specific artifacts and inaccuracies of its training data.
- Increased Difficulty in Debugging
- When a model performs poorly under these conditions, it’s much harder to pinpoint whether the problem lies with the substandard data, the reduced numerical precision, the model architecture, or a combination of all of these. The causes of errors become more complex to isolate.
Conclusion
Yes, the combination of training with compressed images and reduced numerical precision can cause severe problems for an AI model. It’s a “worst-case scenario” where the model has to learn from the outset with flawed information and an impaired “brain.”
Ideally, both should be avoided: - High-Quality Training Data: Always use the best possible, lossless, or at least minimally compressed images for training.
- Controlled Precision Reduction: If reducing numerical precision is necessary for efficiency, it should be done very carefully and using techniques like Mixed Precision Training or Quantization-Aware Training to minimize the impact on the model’s stability and accuracy.
