Extreme Overtraining

Mis à jour le 2025-01-04Confiance : high

overtrainingscaling-lawstest-time-computeliquid-ailfm2-5-350mpre-training

Training methodology that extends pre-training far beyond traditionally compute-optimal token ratios, as demonstrated by liquid-ai with lfm2-5-350m (350M parameters trained on 28T tokens). This approach challenges conventional scaling laws by accounting for test-time computational benefits.

Theoretical Justification

Based on test-time-scaling research by Roberts et al. (April 2026), extreme overtraining becomes compute-optimal when considering inference-time benefits:

Traditional scaling laws focus only on training compute efficiency
Test-time scaling reveals that overtraining improves inference performance
The additional training cost is amortized across all future inference calls

Implementation at Scale

LFM2.5-350M Case Study

Model Size: 350 million parameters
Training Tokens: 28 trillion tokens (far exceeding standard ratios)
Outcome: Superior edge performance despite "inefficient" training compute usage
Key Finding: "More pre-training works, even at the smallest scale"

Practical Benefits

Edge Deployment Advantages

Improved performance on resource-constrained devices
Better task adaptation capabilities
Enhanced inference efficiency per parameter
Reduced need for extensive fine-tuning

Cost-Benefit Analysis

Higher upfront training costs
Lower ongoing inference costs
Better amortization for high-usage deployment scenarios
Particularly valuable for edge models with limited post-training optimization options

Comparison with Traditional Approaches

Traditional Scaling:

Optimize for training compute efficiency
Stop at theoretical compute-optimal point
Focus on parameter scaling over data scaling

Extreme Overtraining:

Optimize for lifetime compute efficiency (training + inference)
Continue training beyond traditional stopping points
Prioritize inference performance improvements

Research Implications

This approach suggests that scaling laws need to be revisited when deployment scenarios favor inference optimization over training efficiency, particularly for edge models where post-training optimization options are limited.