Small Model Training

Specialized training methodologies for models under 3B parameters optimized for edge deployment, as pioneered by liquid-ai. These models face unique challenges compared to scaled-down versions of larger models.

Key Characteristics

Memory-Bound Constraints

• <3B parameters total
• Low knowledge capacity requires careful parameter allocation
• Memory efficiency more critical than raw computational power

Task-Specific Focus

• Optimized for narrow tasks rather than general-purpose capabilities
• Easy to train and adapt to new domain-specific data
• More effective than general improvements for small models

Latency Requirements

• Sub-100ms response times required
• Fast prefill is critical, not just decode speed
• Inference optimization equally important as training

Training Methodology

Massive Overtraining

• liquid-ai's LFM2.5-350M trained on 28T tokens
• Contradicts traditional scaling laws that suggest diminishing returns
• "More pre-training works, even at the smallest scale"

Post-Training Pipeline

1. Supervised Fine-Tuning (SFT): Task-specific adaptation
2. Preference Alignment: Human preference optimization
3. Reinforcement Learning: Advanced policy optimization

Key Insight: Small models benefit more from task-specific training than general improvements beyond benchmarks.

Architecture Innovations

Parameter Distribution Optimization

• LFM2.5-350M: 19% of parameters in embedding layer
• Compare to Gemma 3 270M: 63% in embedding layer
• Effective model size: 287M from 350M total parameters

Gated Short Convolution Blocks

• Novel architecture component for efficient edge inference
• 2.5x better cost ratio vs traditional attention on CPU decode
• Optimized for memory-bound rather than compute-bound scenarios

Common Problems and Solutions

Doom Looping Problem

• Small models with reasoning traces get stuck in repetitive patterns
• Solution: Two-stage approach:
  1. On-policy data generation for DPO with heuristic filtering
  2. Reinforcement learning with n-gram repetition penalty

Cold-Start SFT Data

• Missing critical supervised fine-tuning data for specific domains
• Requires careful curation of task-specific training examples

Performance Characteristics

Inference Speed

• Competitive with llama.cpp 4-bit quantization on CPU
• Strong scaling with SGLang for GPU batch inference
• Mobile-optimized (Galaxy S24 Ultra, Ryzen HX 370)

Efficiency Gains

• Extremely efficient for narrow tasks
• Better task-specific performance than scaled-down general models

See also

• edge-ai-optimization
• doom-looping-problem
• liquid-ai
• maxime-labonne
• Model Architecture Design