Model Parameter Distribution

The allocation of parameters across different components of a language model, particularly critical in small-models where parameter efficiency directly impacts performance and deployment feasibility. liquid-ai's analysis reveals significant differences between traditional and optimized small model architectures.

Traditional Model Distributions

Gemma 3 270M (LLM)

• Embedding parameters: 63% of total model parameters
• Model parameters: 29% of total parameters
• Layers: 24 layers with RMSNorm + 3:1 GDN/Gated Attention + Feedforward

Qwen3.5-0.8B (VLM)

• Embedding parameters: Similar high embedding ratio
• Layers: 18 layers with RMSNorm + 5:1 SWA/GQA + Feedforward
• Effective size: ~600M parameters after accounting for architecture

Optimized Distribution: LFM2.5-350M

lfm2-5-350m achieves dramatically improved parameter efficiency:

• Embedding parameters: Only 19% of total parameters
• Model parameters: 81% allocated to computation
• Effective size: 287M parameters for computation
• Architecture: 16 layers with shortconv/GQA replacing traditional attention

Implications for Edge Deployment

The parameter distribution directly impacts:

1. Memory efficiency: Lower embedding ratios reduce memory overhead
2. Computational capacity: More parameters available for actual computation
3. Knowledge storage: Optimized balance between vocabulary representation and reasoning capacity
4. Inference speed: Fewer embedding lookups improve latency

Architecture Design Principles

Effective small model parameter distribution requires:

• Tied embeddings: Sharing input/output embedding weights
• Efficient attention: Replacing standard attention with optimized mechanisms like shortconv
• Layer optimization: Balancing depth vs width for target parameter count
• Component efficiency: Minimizing overhead in normalization and activation functions

See also

• lfm2-5-350m
• shortconv
• small-model-training
• edge-models
• Parameter Efficiency