~/wiki
Récent+ Ajouter

Liquid Foundation Models

Mis à jour le 2025-01-03Confiance : high
liquid-aifoundation-modelssmall-modelslfmedge-aigated-convolutionshortconvagentic-rlextreme-overtraining

liquid-ai's series of foundation models optimized for edge deployment, featuring innovative architectures and training methodologies for sub-3B parameter models. The LFM series demonstrates that small models require fundamentally different approaches than scaled-down versions of larger models.

Model Series

LFM2.5-350M

The flagship 350 million parameter model featuring:

  • shortconv attention mechanism for CPU optimization
  • extreme-overtraining on 28T tokens (80x compute-optimal ratio)
  • 16 layers with 3:1 ShortConv/GQA architecture
  • 19% embedding parameters (effective size: 287M)
  • Sub-100ms inference on galaxy-s24-ultra and ryzen-hx-370

LFM2.5-1.2B-Thinking

Reasoning-capable model with:

Architectural Innovations

ShortConv Mechanism

Revolutionary attention replacement using gated convolution:

  • 2x computational efficiency vs traditional attention on CPU
  • Optimized for memory-bound edge deployment scenarios
  • Linear scaling with sequence length for prefill operations

Parameter Allocation Strategy

Efficient distribution of model capacity:

  • 19% embeddings: Tied input/output layers
  • ~60% attention: ShortConv/GQA blocks
  • ~21% feedforward: Standard transformer FFN layers

Training Methodology

Pre-training Philosophy

extreme-overtraining approach challenging conventional scaling laws:

  • 28T tokens for 350M model (vs 7B tokens compute-optimal)
  • Justified by test-time-scaling research (Roberts et al., April 2026)
  • "More pre-training works, even at the smallest scale"

Post-Training Pipeline

Specialized three-stage approach:

  1. Supervised Fine-Tuning: Task-specific adaptation
  2. Preference Alignment: on-policy-data-generation with DPO
  3. Reinforcement Learning: agentic-rl with repetition penalties

Training Challenges

Unique small model considerations:

  • doom-looping-problem in reasoning traces
  • Cold-start SFT data requirements
  • Task-specific optimization vs general capabilities

Performance Characteristics

Edge Deployment Advantages

  • Memory-bound: <3B parameters fit edge hardware constraints
  • Task-specific: Easily adaptable to narrow applications
  • Latency sensitive: Sub-100ms response requirements met
  • CPU optimized: Superior performance without GPU acceleration

Benchmark Results

  • CPU inference: 2x faster than comparable attention mechanisms
  • GPU concurrency: Competitive scaling in multi-request scenarios
  • Edge devices: Validated on Samsung Galaxy S24 Ultra and AMD Ryzen HX 370

Applications

Agentic AI

Integration with agentic-rl frameworks enabling:

  • Tool-using capabilities in resource-constrained environments
  • Multi-environment training (Terminal, Search, RLM, OpenClaw)
  • Group computation and advantage estimation

Domain-Specific Deployment

  • Document processing with memory constraints
  • Real-time conversational interfaces
  • Mobile AI applications requiring local inference
  • Embedded systems with strict latency requirements

Research Impact

Scaling Law Challenges

LFM series demonstrates that:

  • Traditional compute-optimal scaling doesn't apply to edge models
  • test-time-scaling justifies extreme overtraining investments
  • Small models benefit from different architectural choices

Edge AI Paradigm

Establishes that edge models are "not just scaled-down versions of bigger models":

  • Require novel architectural innovations
  • Need specialized training methodologies
  • Benefit from task-specific optimization strategies

See also