Concepts — vue longue

Advanced reinforcement learning methodology developed by liquid-ai as the final stage of their three-phase post-training pipeline for liquid-foundation-models. Specifically designed to address the doom-looping-problem in small models with reasoning traces.

Architecture

Core Components

• Policy model (πθ): Current model being optimized
• Reference model (πref): Baseline for comparison and stability
• Group Computation: Batch processing of multiple reasoning paths
• Multiple Environment Types: Diverse training scenarios for robust agent behavior

Environment Types

1. Terminal Env: Command-line and system interaction scenarios
2. Search Env: Information retrieval and knowledge synthesis tasks
3. OpenClaw Env: Integration with openclaw multi-model harness
4. RLM Env: Recursive language modeling environments

Training Process

Input Processing

• Prompt (x): Initial task specification
• *Target output (y)**: Desired completion
• Generation sequences (o1, o2, ..., oG): Multiple candidate outputs

Reward Computation

• Reward signals (r1, r2, ..., rG): Environment-specific feedback
• Advantage estimation (A1, A2, ..., AG): Policy gradient computations
• Group-based optimization: Batch processing for efficiency

Anti-Doom Loop Mechanisms

• N-gram repetition penalty: Prevents repetitive generation patterns
• Reasoning trace validation: Ensures logical consistency
• Multi-environment training: Robust behavior across diverse scenarios

Key Innovations

Doom Loop Mitigation

Specifically addresses the tendency of small models to get stuck in repetitive content generation, particularly problematic when combining <3B parameter models with complex reasoning tasks.

Example Scenario

Input: "I have a great question: what is 2+2?"
Desired: "Fantastic question! Let me unpack this for you. We have 2 + 2 = 4. The final answer is 4."
Ground truth: 4
Result: Correct!

Multi-Environment Training

Exposes models to diverse interaction patterns through different environment types, improving generalization and robustness for agentic applications.

Performance Metrics

Doom Loop Reduction

Significant reduction in doom loop occurrences measured as percentage of failed generations in LFM2.5-1.2B-Thinking model evaluation.

Agentic Capabilities

Enhanced performance on:

• Multi-step reasoning tasks
• Tool use and API interactions
• Complex problem decomposition
• Autonomous task execution

Integration with Liquid AI Pipeline

Post-Training Sequence

1. Supervised Fine-Tuning: Task-specific adaptation
2. Preference Alignment: Human preference optimization via on-policy-data-generation
3. Agentic Reinforcement Learning: Final optimization for autonomous behavior

Data Requirements

• on-policy-data-generation outputs for preference alignment
• Environment-specific reward signals
• Multi-modal interaction scenarios

Applications

Edge Deployment

Optimized for small models requiring autonomous behavior in resource-constrained environments while maintaining reliability and avoiding failure modes.

Tool-Using Agents

Specialized training for models that need to interact with external systems, APIs, and tools in production environments.

See also

• doom-looping-problem
• on-policy-data-generation
• liquid-foundation-models
• small-model-training

A specific failure mode in small models with reasoning traces where the model gets stuck generating repetitive content, particularly problematic when combining small models (<3B parameters) with complex reasoning tasks.

Problem Description

Manifestation

• Model generates repetitive phrases or content blocks
• Example: Recipe request resulting in endless repetition of "1 cup coconut milk"
• Occurs specifically in small models using reasoning traces
• More pronounced in complex, multi-step tasks

Contributing Factors

• Model Size: Problem emerges specifically in small models (<3B parameters)
• Reasoning Traces: Complex reasoning chains increase repetition likelihood
• Task Complexity: Multi-step tasks trigger repetitive patterns more frequently
• Limited Context Handling: Small models struggle with long-range dependencies

Root Causes

Limited Parameter Capacity

• Insufficient model capacity to maintain diverse generation patterns
• Memory constraints lead to simplified repetitive behaviors
• Knowledge compression forces repetitive pattern shortcuts

Training Data Patterns

• Reasoning traces may contain repetitive structures
• Model learns to copy repetitive patterns from training examples
• Insufficient diversity in reasoning demonstration examples

Solution: Two-Stage Approach

Stage 1: On-Policy Data Generation for DPO

1. Policy Model Generation: Generate multiple outputs with temperature > 0
2. Heuristic Filtering: Filter out repetitive or low-quality responses
3. LLM Jury Evaluation: Use larger model to judge response quality
4. Chosen vs Rejected Pairs: Create preference pairs for DPO training
5. Temperature Adjustment: Use temperature = 0 for final policy model

Stage 2: Reinforcement Learning + N-gram Repetition Penalty

1. Reinforcement Learning: Policy optimization against reward model
2. N-gram Repetition Penalty: Explicit penalty for repeated n-gram sequences
3. Ground Truth Validation: Verify correctness of reasoning conclusions
4. Iterative Improvement: Multiple rounds of RL with repetition constraints

Technical Implementation

On-Policy Data Generation

• Generate diverse outputs from current policy model
• Filter using heuristic rules (repetition detection, length constraints)
• Human or LLM evaluation for quality assessment
• Create preference datasets for Direct Preference Optimization

Repetition Penalty Mechanisms

• N-gram repetition detection during generation
• Dynamic penalty scaling based on repetition frequency
• Context-aware penalty adjustment
• Balance between repetition avoidance and coherence

Evaluation Metrics

• Doom loop ratio percentage tracking
• Response quality maintenance during repetition reduction
• Task completion success rates
• Generation diversity measurements

Results and Impact

Performance Improvement

• Dramatic reduction in doom loop ratios
• Maintained task completion accuracy
• Improved generation diversity
• Better user experience for reasoning tasks

Broader Implications

• Demonstrates need for specialized small model post-training
• Shows importance of repetition-aware training objectives
• Highlights trade-offs between model size and capability complexity

See also

• small-model-training
• Reinforcement Learning from Human Feedback
• direct-preference-optimization
• liquid-ai
• Post-Training Methods

Training methodology that extends pre-training far beyond traditionally compute-optimal token ratios, as demonstrated by liquid-ai with lfm2-5-350m (350M parameters trained on 28T tokens). This approach challenges conventional scaling laws by accounting for test-time computational benefits.

Theoretical Justification

Based on test-time-scaling research by Roberts et al. (April 2026), extreme overtraining becomes compute-optimal when considering inference-time benefits:

• Traditional scaling laws focus only on training compute efficiency
• Test-time scaling reveals that overtraining improves inference performance
• The additional training cost is amortized across all future inference calls

Implementation at Scale

LFM2.5-350M Case Study

• Model Size: 350 million parameters
• Training Tokens: 28 trillion tokens (far exceeding standard ratios)
• Outcome: Superior edge performance despite "inefficient" training compute usage
• Key Finding: "More pre-training works, even at the smallest scale"

Practical Benefits

Edge Deployment Advantages

• Improved performance on resource-constrained devices
• Better task adaptation capabilities
• Enhanced inference efficiency per parameter
• Reduced need for extensive fine-tuning

Cost-Benefit Analysis

• Higher upfront training costs
• Lower ongoing inference costs
• Better amortization for high-usage deployment scenarios
• Particularly valuable for edge models with limited post-training optimization options

Comparison with Traditional Approaches

Traditional Scaling:

• Optimize for training compute efficiency
• Stop at theoretical compute-optimal point
• Focus on parameter scaling over data scaling

Extreme Overtraining:

• Optimize for lifetime compute efficiency (training + inference)
• Continue training beyond traditional stopping points
• Prioritize inference performance improvements

Research Implications

This approach suggests that scaling laws need to be revisited when deployment scenarios favor inference optimization over training efficiency, particularly for edge models where post-training optimization options are limited.

See also

• test-time-scaling
• lfm2-5-350m
• small-model-training
• edge-models
• liquid-ai

Novel attention mechanism replacement developed by liquid-ai for their liquid-foundation-models, specifically designed to optimize inference performance on edge devices and CPU-bound environments. ShortConv achieves significant speedups while maintaining model quality.

Architecture

Core Design

B x C → [Linear] → [Linear] → [Conv1D] → Gated Output

Components:

• Gated structure: Controls information flow like attention gates
• Short convolutions: Limited kernel size for efficiency
• Linear projections: Standard feedforward transformations
• Element-wise gating: Selective activation patterns

Integration Pattern

RMSNorm
3:1 ShortConv/GQA  ← Replaces traditional attention
RMSNorm
Feedforward

Performance Advantages

CPU Optimization

• 2.5x faster decode vs traditional attention mechanisms
• Linear complexity vs quadratic attention scaling
• Reduced memory bandwidth requirements
• Cache-friendly access patterns

Edge Device Benefits

• Lower computational overhead for mobile deployment
• Reduced memory pressure during inference
• Optimized for ARM and x86 CPU architectures
• Better thermal efficiency on mobile devices

Comparison with Attention Alternatives

Cost Ratios (M4 Max CPU decode)

• ShortConv: 1.0x baseline
• SWA (Gemma3): 1.2x overhead
• GDN (Qwen3.5): 1.5x overhead
• GLA: 2.0x overhead
• GQA: 2.2x overhead

Implementation Details

Gated Short Convolution Block

• Input processing: Linear transformation of input embeddings
• Convolution: 1D convolution with optimized kernel size
• Gating mechanism: Learned gates for selective activation
• Output projection: Linear transformation to model dimensions

Training Considerations

• Maintains gradient flow during training
• Compatible with standard transformer training pipelines
• Requires minimal architectural changes from attention
• Supports both pre-training and fine-tuning workflows

Applications

LFM2.5 Series

• LFM2.5-350M: Primary attention replacement
• Mobile deployment: Optimized for smartphone inference
• Real-time applications: Sub-100ms response requirements
• Edge AI: Resource-constrained environments

Use Cases

• Conversational AI on mobile devices
• Real-time text processing applications
• Edge-deployed language understanding
• Low-latency completion and generation tasks

Technical Trade-offs

Advantages

• Significant CPU inference speedup
• Linear computational scaling
• Reduced memory requirements
• Hardware-friendly operations

Considerations

• Novel architecture requiring specialized implementation
• Different training dynamics vs attention
• May require architecture-specific optimization
• Limited long-range dependency modeling vs full attention

See also

• liquid-foundation-models
• edge-ai-optimization
• small-language-models
• LLM Inference Optimization

Computational mechanism in liquid-ai's agentic-reinforcement-learning framework that enables efficient batch processing of multiple agent trajectories across diverse environments simultaneously.

Functionality

Group computation processes sequences of observations (o₁, o₂, ..., oG) with corresponding rewards (r₁, r₂, ..., rG) and advantage estimates (A₁, A₂, ..., AG) in parallel, allowing the system to learn from multiple environment types concurrently.

Efficiency Benefits

This approach significantly improves training efficiency for small models by:

• Maximizing batch utilization across different task types
• Enabling simultaneous learning from diverse experience sources
• Reducing computational overhead compared to sequential environment processing

Integration

Works seamlessly with multi-environment-training to provide comprehensive agent training across Terminal, Search, OpenClaw, and RLM environments, ensuring robust performance despite the parameter constraints of edge-models.

See also

• agentic-reinforcement-learning
• multi-environment-training
• Batch Processing
• Training Efficiency

liquid-ai's series of foundation models optimized for edge deployment, featuring innovative architectures and training methodologies for sub-3B parameter models. The LFM series demonstrates that small models require fundamentally different approaches than scaled-down versions of larger models.

Model Series

LFM2.5-350M

The flagship 350 million parameter model featuring:

• shortconv attention mechanism for CPU optimization
• extreme-overtraining on 28T tokens (80x compute-optimal ratio)
• 16 layers with 3:1 ShortConv/GQA architecture
• 19% embedding parameters (effective size: 287M)
• Sub-100ms inference on galaxy-s24-ultra and ryzen-hx-370

LFM2.5-1.2B-Thinking

Reasoning-capable model with:

• On-device reasoning under 1GB memory footprint
• Advanced thinking trace generation
• doom-looping-problem mitigation through n-gram-repetition-penalty
• on-policy-data-generation for preference alignment

Architectural Innovations

ShortConv Mechanism

Revolutionary attention replacement using gated convolution:

• 2x computational efficiency vs traditional attention on CPU
• Optimized for memory-bound edge deployment scenarios
• Linear scaling with sequence length for prefill operations

Parameter Allocation Strategy

Efficient distribution of model capacity:

• 19% embeddings: Tied input/output layers
• ~60% attention: ShortConv/GQA blocks
• ~21% feedforward: Standard transformer FFN layers

Training Methodology

Pre-training Philosophy

extreme-overtraining approach challenging conventional scaling laws:

• 28T tokens for 350M model (vs 7B tokens compute-optimal)
• Justified by test-time-scaling research (Roberts et al., April 2026)
• "More pre-training works, even at the smallest scale"

Post-Training Pipeline

Specialized three-stage approach:

1. Supervised Fine-Tuning: Task-specific adaptation
2. Preference Alignment: on-policy-data-generation with DPO
3. Reinforcement Learning: agentic-rl with repetition penalties

Training Challenges

Unique small model considerations:

• doom-looping-problem in reasoning traces
• Cold-start SFT data requirements
• Task-specific optimization vs general capabilities

Performance Characteristics

Edge Deployment Advantages

• Memory-bound: <3B parameters fit edge hardware constraints
• Task-specific: Easily adaptable to narrow applications
• Latency sensitive: Sub-100ms response requirements met
• CPU optimized: Superior performance without GPU acceleration

Benchmark Results

• CPU inference: 2x faster than comparable attention mechanisms
• GPU concurrency: Competitive scaling in multi-request scenarios
• Edge devices: Validated on Samsung Galaxy S24 Ultra and AMD Ryzen HX 370

Applications

Agentic AI

Integration with agentic-rl frameworks enabling:

• Tool-using capabilities in resource-constrained environments
• Multi-environment training (Terminal, Search, RLM, OpenClaw)
• Group computation and advantage estimation

Domain-Specific Deployment

• Document processing with memory constraints
• Real-time conversational interfaces
• Mobile AI applications requiring local inference
• Embedded systems with strict latency requirements

Research Impact

Scaling Law Challenges

LFM series demonstrates that:

• Traditional compute-optimal scaling doesn't apply to edge models
• test-time-scaling justifies extreme overtraining investments
• Small models benefit from different architectural choices

Edge AI Paradigm

Establishes that edge models are "not just scaled-down versions of bigger models":

• Require novel architectural innovations
• Need specialized training methodologies
• Benefit from task-specific optimization strategies

See also

• maxime-labonne
• liquid-ai
• shortconv
• extreme-overtraining
• edge-deployment
• small-model-training

The allocation of parameters across different components of a language model, particularly critical in small-models where parameter efficiency directly impacts performance and deployment feasibility. liquid-ai's analysis reveals significant differences between traditional and optimized small model architectures.

Traditional Model Distributions

Gemma 3 270M (LLM)

• Embedding parameters: 63% of total model parameters
• Model parameters: 29% of total parameters
• Layers: 24 layers with RMSNorm + 3:1 GDN/Gated Attention + Feedforward

Qwen3.5-0.8B (VLM)

• Embedding parameters: Similar high embedding ratio
• Layers: 18 layers with RMSNorm + 5:1 SWA/GQA + Feedforward
• Effective size: ~600M parameters after accounting for architecture

Optimized Distribution: LFM2.5-350M

lfm2-5-350m achieves dramatically improved parameter efficiency:

• Embedding parameters: Only 19% of total parameters
• Model parameters: 81% allocated to computation
• Effective size: 287M parameters for computation
• Architecture: 16 layers with shortconv/GQA replacing traditional attention

Implications for Edge Deployment

The parameter distribution directly impacts:

1. Memory efficiency: Lower embedding ratios reduce memory overhead
2. Computational capacity: More parameters available for actual computation
3. Knowledge storage: Optimized balance between vocabulary representation and reasoning capacity
4. Inference speed: Fewer embedding lookups improve latency

Architecture Design Principles

Effective small model parameter distribution requires:

• Tied embeddings: Sharing input/output embedding weights
• Efficient attention: Replacing standard attention with optimized mechanisms like shortconv
• Layer optimization: Balancing depth vs width for target parameter count
• Component efficiency: Minimizing overhead in normalization and activation functions

See also

• lfm2-5-350m
• shortconv
• small-model-training
• edge-models
• Parameter Efficiency

Training methodology used in liquid-ai's agentic-reinforcement-learning framework where language models are simultaneously trained across diverse environment types to develop robust agent capabilities.

Environment Types

The framework integrates multiple environment categories:

Terminal Environment: Command-line interface interactions requiring precise syntax and system understanding.

Search Environment: Information retrieval tasks requiring query optimization and result evaluation.

OpenClaw Environment: Physical manipulation and robotics tasks requiring spatial reasoning and motor control.

RLM Environment: Recursive language model environments for complex reasoning chains.

Training Benefits

Multi-environment training ensures that small models like lfm2-5-350m develop generalizable agent skills rather than overfitting to specific task domains. This approach is particularly crucial for edge-models with limited parameter capacity.

Implementation

The system uses group computation to efficiently process trajectories across all environment types simultaneously, generating diverse reward signals and advantage estimates that improve overall agent robustness.

See also

• agentic-reinforcement-learning
• Environment Diversity
• Agent Training
• liquid-foundation-models

A technique used during reinforcement learning training to prevent doom-looping in small language models. Applied alongside RL training to discourage repetitive n-gram patterns that lead to infinite loops or stuck generation states.

Implementation

The penalty is applied during Stage 2 of liquid-ai's training pipeline:

• Combined with reinforcement learning objectives
• Targets specific n-gram patterns that indicate repetitive behavior
• Helps maintain generation diversity while preserving task performance

Context

Part of the comprehensive solution to doom-looping-problem in small-language-models with reasoning traces. Used in conjunction with on-policy data generation and careful preference alignment to maintain model quality while eliminating pathological repetitive behaviors.

Effectiveness

Successfully reduced doom loop ratios in LFM2.5-1.2B-Thinking model, enabling reliable on-device reasoning without getting stuck in repetitive generation patterns.

See also

• doom-looping
• agentic-reinforcement-learning
• liquid-foundation-models
• small-model-training

A training methodology used by liquid-ai to generate preference data for Direct Preference Optimization (DPO) by sampling from the current policy model at different temperatures, then using heuristic filtering and LLM jury evaluation to create chosen/rejected pairs.

Process

Stage 1: Data Generation

1. Policy model π_θ generates multiple outputs at temperature T > 0
2. Same policy model generates additional output at T = 0 (greedy decoding)
3. Heuristic filtering applied to candidate responses
4. LLM jury evaluates and ranks outputs
5. Chosen/rejected pairs selected for DPO training

Purpose

Primary application is addressing doom-looping in small models with reasoning traces:

• Generates training data specifically targeting repetitive behaviors
• Maintains on-policy distribution alignment
• Enables targeted preference learning for problematic generation patterns

Integration

Used as Stage 1 in liquid-ai's two-stage approach:

• Stage 1: On-policy data generation for DPO
• Stage 2: agentic-reinforcement-learning with n-gram-repetition-penalty

Effectiveness

Successfully creates high-quality preference data for mitigating doom-looping-problem while maintaining model performance on target tasks.

See also

• doom-looping
• agentic-reinforcement-learning
• liquid-foundation-models
• small-model-training

Methodology for evaluating how effectively small models utilize their parameters, as demonstrated in maxime-labonne's architectural comparison of frontier edge models. This analysis reveals significant differences in parameter utilization across different small model architectures.

Efficiency Metrics

Effective Parameter Calculation

Measures the actual computational contribution of parameters during inference:

• Total Parameters: Nominal model size
• Effective Parameters: Parameters actively contributing to computation
• Efficiency Ratio: Effective / Total parameters

Comparative Analysis

Architecture Comparison Results

gemma-3-270m (LLM):

• 24 layers with GDN/Gated Attention
• 63% parameter efficiency
• Effective size: ~170M parameters

qwen3-5-0-8b (VLM):

• 18 layers with SWA/GQA
• 29% parameter efficiency
• Effective size: ~230M parameters

lfm2-5-350m (LLM):

• 16 layers with shortconv/GQA
• 19% parameter efficiency
• Effective size: 287M parameters

Key Insights

Architecture Impact

• Layer Count: More layers don't necessarily mean better efficiency
• Attention Mechanism: Different attention types have varying computational overhead
• Specialization Trade-offs: VLMs (like Qwen3.5) show different efficiency patterns than pure LLMs

Design Implications

• Parameter efficiency varies dramatically across architectures
• Effective model size may differ significantly from nominal parameter count
• Optimization should target effective parameter utilization, not just total count

Evaluation Methodology

Profiling Approach

• On-device performance measurement
• Cross-platform benchmarking (galaxy-s24-ultra, ryzen-hx-370)
• Operator-level cost analysis for different attention mechanisms

Distillation Insights

The analysis reveals that effective model sizes after knowledge distillation can be:

• Gemma 3 270M → ~100M effective
• Qwen3.5-0.8B → ~600M effective
• LFM2.5-350M → 287M effective

Practical Applications

Model Selection

Use efficiency analysis to:

• Compare models with similar effective sizes rather than nominal parameters
• Identify architectures suitable for specific deployment constraints
• Optimize for actual computational requirements

Architecture Design

• Focus on improving effective parameter utilization
• Balance specialization vs efficiency trade-offs
• Consider deployment context when optimizing parameter distribution

See also

• edge-models
• lfm2-5-350m
• shortconv
• small-model-training
• gemma-3-270m

Gated Short Convolution attention mechanism developed by liquid-ai for the lfm2-5-350m model, specifically optimized for CPU inference performance in edge deployment scenarios. Replaces traditional attention mechanisms with convolution-based operations that demonstrate significant computational efficiency advantages.

Architecture

Gated Short Convolution Block

• Linear transformations: Two linear layers for gating mechanism
• Conv1D: 1D convolution operation for sequence processing
• Gating mechanism: Controls information flow through the convolution
• Integration: Combined with GQA (Grouped Query Attention) in 3:1 ratio configuration

Performance Characteristics

• CPU optimization: Designed specifically for CPU-bound inference scenarios
• Cost efficiency: Significantly lower computational cost compared to SWA (Sliding Window Attention), GDN (Gated Dense Networks), GLA (Gated Linear Attention), and GQA on M4 Max CPU during decode
• Memory efficiency: Reduced memory footprint for edge deployment
• Latency optimization: Enables sub-100ms response requirements for edge applications

Implementation Details

Architecture Configuration

• Used in lfm2-5-350m with 16 layers
• 19% of model parameters allocated to embeddings (effective size: 287M)
• Integrated with RMSNorm normalization layers
• Tied linear output layer for parameter efficiency

Benchmarking Results

• Tested on galaxy-s24-ultra for mobile deployment
• Evaluated on ryzen-hx-370 for desktop edge scenarios
• CPU inference metrics using Llama.cpp with 4-bit quantization
• Input benchmarks: 2K tokens for prefill performance

Advantages Over Traditional Attention

Computational Efficiency

• Lower FLOPs compared to standard attention mechanisms
• Reduced memory bandwidth requirements
• Better cache locality for CPU inference
• Optimized for sequential processing patterns

Edge Deployment Benefits

• Faster prefill performance critical for edge applications
• Reduced power consumption for mobile deployment
• Better utilization of CPU-specific optimizations
• Scalable across different CPU architectures

See also

• lfm2-5-350m
• cpu-inference-optimization
• liquid-foundation-models
• edge-models

Specialized training methodologies for models under 3B parameters optimized for edge deployment, as pioneered by liquid-ai. These models face unique challenges compared to scaled-down versions of larger models.

Key Characteristics

Memory-Bound Constraints

• <3B parameters total
• Low knowledge capacity requires careful parameter allocation
• Memory efficiency more critical than raw computational power

Task-Specific Focus

• Optimized for narrow tasks rather than general-purpose capabilities
• Easy to train and adapt to new domain-specific data
• More effective than general improvements for small models

Latency Requirements

• Sub-100ms response times required
• Fast prefill is critical, not just decode speed
• Inference optimization equally important as training

Training Methodology

Massive Overtraining

• liquid-ai's LFM2.5-350M trained on 28T tokens
• Contradicts traditional scaling laws that suggest diminishing returns
• "More pre-training works, even at the smallest scale"

Post-Training Pipeline

1. Supervised Fine-Tuning (SFT): Task-specific adaptation
2. Preference Alignment: Human preference optimization
3. Reinforcement Learning: Advanced policy optimization

Key Insight: Small models benefit more from task-specific training than general improvements beyond benchmarks.

Architecture Innovations

Parameter Distribution Optimization

• LFM2.5-350M: 19% of parameters in embedding layer
• Compare to Gemma 3 270M: 63% in embedding layer
• Effective model size: 287M from 350M total parameters

Gated Short Convolution Blocks

• Novel architecture component for efficient edge inference
• 2.5x better cost ratio vs traditional attention on CPU decode
• Optimized for memory-bound rather than compute-bound scenarios

Common Problems and Solutions

Doom Looping Problem

• Small models with reasoning traces get stuck in repetitive patterns
• Solution: Two-stage approach:
  1. On-policy data generation for DPO with heuristic filtering
  2. Reinforcement learning with n-gram repetition penalty

Cold-Start SFT Data

• Missing critical supervised fine-tuning data for specific domains
• Requires careful curation of task-specific training examples

Performance Characteristics

Inference Speed

• Competitive with llama.cpp 4-bit quantization on CPU
• Strong scaling with SGLang for GPU batch inference
• Mobile-optimized (Galaxy S24 Ultra, Ryzen HX 370)

Efficiency Gains

• Extremely efficient for narrow tasks
• Better task-specific performance than scaled-down general models

See also

• edge-ai-optimization
• doom-looping-problem
• liquid-ai
• maxime-labonne
• Model Architecture Design

Research paradigm demonstrating that overtraining models beyond traditional compute-optimal ratios can be justified when accounting for test-time computational benefits. This finding challenges conventional scaling laws and supports extreme-overtraining approaches like those used in lfm2-5-350m.

Core Principle

Traditional scaling laws suggest optimal compute allocation between model size and training tokens. However, test-time scaling research shows that:

• Overtraining smaller models can be more compute-optimal than training larger models with fewer tokens
• Test-time benefits from better-trained smaller models outweigh the additional training cost
• Inference efficiency gains justify the upfront training investment

Research Foundation

Referenced in maxime-labonne's presentation citing Roberts et al. "Test-Time Scaling Makes Overtraining Compute-Optimal" (arXiv:2604.01411, April 2026), supporting the decision to train a 350M parameter model on 28 trillion tokens.

Practical Implications

For Small Models

• Justifies training small-language-models far beyond traditional token counts
• Enables better performance from parameter-constrained models
• Particularly relevant for edge-models where model size is fixed by hardware constraints

For Edge Deployment

• Smaller overtrained models vs larger undertrained models
• Better inference characteristics on resource-constrained devices
• Improved task-specific performance through extended training

Application in LFM Series

liquid-ai applies test-time scaling principles to justify their extreme-overtraining approach:

• 28T tokens for 350M parameter lfm2-5-350m
• Orders of magnitude beyond traditional scaling recommendations
• Results in superior edge performance despite training cost

Relationship to Other Concepts

• Enables extreme-overtraining strategies
• Supports edge-models optimization
• Challenges traditional scaling laws
• Relevant to compute-optimal training discussions

See also

• extreme-overtraining
• lfm2-5-350m
• edge-models
• Scaling Laws