~/wiki
Récent+ Ajouter

Doom Looping

Confiance : high
small-modelsgeneration-problemsrepetitive-outputmodel-pathologiesreinforcement-learningpost-training

A critical pathological behavior observed in small language models where they generate repetitive patterns or get stuck in loops during text generation, particularly when attempting complex reasoning tasks. This phenomenon is especially problematic when combining small-language-models with reasoning traces due to their limited parameter capacity.

Manifestation

Typical Patterns

  • Repetitive content generation: Endlessly repeating the same phrase or ingredient
  • Stuck loops: Model cannot break out of repetitive cycle without intervention
  • Context loss: Model loses track of broader task context while looping
  • Reasoning breakdown: Complex reasoning traces become unusable due to repetition

Example Case

From liquid-ai's research on liquid-foundation-models:

User: Give me a recipe for a delicious biryani
Model: Sure! Here's a classic Indian biryani recipe:

Ingredients (serves 4)
• 1 cup basmati rice
• 1 cup chicken or vegetable broth  
• 1 cup chicken stock
• 1 cup coconut milk
• 1 cup coconut milk
• 1 cup coconut milk
• 1 cup coconut milk
• 1 cup coconut milk
...

Root Causes

Model Size Constraints

  • Limited capacity: <3B parameter models have insufficient capacity to maintain complex state
  • Memory limitations: Cannot effectively track multiple context elements simultaneously
  • Pattern overfit: Models latch onto simple patterns when complexity exceeds capacity

Training Conditions

Doom looping is particularly likely when combining:

  • Small models (<3B parameters)
  • Reasoning trace generation
  • Complex multi-step tasks
  • Insufficient post-training diversity

Solutions

Stage 1: On-Policy Data Generation

liquid-ai's approach uses multi-temperature sampling:

  • Policy model generates outputs at different temperatures (T > 0 and T = 0)
  • LLM jury evaluates response quality
  • Heuristic filtering creates chosen/rejected pairs for DPO training
  • Direct Preference Optimization trains model to avoid repetitive patterns

Stage 2: Reinforcement Learning Integration

  • N-gram repetition penalty: Explicit penalty for repetitive token sequences
  • Multi-environment training: agentic-reinforcement-learning across diverse tasks
  • Reward modeling: Combines correctness with coherence metrics
  • Policy optimization: Balances task performance with repetition avoidance

Technical Implementation

Detection Methods

  • N-gram analysis: Statistical detection of repetitive patterns
  • Entropy monitoring: Tracking information content decline
  • Length anomalies: Identifying abnormally long repetitive sequences
  • Human evaluation: Manual assessment for subtle repetition patterns

Prevention Strategies

  1. Architecture optimization: Use attention mechanisms like shortconv that maintain context better
  2. Training data diversity: Ensure sufficient variety in post-training examples
  3. Temperature scheduling: Appropriate sampling strategies during inference
  4. Repetition penalties: Both training-time and inference-time interventions

Research Findings

Effectiveness Metrics

liquid-ai's LFM2.5-1.2B-Thinking model showed significant doom loop reduction:

  • Before intervention: High repetition rates in complex reasoning tasks
  • After Stage 1 (DPO): Moderate improvement in repetition control
  • After Stage 2 (RL): Substantial reduction in doom looping incidents
  • Performance maintenance: Solutions maintain task accuracy while reducing repetition

Generalization

The doom looping problem appears to be:

  • Universal across small model architectures
  • Task-dependent (more severe with reasoning traces)
  • Solvable with appropriate post-training techniques
  • Preventable through careful architecture and training design

Implications for Edge Deployment

Doom looping is particularly problematic for edge-ai-optimization because:

  • Edge models are typically small and resource-constrained
  • Real-time applications cannot tolerate repetitive failures
  • Limited computational resources make complex mitigation difficult
  • Task-specific deployment requires robust performance

See Also