Doom Looping Problem

A specific failure mode in small models with reasoning traces where the model gets stuck generating repetitive content, particularly problematic when combining small models (<3B parameters) with complex reasoning tasks.

Problem Description

Manifestation

• Model generates repetitive phrases or content blocks
• Example: Recipe request resulting in endless repetition of "1 cup coconut milk"
• Occurs specifically in small models using reasoning traces
• More pronounced in complex, multi-step tasks

Contributing Factors

• Model Size: Problem emerges specifically in small models (<3B parameters)
• Reasoning Traces: Complex reasoning chains increase repetition likelihood
• Task Complexity: Multi-step tasks trigger repetitive patterns more frequently
• Limited Context Handling: Small models struggle with long-range dependencies

Root Causes

Limited Parameter Capacity

• Insufficient model capacity to maintain diverse generation patterns
• Memory constraints lead to simplified repetitive behaviors
• Knowledge compression forces repetitive pattern shortcuts

Training Data Patterns

• Reasoning traces may contain repetitive structures
• Model learns to copy repetitive patterns from training examples
• Insufficient diversity in reasoning demonstration examples

Solution: Two-Stage Approach

Stage 1: On-Policy Data Generation for DPO

1. Policy Model Generation: Generate multiple outputs with temperature > 0
2. Heuristic Filtering: Filter out repetitive or low-quality responses
3. LLM Jury Evaluation: Use larger model to judge response quality
4. Chosen vs Rejected Pairs: Create preference pairs for DPO training
5. Temperature Adjustment: Use temperature = 0 for final policy model

Stage 2: Reinforcement Learning + N-gram Repetition Penalty

1. Reinforcement Learning: Policy optimization against reward model
2. N-gram Repetition Penalty: Explicit penalty for repeated n-gram sequences
3. Ground Truth Validation: Verify correctness of reasoning conclusions
4. Iterative Improvement: Multiple rounds of RL with repetition constraints

Technical Implementation

On-Policy Data Generation

• Generate diverse outputs from current policy model
• Filter using heuristic rules (repetition detection, length constraints)
• Human or LLM evaluation for quality assessment
• Create preference datasets for Direct Preference Optimization

Repetition Penalty Mechanisms

• N-gram repetition detection during generation
• Dynamic penalty scaling based on repetition frequency
• Context-aware penalty adjustment
• Balance between repetition avoidance and coherence

Evaluation Metrics

• Doom loop ratio percentage tracking
• Response quality maintenance during repetition reduction
• Task completion success rates
• Generation diversity measurements

Results and Impact

Performance Improvement

• Dramatic reduction in doom loop ratios
• Maintained task completion accuracy
• Improved generation diversity
• Better user experience for reasoning tasks

Broader Implications

• Demonstrates need for specialized small model post-training
• Shows importance of repetition-aware training objectives
• Highlights trade-offs between model size and capability complexity

See also

• small-model-training
• Reinforcement Learning from Human Feedback
• direct-preference-optimization
• liquid-ai
• Post-Training Methods