On-Policy Data Generation
Confiance : high
data-generationreinforcement-learningdpoon-policyliquid-aidoom-loopingpreference-alignment
A training methodology used by liquid-ai to generate preference data for Direct Preference Optimization (DPO) by sampling from the current policy model at different temperatures, then using heuristic filtering and LLM jury evaluation to create chosen/rejected pairs.
Process
Stage 1: Data Generation
- Policy model π_θ generates multiple outputs at temperature T > 0
- Same policy model generates additional output at T = 0 (greedy decoding)
- Heuristic filtering applied to candidate responses
- LLM jury evaluates and ranks outputs
- Chosen/rejected pairs selected for DPO training
Purpose
Primary application is addressing doom-looping in small models with reasoning traces:
- Generates training data specifically targeting repetitive behaviors
- Maintains on-policy distribution alignment
- Enables targeted preference learning for problematic generation patterns
Integration
Used as Stage 1 in liquid-ai's two-stage approach:
- Stage 1: On-policy data generation for DPO
- Stage 2: agentic-reinforcement-learning with n-gram-repetition-penalty
Effectiveness
Successfully creates high-quality preference data for mitigating doom-looping-problem while maintaining model performance on target tasks.