grpo training

---
title: GRPO Training
category: concepts
created: 2025-01-04
updated: 2025-01-04
tags: [grpo, group-relative-policy-optimization, reinforcement-learning, variance-requirements, reward-distribution, training-efficiency, rollout-ranking, wandb-art]
sources: [raw/conversations/2026-06-03-cursor-workshops-WB-11eba586.md]
confidence: high
---

# GRPO Training

Group Relative Policy Optimization (GRPO) is a reinforcement learning technique that trains agents by comparing rollouts within groups rather than using absolute reward values. Critical for efficient agent training when reward signals are noisy or sparse.

## Core Mechanism

GRPO works by:
1. **Grouping rollouts** from the same training step
2. **Ranking performance** within each group
3. **Learning from comparisons** rather than absolute rewards
4. **Updating policy** based on relative performance

This approach is more robust to reward scale and distribution than traditional policy gradient methods.

## Critical Success Factor: Reward Variance

### The Variance Requirement
GRPO fundamentally depends on **reward variance within groups**. Key metrics:

- **`num_groups_trainable`**: Groups with sufficient reward variance for learning
- **`num_groups_submitted`**: Total groups generated
- **`reward_std`**: Standard deviation of rewards within groups

### Failure Mode: Uniform Rewards
When all rollouts in a group receive similar rewards:

Group A: [0.93, 0.92, 0.94] → reward_std ≈ 0 → non-trainable Group B: [0.4, 0.8, 0.1] → reward_std > threshold → trainable

Groups with low variance provide no ranking signal, making training inefficient.

## Common Training Issues

### Low Trainability Ratio
Workshop example showing efficiency problem:

data/cum/num_groups_submitted 28 data/cum/num_groups_trainable 5 ← only 17% usable

When most groups become non-trainable:
- **Training efficiency drops** dramatically
- **Learning signal weakens** despite computational cost
- **Convergence slows** or stalls entirely

### Causes of Variance Collapse
1. **Binary rewards**: All-or-nothing scoring (0 vs 1) reduces variance
2. **Easy tasks**: When most rollouts succeed, rewards cluster
3. **Poor prompting**: Deterministic agent behavior across rollouts
4. **Insufficient rollouts**: Too few samples per group for variance

## Optimization Strategies

### Increasing Reward Variance
- **Continuous metrics**: Use F1 scores, not just binary accuracy
- **Multiple rollouts per group**: More samples increase variance probability
- **Composite scoring**: Combine multiple metrics with different distributions
- **Task difficulty tuning**: Ensure reasonable success/failure rates

### Monitoring Training Health
Key indicators to watch:
```python
# Healthy GRPO training shows:
reward_std > 0.1  # Sufficient variance for ranking
trainable_ratio > 0.3  # Most groups contribute to learning
train_reward_trend = "increasing"  # Overall improvement

Weight Adjustment Strategy

When facing low trainability:

1. Identify uniform components: Which metrics show no variance?
2. Increase continuous metric weights: Boost F1, distance, or similarity scores
3. Reduce binary metric weights: De-emphasize all-or-nothing measurements
4. Add penalty variance: Introduce graduated penalties, not just -1/0

Time-Constrained Optimization

Competition Context

In workshop scenarios with time pressure:

• Conservative step increases: 3→20 steps safer than 3→50
• Proven weight adjustments: Target lowest-performing metrics
• Avoid experimental changes: Stick to validated approaches

Practical Implementation

Workshop-proven approach:

# Before: Poor performance
training_steps = 3  # Insufficient for learning
citation_weight = 0.3  # Weak signal for citing sources

# After: Improved performance  
training_steps = 20  # Adequate learning time
citation_weight = 0.65  # Strong signal for evidence attribution

Integration with Composite Scoring

GRPO pairs naturally with composite-scoring because:

• Multiple metrics provide variance when individual components don't
• Continuous scores (F1, efficiency) create ranking opportunities
• Weight balancing can optimize for both performance and trainability

Best Practices

Setup Phase

1. Validate reward distribution before training
2. Test rollout variance with small pilot runs
3. Monitor trainability metrics during early steps
4. Adjust weights if variance collapses

During Training

1. Watch data/step_num_groups_trainable for trend analysis
2. Stop early if trainability drops to near-zero
3. Adjust weights mid-training if needed for variance recovery

Post-Training Analysis

1. Calculate overall trainability ratio: trainable_groups / total_groups
2. Identify variance bottlenecks: Which metrics created uniformity?
3. Document successful weight combinations for future runs

Troubleshooting

Symptoms: train: 0it [00:02, ?it/s]

This indicates zero trainable groups for that step.

Quick fixes:

• Increase rollouts per group
• Boost weights on continuous metrics
• Reduce weights on binary metrics
• Check if task difficulty appropriate

Symptoms: Declining trainability over time

As agents improve, tasks become easier, reducing variance.

Solutions:

• Curriculum learning with harder tasks
• Dynamic reward thresholds
• Multi-level evaluation scenarios

GRPO training requires careful attention to reward distribution and variance management, but when properly configured, provides robust and efficient agent optimization particularly suitable for competitive scenarios and time-constrained development.