performance metrics

---
title: Performance Metrics
category: concepts
created: 2025-12-22
updated: 2025-12-22
tags: [metrics, evaluation, performance, measurement, benchmarks, scoring]
sources: [raw/papers/the-llm-evaluation-guidebook.pdf]
confidence: high
---

# Performance Metrics

Quantitative measures used to assess the quality and effectiveness of Large Language Model outputs across various tasks and domains. Essential for objective comparison between models and tracking improvement over time.

## Types of Metrics

### Accuracy-Based Metrics
- **Exact Match**: Perfect alignment between predicted and reference answers
- **Token Accuracy**: Percentage of correctly predicted tokens
- **Top-k Accuracy**: Target answer appears in top-k predictions

### Text Quality Metrics
- **BLEU**: Bilingual Evaluation Understudy for translation quality
- **ROUGE**: Recall-Oriented Understudy for text summarization
- **METEOR**: Metric for machine translation evaluation
- **BERTScore**: Semantic similarity using contextual embeddings

### Task-Specific Metrics
- **Code Evaluation**: Pass@k, execution success rate, functional correctness
- **Mathematical Reasoning**: Problem-solving accuracy, step-by-step correctness
- **Question Answering**: F1 score, exact match, semantic equivalence

### Efficiency Metrics
- **Latency**: Response time for generation
- **Throughput**: Tokens generated per second
- **Memory Usage**: Computational resource requirements
- **Energy Consumption**: Environmental impact assessment

## Selection Criteria

### Task Alignment
- **Domain relevance**: Metrics should match intended use case
- **Evaluation granularity**: Token-level vs. sequence-level assessment
- **Reference requirements**: Availability of ground truth data

### Statistical Properties
- **Reliability**: Consistent results across runs
- **Validity**: Measures what it claims to measure
- **Sensitivity**: Ability to detect meaningful differences
- **Interpretability**: Clear meaning and actionable insights

## Implementation Considerations

### Metric Computation
- **Preprocessing**: Text normalization and tokenization
- **Aggregation**: Averaging across samples and batches
- **Confidence intervals**: Statistical significance testing
- **Multiple references**: Handling diverse acceptable answers

### Common Pitfalls
- **Gaming**: Optimizing specifically for metrics rather than quality
- **Metric brittleness**: Small changes causing large score variations
- **Cultural bias**: Metrics favoring particular language patterns
- **Limited coverage**: Missing important quality dimensions

## Best Practices

### Comprehensive Assessment
- **Multiple metrics**: No single metric captures all aspects
- **Human validation**: Correlation with human judgment
- **Ablation studies**: Understanding component contributions
- **Error analysis**: Qualitative examination of failures

### Reporting Standards
- **Methodology transparency**: Clear description of evaluation setup
- **Statistical significance**: Proper testing of differences
- **Limitation acknowledgment**: Known biases and constraints
- **Reproducibility**: Sufficient detail for replication

## See also

- [llm-evaluation](/concepts/llm-evaluation)
- [benchmark-selection](/concepts/benchmark-selection)
- [human-evaluation](/concepts/human-evaluation)
- Statistical Testing