Evaluation Framework Design

Mis à jour le 2025-12-22Confiance : high

evaluation-frameworkllm-evaluationproduction-systemsregression-testingreference-datasetscriticality-weightsbacktest-monitoringfield-level-analysisclassification-diffextraction-diffpipeline-validationmeasure-before-shipsafety-netchange-validation

Systematic approach for measuring LLM system changes before production deployment, preventing silent regressions and ensuring quality improvements. Essential for production systems where changes must be validated against real-world performance, as implemented at alan-health with comprehensive backtest monitoring and field-level analysis.

Core Principle: Measure Before You Ship

Pre-Deployment Validation: Every change (new LLM model, updated instructions, different few-shot strategy) measured against reference datasets before reaching production.

Regression Prevention: Avoid scenarios where pharmacy invoice extraction improvements silently degrade hospital invoice accuracy.

Confidence Building: Teams can iterate on extraction instructions, switch models, or adjust strategies with quantified impact visibility.

Reference Dataset Requirements

Ground Truth Quality: Real production documents with verified extractions serving as gold standard for comparison.

Representative Coverage: Documents spanning all categories, edge cases, and quality levels encountered in production.

Versioned and Immutable: Reference datasets remain stable for consistent measurement across evaluation runs.

Evaluation Components

Classification Diff: Measure whether system predicts correct document category and sub-classes compared to expected results.

Extraction Diff: Field-by-field comparison between actual extracted values and expected results across entire schema.

Criticality Weighting: Different importance levels for extraction errors:

High criticality: Amount paid, dates, critical financial values
Medium criticality: Healthcare professional names, secondary fields
Low criticality: Minor formatting or optional field variations

Backtest Execution

Full Pipeline Re-run: Complete processing from document input through transcription, classification, and extraction steps.

Automated Analysis: Systematic comparison storing results for analysis per category, per field, or in aggregate.

Dashboard Monitoring: Visual tracking of accuracy metrics, regression alerts, and improvement validation across document types.

Production Implementation

alan-health uses this framework for:

LLM model upgrades and downgrades
Prompt engineering iterations
Few-shot strategy modifications
Pipeline architecture changes
Performance optimization validation