human data quality

---
title: Human Data Quality
category: concepts
created: 2026-06-11
updated: 2025-01-03
tags: [human-data-quality, data-annotation, machine-learning, rlhf, classification-tasks, training-data, data-collection, model-performance, quality-control, inter-annotator-agreement, deep-learning, fuel-metaphor, ml-techniques, cultural-bias, sambasivan-et-al, model-work-bias]
sources: [raw/feeds/2026-06-11-thinking-about-high-quality-human-data.md]
confidence: high
---

# Human Data Quality

The challenge of obtaining high-quality labeled data from human annotators for machine learning model training. Critical foundation for modern deep learning systems, particularly in classification tasks and [rlhf](/concepts/rlhf) alignment training for language models. As emphasized by lilian-weng, high-quality human data serves as the "fuel" for modern deep learning model training - just as high-quality fuel enables optimal engine performance, high-quality training data enables optimal model performance.

Despite widespread recognition of its importance, the ML community exhibits a cultural bias where "Everyone wants to do the model work, not the data work" as documented by [sambasivan-data-work-research](/concepts/sambasivan-data-work-research).

## Core Challenges

### Data Collection Fundamentals
- Attention to detail in annotation design
- Careful execution of collection protocols
- Systematic quality control measures
- Inter-annotator agreement validation

### Technical Solutions
While ML techniques can assist with data quality improvement, the fundamental challenge remains human-centered:
- Proper task design and structuring
- Clear annotation guidelines
- Robust validation frameworks
- Cultural recognition of data work importance

### RLHF Applications
Most RLHF labeling can be structured as classification tasks, making data quality principles directly applicable to language model alignment training.

## Community Dynamics

The [ml-community-culture](/concepts/ml-community-culture) systematically undervalues data work despite its fundamental importance, creating ongoing challenges for:
- Resource allocation to data quality efforts
- Career incentives for data-focused work
- Recognition of annotation expertise
- Investment in data infrastructure

## See also

- [data-annotation](/concepts/data-annotation)
- [classification-tasks](/concepts/classification-tasks)
- [rlhf](/concepts/rlhf)
- [fuel-metaphor](/concepts/fuel-metaphor)
- [ml-community-culture](/concepts/ml-community-culture)