Model Dependency Tracing

Methodology for mapping the complex dependency graphs of modern large language models, revealing how contemporary LLMs rely on extensive chains of other models and datasets rather than being trained from scratch on raw data.

Core Concept

Modern LLM training has evolved from simple "model trained on web data" to deeply compositional processes involving multiple model generations, synthetic data creation, and complex dependency chains.

AllenAI ModSleuth Implementation

allenai-modsleuth pioneered systematic dependency tracing, revealing:

Dependency Scale

• Olmo 3: Depends on 89 models and 183 datasets
• Nemotron 3: Depends on 273 models and 560 datasets

Dependency Types

• Model dependencies: Teacher models used for distillation, reward models, safety filters
• Dataset dependencies: Synthetic datasets generated by other models, processed datasets, filtered corpora
• Indirect dependencies: Transitive dependencies through multiple model generations

Implications

For Model Understanding

• Challenges simplistic narratives about LLM training
• Reveals compositional nature of modern model development
• Highlights interconnectedness of model ecosystem

For Reproducibility

• Dependencies may not be fully documented or available
• Reproducibility requires recreating entire dependency chains
• Licensing and access issues compound across dependencies

For Security and Safety

• Attack surfaces include entire dependency chain
• Safety properties may depend on upstream model behaviors
• Audit requirements expand to include all dependencies

Technical Challenges

Tracing Complexity

• Dependencies often undocumented or implicit
• Synthetic data generation obscures original sources
• Multi-step processing pipelines create complex lineages

Scale Management

• Dependency graphs grow exponentially with model generations
• Tracking becomes computationally intensive
• Storage and querying of dependency information

Industry Response

Recognition that:

• Transparency needed: Clear documentation of model dependencies
• Tooling required: Systematic approaches to dependency tracking
• Standards emerging: Common formats for dependency specification

Future Directions

• Automated tracing: Tools that automatically discover dependencies
• Dependency optimization: Reducing unnecessary dependencies for efficiency
• Security scanning: Auditing entire dependency chains for vulnerabilities

See also

• allenai-modsleuth
• Model Lineage
• LLM Training
• Synthetic Data
• Model Composition