Knowledge Compilation
The process of systematically transforming raw source documents into structured, cross-referenced knowledge artifacts that persist between interactions. Core innovation of the llm-wiki-pattern that contrasts sharply with query-time retrieval approaches in traditional RAG systems.
Compilation vs Retrieval
Traditional RAG: Fragments documents into chunks, retrieves relevant pieces at query time, and re-synthesizes knowledge for each interaction. Knowledge is rediscovered from scratch repeatedly.
Knowledge Compilation: Pre-processes sources into integrated wiki structures where cross-references exist, contradictions are flagged, and synthesis reflects accumulated reading. Knowledge is compiled once and maintained continuously.
Compilation Process
Initial Integration
When ingesting new sources, the LLM:
- Reads and extracts key information from raw documents
- Integrates across existing pages - typically touching 10-15 wiki pages per source
- Creates new entity/concept pages for previously uncovered topics
- Updates cross-references to maintain knowledge graph connectivity
- Notes contradictions where new information challenges existing claims
- Strengthens synthesis by incorporating supporting evidence
Incremental Refinement
Each compilation cycle builds on previous work:
- Concept pages deepen with additional sources and perspectives
- Entity profiles expand with new activities, relationships, and attributes
- Cross-references multiply as connections between topics emerge
- Synthesis evolves reflecting cumulative understanding rather than isolated insights
Architectural Benefits
Persistent Structure: Knowledge exists in organized form between sessions, eliminating need to rebuild understanding from raw sources.
Cumulative Intelligence: Each new source makes the entire knowledge base more valuable through integration rather than simple addition.
Query Efficiency: Questions answered by synthesizing from pre-structured content rather than assembling fragments in real-time.
Maintenance Automation: LLMs handle tedious cross-referencing and consistency work that causes humans to abandon personal wikis.
Implementation Patterns
Three-Layer Architecture: Raw sources remain immutable while compiled wiki layer evolves continuously under LLM management guided by schema configuration.
Batch Integration: Single sources can update multiple concept areas simultaneously, creating natural knowledge clustering and relationship discovery.
Version Evolution: Compiled knowledge improves over time as more sources provide additional perspectives and corrections to initial understanding.
Distinction from Document Storage
Unlike traditional document management systems where each source exists in isolation, knowledge compilation creates semantic integration where information from multiple sources synthesizes into coherent, cross-referenced understanding that compounds in value.
The compiled knowledge base becomes greater than the sum of its sources through systematic integration rather than simple accumulation.
See also
- llm-wiki-pattern - Overall framework for persistent knowledge management
- persistent-learning - Paradigm for accumulating rather than rediscovering knowledge
- incremental-knowledge-building - Methodology for systematic information integration
- compounding-artifacts - Knowledge artifacts that grow in value over time