Git Diff Tracking

Version control pattern for detecting and processing only changed content in knowledge management systems. Eliminates the need for complex manifest systems by leveraging Git's native change detection capabilities.

Core Concept

Instead of building custom tracking mechanisms, use Git diff to identify exactly what content has changed since the last processing run. This approach provides reliable change detection with minimal overhead and perfect accuracy.

Implementation Pattern

Basic Change Detection

# Identify all changed files since last successful ingest
git diff HEAD~1 --name-only raw/projects/ | while read file; do
    echo "Processing changed file: $file"
    # Process only this specific file
done

Incremental Project Synchronization

# Sync projects from development environment
rsync -av --include='*.md' --exclude='node_modules' ~/code/*/ raw/projects/

# Git shows exactly what changed
git add raw/projects/
if ! git diff --cached --quiet; then
    echo "New content detected, processing changes..."
    git diff --cached --name-only | process_changed_files
    git commit -m "Sync projects: $(date)"
fi

Architecture Benefits

Eliminates Manifest Complexity

Traditional approaches require maintaining separate tracking files:

• Timestamp-based systems prone to clock skew
• Hash-based manifests requiring custom logic
• Database tracking adding system complexity

Git diff provides all tracking functionality with zero custom code.

Atomic Processing

Git commits create natural processing boundaries:

• Each commit represents one complete ingestion cycle
• Failed processing can be retried from last successful commit
• Processing history provides complete audit trail

Distributed Reliability

Git's distributed nature provides automatic backup and synchronization:

• Processing history preserved across machines
• Remote repository serves as authoritative state
• Merge conflicts reveal concurrent processing issues

Advanced Patterns

Selective Processing by File Type

# Process only markdown changes
git diff HEAD~1 --name-only --diff-filter=AM | grep '\.md$' | process_files

# Handle deletions separately  
git diff HEAD~1 --name-only --diff-filter=D | cleanup_deleted_files

Content-Aware Change Detection

# Get actual content changes, not just file list
git diff HEAD~1 raw/projects/ | while read line; do
    if $line =~ ^\+\+\+; then
        # New file or significant addition
        process_file_addition
    elif $line =~ ^\---; then
        # Deleted content
        process_file_removal  
    fi
done

Automated Commit Integration

# Combine sync and change detection
sync_projects() {
    rsync -av ~/code/*/ raw/projects/
    
    if git add raw/projects/ && ! git diff --cached --quiet; then
        # Process only changed content
        git diff --cached --name-only | process_incremental_changes
        
        # Commit after successful processing
        git commit -m "Auto-sync: $(date) - $(git diff --cached --numstat | wc -l) files"
        
        echo "✓ Processed $(git diff HEAD~1 --numstat | wc -l) changed files"
    else
        echo "No changes detected"
    fi
}

Error Handling

Processing Failure Recovery

# If processing fails, reset to last known good state
if ! process_changes; then
    echo "Processing failed, resetting to last commit"
    git reset --hard HEAD
    exit 1
fi

Partial Success Handling

# Process files individually to isolate failures
git diff HEAD~1 --name-only | while read file; do
    if process_single_file "$file"; then
        git add "$file"
        echo "✓ Processed $file"
    else
        echo "✗ Failed to process $file"
        # Continue with other files
    fi
done

# Commit whatever succeeded
if ! git diff --cached --quiet; then
    git commit -m "Partial sync: $(date)"
fi

Integration Patterns

Daily Automation Integration

# Daily agent workflow
cd /path/to/knowledge-base

# Sync latest content
./scripts/collect-projects.sh

# Process only changes
if ! git diff --quiet raw/projects/; then
    echo "Changes detected, starting processing..."
    git add raw/projects/
    changed_files=$(git diff --cached --name-only | wc -l)
    
    # Process incrementally
    process_git_changes
    
    # Update wiki index
    update_index_with_changes
    
    git commit -m "Daily sync: $changed_files files updated"
    echo "✓ Processed $changed_files changed files"
else
    echo "No changes since last sync"
fi

Multi-Repository Coordination

# Track changes across multiple source repositories
for repo in ~/code/*/; do
    repo_name=$(basename "$repo")
    
    # Sync and track per-repository changes