~/wiki
Récent+ Ajouter

TF-IDF Text Search

Confiance : medium
tf-idftext-searchinformation-retrievalterm-weightingsparse-vectorskeyword-searchdocument-similaritytokenizationcachingperformance-optimization

Term Frequency-Inverse Document Frequency is a classical text search algorithm that weights terms based on their importance within a document and rarity across a corpus. Fundamental to many information retrieval systems and still effective for domain-specific search applications.

Mathematical Foundation

Term Frequency (TF)

Measures how frequently a term appears in a document:

TF(t,d) = count(t in d) / total_terms(d)

Inverse Document Frequency (IDF)

Measures the rarity of a term across the entire corpus:

IDF(t) = log(total_documents / documents_containing(t))

Combined TF-IDF Score

TF-IDF(t,d) = TF(t,d) × IDF(t)

Implementation Patterns

Tokenization and Preprocessing

Standard preprocessing pipeline:

  1. Text Normalization: Lowercase conversion, punctuation removal
  2. Tokenization: Split text into individual terms
  3. Stop Word Removal: Filter common words with little semantic value
  4. Stemming/Lemmatization: Reduce words to root forms

Caching Strategies

For performance optimization in systems like csv-based-retrieval:

  • Token Structure Caching: Pre-computed term-document matrices
  • IDF Score Caching: Pre-calculated inverse document frequencies
  • Similarity Matrix Caching: Pre-computed document-document similarities for small corpora

Advantages

Computational Efficiency

  • Sparse Vectors: Most documents contain only a small subset of vocabulary
  • Fast Similarity Computation: Efficient cosine similarity calculation
  • Scalable Indexing: Can be pre-computed and cached

Domain Effectiveness

  • Keyword Relevance: Excellent for exact term matching
  • Corpus Adaptation: Automatically adjusts to domain-specific terminology
  • Interpretability: Clear understanding of why documents are retrieved

Modern Applications

RAG Systems

TF-IDF serves as effective baseline or fallback for retrieval-augmented-generation:

  • Hybrid Retrieval: Combined with semantic search for comprehensive coverage
  • Keyword Filtering: Pre-filtering large corpora before expensive semantic similarity
  • Fallback Mechanism: When vector embeddings are unavailable

Particularly effective for legal text search:

  • Exact Term Matching: Critical for finding specific legal concepts
  • Citation Retrieval: Finding documents containing specific references
  • Regulatory Compliance: Searching for specific regulatory language

Performance Characteristics

Memory Usage

  • Sparse Matrices: Efficient storage for large vocabularies
  • Incremental Updates: Can add new documents without full recomputation
  • Vocabulary Management: Memory scales with unique terms, not document count

Query Performance

  • Sub-linear Search: With proper indexing, search time scales better than linear
  • Batch Processing: Efficient for multiple queries against same corpus
  • Real-time Updates: Fast incremental updates for dynamic corpora

Limitations

Semantic Blindness

  • Synonym Problem: Doesn't recognize semantically similar terms
  • Context Ignorance: Same word in different contexts treated identically
  • Word Order: Ignores phrase structure and word relationships

Statistical Assumptions

  • Bag of Words: Ignores word order and grammatical structure
  • Independence Assumption: Treats words as independent features
  • Linear Scoring: May not capture non-linear term interactions

Integration with Modern Systems

Streamlit Applications

Common pattern for interactive search interfaces:

def tf_idf_search(query, corpus, top_k=10):
    # Tokenize and weight query terms
    # Compute similarity scores
    # Return ranked results
    return ranked_chunks

Hybrid Architectures

TF-IDF often complements semantic search:

  1. First Stage: TF-IDF for broad relevance filtering
  2. Second Stage: Neural embeddings for semantic refinement
  3. Result Fusion: Combine scores from both approaches

See also