llm retrieval ai_generated true

RAG retrieval returns irrelevant results despite relevant documents existing in the index

ID: llm/rag-chunk-size-vs-embedding-max

Also available as: JSON · Markdown
82%Fix Rate
88%Confidence
3Evidence
2023-01-01First Seen

Version Compatibility

VersionStatusIntroducedDeprecatedNotes
any active — — —

Root Cause

Embedding models have max token limits (e.g., 8191 for ada-002). Text beyond the limit is silently truncated. If chunks are larger than the embedding model's limit, only the beginning is semantically captured. Small chunks lose context.

generic

Workarounds

  1. 92% success Set chunk size to 50-80% of embedding model's max token limit with overlap 
    chunk_size=512, chunk_overlap=64  # for models with 8191 max tokens. Overlap preserves context boundaries.
  2. 88% success Use parent-child chunking: embed small chunks but retrieve parent documents 
    Embed 256-token chunks. On retrieval, return the 2048-token parent document containing the matched chunk.
  3. 85% success Add metadata (title, section headers) to each chunk before embedding 
    chunk_text = f'Document: {title}\nSection: {header}\n\n{chunk}'  # provides context for embedding

Dead Ends

Common approaches that don't work:

  1. Use large chunk sizes (10K+ tokens) for more context per chunk 85% fail

    Embedding model silently truncates beyond its max tokens. A 10K token chunk with 8191 limit embeds only the first 82%. The answer might be in the truncated part.

  2. Use very small chunks (50-100 tokens) for precision 78% fail

    Small chunks lose surrounding context. 'It was resolved in v2.3' means nothing without knowing what 'it' refers to.