# RAG 混合搜索返回空结果，因为稠密和稀疏分数不在同一尺度上

- **ID:** `llm/rag-hybrid-search-score-mismatch`
- **领域:** llm
- **类别:** data_error
- **验证级别:** ai_generated
- **修复率:** 80%

## 根因

当使用混合搜索（稠密 + 稀疏）时，稠密嵌入相似度分数（例如，余弦相似度 0.0-1.0）和稀疏 BM25/SPLADE 分数（例如，0-20）在融合前未归一化到同一范围，导致一个组件占主导地位或在阈值处理后返回空结果。

## 版本兼容性

| 版本 | 状态 | 引入 | 弃用 |
|------|------|------|------|
| langchain==0.2.5 | active | — | — |
| chromadb==0.5.0 | active | — | — |
| qdrant-client==1.9.0 | active | — | — |
| elasticsearch==8.13.0 | active | — | — |

## 解决方案

1. ```
   在融合前使用最小-最大缩放将两组分数归一化到 [0,1]。示例：

def normalize_scores(scores):
    min_s, max_s = min(scores), max(scores)
    if max_s == min_s:
        return [0.0] * len(scores)
    return [(s - min_s) / (max_s - min_s) for s in scores]

dense_scores = normalize_scores(dense_scores)
sparse_scores = normalize_scores(sparse_scores)
hybrid_scores = [alpha * d + (1-alpha) * s for d, s in zip(dense_scores, sparse_scores)]
   ```
2. ```
   使用基于排名的融合（RRF）而不是基于分数的融合。RRF 直接合并排名，完全避免尺度问题。示例：

def reciprocal_rank_fusion(dense_ranks, sparse_ranks, k=60):
    scores = {}
    for rank, doc_id in enumerate(dense_ranks):
        scores[doc_id] = 1 / (k + rank + 1)
    for rank, doc_id in enumerate(sparse_ranks):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)
   ```
3. ```
   配置向量数据库使用内置分数归一化。对于 Qdrant，设置 `quantization_config=models.ScalarQuantization(scalar=models.ScalarQuantizationConfig(type=models.ScalarType.INT8))`，这会在内部归一化分数。
   ```

## 无效尝试

- **Lowering the similarity threshold to 0.0 to force results** — This returns all documents regardless of relevance, defeating the purpose of hybrid search and polluting the context with irrelevant documents. (80% 失败率)
- **Switching to only dense search** — This eliminates the sparse component, which may be crucial for keyword-based retrieval in domains like legal or medical where specific terms matter. (65% 失败率)
- **Increasing the alpha weight for the sparse component to 0.9** — This does not solve the scale mismatch; it only shifts dominance. The scores are still on different scales, so the fusion remains skewed. (70% 失败率)