ragdiff

A minimal retrieval-quality evaluator and differ for RAG pipelines.

Most retrieval eval tools give you a number. ragdiff's real job is answering a different question: when I change my chunking, embedding model, top-k, or reranker, what actually got better or worse — and for which queries?

$ ragdiff diff --golden examples/golden_set.jsonl \
    --retriever-a ragdiff.adapters.inmemory:build_raw \
    --retriever-b ragdiff.adapters.inmemory:build_normalized \
    --k 3

              Diff summary — A → B
┏━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━┳━━━━━━━┓
┃ Metric      ┃     A ┃     B ┃     Δ ┃
┡━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━╇━━━━━━━┩
│ Precision@k │ 0.278 │ 0.333 │ +0.056│
│ Recall@k    │ 0.833 │ 1.000 │ +0.167│
│ MRR         │ 0.833 │ 1.000 │ +0.167│
│ Hit Rate    │ 0.833 │ 1.000 │ +0.167│
└─────────────┴───────┴───────┴───────┘

Why

Every RAG eval framework wants to own your embedding pipeline, your vector store, and your config format. ragdiff doesn't. It defines one Protocol:

class Retriever(Protocol):
    def retrieve(self, query: str, k: int) -> list[str]: ...

Implement that method on anything — a Qdrant wrapper, a hybrid search pipeline, a single Python function wrapped in a class — and ragdiff can score it or diff it against another config. No base classes, no framework lock-in, no vendor SDK required to run pip install.

Install

uv pip install -e .

Golden set format

JSONL, one query per line:

{"query": "how do I reset my password", "relevant_ids": ["doc_01"]}

relevant_ids are whatever IDs your corpus already uses.

Usage

Score a single retriever:

ragdiff eval --golden path/to/golden_set.jsonl \
  --retriever mymodule:build_retriever \
  --k 5

Diff two configs:

ragdiff diff --golden path/to/golden_set.jsonl \
  --retriever-a mymodule:config_a \
  --retriever-b mymodule:config_b \
  --k 5

A retriever spec is module.path:callable_name — a zero-argument callable that returns an object implementing Retriever.

Metrics

• Precision@k — fraction of top-k results that are relevant
• Recall@k — fraction of all relevant docs found in top-k
• MRR — mean reciprocal rank of the first relevant hit
• Hit Rate — fraction of queries with at least one relevant doc in top-k

Try it now

The repo ships with a toy corpus and golden set plus two demo retriever configs, so you can see a real diff without wiring up your own pipeline:

ragdiff diff --golden examples/golden_set.jsonl \
  --retriever-a ragdiff.adapters.inmemory:build_raw \
  --retriever-b ragdiff.adapters.inmemory:build_normalized \
  --k 3

Development

uv pip install -e ".[dev]" --group dev
pytest

License

MIT