Choose a RAG configuration on evidence: quality, latency and cost, with confidence intervals.

Quickstart

Requires Python 3.11+ and uv. One command runs the whole demo offline, with no API keys:

git clone https://github.com/seanmcrae/ragbench.git && cd ragbench && make demo

make demo runs these three steps, which you can also run one at a time:

uv run rag-eval run configs/example.yaml                 # 8 pipelines x 55 queries
uv run rag-eval report --img-dir docs/img                # runs/example/report.{md,html} + chart
uv run rag-eval compare bm25-k5 hybrid-rerank-agent3-k5  # paired deltas with 95% CIs

Other targets: make ci (lint, format check, type check, tests), make site (the static docs site in site/), make scifact (optional public benchmark, downloads about 3 MB). To use a hosted model instead of the mock, install an extra and export its key:

uv sync --frozen --extra anthropic && export ANTHROPIC_API_KEY=...
# in the experiment YAML: generator: {provider: anthropic, model: claude-haiku-4-5}
#                         judge:     {provider: openai, model: gpt-4o-mini}

Results

Generated by make site from rag-eval run configs/example.yaml (experiment example). Nothing below is typed by hand.

Dataset: synthetic-helpcenter (176 documents, 55 queries, fingerprint 32cc21df822d5c4e). Prices: illustrative table dated 2026-10-07. Run created 2026-10-08T07:36:22+00:00.

Generation latency comes from the offline mock's token-based latency model, not from a live API; retrieval latency is measured wall-clock.

Recommendation

Budget: cost <= $1.00 per 1k queries and p95 latency <= 1,500 ms. Selection: highest answer_f1 on the in-budget Pareto frontier.

Recommended: dense-lsa-k3 (answer_f1 0.606, $0.475 per 1k queries, p95 862 ms).

Versus the baseline bm25-k5: +0.057 answer_f1 [-0.013, +0.145], no significant difference at 95% over 55 queries.

Best answer_f1 regardless of budget: hybrid-rerank-agent3-k5 (0.622, +0.016 vs the recommendation) at 3.9x the cost and 2.3x the p95 latency.

In-budget Pareto frontier (cheapest first): dense-lsa-k3.

In budget but dominated: bm25-k5, hybrid-k5, hybrid-rerank-k5, dense-lsa-k5.

Over budget: hybrid-rerank-agent3-k5 (cost $1.868/1k > $1.00; p95 1961 ms > 1500 ms); dense-lsa-agent3-k3 (cost $1.337/1k > $1.00; p95 1877 ms > 1500 ms); dense-lsa-agent3-k5 (cost $1.916/1k > $1.00; p95 1928 ms > 1500 ms).

All pipelines

Versus the baseline (bm25-k5)

Paired bootstrap over queries: difference in mean answer_f1 with a 95% percentile interval.

answer_f1 by query type

hybrid-rerank-agent3-k5 vs bm25-k5, every metric

The highest-quality configuration regardless of budget, against the baseline. Same output as rag-eval compare bm25-k5 hybrid-rerank-agent3-k5.

How evaluation works

1. Run. Each pipeline answers every query: retrieve top-k (optionally rerank from a deeper pool), build a prompt from the passages, generate an answer with [doc-id] citations. An agentic pipeline may ask for another search, up to max_steps. Every stage is timed and every completion records its token usage.
2. Score each query.
   • Retrieval: recall@k, precision@k, MRR and nDCG@k against graded qrels (2 = states the answer, 1 = supporting or bridge document).
   • Answer: SQuAD-style token F1 and exact match against the reference answers.
   • Groundedness: share of answer sentences that cite a retrieved document, times the share of the answer's content terms found in the documents it cites. Uncited answers score 0.
   • Judge: a 1-5 rubric (correct, complete, supported by cited passages) normalised to [0, 1]; the offline judge maps F1 and citation coverage onto it, an LLM judge reads the passages.
   • Latency and cost: per-stage wall-clock (modeled for the mock generator), and tokens priced from configs/prices.yaml.
3. Summarise. Means per pipeline, p50/p95 latency, cost per 1k queries, quality by query type.
4. Decide. Drop configurations over the cost or p95 budget (reasons recorded), take the Pareto frontier of (quality, cost, p95) among the rest, and pick the highest quality, ties to cheaper then faster. Deltas against the baseline use a paired bootstrap over queries.
5. Gate. rag-eval compare A B --fail-on-regression exits 1 only when B's gate metric is significantly worse than A's (the CI excludes zero) and the drop exceeds --tolerance.

Architecture

flowchart LR
    cfg["Experiment YAML: pipelines, matrix, budget"] --> run
    data["Dataset: corpus, queries, qrels, answers"] --> run
    run["Runner: each pipeline x each query"] --> ret
    ret["Retriever: BM25, dense or hybrid RRF"] --> rr["Reranker (optional)"]
    rr --> gen["Generator: mock, Anthropic or OpenAI"]
    gen -->|"agentic: next search"| ret
    gen <--> cache[("Response cache: SQLite, content-hashed")]
    gen --> score["Scoring: retrieval, answer, groundedness, judge, latency, cost"]
    prices["Price table YAML"] --> score
    score --> results["runs/NAME: per_query.jsonl, summary.csv, manifest.json"]
    results --> report["rag-eval report: budgeted Pareto pick, bootstrap CIs, MD, HTML, PNG"]
    results --> compare["rag-eval compare: paired deltas, release gate"]

Design decisions

• Offline by default, same code path online. The mock generator receives the exact prompt an API model would, so token counts and cost scale with context size and step count the way they would in production. Keys are only read when a hosted provider is configured.
• Latency is honest about where it comes from. Retrieval and reranking are timed wall-clock. API generations are timed wall-clock; the mock reports a modeled latency (350 ms + 0.08 ms per input token + 15 ms per output token) and every report says so.
• Cache keys hash content, not ids. A key covers the generator's identity and parameters, the question, prior searches and each passage's id and text hash. Editing a document invalidates exactly the generations that saw it. Cached entries keep their original latency so a warm re-run does not flatter a pipeline.
• Paired bootstrap over queries. Pipelines answer the same questions, so the harness resamples per-query differences rather than comparing two independent means; with 55 queries that is the difference between a usable interval and noise.
• A release gate needs significance and size. compare --fail-on-regression fails only when the CI of the delta is entirely below zero and the point estimate is worse than --tolerance. Either condition alone either blocks on noise or ships real regressions.
• The recommendation is constrained, then Pareto. Configs over the cost or p95 budget are dropped with the reason recorded; the pick is the highest-quality point on the in-budget frontier of (quality, cost, p95), ties to cheaper then faster.
• Rank fusion, not score fusion. RRF (k=60) ignores raw scores, which live on incomparable scales for BM25 and cosine similarity.
• Own BM25 and BEIR's file format. A 60-line BM25 over an inverted index with Lucene's non-negative IDF is easier to verify (there is a hand-computed test) than a dependency, and the BEIR layout means public benchmarks load without conversion.
• Prices are configuration. configs/prices.yaml is labeled illustrative and dated; a missing model is a loud error naming the known models.

Data

• Bundled synthetic corpus (data/synthetic_helpcenter/, generated by scripts/generate_synthetic.py from src/rag_eval/datasets/synthetic.py, seed 7): the help center of Tallyhive, a fictional invoicing and time-tracking SaaS. 176 short documents (plan limits, feature setup, plan availability, integrations, error codes, policies, release notes) and 55 queries across 10 types, 8 of them multi-hop ("What is the seat cap on the cheapest plan that includes SAML single sign-on?"). Qrels are graded (2 = states the answer, 1 = supporting or bridge document) and every query has a reference answer. All content is invented; real product names appear only as integration targets. A test asserts the committed files hash identically to fresh generator output.
• BEIR SciFact (optional): scripts/download_scifact.py fetches https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scifact.zip and checks the MD5 published in the BEIR README. SciFact (Wadden et al., 2020) is licensed CC BY-NC 2.0; it is never downloaded by tests or CI and is git-ignored. It has no reference answers, so answer metrics are skipped and configs/scifact.yaml ranks on nDCG@10. As a sanity check, a local run of that config gave BM25 (k1=0.9, b=0.4) nDCG@10 0.641 on the 300 test queries, against 0.665 reported for BM25 in the BEIR paper (Table 2).

Configuration

An experiment is one YAML file; configs/example.yaml is annotated. Top-level keys:

Per pipeline: retriever (bm25 with k1/b, dense with an embedder, hybrid with components, rrf_k, weights), reranker (query-likelihood or cross-encoder, depth), generator (mock, anthropic, openai; model, max_tokens, and price_as for the mock), top_k, max_steps. Unknown keys are rejected. Hosted providers read ANTHROPIC_API_KEY or OPENAI_API_KEY from the environment and are only imported when used.

Where it fails

Failure modes from the runs above, split by where the limit sits. Issue links track the ones with a planned fix.

Model and data limits (the offline generator, the corpus, the sample size)

Design and scaffolding limits (choices in the harness itself)

Considered and rejected: a t-test for comparing pipelines. Answer metrics are bounded and mostly right-or-wrong per query, so normality is a poor assumption at n=55. The paired bootstrap makes no distributional assumption, pairing removes per-question difficulty, and it yields the interval a ship decision needs. The cost is 2,000 resamples per comparison, which is negligible at this size (docs/PRODUCT.md, trade-offs).

The known limitations follow.

Limitations

• The mock generator and planner are lexical heuristics. They make relative comparisons between retrieval configurations meaningful offline, but absolute answer quality and the agentic results describe the heuristics, not an LLM. Lexical gaps defeat them: every pipeline scores 0 on "what data does X sync" because the answer sentence says "synced".
• The synthetic corpus is templated, single-domain and small. Its phrasing patterns favour copular fact sentences, and n=55 gives wide intervals (the recommended config's lift is not significant). Use it to exercise the harness; decide on your own queries.
• The heuristic judge maps token F1 and citation coverage onto the rubric, so it agrees with answer F1 by construction. An LLM judge adds signal but brings its own biases (see docs/PRODUCT.md).
• For agentic pipelines the retrieval metrics score passages in the order the generator saw them, so hop results can sit beyond rank k.
• Offline token counts are an approximation (about 4 characters per token); API adapters use provider-reported usage. Cost-to-serve excludes corpus indexing and the judge.
• No live traffic: these are offline metrics. Click-through, deflection or CSAT need an online experiment.

Product brief

The problem, users, scope, success metrics, trade-offs, risks and roadmap are in the product brief.