Dataset: synthetic-helpcenter (176 documents, 55 queries, fingerprint 32cc21df822d5c4e). Prices: illustrative table dated 2026-10-07. Run created 2026-10-08T07:36:22+00:00.
Generation latency comes from the offline mock's token-based latency model, not from a live API; retrieval latency is measured wall-clock.
Budget: cost <= $1.00 per 1k queries and p95 latency <= 1,500 ms. Selection: highest answer_f1 on the in-budget Pareto frontier.
Recommended: dense-lsa-k3 (answer_f1 0.606, $0.475 per 1k queries, p95 862 ms).
Versus the baseline bm25-k5: +0.057 answer_f1 [-0.013, +0.145], no significant difference at 95% over 55 queries.
Best answer_f1 regardless of budget: hybrid-rerank-agent3-k5 (0.622, +0.016 vs the recommendation) at 3.9x the cost and 2.3x the p95 latency.
In-budget Pareto frontier (cheapest first): dense-lsa-k3.
In budget but dominated: bm25-k5, hybrid-k5, hybrid-rerank-k5, dense-lsa-k5.
Over budget: hybrid-rerank-agent3-k5 (cost $1.868/1k > $1.00; p95 1961 ms > 1500 ms); dense-lsa-agent3-k3 (cost $1.337/1k > $1.00; p95 1877 ms > 1500 ms); dense-lsa-agent3-k5 (cost $1.916/1k > $1.00; p95 1928 ms > 1500 ms).
| pipeline | answer_f1 | exact_match | judge_score | groundedness | nDCG@5 | recall@5 | MRR | p50 ms | p95 ms | $/1k q | steps |
|---|---|---|---|---|---|---|---|---|---|---|---|
| bm25-k5 | 0.549 | 0.418 | 0.573 | 0.909 | 0.802 | 0.818 | 0.891 | 652 | 884 | 0.646 | 1.00 |
| hybrid-k5 | 0.552 | 0.436 | 0.573 | 0.909 | 0.810 | 0.818 | 0.938 | 652 | 874 | 0.650 | 1.00 |
| hybrid-rerank-k5 | 0.549 | 0.418 | 0.573 | 0.909 | 0.795 | 0.809 | 0.903 | 679 | 885 | 0.651 | 1.00 |
| hybrid-rerank-agent3-k5 | 0.622 | 0.491 | 0.645 | 0.927 | 0.795 | 0.809 | 0.903 | 1,603 | 1,961 | 1.868 | 1.67 |
| dense-lsa-k3 | 0.606 | 0.491 | 0.627 | 0.909 | 0.830 | 0.818 | 0.979 | 635 | 862 | 0.475 | 1.00 |
| dense-lsa-agent3-k3 | 0.570 | 0.455 | 0.591 | 0.927 | 0.827 | 0.800 | 0.979 | 1,545 | 1,877 | 1.337 | 1.73 |
| dense-lsa-k5 | 0.606 | 0.491 | 0.627 | 0.909 | 0.830 | 0.818 | 0.979 | 647 | 875 | 0.647 | 1.00 |
| dense-lsa-agent3-k5 | 0.570 | 0.455 | 0.591 | 0.927 | 0.830 | 0.818 | 0.979 | 1,599 | 1,928 | 1.916 | 1.65 |
Paired bootstrap over queries, 95% percentile intervals on the difference in mean answer_f1.
| pipeline | delta answer_f1 | 95% CI | p | verdict |
|---|---|---|---|---|
| hybrid-k5 | +0.003 | [-0.047, +0.055] | 0.956 | no significant difference |
| hybrid-rerank-k5 | +0.000 | [+0.000, +0.000] | 1.000 | no significant difference |
| hybrid-rerank-agent3-k5 | +0.073 | [+0.018, +0.145] | 0.038 | better |
| dense-lsa-k3 | +0.057 | [-0.013, +0.145] | 0.095 | no significant difference |
| dense-lsa-agent3-k3 | +0.021 | [-0.031, +0.091] | 0.455 | no significant difference |
| dense-lsa-k5 | +0.057 | [-0.013, +0.145] | 0.095 | no significant difference |
| dense-lsa-agent3-k5 | +0.021 | [-0.031, +0.091] | 0.455 | no significant difference |
| pipeline | error_cause | error_fix | feature_limit | feature_location | feature_plan | integration_objects | integration_sync | multi_hop | plan_attribute | policy |
|---|---|---|---|---|---|---|---|---|---|---|
| bm25-k5 | 0.85 | 0.69 | 1.00 | 0.56 | 0.80 | 0.00 | 0.40 | 0.00 | 0.71 | 0.46 |
| hybrid-k5 | 0.85 | 0.69 | 1.00 | 0.39 | 0.80 | 0.00 | 0.60 | 0.00 | 0.71 | 0.46 |
| hybrid-rerank-k5 | 0.85 | 0.69 | 1.00 | 0.56 | 0.80 | 0.00 | 0.40 | 0.00 | 0.71 | 0.46 |
| hybrid-rerank-agent3-k5 | 0.85 | 0.69 | 1.00 | 0.56 | 0.80 | 0.00 | 0.40 | 0.50 | 0.71 | 0.46 |
| dense-lsa-k3 | 0.85 | 0.69 | 1.00 | 0.39 | 0.80 | 0.00 | 1.00 | 0.00 | 0.81 | 0.46 |
| dense-lsa-agent3-k3 | 0.85 | 0.69 | 1.00 | 0.39 | 0.80 | 0.00 | 0.40 | 0.12 | 0.81 | 0.46 |
| dense-lsa-k5 | 0.85 | 0.69 | 1.00 | 0.39 | 0.80 | 0.00 | 1.00 | 0.00 | 0.81 | 0.46 |
| dense-lsa-agent3-k5 | 0.85 | 0.69 | 1.00 | 0.39 | 0.80 | 0.00 | 0.40 | 0.12 | 0.81 | 0.46 |