~ $ cat demos/rag/README
retrieval-augmented generation
Hybrid retrieval over Azure docs with a RAGAS faithfulness guardrail that retries on weak grounding — ask it something and watch the pipeline work.
// interactive
● live · your query// what's happening
Your question fans out to dense (vector) and bm25 (keyword) search, fused by RRF, reranked by Azure's semantic ranker; a gpt-4o agent answers using only those sources. RAGAS scores faithfulness — weak answers trigger a retry. Run one and the diagram follows along.
// pipeline
The query fans out to two retrievers in parallel — dense vector search and bm25 keyword search. Their results are fused by reciprocal rank fusion, reranked, and passed to the generator. A faithfulness score gates the answer: below 0.7 the pipeline retries, otherwise it returns the answer.
// live run
// benchmark
offline · fixed test set · not your runPre-computed RAGAS scores from replaying the pipeline over a fixed test set — independent of the query above; it won't change when you run one.
// answer quality by mode
abstained
answer_relevancy
context_precision
context_recall
faithfulness
Bars are means; the line across each is its 95% bootstrap interval. Overlapping lines mean no measurable difference, however tempting the longer bar looks.
// paired vs baseline
contextual − baseline
- abstained
- +0.00 [−0.07, +0.07]
- p 1.000 · within noise
- answer_relevancy
- −0.01 [−0.07, +0.02]
- p 0.893 · within noise
- context_precision
- +0.00 [−0.05, +0.07]
- p 0.917 · within noise
- context_recall
- −0.08 [−0.17, +0.01]
- p 0.128 · within noise
- faithfulness
- −0.04 [−0.09, +0.00]
- p 0.071 · within noise
raptor_sac − baseline
- abstained
- −0.07 [−0.16, +0.00]
- p 0.240 · within noise
- answer_relevancy
- +0.01 [−0.03, +0.05]
- p 0.813 · within noise
- context_precision
- +0.02 [−0.06, +0.10]
- p 0.666 · within noise
- context_recall
- −0.05 [−0.13, +0.03]
- p 0.280 · within noise
- faithfulness
- −0.00 [−0.04, +0.03]
- p 0.957 · within noise
baseline_agentic − baseline
- abstained
- +0.00 [−0.09, +0.09]
- p 1.000 · within noise
- answer_relevancy
- −0.02 [−0.05, +0.01]
- p 0.294 · within noise
- context_precision
- −0.21 [−0.30, −0.11]
- p 0.000 · significant
- context_recall
- −0.15 [−0.27, −0.05]
- p 0.007 · significant
- faithfulness
- −0.03 [−0.11, +0.03]
- p 0.444 · within noise
raptor_sac_agentic − baseline
- abstained
- −0.05 [−0.14, +0.05]
- p 0.620 · within noise
- answer_relevancy
- −0.02 [−0.08, +0.01]
- p 0.457 · within noise
- context_precision
- −0.17 [−0.27, −0.07]
- p 0.001 · significant
- context_recall
- −0.11 [−0.22, −0.01]
- p 0.054 · within noise
- faithfulness
- −0.01 [−0.06, +0.03]
- p 0.624 · within noise
graphrag − baseline
- abstained
- +0.19 [+0.05, +0.33]
- p 0.040 · significant
- answer_relevancy
- −0.15 [−0.28, −0.03]
- p 0.038 · significant
- context_precision
- −0.26 [−0.38, −0.13]
- p 0.001 · significant
- context_recall
- −0.55 [−0.69, −0.40]
- p 0.000 · significant
- faithfulness
- −0.06 [−0.16, +0.03]
- p 0.224 · within noise
graphrag_agentic − baseline
- abstained
- +0.21 [+0.07, +0.37]
- p 0.023 · significant
- answer_relevancy
- −0.05 [−0.16, +0.04]
- p 0.388 · within noise
- context_precision
- −0.26 [−0.39, −0.13]
- p 0.001 · significant
- context_recall
- −0.49 [−0.63, −0.34]
- p 0.000 · significant
- faithfulness
- −0.08 [−0.18, +0.01]
- p 0.125 · within noise
combined − baseline
- abstained
- +0.05 [−0.07, +0.19]
- p 0.725 · within noise
- answer_relevancy
- +0.00 [−0.04, +0.04]
- p 0.967 · within noise
- context_precision
- −0.14 [−0.24, −0.03]
- p 0.013 · significant
- context_recall
- −0.20 [−0.33, −0.08]
- p 0.004 · significant
- faithfulness
- +0.00 [−0.07, +0.06]
- p 0.998 · within noise
combined_agentic − baseline
- abstained
- −0.05 [−0.16, +0.07]
- p 0.690 · within noise
- answer_relevancy
- +0.02 [−0.02, +0.06]
- p 0.448 · within noise
- context_precision
- −0.14 [−0.26, −0.03]
- p 0.022 · significant
- context_recall
- −0.21 [−0.35, −0.09]
- p 0.004 · significant
- faithfulness
- −0.00 [−0.04, +0.04]
- p 0.869 · within noise
Per-item differences on the same test items, sign-flip randomization test. Bracketed: 95% interval on the mean difference.