~ $ cd #content

~ $ cat demos/rag/README

retrieval-augmented generation

Hybrid retrieval over Azure docs with a RAGAS faithfulness guardrail that retries on weak grounding — ask it something and watch the pipeline work.

back to demos

// interactive

● live · your query

// what's happening

Your question fans out to dense (vector) and bm25 (keyword) search, fused by RRF, reranked by Azure's semantic ranker; a gpt-4o agent answers using only those sources. RAGAS scores faithfulness — weak answers trigger a retry. Run one and the diagram follows along.

// pipeline

The query fans out to two retrievers in parallel — dense vector search and bm25 keyword search. Their results are fused by reciprocal rank fusion, reranked, and passed to the generator. A faithfulness score gates the answer: below 0.7 the pipeline retries, otherwise it returns the answer.

// live run

~/demos/rag $

// benchmark

offline · fixed test set · not your run

Pre-computed RAGAS scores from replaying the pipeline over a fixed test set — independent of the query above; it won't change when you run one.

// answer quality by mode

abstained

answer_relevancy

context_precision

context_recall

faithfulness

Bars are means; the line across each is its 95% bootstrap interval. Overlapping lines mean no measurable difference, however tempting the longer bar looks.

// paired vs baseline

contextual − baseline

abstained
+0.00 [−0.07, +0.07]
p 1.000 · within noise
answer_relevancy
−0.01 [−0.07, +0.02]
p 0.893 · within noise
context_precision
+0.00 [−0.05, +0.07]
p 0.917 · within noise
context_recall
−0.08 [−0.17, +0.01]
p 0.128 · within noise
faithfulness
−0.04 [−0.09, +0.00]
p 0.071 · within noise

raptor_sac − baseline

abstained
−0.07 [−0.16, +0.00]
p 0.240 · within noise
answer_relevancy
+0.01 [−0.03, +0.05]
p 0.813 · within noise
context_precision
+0.02 [−0.06, +0.10]
p 0.666 · within noise
context_recall
−0.05 [−0.13, +0.03]
p 0.280 · within noise
faithfulness
−0.00 [−0.04, +0.03]
p 0.957 · within noise

baseline_agentic − baseline

abstained
+0.00 [−0.09, +0.09]
p 1.000 · within noise
answer_relevancy
−0.02 [−0.05, +0.01]
p 0.294 · within noise
context_precision
−0.21 [−0.30, −0.11]
p 0.000 · significant
context_recall
−0.15 [−0.27, −0.05]
p 0.007 · significant
faithfulness
−0.03 [−0.11, +0.03]
p 0.444 · within noise

raptor_sac_agentic − baseline

abstained
−0.05 [−0.14, +0.05]
p 0.620 · within noise
answer_relevancy
−0.02 [−0.08, +0.01]
p 0.457 · within noise
context_precision
−0.17 [−0.27, −0.07]
p 0.001 · significant
context_recall
−0.11 [−0.22, −0.01]
p 0.054 · within noise
faithfulness
−0.01 [−0.06, +0.03]
p 0.624 · within noise

graphrag − baseline

abstained
+0.19 [+0.05, +0.33]
p 0.040 · significant
answer_relevancy
−0.15 [−0.28, −0.03]
p 0.038 · significant
context_precision
−0.26 [−0.38, −0.13]
p 0.001 · significant
context_recall
−0.55 [−0.69, −0.40]
p 0.000 · significant
faithfulness
−0.06 [−0.16, +0.03]
p 0.224 · within noise

graphrag_agentic − baseline

abstained
+0.21 [+0.07, +0.37]
p 0.023 · significant
answer_relevancy
−0.05 [−0.16, +0.04]
p 0.388 · within noise
context_precision
−0.26 [−0.39, −0.13]
p 0.001 · significant
context_recall
−0.49 [−0.63, −0.34]
p 0.000 · significant
faithfulness
−0.08 [−0.18, +0.01]
p 0.125 · within noise

combined − baseline

abstained
+0.05 [−0.07, +0.19]
p 0.725 · within noise
answer_relevancy
+0.00 [−0.04, +0.04]
p 0.967 · within noise
context_precision
−0.14 [−0.24, −0.03]
p 0.013 · significant
context_recall
−0.20 [−0.33, −0.08]
p 0.004 · significant
faithfulness
+0.00 [−0.07, +0.06]
p 0.998 · within noise

combined_agentic − baseline

abstained
−0.05 [−0.16, +0.07]
p 0.690 · within noise
answer_relevancy
+0.02 [−0.02, +0.06]
p 0.448 · within noise
context_precision
−0.14 [−0.26, −0.03]
p 0.022 · significant
context_recall
−0.21 [−0.35, −0.09]
p 0.004 · significant
faithfulness
−0.00 [−0.04, +0.04]
p 0.869 · within noise

Per-item differences on the same test items, sign-flip randomization test. Bracketed: 95% interval on the mean difference.