Source-linked AI summary
AGO AI Quality Gate: Evidence-First Release Decisions for Retrieval-Augmented Generation
Giulio Zeloni, Enrico Lo Conte, Salvatore Rionero, Giuseppe Santoro, Alessandro Rastelli, Fabio Sorrentino
TL;DR
Enterprise RAG release decisions must be made with incomplete evidence and fallible LLM judges. AGO combines explicit evidence-aware decisions, statistical regression-risk estimates, and per-engagement judge validation. Its evaluations show that judge reliability varies and that the decision-grade profile lowers unsafe promotion across tested sample sizes.
Problem
RAG release decisions rely on incomplete evidence, uncertain point estimates, and judges whose reliability can vary by task and domain.
Source: §1 · p. 2Method
AGO combines four-state release decisions, a stratified beta-binomial gate, layered scoring, and mandatory per-engagement validation before judge scores influence decisions.
Source: §1 · p. 2, §3.3 · p. 5, §3.4 · p. 5Results
The decision-grade profile lowers unsafe promotion at every tested sample size, while judge reliability is model- and domain-dependent, with MCC 0.040–0.618 at a fixed threshold.
Source: §6 · p. 13Takeaways & Limitations
Evidence completeness, judge validity, posterior regression risk, and fixed critical precedence are first-class release criteria, not assumptions behind a score.
Source: §1 · p. 2, §3.3 · p. 5, §6 · p. 13The judge evaluation uses public RAGBench and the gate operating-characteristic study uses synthetic binary outcomes; non-promoted cases require human review.
Source: Ethical Considerations · p. 13
Abstract
from arXiv · showhide
Enterprises adopting retrieval-augmented generation (RAG) face a recurring operational decision: promote, revise, or block a system version. The evidence is incomplete and the metrics come from fallible LLM judges. We report on AGO AI Quality Gate (AGO), an evidence-first quality-gate framework deployed in industrial RAG assessment engagements. AGO integrates four key components: a four-state decision model that treats missing data and judge errors as explicit outcomes; layered scoring combining deterministic checks, local guardrails, and structured LLM evaluation; a stratified beta-binomial gate that quantifies regression risk probabilistically; and a mandatory meta-evaluation protocol to validate the LLM judge before it influences decisions. Since engagement data is proprietary, we evaluate the judge layer on RAGBench, a public benchmark of 100k annotated RAG traces across 12 datasets. On identical stratified test samples (N=1200 per judge), a low-cost judge (gpt-4.1-nano) detects non-adherent answers barely above chance (AUROC 0.603 [0.570, 0.634]), despite producing flawless protocol output, while gpt-4o reaches 0.783 [0.756, 0.807] -- yet its per-domain performance still ranges from 0.62 to 0.88. A fixed-seed gate study spanning regression, no change, and improvement quantifies unsafe promotion, false-alarm cost, and improvement throughput. Under regression, the decision-grade profile reduces unsafe promotion to 22.2%-35.1%, against 29.3%-41.8% for a naive gate. These results support the design choices that judge quality must be measured per engagement and that point estimates alone are not a release decision.
1 Introduction
Enterprise RAG releases require decisions despite incomplete evidence and fallible judges. AGO makes those uncertainties explicit in an auditable release-decision layer, alongside statistical gating and judge validation.
1 Introduction
- AGO is deployed in enterprise RAG assessment engagements where client data must remain on premises, observability is often incomplete, and evaluation sets contain tens to hundreds of stratified cases.Source: §1 · p. 2
- RAG scoring frameworks and benchmarks do not by themselves resolve accountable release decisions when evidence is missing, estimates are uncertain, and judge reliability varies by task and domain.Averaging available evidence can conflate “measured as bad” with “not measured.”Source: §1 · p. 2
- AGO combines four-state decision semantics, stratified beta-binomial regression-risk estimates, mandatory per-engagement judge validation, and public evaluations of judges and gate behavior.Its decision states are promote, manual review, block, and not evaluable.Source: §1 · p. 2
- AGO’s contribution is a decision layer that makes missing evidence, judge protocol errors, statistical uncertainty, and per-engagement judge validity explicit inputs rather than hidden assumptions behind scores.Source: §1 · p. 2
2 Related Work
Prior work measures RAG quality and evaluator reliability, but AGO focuses on converting scores into accountable release decisions under incomplete evidence.
RAG evaluation frameworks
- RAGAs and ARES score RAG quality with LLM judges, while AGO embeds evaluator validation within a decision pipeline that explicitly handles missing evidence.Existing toolkits generally provide scores or evaluator predictions rather than evidence-aware release semantics.Source: §2 · p. 3
Benchmarks for RAG and its evaluators
LLM-as-a-judge reliability
3 The AGO AI Quality Gate Framework
AGO turns RAG evaluation into an auditable release gate by making evidence gaps, decision rules, uncertainty, and judge validation explicit. Its framework combines layered scoring, conservative decision semantics, stratified statistical comparison, and per-engagement judge checks.
3.1 Setting and Evidence Model
- Each evaluation case can include expected behavior, retrieval requirements, and forbidden phrases; datasets also carry category and criticality metadata.Metric results distinguish observed, declared, inferred, and missing evidence, and missing evidence must surface with an impact statement.Source: §3.1 · pp. 3–4
3.2 Layered Scoring
- Scoring combines deterministic checks, local guardrails, and LLM-judge measures, with adherence rate reported as the fraction of evaluated cases passing.Malformed or out-of-range judge output is treated as a protocol error and routed to manual review; adherence is omitted for cases not requiring retrieval.Source: §3.2 · p. 4
3.3 Decision Semantics
- Missing evidence cannot produce a promotion, and critical safety findings block regardless of a metric’s configured decision mode.Non-promote outcomes carry machine-readable failure reasons and are routed to consultant review.Source: §3.3 · p. 5
3.4 The Stratified Hierarchical Beta-Binomial Gate
- The stratified gate aggregates per-category beta-binomial posteriors to compare baseline and candidate run quality, rather than relying on point pass-rates.Evaluation-design coverage weights are the default; explicit target weights may be used for production-facing estimates.Source: §3.4 · pp. 5–6
3.5 Per-Engagement Judge Meta-Evaluation
- Before judge metrics can influence decisions in a new domain, AGO requires a human-labeled golden set and reports agreement, discrimination, and bootstrap uncertainty.Judge-dependent metrics may run in block mode only when validation meets profile criteria; changing the judge configuration invalidates the report.Source: §3.5 · pp. 6–7
4 Deployment
AGO is deployed as a self-hosted platform for enterprise RAG assessment, with data handling and release decisions kept within an auditable engagement workflow.
4 Deployment
- Client data stays within the client or consultant perimeter, with the judge endpoint as the only outbound call and a self-hosted option for deployments forbidding egress.Traces are normalized, PII and secrets are redacted, and blocking dataset-curation findings prevent a gate run.Source: §4 · p. 7
- The repeatable engagement workflow imports conversations, curates a versioned dataset and profile, validates judges, runs the gate, and delivers a report.Every decision records failure reasons, evidence types, per-stratum uncertainty, and gaps.Source: §4 · p. 7
- Content-hashed dataset and profile snapshots preserve an auditable record as either evolves, while consultants review non-promoted cases and make the final decision.The gate escalates; it does not decide autonomously.Source: §4 · p. 7
5 Evaluation
The evaluation tests the judge layer on RAGBench and the statistical gate in controlled simulations; it does not validate AGO’s full release gate end to end. Results show judge performance varies by model and domain, while gate policies trade regression protection against operational costs.
5.1 Judge-Layer Validation on RAGBench
- RAGBench supplies 12 datasets across five industry domains, and adherence is compared with reference labels using AUROC and MCC/F1 at θ = 0.8.Holistic context relevance is compared with TRACe relevance by rank correlation because their definitions differ; answer relevance is excluded.Source: Benchmark and Metric Mapping · p. 8
- The frozen protocol scores identical stratified samples of 100 cases per dataset; gpt-4o was selected using validation-split MCC at the deployed threshold.The full test run covers all 12 datasets, with N=1200 per judge and 95% bootstrap confidence intervals; protocol errors are reported separately and none occurred.Source: Protocol · p. 8, Results · p. 9, Table 1 · p. 9
- In Table 2’s identical test instances (N=1200 per judge), gpt-4o exceeds the low-cost judge by +0.180 AUROC and +0.263 MCC at the deployed threshold; the low-cost judge’s AUROC is 0.603.The confidence intervals do not overlap; across a 0.1–0.9 threshold sweep, MCC never exceeds 0.17, despite zero protocol errors.Source: Table 2 · p. 9, Results · p. 9
- Published RAGBench baselines are for orientation, not protocol-identical comparison; gpt-4o’s per-dataset mean AUROC is 0.77 and micro AUROC is 0.783.The published results use full test splits and earlier GPT-3.5-backed evaluators, whereas AGO uses fixed stratified samples under a frozen protocol.Source: Results · p. 10
- gpt-4o’s AUROC ranges from 0.620 to 0.883 across datasets, while its thresholded MCC ranges from 0.040 to 0.618.The authors treat relevance rank correlations as exploratory because they are low and unstable across datasets.Source: Results · p. 9, Table 3 · p. 11
5.2 Operating Characteristics of the Statistical Gate
- The gate simulation tests regression, no change, and improvement across N = 20–200, comparing a naive rule with two AGO profiles on shared observations.Each scenario and sample size has 1,000 replicates; the profiles differ in their review threshold.Source: §5.2 · p. 11
- Under regression, decision_grade promotes 22.2%–35.1% of candidates versus 29.3%–41.8% for the naive gate.Decision_grade promotes 51.4%–76.2% under no change and 71.5%–99.4% under improvement, compared with 76.7%–99.8% for the naive rule under improvement.Source: Table 4 · p. 12, §5.2 · p. 12
- The balanced profile promotes 29.8%–43.9% of regressed candidates, which can match or exceed the naive rate despite using the same simulated evidence.Its promotion rates are 60.6%–83.0% under no change and 77.4%–99.8% under improvement.Source: §5.2 · p. 12
6 Conclusion
AGO turns RAG evaluation from score production into an auditable release decision; its results show judge reliability must be measured under a frozen meta-evaluation protocol, not inferred from well-formed output.
6 Conclusion
- Across three scenarios, the decision-grade profile lowers unsafe promotion at every tested sample size, with false-alarm and throughput costs.The study reports the release policy’s operating characteristics across regression, no change, and improvement.Source: §6 · p. 13
- The low-cost judge produced flawless, plausibly explained output while discriminating barely above chance; labelled references exposed the failure.The N=300 pilot’s AUROC 0.563 [0.495, 0.631] could not be distinguished from chance, and only the full run supported a verdict.Source: §6 · p. 13
- Judge reliability varied by model and domain (MCC 0.040–0.618 at a fixed threshold), supporting per-engagement threshold calibration rather than hard-coded thresholds.The passage identifies confidence intervals and full-run evaluation as necessary to support a verdict.Source: §6 · p. 13
- Distinguishing missing evidence from measured-bad results made the operational failures visible.The conclusion argues that missing evidence must be a first-class outcome.Source: §6 · p. 13
- The authors identify evidence completeness, judge validity, posterior regression risk, and fixed critical precedence as first-class release criteria.Future work includes paired Bayesian comparison, native evaluation of remaining TRACe dimensions, and repeated-sampling analysis of per-case variance.Source: §6 · p. 13
- The judge-layer study uses public RAGBench, the gate study uses synthetic binary outcomes, and non-promoted cases require human review.Neither gate-level experiment uses personal, client, or proprietary data or calls an external model service.Source: Ethical Considerations · p. 13