Incremental Retrieval-Augmented Generation for Human-in-the-Loop Decision Support

• Standard RAG assumes a static, precompiled Knowledge Base, unlike domains where human deliberations incrementally populate an initially empty one.Source: §1 · p. 1
• iRAG stores prior human decisions as semantically retrievable precedents and progresses from observation to evidence-grounded contestation, then conditional autonomy.Source: §1 · p. 2
• The paper formalises incremental deliberative RAG and specifies behavior from silent observation through Skeptical Learning-based contestation to Learning-to-Defer-inspired autonomy.Source: §1 · p. 2
• The evaluation compares iRAG against an independent Controller-free RAG-with-defer rollout.Source: §1 · p. 2
• The claims are system-level: evaluation uses controlled decision-maker and acceptance policies without real human involvement.Source: §1 · p. 2

• Hybrid Decision-Making combines human and model decisions, with machine-driven systems deferring to people and human-driven systems retaining the human’s primary role [20].Source: §2 · p. 3
• Learning-to-Defer routes ambiguous cases to human supervisors, but the machine decides when to consult them, leaving the human largely passive.Source: §2 · p. 3
• Skeptical Learning keeps humans primary, but early-decision bias and concept drift make stability–plasticity a concern.Source: §2 · p. 3
• Retrieval-Augmented Generation retrieves passages from an external Knowledge Base at inference time, but its original formulation keeps the index fixed.Source: §2 · p. 3
• Unlike approaches that ingest external document streams, iRAG’s Knowledge Base grows from structured records of human decisions made over time.Source: §2 · p. 3

3 System Overview

• Each interaction adds its input and final decision to the knowledge base, and iRAG retrieves top-K records using similarity and insertion-based temporal decay.Only records with positive cosine similarity meeting θsem are ranked; decay makes older records harder, but not impossible, to retrieve.Source: §3 · pp. 4–5
• FEA measures agreement with final human decisions rather than objective accuracy, so using it to authorize autonomy assumes those decisions are sufficiently reliable.FEA fades across reliability observations; its count can differ from the number of stored records.Source: §3 · pp. 4–5
• The system begins in Silent Observer, where human decisions remain final and model outputs are hidden, then enters Skeptical Contestator after FEA and observation-count thresholds are met.In contestation, a non-abstaining model disagreement is presented as an alternative; the human retains the final decision and FEA is updated against it.Source: §3 · p. 5
• In Deferring Surrogate, the model decides unless it abstains, but selective feedback can leave FEA stale and periodic review is required if it continues authorizing autonomy.A human override flagged as incorrect records a negative observation and can return the system to Silent Observer.Source: §3 · p. 6

3.1 Comparison with HDM systems

• Unlike HDM approaches that retrain or incrementally update a predictive model, iRAG keeps LLM parameters fixed and appends decisions to an external knowledge base.Retrieved records inform later decisions without additional model training.Source: §3.1 · p. 6
• Because decisions are natural-language answers, iRAG relies on an auxiliary model to assess answer coverage, making its reliability estimate dependent on that assessment's quality.Abstention separately allows deferral when retrieved evidence is insufficient.Source: §3.1 · p. 6
• The comparison with incremental-learning HDM systems concerns what is updated and how past observations remain available.Both approaches adapt sequentially to newly observed decisions.Source: §3.1 · p. 6, Table 1 · p. 6

4 Evaluation

• The evaluation used a synthetic 2,000-ticket CRM benchmark with 150 or 600 changed-answer tickets, simulated human profiles, and an independently rolled-out Controller-free RAG-with-defer baseline.The baseline used its own decisions to build its non-decaying KB and had no state controller or FEA.Source: §4 · pp. 7–8

4.3 Overall Contribution of iRAG

• At 40% drift, iRAG’s final-decision error is 9.51–10.52%, 3.52–4.53 points below Controller-free RAG-with-defer across all configurations.The gain coincides with lower stable- and drift-ticket error, while iRAG finalizes fewer tickets than the baseline.Source: Table 2 · p. 10, §4.3 · p. 12

4.1 Contribution of RAG Decay

• Against R:ND, retrieval decay lowers drift-ticket error by 7.87–8.87 points at 10% drift and 2.63–3.78 at 40%, while increasing stable-ticket error by 2.31–2.62 and 1.29–1.85 points, respectively.Down-weighting older precedents reduces interference from superseded decisions but weakens still-valid evidence when knowledge has not changed.Source: §4.1 · p. 10

4.2 Contribution of FEA Decay

• FEA decay raises the model-finalized proportion by 5.24–6.89 points under 10% drift and 5.98–8.38 under 40%, while increasing error under 40% drift.Its primary effect is the timing and extent of delegation; final-state counts are threshold-sensitive and do not show how long runs spent under autonomous authority.Source: §4.2 · p. 11

4.3 Overall Contribution of iRAG

• At 10% drift, the two R:ND settings score 5.52–5.75% versus the baseline’s 5.50±0.62%, with paired confidence intervals crossing zero.With retrieval decay, aggregate error rises to 6.99–7.59%.Source: Table 2 · p. 10, §4.3 · p. 11

5 Conclusion

• iRAG builds its knowledge base from decisions supported by the system, progressing from human-decision observation to contestation and conditional autonomy.Source: §5 · p. 12
• Under stronger drift, iRAG was clearly more accurate than independently rolled-out Controller-free RAG-with-defer, while matching it under mild drift without retrieval decay.Source: §5 · p. 12
• Retrieval decay consistently improves changed-knowledge accuracy while sacrificing some stable-knowledge accuracy, whereas FEA decay chiefly increases delegated authority.Source: §5 · p. 12
• The design inherits reliance on human decisions, selective feedback during autonomy, and potentially arbitrary user-specified transition thresholds from its HDM paradigms.Source: §5 · p. 12
• The experiments assign a substantial share of tickets to an oracle decision-maker, leaving fallible-bootstrap and narrower-review settings for future study.Source: §5 · p. 13
• The controlled evaluation is not a human-participant study; planned work will examine how users understand states, inspect evidence, contest decisions, and delegate control.Source: §5 · p. 13