PaperlayerView exact revision on arXiv

Source-linked AI summary

MuninAI: A Self-Hosted Agentic Framework for Independent Research Groups

Markus Fischer, Jens Meiler, Matthias Elgeti

arXiv:2610.23480v1cs.CL

TL;DR

Research groups must choose between hosted services that receive unpublished work and self-hosted deployments that lack access to their literature corpus. Munin provides a self-hosted agent harness and evaluation suite; the harness approximately doubles base-model accuracy, and both Qwen models exceed the published PaperQA2 result with outbound access disabled.

  • Problem

    Research groups need answers grounded in private literature that hosted services may receive as input, while self-hosted chat frontends do not retrieve, read, or check answers against local documents.

    Source: §1 · p. 1, §7 · p. 31

  • Method

    Munin indexes a group’s corpus and citation graph, wraps a configurable open-weight model in an agent harness, and ships a rerunnable evaluation suite.

    Source: §1 · p. 2

  • Results

    The harness approximately doubles base-model accuracy and answer-claim traceability; offline, both Qwen models exceed PaperQA2’s 0.660, but gpt-oss-20b does not.

    Source: §7 · p. 31

  • Takeaways & Limitations

    The harness improves results across the two Qwen models and gpt-oss-20b, including a 20-billion-parameter model running on a single consumer-grade GPU.

    Source: §7 · p. 31

    Because gpt-oss-20b differs from Qwen in developer, architecture, quantisation, and sampling configuration, the smaller harness effect cannot be attributed to any single cause.

    Source: §6.2 · p. 30

Abstract

from arXiv · show

Research groups that want a language model grounded in their own literature face a choice between hosted services that receive their unpublished work and self-hosted deployments that keep the data local but give the model no access to the corpus. Munin is an open-source framework for a self-hosted scientific assistant that indexes a group's corpus and its citation graph, and wraps whichever open-weight model the group can serve in an agent harness with specialised reading, search, and computation tools. We claim that the harness, rather than the specific model, is what makes the system useful on domain literature, and we ship the evaluation suite that tests this claim with the software. On 199 paired literature questions, the harness more than doubles the accuracy of the base model, from 0.387 to 0.874 (+0.487, 95% CI [0.407, 0.568]), while naive retrieve-then-answer scores below the unaided model on every backbone tested. Under the harness, roughly twice as many answer claims trace to a passage the model read itself as with fixed top-5 retrieval. When the source paper for a question is removed from the corpus, the system abstains on about nine in ten of the questions it had answered correctly on the Qwen backbones and eight in ten on gpt-oss-20b, and it produces no fabricated local citation across 300 fabricated references. The pattern holds on three backbones from two developers, including a 20-billion-parameter model that runs on a single consumer-grade GPU. With all outbound network access disabled, both Qwen backbones still exceed the published PaperQA2 accuracy. The evaluation suite runs as a single command, so an adopting group can re-run it on its own corpus and model.

1 Introduction

Research groups need to retrieve details from growing private literature collections that public services cannot access, while self-hosted chat frontends keep data local but lack corpus-grounded retrieval. Munin pairs a self-hosted, model-agnostic assistant with a re-runnable evaluation suite for measuring its performance on a group’s own corpus.

1 Introduction

  • The private-literature problem includes locating specific values, figures, and methods across growing collections, a task that often depends on scientists’ memory and manual extraction.Those collections can include licensed articles, unpublished drafts, instrument-specific datasets, and internal reports outside public indexes.Source: §1 · pp. 1–2
  • Munin combines a self-hosted agentic assistant with an evaluation suite groups can rerun on their own corpus and after component changes.The framework supports open-weight models exposed through an OpenAI-compatible API and scales down to a single GPU.Source: §1 · p. 2
  • The backend combines an agent loop, knowledge base, and inference client on institutional infrastructure, while the frontend handles authentication, uploads, and chat.The halves can be deployed independently, and the model is configurable rather than a hard dependency.Source: §1 · p. 2
  • The evaluation suite compares the full system with bare-model and naive top-k retrieval baselines to isolate the iterative tool-use loop’s contribution.Groups can rerun the evaluation after changing the model, encoder, corpus, or harness.Source: §1 · p. 2
  • The stated contributions include a 45-tool harness, a vector-and-graph knowledge base, a corpus host with no inbound traffic, per-question results, and DOI-grounded tests of abstention and confabulation.The evaluation also treats outbound egress level as an experimental variable.Source: §1 · p. 2

2 Related Work

Prior systems address literature search, abstention, or citation fidelity, but differ in corpus access, task scope, and evaluation. Munin combines corpus-grounded abstention and confabulation testing with a self-hosted agent framework, while treating outbound access and threshold certification as explicit design and evaluation choices.

2.1 Agentic systems for scientific literature

  • Hosted scientific QA systems use large public corpora, while local chat frontends generally lack citation graphs, full-text reading, not-in-corpus routing, and rerunnable evaluation.PaperQA2 [1] reports LitQA2 accuracy 0.660 ± 0.012; LitSearch [4] supplies 597 queries over 64,183 papers, averaging 1.07 relevant papers per query.Source: §2.1 · pp. 2–4
  • AstaBench [22] reports that specialised agents can outperform generic ReAct and that stronger models do not uniformly improve specialised-agent results; PaperArena [23] tests harder cross-paper synthesis, while AutoResearchBench [24] evaluates paper discovery.Munin adopts cost-aware reporting of wall-clock time and tool calls alongside accuracy, and its single-source questions differ from PaperArena’s cross-paper task.Source: §2.1 · p. 3

2.2 Abstention and selective prediction

  • Prior abstention work tests flawed or unanswerable questions, whereas Munin withholds real source papers even though an answer exists; its agentic loop retrieves iteratively and requires a supporting sentence before answering.KnowOrNot [9] also evaluates out-of-knowledge-base abstention; Sufficient Context [10] uses a post-hoc sufficiency classifier, while RefusalBench [11] damages document content and AbstentionBench [12] collects unanswerable questions.Source: §2.2 · p. 4

2.3 Citation fabrication

  • Prior citation-fidelity work examines fabricated references or attribution, while Munin pairs citation confabulation testing with corpus-withholding abstention for a paper-search agent.The cited literature includes existence and attribution checks [17] [18] [19] and DOI field accuracy [20].Source: §2.3 · pp. 4–5

2.4 Faithfulness judging

  • Munin uses an on-premise MiniCheck-Flan-T5-Large judge [7] to assess claims against retrieved contexts and validates it on RAGTruth [8] before experimental use.A hosted frontier-model judge was rejected to avoid sending evaluation queries and answers off-premise.Source: §2.4 · p. 5

2.5 What is new

  • Munin pairs real-paper withholding with tests on nonexistent papers, checking emitted DOIs against corpus membership without a judge.The authors describe this pairing as their novel contribution for a paper-search agent.Source: §2.5 · p. 5
  • Munin varies outbound access between runs; its deployment keeps inference, agents, and the corpus on cluster machines reached through a cluster-initiated reverse SSH tunnel with no inbound port exposed.The authors report that egress settings are worth between 0.17 and 0.21 accuracy on the same questions.Source: §2.5 · p. 5, Figure 1 · p. 6
  • The shipped threshold-based certification implementation covers one model, harness, and corpus, but its thresholds are provisional and unvalidated.The authors note that whether such thresholds predict field behaviour remains an open question [36] [37].Source: §2.5 · pp. 5–6

3 The Munin Framework

This section describes Munin’s locally deployable research framework, including its backend, profiled agent harness, reading, search, computation, and synthesis tools, and knowledge base. It also covers evidence traceability and operational limits, corpus quality as part of measurement, five settings for adaptation, and which components remain within the deploying institution or may be external.

3.1 Deployment topology

  • The backend keeps the harness and knowledge base within the institution while a decoupled frontend provides authentication, uploads, and chat or can be replaced by an OpenAI-compatible API client.The frontend may be hosted externally; the backend initiates a reverse tunnel so the corpus machine need not accept inbound connections.Source: §3.1 · p. 6

3.2 The agent harness

  • The architecture selects a profile before the first model call, using explicit slash commands or a distance-weighted vote over 244 labelled queries with a pinned-profile prior.The vote is accepted only when the winning profile leads the runner-up by at least 0.10 of total vote mass and nearest-neighbour cosine similarity is at least 0.45; otherwise, the pinned profile applies, or chat if none is set. The 18-case anchor evaluation’s trajectory pass rate exceeded the 0.95 threshold.Source: §3.2.1 · p. 7

3.2.2 source: full-text reading

  • The source agent resolves references, reads full text in one model call when it fits, and returns an answer supported by an exact sentence or abstains.Multiple outcome flags distinguish an informative negative from addressable system failures; accepting title references reduces pressure to fabricate DOIs, but a plausible DOI can still resolve to the wrong paper without detection.Source: §3.2.2 · pp. 7–8

3.2.3 search: tiered evidence retrieval

  • Search combines local, scholarly, and web evidence according to egress settings, ranks sources in tier-specific slots, and reports scope coverage and sparse evidence rather than padding weak results.External results must clear a 0.62 relevance floor calibrated for the BGE encoder; deployments using another encoder or domain must recalibrate it.Source: §3.2.3 · pp. 8–9

3.2.4 compute: sandboxed code generation

  • The compute agent applies language-specific checks and labels outputs as executed, compiled, or linted, while running Python in a sandbox active from the first import.Execution verifies that code runs, not that its result is correct; returning the rendered figure alongside code helps users inspect the result.Source: §3.2.4 · pp. 9–10

3.2.5 deep_research: planned multi-source synthesis

  • Deep research uses an explicit plan of sub-questions, screens retrieved candidates for recall, and records structured claim-value-evidence notes so conflicting findings can be flagged.It follows citations only to depth one, a configurable limit chosen to bound search space and per-run cost; sections retrieve notes by sub-question ID rather than similarity.Source: §3.2.5 · pp. 10–11

3.2.6 Guardrails and audits

  • Guardrails make some failures structurally impossible and detect others before delivery, while traces record sources, decisions, model calls, and abstention or failure reasons.Code-level tool-call caps and identifier resolution constrain actions; phantom-URL, phantom-paper, and citation-claim audits flag suspect outputs.Source: §3.2 · p. 7, §3.2.6 · p. 11

3.3 The knowledge base

  • The knowledge base grows from contributor uploads through a citation-graph crawler, while ingestion checks registry metadata against parsed titles and rejects mismatches.The reference deployment contains 3,922 direct-upload records and 68,500 total after citation crawling; scheduled checks quarantine flagged records, and an author-metadata repair raised clean records from 80.3% to 98.7%.Source: §3.3 · p. 12, §3.3.1 · p. 12, §3.3.2 · p. 12

3.3.3 Corpus quality is a measurement concern

  • Apparent model failures in private-corpus retrieval can also be corpus failures, so distinguishing them requires correct ingestion metadata and continuous audits.The framework therefore ships corpus tooling rather than assuming the corpus is clean.Source: §3.3.3 · p. 13

3.4 Configuration surface

  • Porting the framework to a different corpus or model requires deliberate attention to five settings.Table 3.3 identifies the settings requiring attention.Source: §3.4 · p. 13, Table 3.3 · p. 13

3.5 Deployment boundary

  • Inference, the corpus, vector store, citation graph, code sandbox, and faithfulness judge run inside the deploying institution; external placement of two components is a deployment choice.The reference frontend runs on rented infrastructure, but backend-only deployment avoids that third-party host; enabling web search sends query text to a commercial search API.Source: §3.5 · p. 13

4 Experimental Setup

The evaluation uses paired comparisons across three model backbones and validates both retrieval and measurement procedures. It also defines on-premise faithfulness judging and two corpus-grounded abstention tests.

4.2 Backbones, arms and protocol

  • The three backbones include two Qwen models and gpt-oss-20b; the latter changes developer, architecture family, and quantisation together, so its smaller harness effect cannot be assigned to one factor.The two Qwen models use the same context window and concurrency; gpt-oss-20b is served natively in MXFP4 on a single GPU.Source: §4.2 · p. 14
  • 199 source-present LitQA2 questions [1] compare bare-model, top-5 naive-RAG, and agentic-harness arms, with the harness tested both with and without outbound access.Questions whose source paper is absent are reserved for abstention tests; runs use concurrency 1, a 16,384-token output budget, and a 900-second deadline.Source: §4.2 · p. 14

4.3 Retrieval datasets

  • The retrieval gate found SPECTER below the 0.5 nDCG@10 threshold, while BM25 scored 0.652 versus the published 0.665, supporting the pipeline’s expected ranking behavior.SPECTER scored 0.479 as deployed and 0.494 with canonical [SEP] formatting; the 0.013 BM25 difference is described as consistent with tokenisation and implementation variation.Source: §4.3 · p. 15

4.4 Metrics and statistics

  • Confidence intervals use query-level paired bootstrap resampling, and comparisons also report paired Wilcoxon tests; Wilson intervals supplement estimates for small binary-outcome subsets.The bootstrap uses 1,000 resamples, or 2,000 for abstention tests, with a fixed seed.Source: §4.4 · p. 16

4.5 Faithfulness judging

  • Faithfulness is scored on-premise with MiniCheck-Flan-T5-Large [7], whose question-answering validation on RAGTruth [8] reached AUROC 0.950.The judge assesses claims against passages retrieved for each turn; validation on the other two RAGTruth domains was lower, at 0.708 and 0.723.Source: §4.5 · p. 16

4.6 Abstention test sets

  • C1 treats any DOI returned for a nonexistent paper as fabricated, while C2b tests abstention after removing 49 source papers from a shadow corpus with outbound requests disabled.Both retrieval indexes are shadowed; abstentions and unparseable responses count as unanswered.Source: §4.6 · p. 16, Table 4.6 · p. 17

5 Results

The results evaluate the tool-use harness across backbones for literature-question answering, grounding, and abstention, alongside fixed-retrieval and published-baseline comparisons. They also examine retrieval performance, recovery from tool failures, and costs across backbones and serving setups.

5.1 Does the harness improve the base model?

  • 0.874 of 199 questions were answered correctly by the Qwen3.8-27B harness, versus 0.387 by the bare model, a paired gain of +0.487 (95% CI [0.407, 0.568]).The harness enables the model to retrieve papers, reread passages, and cross-check values before answering.Source: Table 5.1 · p. 18, §5.1 · p. 18
  • +0.367 and +0.276 of the harness gains came from the corpus-only loop on Qwen3.6 and Qwen3.8, while external tiers added +0.166 and +0.211, respectively.On gpt-oss-20b, the corpus-only gain was +0.075 (p = 0.10), while external tiers added +0.080.Source: Table 5.2 · p. 19, §5.1 · p. 19

5.1.1 Naive retrieval is worse than no retrieval

  • -0.211, -0.176, and -0.306 were the paired accuracy differences for naive RAG versus the bare model on Qwen3.6, Qwen3.8, and gpt-oss-20b, respectively.The Qwen models abstained more under fixed-context retrieval; gpt-oss-20b ended 162 of 199 turns without a final answer.Source: §5.1.1 · p. 20

5.2 Does the harness improve grounding?

  • 0.627 and 0.540 of Qwen3.6 and Qwen3.8 harness claims were supported by retrieved passages, versus 0.290 and 0.282 under RAG.The share of answers with no supported claim fell from 38% to 3% on Qwen3.6 and from 27% to 7% on Qwen3.8; gpt-oss-20b’s +0.253 grounding change was not significant (p = 0.056) on 21 matched questions.Source: Table 5.3 · p. 20, Figure 3 · p. 21, §5.2 · p. 22

5.3.1 Fabricated papers (C1)

  • 0.98 and 1.00 were the fabricated-paper refusal rates for the two Qwen backbones, and all three backbones produced zero fabricated local citations.The gpt-oss-20b refusal rate of 0.720 is a lower bound because 20 of 24 undecided items were empty turn-ends.Source: Table 5.4 · p. 23, §5.3.1 · p. 23

5.3.2 Paired shadow corpus (C2b)

  • 0.875, 0.889, and 0.800 of previously correctly answered questions led to explicit refusals when their source paper was removed on Qwen3.6, Qwen3.8, and gpt-oss-20b.Accuracy on the absent-source arm fell to 0.040 across all three backbones, and abstention rose to 0.80–0.92.Source: Table 5.6 · p. 23, §5.3.2 · pp. 23–24

5.3.3 Risk-coverage operating points

  • 0.070 and 0.054 were the harness error rates among attempted answers on source-present questions for Qwen3.6 and Qwen3.8, versus 0.54–0.60 for bare-model attempts.For gpt-oss-20b, attempted-answer error fell from 0.518 bare to 0.104 under the harness, while coverage fell from 0.844 to 0.628.Source: Table 5.7 · p. 24, §5.3.3 · p. 24

5.4 Does the harness hold across backbones?

  • 0.869 and 0.874 were the harness accuracies for Qwen3.6 and Qwen3.8, while gpt-oss-20b reached 0.563 with a smaller gain and a different contribution from external retrieval.The Qwen backbones converged in abstention, precision, and correct refusal when the source was withheld; gpt-oss-20b’s corpus-only gain was +0.075 (p = 0.10).Source: Table 5.8 · p. 25, §5.4 · p. 25

5.5 Retrieval on external anchors

  • BGE-dense scores 0.485 nDCG@10 versus 0.378 for BM25 on LitSearch, while citation-graph reranking costs 0.016 nDCG@10 and RRF adds no measurable gain.BGE-dense also raises R@10 from 0.511 to 0.637 and R@100 from 0.699 to 0.829; RRF scores 0.490, statistically indistinguishable from BGE alone. These queries cover machine learning and NLP, so the observed retriever ordering may not generalize to the production domains.Source: §5.5 · pp. 26–27

5.6 Reliability and cost

  • Every query encountering a failed or unusable tool call still produced a final answer across all three backbones and corpus-only arms, with unrestricted-arm recovery rates of 1.000.When outbound requests were blocked, recovery was 1.000 on both Qwen backbones and 0.965 on gpt-oss-20b, where two of 57 queries with refused calls ended without a final answer; blocked calls were intercepted before leaving the machine.Source: §5.6 · p. 27
  • gpt-oss-20b keeps 85% of tool calls within local corpus search and uses web search on 17 of 199 questions, while Qwen3.8 and Qwen3.6 issue 331 and 362 web searches, respectively.The external tiers contribute +0.080 to accuracy on gpt-oss, +0.166 on Qwen3.8, and +0.211 on Qwen3.6.Source: §5.6 · p. 28
  • Under injected retrieval-tier faults, the harness raises calls per query from 8.6 to 13.9–16.1, keeps grounding within 0.04 of fault-free performance, and still answers every query.On Qwen3.8, more than half of the total harness improvement remains with outbound requests blocked, indicating that substantial supporting evidence comes from the local corpus.Source: §5.6 · p. 28
  • The agentic loop takes approximately four times as long as a bare call on Qwen3.6, twelve times as long on gpt-oss-20b, and twenty times as long on dense Qwen3.8.Costs across backbones are not directly comparable because Qwen3.8 ran on a shared production service while the other arms used dedicated hardware; the loop averages seven to nine tool calls per query.Source: §5.6 · p. 28

5.7 Comparison with published baselines

  • Both Qwen backbones exceed published PaperQA2 accuracy by approximately 0.22 and the human-expert baseline by approximately 0.20, with higher precision than either.With outbound requests disabled, the corpus-only harness scores 0.704 on Qwen3.6 and 0.663 on Qwen3.8, at precision above 0.91; gpt-oss-20b scores 0.528, trailing PaperQA2 on accuracy but matching its precision.Source: Table 5.12 · p. 28, §5.7 · p. 29
  • The published-baseline comparison covers 199 questions with source papers in the local corpus, and run-to-run variability of approximately ±0.035 means the two Qwen results should be treated as one value.PaperQA2 was trained on the LitQA2 training and evaluation splits, unlike the models reported here.Source: §5.7 · pp. 28–29

6 Discussion

Munin’s model-agnosticism requires empirical re-certification because backbone changes can affect tool handling and answer scoring. The reported results also have limits in benchmark scope, deployment privacy, and measurement.

6.1 Model-agnosticism and re-certification

  • Model-agnosticism must be tested for each model; cross-family swaps can require tool-contract fixes, and output parsers, scorers, and routers need inspection before trusting results.A model upgrade can shift operating points, so the authors recommend re-running the evaluation suite; development checks also surfaced system and evaluation-harness defects.Source: §6.1 · p. 29

6.2 Limitations

  • The reported harness effect spans three models from two developers, but does not establish future-model generality, and the gpt-oss-20b gain is roughly one third of the Qwen figure.The harness is not entirely model-agnostic: family-specific changes addressed Qwen clarification handling and routing, plus gpt-oss tool-name parsing and answer scoring.Source: §6.2 · p. 30
  • The answering evaluation uses 199 in-corpus, multiple-choice LitQA2 questions rather than the full benchmark, and many are answerable from model knowledge alone.PaperQA2 was trained on LitQA2’s training and evaluation splits, unlike the models reported here; its adjacent human-expert figure averages nine experts.Source: §6.2 · p. 30
  • Repeated 199-question arms vary by up to 0.035 accuracy, so paired differences below about three points fall within the measurement floor; the model comparison also has confounds and small abstention subsets.The Qwen and gpt-oss-20b configurations differ in several respects, and bare/RAG versus agentic arms use different sampling settings.Source: §6.2 · p. 30
  • The faithfulness judge gives no credit to claims requiring multiple passages, was validated on general-domain RAGTruth rather than scientific answers, and makes reported faithfulness figures lower bounds.Source: §6.2 · p. 30
  • The abstention scorer relies on a text marker and can miss wrong-paper retrieval; several gpt-oss-20b arms also lack final messages, while binary abstention yields only single-point risk-coverage results.For gpt-oss-20b, bare and RAG accuracies and the C1 refusal rate are floors; precision over attempted answers is fairer for those arms.Source: §6.2 · p. 31
  • The public benchmarks differ from Munin’s chemistry, biophysics, and membrane-biology deployment corpus, while retrieval anchors use query variants from one expander model.Source: §6.2 · p. 31
  • The sovereignty claim has boundaries: chat traffic uses a rented web host, web-search queries reach a commercial API, and all reported numbers come from one unpublished consortium corpus.Evaluation on the group’s own queries is deferred until a curated local query pool exists; released scorecards allow recomputation, but the corpus itself is not published.Source: §6.2 · p. 31

7 Conclusion

Munin combines a self-hosted literature assistant with an evaluation suite intended to test its harness on domain questions. On the reference deployment, the harness improves accuracy and evidence tracing, while the released suite lets groups assess their own corpus and model.

7 Conclusion

  • The harness approximately doubles base-model accuracy on literature questions and doubles the fraction of answer claims traceable to a passage the model read.It declines to answer rather than fabricate when source papers are absent, with no invented local citation across three backbones and 300 fabricated references.Source: §7 · p. 31
  • With outbound requests blocked, both Qwen models exceed PaperQA2’s published benchmark result at 0.660 accuracy, while gpt-oss-20b does not.The harness improves both Qwen backbones and gpt-oss-20b, which runs on a single consumer-grade GPU; the gain is attributed to the retrieval-and-verification loop.Source: §7 · p. 31
  • Groups can run the shipped evaluation suite on their own corpus and model, then re-run it after model or harness changes to track performance over time.Source: §7 · p. 32

Availability of code, data and scorecards

  • The Apache-2.0 release includes a one-command evaluation suite, comparison and certification commands, 54 scorecards, and frozen question sets; the reference corpus is not redistributed.Following the same procedure on another corpus yields comparable rather than identical figures because retrieval difficulty, citation graphs, and shadow-corpus outcomes vary.Source: Availability of code, data and… · p. 32

Competing interests

The authors report no competing interests and disclose language-model use in research, software development, evaluation, and manuscript preparation. The appendices describe the tool inventory and the design rationale behind architectural choices.

Competing interests

Declaration of AI use

  • Language models served as study backbones, an on-premises faithfulness judge, coding assistants, and manuscript editors; the authors reviewed and revised all text.Results were computed by committed evaluation code from recorded outputs, and authors checked reported numbers against scorecards.Source: Declaration of AI use · pp. 32–33

Appendix A Tool Inventory

  • Munin exposes four agents with multi-step loops and 41 standalone tools that perform deterministic single steps; most operations need no agent layer.Tools run in parallel by default, with explicit serialization for eleven state mutators; the full inventory is Table A1.Source: Appendix A · p. 35, Table A1 · p. 35

Appendix B: Design rationale

  • Appendix B records the measurement results during development that informed the system’s architectural choices.Source: Appendix B · p. 35

B.1 Encoder migration

Replacing SPECTER-dense with BGE-large substantially raised local-pool retrieval recall, but the answer-accuracy increase was smaller, redirecting optimization effort to reading.

B.1 Encoder migration

  • Recall@10 rose from 0.44 to 0.73 (+0.29, p < 0.001), while end-to-end answer accuracy increased by only 0.075, prompting a shift from retrieval optimization to the reading step.The correct paper was already in the candidate set before the encoder change.Source: §B.1 · p. 35

B.2 Removal of passage summarisation

Removing passage summarisation preserved numerical evidence needed to answer questions, improving accuracy when full text was available. The reading agent now receives raw tagged reference text, with its token cost accepted.

B.2 Removal of passage summarisation

  • 0.82 accuracy in an oracle run and 0.864 for the pinned full-text reading agent under production conditions followed supplying unsummarised full text.The original implementation's passage summaries removed numerical values required by questions.Source: §B.2 · p. 35
  • The summaries' missing numerical evidence led to more abstentions, not incorrect answers, because the model recognized the information was absent from context.The summarisation step was removed.Source: §B.2 · p. 35
  • The reading agent receives raw tagged reference text, and its associated token cost is accepted to preserve required evidence.Source: §B.2 · p. 36

B.3 Introduction of the agentic loop

Fixed top-5 context lowered accuracy below the bare model on both backbones by steering it toward passages without answers. The iterative agentic loop replaced fixed context with model-directed retrieval and stopping.

B.3 Introduction of the agentic loop

  • Prepending top-5 retrieved documents lowered accuracy below the bare model on both backbones, directing it to irrelevant passages and reducing attempts from parametric knowledge.Source: §B.3 · p. 36
  • The iterative agentic loop lets the model select what and how many documents to retrieve, whether to use results, and when to stop searching.Source: §B.3 · p. 36
  • The fixed-context approach was replaced by the iterative agentic loop.Source: §B.3 · p. 36

B.4 Guardrail implementation via code counters

Prompt-level guardrail interventions either had no measurable accuracy effect or reversed, while a hard execution-loop counter bounded tool calls without detectable grounding-score change. The authors implement behavioural guardrails as code-level constraints.

B.4 Guardrail implementation via code counters

  • A hard CHAT_MAX_TOOL_CALLS counter set to 30 reduced maximum calls from 43 to 31 and the 90th percentile from 40 to 30, with grounding scores within noise.Source: §B.4 · p. 36
  • A research-fragment prompt raised median tool calls per query from 10.5 to 20.5 without changing accuracy.Source: §B.4 · p. 36
  • A second prompt modification reversed itself within two runs, so behavioural guardrails are implemented as code-level constraints rather than prompt instructions.Source: §B.4 · p. 36

B.5 Pre-deployment bug detection by the evaluation suite

Controlled evaluation exposed a citation re-ranker score-scale mismatch and improved correct abstention on missing-source questions after explicit detection was introduced. The evaluation suite is shipped so groups can verify fixes and detect regressions on their own corpus.

B.5 Pre-deployment bug detection by the evaluation suite

  • 0.117 to 0.469: a controlled LitSearch A/B comparison raised nDCG@10 after fixing citation re-ranker comparisons across different normalisation bases.The mismatch had compressed the ranking distribution.Source: §B.5 · p. 36
  • 0.200 to 0.667 (+0.467, p < 0.001): correct abstention on missing-source questions improved across the harness upgrade introducing explicit not-in-corpus detection.Source: §B.5 · p. 36
  • The shipped evaluation suite lets adopting groups verify fixes on their own corpus and detect regressions from later upgrades.Neither defect would have been apparent from a single-query demonstration or accuracy measurements alone.Source: §B.5 · p. 36

B.6 A defect of the faithfulness suite

An incomplete retrieval-tool list caused the initial faithfulness evaluation to score claims against only a fraction of passages read. Re-capturing all tools revealed positive paired faithfulness differences on both Qwen backbones.

B.6 A defect of the faithfulness suite

  • +0.258 on Qwen3.8 and +0.336 on Qwen3.6 (both p < 0.001) were the paired faithfulness differences after complete tool capture.These are the figures reported in Section 5.2.Source: §B.6 · p. 36
  • The first capture omitted the corpus search ladder and grounded-read steps, so the judge compared claims against only a fraction of passages the agent had read.Source: §B.6 · p. 36
  • The incomplete capture produced a near-zero paired difference (+0.010 to +0.023) across two backbones and three runs, illustrating that replication can repeat an upstream error.Source: §B.6 · p. 36

Appendix C Metric definitions

The appendix defines retrieval, answer-quality, and faithfulness metrics, along with procedures for uncertainty intervals and paired tests.

Appendix C Metric definitions

  • nDCG@k discounts qrels relevance grades by log2(rank + 1), while Recall@k measures relevant documents in the top k and MRR averages the reciprocal rank of the first relevant document.Documents absent from the qrels receive grade 0; nDCG's ideal DCG is computed from qrels grades.Source: Appendix C · pp. 36–37
  • Accuracy counts correct responses over all questions, while coverage is the answered fraction and selective risk is the incorrect fraction among answered responses.Abstentions and unparseable outputs count against accuracy; unparseable counts are reported separately.Source: Appendix C · p. 37
  • Faithfulness is the mean per-answer fraction of claims judged supported by passages retrieved on that turn; paired differences use questions answered by both arms.Each answer contributes equally regardless of its number of claims.Source: Appendix C · p. 37
  • Intervals use query-index percentile bootstraps; small answerable subsets also use Wilson score intervals when wider, and paired comparisons include bootstrap and Wilcoxon tests.The bootstrap uses 1,000 resamples for ablation comparisons and 2,000 for abstention tests.Source: Appendix C · p. 37
Loading 2610.23480v1…