PaperlayerView exact revision on arXiv

Source-linked AI summary

Not Until the Evidence Says So: Teaching LLM Investigators When to Close a Case

Tingzhu Bi, Ping Wang, Meng Ma

arXiv:2610.03190v1cs.CLcs.AIcs.LG

TL;DR

Investigative LLMs must decide not only what caused an incident, but whether gathered evidence is sufficient to close the case. The paper builds Nautil and trains investigators for that decision; supervised fine-tuning improves evidence grounding, while closure-only reinforcement learning raises closure accuracy to the teacher’s level, with some cost in evidence dependence.

  • Problem

    Investigators must decide whether gathered evidence supports closing a case, but cause-finding ability alone does not establish when to close.

    Source: §1 · p. 1

  • Method

    The authors build Nautil, a dataset of audited investigation trajectories, and evaluate closure with accuracy, evidence-dependence, and judged conclusion-and-gap tests.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Overstatement falls from 97% for the base model to 35% for Nautil-SFT, and answers both right about the cause and not overstated rise from 3% to 43%.

    Source: §6.3 · p. 8

  • Takeaways & Limitations

    Nautil-SFT closures move with the evidence, while closure-only RL raises accuracy to the teacher’s level at some cost in evidence dependence.

    Source: §8 · p. 9

    The main judge is also the teacher, and two additional judges agree on direction but not absolute rates.

    Source: §7 · p. 9

Abstract

from arXiv · show

Accident, defect and outage investigations end with a decision that ordinary question answering never faces: whether the evidence gathered so far is enough to close the case. We study this decision for LLM investigators, which request evidence from a case file, revise their hypotheses, and either close the case with a conclusion grounded in what they read or leave it open and name what is missing. This judgment does not come with capability: an untrained 9B model overstates its evidence in 97% of its answers, and a frontier model that identifies the right cause in 84% of cases still overstates in 91% and closes 17 of the 41 cases whose official finding is "cause undetermined". Measuring it is also non-trivial: the source of a case largely predicts its label, and a rule that reads only the source reaches 83.0 balanced accuracy on our test cases. We therefore evaluate closure with three tests: closure accuracy, reported against this rule and within each source; evidence dependence, which removes the grounds of a conclusion and checks whether the model stops closing; and conclusion and gap quality, a judged checklist of what the model asserts and what it says is missing. We build Nautil, 731 audited cases from aviation, rail, maritime, chemical-safety and vehicle-defect reports and production server incidents, with teacher trajectories, an out-of-distribution test set and counterfactual evidence versions. Fine-tuning a 9B model on these trajectories makes its closures follow the evidence: removing the grounds lowers its closure rate by 26 points relative to a matched control, overstatement falls from 97% to 35%, and correct, non-overstated conclusions rise from 3% to 43%. Reinforcement learning that rewards only the closure decision then raises balanced accuracy from 69.2 to 83.3, on par with the teacher, and within-source accuracy from 60.4 to 74.1, at some cost in evidence dependence.

1 INTRODUCTION

Investigative closure asks whether gathered evidence is sufficient to state a cause, a decision distinct from identifying the cause itself. Nautil formalizes and evaluates that decision, building audited cases and training investigators to ground their closures in evidence.

1 INTRODUCTION

  • A frontier model identifies the reference cause in 84% of test cases but overstates evidence in 91% and closes 17 of 41 officially undetermined cases.An untrained 9B model overstates in 97% of its answers.Source: §1 · p. 1
  • The task treats “the cause cannot be determined from this record” as a correct, scored outcome after evidence is gathered over multiple turns.This differs from single-question abstention settings where information is given up front.Source: §1 · pp. 1–2
  • Nautil evaluates closure through closure accuracy, evidence dependence, and judged conclusion and gap quality.Accuracy is compared with a source-only rule and within each source; evidence dependence removes a conclusion’s grounds against a matched control.Source: §1 · p. 2
  • Nautil1 contains 731 audited cases from four sources, including production server incidents, with teacher trajectories, an OOD test set, and counterfactual evidence versions.The OOD test set contains 141 cases.Source: §1 · p. 2
  • Training on audited trajectories teaches a 9B model evidence-following closure, while closure-only RL raises closure accuracy to the teacher’s level.The introduction also reports that judged conclusion-quality rewards collapse the policy into never closing, while outcome-only reward trades evidence dependence for accuracy.Source: §1 · p. 2

2 RELATED WORK

Prior abstention and diagnosis work typically evaluates answering a question with information already available or finding a cause. This paper instead scores whether an investigator should close after gathering evidence across turns.

2 RELATED WORK

  • Prior abstention settings provide information up front, whereas this task makes the decision after multi-turn evidence gathering and scores “cannot determine” as a valid outcome.The case files contain tens to hundreds of evidence items requested over several turns.Source: §2 · pp. 2–3
  • Incident-diagnosis and root-cause benchmarks score cause finding; this work adds closure itself as a scored, trainable decision across domains.The cited examples include cloud incidents, auditable diagnosis agents, evidence-first investigators, and sequential medical diagnosis.Source: §2 · p. 3
  • The paper makes “not closed” a verifiable RL outcome and reports that judge-scored rubric rewards collapse closure in this setting.The related work contrasts this with RL approaches that assume a solvable task with a checkable answer.Source: §2 · p. 3
  • A source-only baseline, matched control, counterfactual edits, and cross-judge checks address shortcut learning and judge bias concerns.The cited work notes that contrast sets and counterfactual edits test whether predictions follow input, while LLM judges can be biased toward their own outputs.Source: §2 · p. 3

3 INVESTIGATIVE CLOSURE

Investigative closure is a two-part judgment: whether the case should close and whether its conclusion is supported by the evidence gathered. The evaluation combines accuracy, evidence-dependence, and conclusion-and-gap checks.

3 INVESTIGATIVE CLOSURE

  • Each investigation begins with a brief, an opening record, and a neutral-title index, then requests evidence over turns before returning a cited closed or not-closed answer.The episode has at most 15 turns; an undetermined report remains open even when it establishes an event but not its cause.Source: §3 · p. 3
  • Three tests address different failure modes: closure accuracy, evidence dependence, and conclusion and gap quality.They guard against source recognition, habitual or scarcity-driven closure, and correct decisions paired with wrong or overstated conclusions.Source: §3 · p. 3
  • Balanced closure accuracy averages closing rates on cases that should close and non-closing rates on cases that should remain open.Every result is paired with a source-only rule, and within-source balanced accuracy gives that rule exactly 50.Source: §3 · p. 3
  • Evidence dependence compares closure after removing a conclusion’s cited grounds with a control that removes the same number of other items.The brief, question, and source stay fixed; a more negative removed-minus-control closure-rate difference means closure follows evidence.Source: §3 · pp. 3–4
  • A fixed judged checklist tests causal correctness, chain fit, overstatement, citation support, alternatives, and whether an unclosed answer names a specific absent gap.The summary metric is the share of answers whose cause is right and not overstated; the authors also repeat the checklist with two further judges.Source: §3 · p. 4

4 THE NAUTIL DATASET

Nautil combines report-based investigations with production server incidents in a 731-case dataset and evaluates whether closure follows evidence rather than source labels. It also provides unseen-domain and counterfactual test cases.

4 THE NAUTIL DATASET

  • Nautil draws on four source groups: transport and chemical-safety boards, NTSB aviation reports, NHTSA vehicle-defect investigations, and production server incidents.Report-case labels reflect whether the official report determined the cause; host labels come from the teacher’s closure decision and are reported only in the appendix.Source: §4 · p. 4
  • Report case files use neutral-title atomic evidence items and withhold conclusion-hinting text while preserving a hidden reference to determination, official conclusions, rejected explanations, and open alternatives.For server incidents, metric-series charts are described using numbers from their accompanying tables, with a program flagging unsupported numbers.Source: §4 · p. 4
  • Teacher trajectories are assembled and checked programmatically, then supplemented with citations and a ledger that tracks hypotheses as open, favoured, or weakened.The teacher sees the report analysis and label, and fetched items must support each cited hypothesis change.Source: §4 · p. 5
  • Of 1,345 generated cases, 518 passed the audit; supplements and removals produced a final set of 731 cases split 545 / 74 / 112 within each source.The split balances labels and keeps each incident in one split.Source: §4 · p. 5
  • Boards and host cases mostly close, while NTSB and NHTSA cases mostly remain open; a source-only rule reaches 83.0 balanced accuracy on 64 non-host test cases.The headline test set is non-host because host labels are teacher judgments.Source: §4 · p. 5
  • The 141-case OOD set spans unseen report domains, medical cases, microservice faults, and Tennessee Eastman process runs.The authors report OOD results by domain and as a secondary column because domain and label are tied.Source: §4 · p. 5
  • For 62 test cases the teacher closes, counterfactual versions remove 40%, 75%, or 100% of cited grounds, while controls remove the same number of other items.A leak check flags removed items that may be restated elsewhere; 20 cases have none.Source: §4 · p. 5

5 TRAINING NAUTIL INVESTIGATORS

Nautil-SFT is trained on audited investigation trajectories, and Nautil-RLVR starts from that model with a program-checked reward focused on closure. The RL setup balances case sources and labels, but its selected checkpoint used a screening set that included test cases.

Supervised fine-tuning

  • Nautil-SFT fine-tunes Qwen3.5-9B for two epochs on 545 training trajectories, selecting epoch 2 before touching the test set.Training uses LoRA, a 32k-token context, assistant-turn-only loss, and the native tool-call format.Source: §5 · p. 5

RLVR with a closure-only reward

  • RL assigns +1 when the closure decision matches the label and −1 when it does not, or when no closure marker appears.No judge is called during training.Source: §5 · pp. 5–6
  • Nautil-RLVR is selected at step 30 using a screening evaluation that included 30 test cases, 22 non-host.Appendix B reports results with those cases removed.Source: §5 · p. 6
  • The format penalty s is nonpositive and centered separately among closing and non-closing rollouts, leaving D as the only reward term separating those decisions.The penalty covers missing citations, unread or nonexistent citations, excess turns, repeated requests, malformed calls, or a missing marker.Source: §5 · p. 6
  • RL training samples cases uniformly across eight source-by-label strata, with host cases making up a quarter of draws.Each step samples 22 cases with eight rollouts each; Table 2 evaluates 64 non-host test cases and 141 OOD cases across four unseen domains.Source: §5 · p. 6, Table 2 · p. 6

6 EXPERIMENTS

The experiments compare investigators under a shared protocol and test whether training improves closure decisions, makes closures follow evidence, and changes conclusion quality. They also examine training components and out-of-distribution and long-file cases.

6 EXPERIMENTS

  • Investigators use a shared protocol, prompt, and tool; comparisons include base and prompted models, abstention models, Nautil variants, Gemini, and a teacher reference.Unless otherwise noted, results use 64 non-host test cases averaged over samples per case; intervals and host results are in the appendix.Source: §6 · p. 6

6.1 CLOSURE ACCURACY: ARE CLOSURE DECISIONS RIGHT, AND NOT BY SOURCE?

  • Nautil-RLVR reaches 83.3 balanced accuracy, matching the source-only rule, and 74.1 within-source accuracy; Nautil-SFT reaches 69.2 and 60.4, respectively.Gemini reaches 79.3 balanced accuracy, while Nautil-RLVR is on par with the teacher at 81.8.Source: §6.1 · pp. 6–7

6.2 EVIDENCE DEPENDENCE: DO CLOSURES FOLLOW THE EVIDENCE?

  • On 23 non-host counterfactual cases, removing all conclusion grounds lowers Nautil-SFT’s closure rate 26 points relative to the matched control; Nautil-RLVR’s effect is −15.9.The test compares versions with the same source, brief, and number of items; Figure 3 plots closure as grounds are removed and includes the matched control.Source: §6.2 · p. 7, Table 3 · p. 7, Figure 3 · p. 7

6.3 CONCLUSION AND GAP QUALITY: WHAT DOES TRAINING CHANGE IN THE ANSWERS?

  • Overstatement falls from 97% for the base model to 35% for Nautil-SFT, while correct, non-overstated conclusions rise from 3% to 43%.The right-cause rate changes only from 54% to 56%; Nautil-SFT leaves 15 of the 17 undetermined cases closed by Gemini open, but often gives generic gaps.Source: §6.3 · p. 8, Table 4 · p. 8

6.4 WHICH COMPONENTS MATTER, AND WHAT DOES RL ADD?

  • Not-closed demonstrations teach restraint; removing them leaves no cases open and 84% of answers overstated, while removing the hypothesis ledger weakens evidence dependence.With 10%, 50%, and 100% of trajectories, balanced accuracy is 45.9, 64.9, and 69.2, respectively.Source: §6.4 · p. 8
  • A closure-only RL reward raises balanced accuracy by 14.1 points and within-source accuracy from 60.4 to 74.1, but weakens counterfactual evidence dependence.Adding a judged checklist to the reward either drives reluctance to close or yields only a 3.8-point gain; closure-only RL also shortens investigations from 5.5 to 4.6 turns.Source: Table 5 · p. 8, §6.4 · p. 9

6.5 FURTHER RESULTS

  • ReAct prompting and a prompted ledger leave the base model’s closing habit intact, while single-turn abstention models often fail to finish investigations; OOD gains do not establish above-chance performance within domains.On the 20 largest files, Nautil-RLVR closes 93% of should-close cases when fetching its own evidence and 61% when given the whole file.Source: §6.5 · p. 9

7 LIMITATIONS

The paper notes that case source predicts labels, while several data, evaluation, and modeling choices constrain interpretation of its results.

Labels and sources

  • The source of a case still predicts its label, so accuracy is paired with within-source and counterfactual tests; host labels are teacher judgments the teacher does not reproduce, and host results are in the appendix.Source: §7 · p. 9
  • Within-source accuracy rests mostly on board cases, and host labels are not reproducible under the study protocol.The teacher is both teacher and main judge; two further judges agree on direction but not absolute rates.Source: §7 · p. 9
  • Because case source still predicts its label, the study pairs accuracy with within-source and counterfactual tests; the non-host counterfactual set contains 23 cases, grounded in teacher citations.The Nautil-RLVR checkpoint was screened on 30 test cases, and a 33-case pilot set consulted while choosing the SFT epoch overlaps the OOD test set.Source: §7 · p. 9

Judge, teacher and samples

  • API models are mostly sampled once, and the non-host counterfactual set has 23 cases whose grounds come from teacher citations.The passage also notes that Nautil-RLVR depends less on evidence than Nautil-SFT.Source: §7 · p. 9

Data

  • About 4% of references contain extraction errors, and some anonymised words are damaged.Investigation trajectories retrace the report rather than explore it.Source: §7 · p. 9

Open side of the decision

  • When models leave a case open, they often fail to name specific missing evidence, unlike frontier models.The passage also identifies reduced evidence dependence for Nautil-RLVR relative to Nautil-SFT.Source: §7 · p. 9

8 CONCLUSION

The conclusion argues that case closure is a distinct judgment from cause identification and that training can improve evidence-grounded restraint. It also cautions that closure accuracy needs source-aware and evidence-dependence checks.

8 CONCLUSION

  • Trained on audited investigation trajectories, a 9B model becomes more restrained and grounded, with closures that move with the evidence.Its claims cite what it read, and its conclusions are less often overstated.Source: §8 · p. 9
  • A closure-only RL reward lifts accuracy to the teacher’s level, but with some cost in evidence dependence.Rewarding judged conclusion quality instead collapses the policy into never closing.Source: §8 · p. 9
  • Because case source can stand in for evidence, the authors recommend reporting closure accuracy with a source-only baseline, within sources, and with evidence-dependence tests.Source: §8 · p. 9

AI USE STATEMENT

The authors used generative AI across research and manuscript preparation, reviewed the AI-assisted work, and accept responsibility for the final content.

AI USE STATEMENT

A DATASET CONSTRUCTION

The dataset combines investigation reports and production-server incidents, converts them into audited evidence-based cases, and includes counterfactual and out-of-distribution evaluations. Its construction also entails limits in host labels, extraction, and anonymisation.

Sources and licences

  • Nautil’s source pools include board reports, NTSB aviation and defect investigations, and host anomaly windows; medical reports were reserved for OOD testing because their licences are mixed.Host windows were detector-flagged anomalies, not verified root-cause labels; report licences vary by source.Source: A DATASET CONSTRUCTION · p. 15

Report extraction

  • Report extraction separates factual, analytical, and conclusion passages, turns facts into atomic evidence items, withholds conclusion-hinting sentences, and records a hidden reference.E1 is fixed to the first four observation or measurement items of the primary source.Source: A DATASET CONSTRUCTION · p. 15

Host textualisation

  • Host cases are textualised from selected metric charts and their numbers, with a program flagging values in descriptions that are absent from the table.Source: A DATASET CONSTRUCTION · p. 15

Anonymisation

  • Anonymisation replaces report identifiers in two passes followed by a sweep, but substring matching damaged ordinary words in at least 63 cases.Host service names become neutral roles, while IP addresses, alert IDs, links, and calendar dates are removed.Source: A DATASET CONSTRUCTION · p. 15

Host labels

  • Host closure labels are teacher judgments rather than ground truth, and a rerun closed only 5 of 39 test cases labelled closed.The host results are therefore presented only in the appendix.Source: A DATASET CONSTRUCTION · p. 15

Trajectories and audit

  • Teacher trajectories are planned, assembled, and audited for evidence grounding and honest closure; the resulting dataset contains 731 cases.After staged checks and revisions, the pipeline added supplements and rewritten failed cases, while removing 26 mismatched board cases.Source: A DATASET CONSTRUCTION · pp. 15–16

Splits and OOD test set

  • The split is balanced by source subtype and label, while the OOD set spans report, medical, microservice, and Tennessee Eastman cases whose domains strongly predict labels.A domain-majority rule reaches 90.3 balanced accuracy, so OOD results are reported by domain.Source: A DATASET CONSTRUCTION · p. 16

Counterfactual versions

  • Counterfactual versions remove 40%, 75%, or 100% of teacher-cited grounds, with controls removing the same number of uncited items; about 10 of 247 audited references contain extraction errors.The leakage-clean subset has 20 cases without flagged items in either the fully removed or control version.Source: A DATASET CONSTRUCTION · p. 16

B TRAINING DETAILS

The training setup pairs supervised investigation trajectories with reinforcement-learning variants that test different reward designs, then evaluates closure under a shared protocol and judged quality criteria. Checkpoint screening and judge choice shape how results should be interpreted.

B TRAINING DETAILS

  • Supervised fine-tuning uses Qwen3.5-9B with LoRA rank 16 for two epochs, chosen before test-set use based on validation cross-entropy.Training used a 6.0M-token split, 1.4M supervised tokens, and 273 steps per epoch.Source: B TRAINING DETAILS · p. 16
  • RLVR uses synchronous GRPO on 544 cases, sampling 22 cases per step with eight rollouts each and four AdamW updates.Format penalties discourage missing citations, unread or nonexistent citations, excessive turns or repeated items, malformed calls, and missing markers.Source: B TRAINING DETAILS · p. 16
  • Judged reward variants score conclusion quality for closed answers and conclusion plus gap quality for unclosed answers; exploratory variants were stopped after poor or screened results.The authors report that rewarding hypothesis updates led the judge to reward not updating, while source-weighted sampling drifted along source priors.Source: B TRAINING DETAILS · pp. 16–17
  • Step 30 was selected for accuracy with a counterfactual effect on par with Nautil-SFT, but screening included 22 test cases and left a source-only rule score of 98.3 on the remainder.The authors explicitly say the remaining-set score is not evidence of robustness; Table 6 reports 42 non-host cases not used in checkpoint screening.Source: B TRAINING DETAILS · p. 17, Table 6 · p. 17

C EVALUATION PROTOCOL AND JUDGES

  • All investigators use a shared protocol; local 9B models receive three samples per case, while API models generally receive one, with 95% case-bootstrap intervals.Runs without a closure marker count as wrong on both sides.Source: C EVALUATION PROTOCOL AND JUDGES · p. 17
  • GPT-6 Sol judges first samples without seeing condition names; its own investigations are judged separately, and three judges agree on cause-item decisions 88% of the time.Conclusion-score correlations across judge pairs range from 0.71 to 0.82, while absolute overstatement rates differ substantially; all judges find trained models overstate less than untrained ones.Source: C EVALUATION PROTOCOL AND JUDGES · p. 17

D ADDITIONAL RESULTS

The additional results show how performance and evidence use vary across board cases, long files, and illustrative counterfactuals. They also document limits from case-source labels, sampling, and judge or domain coverage.

D ADDITIONAL RESULTS

  • On boards, Nautil-RLVR’s balanced-accuracy differences are +8.7 versus Gemini and +8.5 versus Nautil-SFT, with intervals spanning zero for both comparisons.The reported paired differences are +8.7 [−12.2, +31.1] and +8.5 [−1.3, +18.3], respectively.Source: D ADDITIONAL RESULTS · p. 17

E OWN FETCHING AND LONG CASE FILES

  • Nautil-SFT fetches about two thirds of items in cases with up to 50 items, but only 30–40 items (7–23%) in cases with 100 to over 1,000 items.On the 20 largest files, the untrained model closes every case when given the full file; Nautil-RLVR closes more should-close cases with its own fetching (93% vs. 61%).Source: E OWN FETCHING AND LONG CASE FILES · p. 18, Figure 4 · p. 18

G RELEASE

  • Gemini’s counterfactual results vary across samples even on byte-identical inputs, so the reported change is attributed to single-sample variance rather than the data build.On 49 cases with byte-identical V1 and V2 inputs, the reported values are −14.3 versus −4.1.Source: G RELEASE · p. 20
  • Judge agreement is imperfect: absolute overstatement rates depend strongly on the judge, although all judges find trained models overstate less than untrained ones.The appendix also reports that OOD domain and label are tightly coupled, with only the medical domain having enough cases on both sides.Source: Table 12 · p. 21, Table 14 · p. 21

E OWN FETCHING AND LONG CASE FILES

  • The long-file comparison gives all evidence in one message versus model fetching, but the two modes also differ in prompt wording, turns, and context limits.The comparison’s table defines the judge score and main-cause categories, while noting these protocol differences.Source: Table 15 · p. 22

G RELEASE

  • In the coat case, Nautil-SFT closes on CCTV and test evidence while leaving what the driver saw open; the base model cites unopened and nonexistent items.The frontier model treats hedged monitor-contrast and sunlight records as an established reason the driver missed the passenger.Source: Figure 5 · p. 22
  • In the connector case, Nautil-SFT closes when the repair campaign and crash tests remain, but leaves the case open when both grounds are removed; the base model closes on warranty counts alone.The figure contrasts removing unrelated evidence with removing the conclusion’s grounds.Source: Figure 6 · p. 23
  • A ferry-tank example shows a residual failure: Nautil-SFT misreads the continuity test and keeps contamination open, while Nautil-RLVR reads it correctly but still calls the valve’s role unproven.The frontier model reads the valve evidence and closes.Source: Figure 7 · p. 23

Related papers

Loading 2610.03190v1…