Source-linked AI summary
What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark
Seyed Mosayeb Alam
TL;DR
Language models can give different answers to repeated questions about the same document, while excerpt-built benchmarks may omit the evidence needed to answer. Probity audits this provenance problem and finds flagged items wobble more, but re-cutting windows did not reduce instability to the registered level.
Problem
Excerpt-built document benchmarks can show models text that omits the evidence underlying labels, making it important to distinguish provenance defects from genuinely difficult questions.
Method
Probity measures repeated-answer instability across financing-document questions and audits label provenance, separating genuinely absent evidence from answers computable from evidence present.
Results
Flagged items wobble at 0.255 versus 0.087 for 427 clean items, while re-cutting windows moved wobble by 0.058 with an interval containing zero.
Takeaways & Limitations
The audit identifies defective windows regardless of model stability, but the reported association between missing evidence and instability remains correlational and the repair mechanism is untested.
Takeaways & Limitations
All results use a single temperature, 0.7; temperature effects and instability at temperature 0 are out of scope.
Abstract
from arXiv · showhide
Ask a language model the same question about the same document twenty times, and it sometimes returns two different answers. We built Probity, a benchmark of 60 tasks and 470 items from real venture-financing filings, to measure how often this happens. Then we audited our own corpus and found a defect any excerpt-built benchmark can carry: items whose evidence is missing from the window of text the model is shown. The audit flags 36 items and separates two failures a single flag would conflate: evidence genuinely absent from the window and answers that must be computed from numbers the window does supply. Flagged items change their answers far more often, wobbling at 0.255 against 0.087 on the 427 clean items, and excluding them cuts apparent cross-model agreement by about a fifth. Before testing whether the missing evidence explains the instability, we registered a prediction: re-cut each window to hold its evidence, and instability should fall below a set threshold. It failed: the repair moved wobble by 0.058, with an interval containing zero. We report the association as correlational. Almost all measurements sit where instability cannot show, which bounds what a corpus built for accuracy can say about stability. We release the corpus, all 112,800 raw responses, and the audit as a runnable check for any document benchmark.
1 Introduction
The paper studies answer instability in document-grounded language-model evaluation and audits whether benchmark windows contain the evidence needed to answer their questions. It finds that missing or mischaracterized provenance is a benchmark defect, while re-windowing did not establish that it causes instability.
- Benchmark and motivation: Probity measures answer instability across 60 tasks and 470 real filing-based items, with twelve configurations answering each item 20 times.An item wobbles when a configuration does not return the same answer on all 20 runs.
- Benchmark provenance: Windowed benchmarks can present models with excerpts that omit the evidence used by human annotators to write labels.Such items are scored as difficult even though their supplied input does not contain the required evidence.
- Repair and interpretation: Re-cutting windows to include evidence did not reduce instability below the preregistered threshold, so the paper reports the relationship as correlational.The audit remains consequential because evidence outside the window is a construction defect regardless of model stability.
- Benchmark provenance: The audit separates evidence-absent items from items whose answers require computation using numbers present in the window.This distinction prevents a single provenance flag from conflating missing evidence with computed labels.
- Contributions: The paper releases a provenance audit, a reimplementable wobble protocol, and a checklist for evaluating document benchmarks before release.Its contributions also include measuring the defect’s effects on instability and apparent cross-model agreement.
2 Related work
Prior work measures model instability, task-level variation, reliability, and disaggregated performance, but generally treats benchmark corpora as given. This paper shifts attention to whether document-benchmark inputs contain the evidence needed for their labels.
- Instability as an evaluation axis: Earlier studies show that repeated model runs can vary in accuracy, that non-determinism belongs in evaluation protocols, and that one observed score can misrank systems.These studies measure how models move on an accepted corpus rather than auditing the corpus itself.
- Where instability concentrates: Task-category and clause-level reporting motivate measuring instability at item granularity rather than relying on aggregate scores.The paper asks each item repeatedly so answer changes can be observed directly.
- Evaluation reliability: Prior reliability work uses split-half correlation with Spearman-Brown correction, which this paper adopts as a control rather than a contribution.The cited nearest prior result measures correctness changes between model versions, a different quantity from repeated-answer wobble.
- Financial and legal benchmarks: Existing financial and legal document suites do not cover private financing instruments and stability with per-item answer spread.The paper positions Probity as addressing both the document domain and the repeated-answer stability measurement gap.
- Disaggregated reporting: Disaggregated and worst-slice reporting is established, but this work emphasizes intervals and task counts when interpreting slices.Applying those qualifiers can withdraw a headline result that would otherwise be reported.
3 The benchmark
Probity combines filing-based questions, human-verified oracle labels, repeated model runs, and a provenance audit that distinguishes missing evidence from computable answers. Flagged items show higher instability and lower accuracy, but the audit and repair results constrain causal interpretation.
- Corpus: Probity uses regulatory financing documents to build 60 tasks across eight categories, with 470 document-question items overall.Items cover certificates, term sheets, convertible instruments, and Form D filings.
- Oracle protocol: Human annotators record each oracle answer separately from the question and attach a validating quote and source-filing pointer.Items are excluded rather than guessed when the supplied evidence cannot settle an answer.
- Provenance audit: The provenance audit checks whether each validating quote appears in the source filing and inside the model’s window, reporting verified-present, verified-absent, and not-checkable states.It flags 36 of 470 items, including window-only failures and cases where the quote is absent from the held source.
- Failure types: The audit separates genuinely absent evidence from computational labels: the former wobble at 0.338, while the latter wobble at 0.186.Both exceed the clean rate, but only genuinely absent evidence is repairable by re-windowing.
- Flagged-item profile: Flagged items have majority accuracy of 0.722 versus 0.909 for clean items, and excluding them removes most arithmetic-heavy cap-table and waterfall tasks.The exclusion removes eight tasks, seven of which are cap-table or waterfall computations.
- Stratified contrast: Within numeric items, flagged wobble is 0.283 versus 0.093 for clean items, while the difference remains positive after dropping the two 1B configurations.A category-matched comparison is not constructible because flagged items are concentrated in cap-table and exit-waterfall categories.
- Alternative instability measure: The flagged-against-clean gap also appears in dispersion, which is 0.104 versus 0.019 pooled.Dispersion measures the share of runs disagreeing with the modal answer across model-item pairs.
- Answer presence: Where the window states the answer, wobble is 0.085; where it does not, wobble is 0.264, including some items not flagged by the audit.This property carries signal that the provenance flag misses.
4 What the instrument can resolve
The instrument is reliable for ranking task instability, but the corpus offers little opportunity to observe wobble because most answers are consistently correct or identical.
- Reliability: 0.903 is the median split-half reliability across configurations, indicating reproducible task-instability rankings.The correction uses the Spearman-Brown procedure.
- Observability: 91.3% of model-item pairs are identical across 20 runs, while 86.0% are correct across all 20 runs.An additional 6.3% are never correct, though some of those answers still vary across runs.
- Observability: Only 1.9% of 5,116 model-item pairs fall in the 0.3 ≤p ≤0.7 band where flipping can occur.Using the half-open band 0.3 ≤p < 0.7 yields 85 pairs, or 1.7%.
- Observability: Among tasks answered correctly at least 90% of the time, the largest structurally reachable wobble is 0.115, below the 0.30 instability threshold.This ceiling is determined by the observed accuracy distribution before model behavior is evaluated.
5 What the benchmark shows
The benchmark finds shared instability across configurations and inflated apparent agreement from flagged items, but its preregistered window repair does not establish that missing evidence causes wobble.
- Shared instability: 0.187 is the mean cross-model transfer of instability across 90 ordered pairs and 425 clean items.The configurations share 84 items against 9.4 expected by chance, a ratio of 8.9.
- Shared instability: 5.6 times chance is the transfer within the accuracy stratum containing 407 of 425 items, showing that shared instability is not explained by difficulty alone.The analysis controls for item difficulty while examining cross-model transfer.
- Audit effect: Excluding flagged items reduces apparent cross-model sharing from 0.238 and 11.0 to 0.187 and 8.9, removing about one-fifth of the apparent agreement.The comparison is at k = 5 and contrasts the pre-audit and post-audit populations.
- Window repair: None of the eight repairable numeric items stated its answer in the original window, yet configurations continued disagreeing after the re-cut windows supplied those answers.The repaired windows were the same character length and contained the validating quotes.
- Window repair: 0.058 is the change from 0.267 original-window wobble to 0.208 repaired-window wobble, with an interval of [−0.049, 0.165] containing zero.The repair therefore failed the preregistered threshold of 0.161, and the authors report the association as correlational.
- Unsupported claims: The conservative analysis does not support the worst-category claim: no configuration’s worst category clears its mean.The point-estimate median worst-to-mean ratio is 2.64, but conservative Wilson-bound comparisons eliminate the claim.
6 Using the benchmark
The paper turns its audit findings into release checks, reporting rules, corpus-design guidance, and reproducible artifacts for future document benchmarks.
- Provenance checklist: Eight mechanical checks verify source identity, quote presence, window containment, three-way coverage, and related provenance conditions before release.The checks are designed to catch the failure modes found in Probity.
- Reporting rule: Per-category wobble should be reported with intervals and task counts, and worst-category figures should require at least five tasks.Applying these qualifiers caused the authors to withdraw one unsupported claim.
- Corpus design rule: Successor corpora should deliberately target 0.3 ≤p ≤0.7 and record computation answers as sets of provenance spans.Only 1.9% of this corpus’s measurements fall in that observable band, and multi-span computations are difficult to window correctly.
- Release: The release includes the corpus, oracle quotes, 112,800 raw responses, scoring engine, standalone verifier, audit self-tests, and blind re-annotation materials.The verifier reproduces the reported numbers and demonstrates both accepted and rejected inputs.
7 Conclusion
The audit identifies provenance defects in windowed document benchmarks and separates them from computation-based items, but the benchmark cannot strongly test instability where only about 2% of measurements can show it.
- Audit: 36 of 470 items fail the provenance audit, including window-missing evidence and evidence absent from the held document.The audit also reports items that could not be checked rather than forcing a binary determination.
- Audit: 0.338 versus 0.087 is the wobble comparison between evidence-absent items and the clean set.The conclusion reports this association without treating it as causal.
- Interpretation: Excluding flagged items cuts apparent cross-model agreement by about a fifth, while re-cutting windows changes wobble by 0.058 with an interval containing zero.The mechanism linking missing evidence to instability remains untested.
- Scope: Only about 2% of measurements lie where instability can show, limiting what this accuracy-built corpus can resolve about correct-but-unstable answers.The release provides the corpus, raw responses, audit, and reproducible checking tools.
Limitations
The benchmark’s conclusions are bounded by annotation, provenance, reproducibility, sampling, and scope limitations. Several checks remain narrow or incomplete, so the authors avoid stronger claims about generalization, causation, and category ordering.
- Annotation: A single annotator produced every oracle label, and no inter-annotator agreement coefficient was reported.The released blind pack and scoring protocol enable an independent agreement study, but the corpus itself has not received a second reading.
- Annotation: The audit checks source and window provenance, not whether each label is correct; model disagreement surfaced 7 candidates, none upheld as an oracle error.A label reproduced by every configuration could evade this review entirely.
- Repair scope: The repair experiment covers only 10 repairable items, and its failure does not establish that re-windowing never helps.Three other evidence-absent items lacked source text, while four computational items were excluded from the repair.
- Repair measurement: The original repair comparison has incomplete response counts, which makes the original condition look more stable and the repair effect smaller.Twelve of 120 paired cells have fewer than 20 original responses, including one with only 6.
- Experimental scope: All results use temperature 0.7; temperature effects and instability at temperature 0 are outside this study’s scope.Those settings measure instability with different causes.
- Sampling: Category counts range from 1 to 16 after exclusion, with two categories too small to estimate and no claim resting on them.The largest apparent category effect is among the smallest categories.
- Scope: The corpus is English-only and drawn from United States filings, with no transfer claim to other jurisdictions without re-annotation.Terms, drafting conventions, and provision meanings can differ elsewhere.
- Scope: The audit has been run on one corpus, so whether its failure classes generalize to other windowed benchmarks remains untested.The released checker supports a second-benchmark test.
A Data statement
Probity is a U.S.-focused corpus of real venture-financing filings, organized into 60 tasks and 470 items with repeated model responses. Its human oracle, provenance links, exclusions, and intended use define it as an instability-measurement instrument rather than a training or decision tool.
- Sources: The corpus contains 965 distinct filings from 863 filer CIKs, with each item identified by CIK_accession for retrieval from EDGAR Archives.The release stores full retrieved documents and windowed extracts.
- Evaluation: Twelve configurations spanning 11 distinct base models produced 112,800 responses from 20 samples per item at temperature 0.7.After 1,000 parse failures, 111,800 responses were analysed.
- Exclusions: The audit and adjudication reduce the corpus from 470 items over 60 tasks to 427 items over 52 tasks.The audit removes 36 items and author adjudication removes 7 more.
- Scope: The corpus is English-only and limited to United States venture-financing instruments under U.S. securities law.The authors make no transfer claim to other jurisdictions without re-annotation.
- Data statement: The release includes 1,145 retrieved filing texts whose redistribution licence is unresolved, although extracts, labels, quotes, audit outputs, and code are covered by stated licences.Tables and figures can be reproduced without the full texts; the full texts are needed only for source-document checks.
- Intended use: Probity is intended to measure answer instability, not provide legal advice, compliance guidance, financing decisions, or a training-set capability score.A high score indicates stability, not correctness.
B What the flagged items have in common
Flagged items are concentrated in numeric questions, especially cap-table and exit-waterfall clauses. The instability analysis uses repeated-run wobble and a separate split-half reliability calculation, while the answer-stated-in-window property is decidable for only part of the corpus.
- Composition: 83.3% of provenance flags occur on numeric-answer items, versus 32.0% of the corpus.The corresponding flag rates are 0.203 for numeric-answer items and 0.009 for binary-answer items.
- Composition: 25 of 33 cap-table items and 6 of 8 exit-waterfall items are flagged, while four categories contain no flagged item.The concentration is therefore uneven across clause categories.
- Audit measure: The answer-stated-in-window property is decidable for 137 items using only each answer and its window.A pass that reads numerals written as words preserves the separation between the two rates.
- Audit measure: Dropping the number-word pass changes the two rates from 0.085 and 0.264 to 0.079 and 0.252, so the separation remains.The property remains measurable on filings despite the alternate parsing choice.
- Reliability: Reliability splits each item’s 20 runs into odd and even halves, correlates each configuration’s task vectors, and applies the Spearman-Brown correction.Figure 5 reports the per-configuration values underlying the Section 4 median.
C A correction to the repairable count, against our own interest
The registered repair count was corrected from 14 to 10 after four items were found to be computational rather than evidence-absent. Those four had self-sufficient derivations, so re-windowing them would have replaced usable computation with source text lacking the operands.
- Correction: The preregistration initially recorded 14 repairable items, but retrieving cited filings resolved 4 as computational.Their windows contained complete labelled derivations whose arithmetic matched the oracle to two decimals.
- Correction: The corrected repair target is 10, exactly the registered floor, so it clears the threshold without margin.The four computational items were excluded from repair in advance by the preregistration’s definition, although their runs had already been collected and discarded.
- Rationale: Re-cutting those four windows would have replaced self-sufficient derivations with source text that did not contain the operands.The reclassification exposed a deviation that the authors report rather than silently altering the preregistration.
D Full results
The appendix disaggregates the benchmark’s full results, showing how provenance exclusion changes wobble, reliability, transfer, and model-flag analyses. It also reports the re-windowing repair and the limitations of sparse categories and configurations.
- Category-level results: Two of the eight clause categories retain fewer than three tasks after provenance exclusion and are shown without supporting claims.The reporting rule prints these categories because their small size is itself relevant to interpretation.
- Transfer and model flags: Transfer analyses compare ranking units and lineup splits, while model-flag tables expose adjudications and sparse-support cases rather than hiding them.The transfer tables include frontier-drop checks and ranking-unit comparisons; the model-flag table records both raised items and settlements.
- Reliability: 0.903 is the median split-half reliability across configurations, while Figure 5 provides per-configuration values behind that summary.Reliability uses odd/even run splits with Spearman-Brown correction.
- Provenance exclusion: The provenance audit flags 36 of 470 items; excluding them leaves 427 audited items, compared with 463 items including flagged rows.Table 6 reports both readings after removing seven items at author adjudication.
- Provenance exclusion: 0.255 wobble on flagged items contrasts with 0.087 on the remaining items, a difference of 0.167.The difference is reported with a Newcombe interval of [0.128, 0.211].
- Re-windowing repair: 0.208 repaired wobble remained above the registered 0.161 threshold, while the change from original windows was 0.058 with an interval containing zero.The repair used 10 items measured in both conditions across 120 paired cells.