PaperlayerView exact revision on arXiv

Source-linked AI summary

What Can a Leaderboard Certify? Compositional Controllability for Fair Evaluation and Training of Biomedical Literature-Review Agents

Zhaowei Han, Xiang Zhang, Lingxiao Guan, Danqi Hu, Kai Liu, Kevin Chang, Jie Liu

arXiv:2609.32318v1cs.LGcs.AIcs.CL

TL;DR

Final-output leaderboards do not by themselves establish whether systems are comparable or which stage explains a score difference. The paper introduces window-based attribution bounds and pre-score admissibility tests, then reports benchmark comparisons and stage-reward training for SCRIBE. Under fixed input, SCRIBE receives a certified rank interval of [1,2].

  • Problem

    Final-report scores can reflect unequal evidence, inputs, or budgets, and do not establish whether systems are comparable or which stage accounts for a difference.

    Source: §1 · pp. 1–2

  • Method

    The framework defines comparison windows and bounds the observed-versus-controlled score gap using nuisance outside each window, enabling admissibility tests and stage-level rewards.

    Source: §1 · pp. 2–3, §3.3 · p. 5

  • Results

    Under fixed input, SCRIBE (Qwen3.8-27B) reaches certified rank interval [1,2], above every reported published pipeline and commercial agent.

    Source: §1 · p. 3, §7 · p. 10

  • Takeaways & Limitations

    The test refuses comparisons whose outside-window mismatch cannot be bounded within tolerance, so score-based ordering is supported only for admissible pairs.

    Source: §1 · p. 2

    Certificates measure agreement with human review content, not evidence use or clinical safety, and compare systems up to an exit rather than modules in isolation.

    Source: §7 · p. 10, ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Leaderboards rank long-horizon agents by their final outputs. Yet a higher score alone does not establish whether two systems are comparable or which stage accounts for the difference. Unequal evidence, inputs, or budgets can affect scores, and statistical corrections do not remove this mismatch. We introduce compositional controllability to address these questions. A comparison window covers one stage, several stages, or the whole agent. Our central result bounds the gap between observed and controlled score differences using only nuisance outside the window. This yields an admissibility test applied before scores are inspected. Inadmissible comparisons are refused. For admissible pairs, an ordering is certified only when the score gap exceeds the combined sampling and nuisance radii; otherwise, it remains undecided. These decisions give each system a rank interval. We introduce BioLitBench, a benchmark of 2,042 biomedical articles represented as structured claim graphs. Among seven published pipelines, a conventional statistical analysis declares a winner in 14 of 21 pairwise comparisons. Yet the top-ranked system alone received the target review's bibliography. To isolate pipeline performance, our test requires matched inputs and a fixed backbone model. It refuses 11 of the 21 comparisons, including every comparison involving the top-ranked system. Seven of the 14 conventional conclusions fall within these refused pairs. The same comparison windows support stage-level training. We train SCRIBE on Qwen3.8-27B using rewards measured at each stage's exit. Under matched evidence, SCRIBE achieves a certified rank interval of [1,2], with certified advantages over all evaluated published pipelines and the evaluated Claude and OpenAI agents. Under same pool, SCRIBE matches the strongest published retriever and is certified above three published pipelines.

1 INTRODUCTION

Final-report scores can reflect unequal evidence and inputs, so they do not by themselves establish comparability or identify which stage accounts for a difference. The paper introduces comparison windows and an admissibility test, then applies them to biomedical-review pipelines and stage-level training.

1 INTRODUCTION

  • Long-horizon agents may receive different evidence, inputs, or budgets, while a final score cannot separate stage contributions such as retrieval and writing.More tasks, clustering, and multiplicity correction do not remove the mismatch.Source: §1 · pp. 1–2
  • Comparison windows cover one stage, several stages, or the whole agent, and the attribution bound limits the observed-versus-controlled score gap using nuisance outside the window.An admissibility test is applied before scores are inspected; an ordering is certified only when the score gap exceeds sampling and nuisance radii.Source: §1 · p. 2
  • A systematic-review evaluation illustrates the issue: its 96.9% sensitivity estimate and human estimates of 86.6% and 97.5% came from different reviews, abstracts, and reference standards.The comparison reported no mismatch bound or sampling radius, and two reviews provide little basis for estimating between-review variation.Source: §1 · p. 2
  • BioLitBench represents 2,042 biomedical articles as claim graphs and uses same-topic reviews as set-valued references for evaluation and training.It includes claims, typed relations, hard negatives, and section hierarchies.Source: §1 · pp. 2–3
  • 11 of 21 published-pipeline comparisons are refused, including every comparison involving the top-ranked system; 7 of 14 statistically significant orderings fall within refused pairs.The top-ranked system alone received the target review’s bibliography, so its score advantage does not isolate pipeline quality from supplied evidence.Source: §1 · p. 2
  • Under matched evidence, SCRIBE has certified rank interval [1, 2] and certified advantages over evaluated published pipelines and commercial agents.Its comparison with its own untrained harness remains undecided.Source: §1 · p. 3

2 RELATED WORK

The paper relates its comparison framework to process abstraction and partial identification, distinguishing cross-system stage attribution from within-system credit assignment. It also positions admissibility as an addition to existing measurement and rank-inference practices.

2 RELATED WORK

  • Unlike within-system stage-level credit assignment, this framework attributes a difference between two systems to the stage in which they differ.Its bound answers when an observed contrast across separate pipelines identifies the effect of replacing one stage.Source: §2 · p. 3
  • A comparison window supplies a declared boundary for report-score interpretation, while an admissibility test addresses whether the pair was entitled to be compared.The paper distinguishes this test from rigorous measurement practice and confidence sets for ranks.Source: §2 · p. 3
  • The paper contrasts cross-system comparison with methods that inspect one run’s logs or trajectory, and with ADRA-Bank’s stage inputs for backbone models.Published pipelines expose no entry other than the task, according to the passage.Source: §2 · p. 3

3 METHOD: COMPOSITIONAL CONTROLLABILITY

Compositional controllability defines a comparison window and bounds how much conditions outside it can distort an observed contrast. The resulting admissibility and certification rules support partial rankings rather than requiring a total order.

3 METHOD: COMPOSITIONAL CONTROLLABILITY

  • The framework uses an attribution bound to connect an observed contrast to a controlled contrast for a window, from one stage to the whole agent.A certificate combines the bound with sampling uncertainty.Source: §3 · p. 4

3.1 A MOTIVATING EXAMPLE

  • With unequal evidence bundles, pipeline scores favor B by 0.04, but at the same coverage e = 0.5, A leads by 0.15.The example shows that more tasks alone do not remove a systematic input mismatch.Source: §3.1 · p. 4

3.2 COMPARISON WINDOWS AND READOUTS

  • A comparison window runs from an entry frontier to an exit frontier; a stage window starts at its input artifact, while a system window covers task input through final report.A semantic interface maps system states into a common space, and a readout scores the window’s exit through an observation channel.Source: §3.2 · p. 4

3.3 CONTRASTS AND THE ATTRIBUTION BOUND

  • Under the stated conditions, the attribution bound limits the gap between observed and controlled contrasts using outside-window mismatches and their effects on scores.The example’s nuisance bound is 0.29, covering the 0.19 discrepancy between observed −0.04 and controlled +0.15 contrasts.Source: §3.3 · p. 5

3.4 CERTIFICATION AND RANK INTERVALS

  • A pair is admissible when the declared conditions hold and its nuisance radius is within a tolerance fixed before rankings are observed.An admissible ordering is certified only when the score gap exceeds sampling and nuisance radii; otherwise, it remains undecided, and systems receive rank intervals.Source: §3.4 · pp. 5–6
  • When entries, channels, and protocols are exactly matched, the nuisance radius is zero and certification rests on the sampling radius alone.In the example, matching the evidence bundle yields observed and controlled contrasts of 0.15.Source: §3.4 · p. 6

4 HUMAN REVIEWS AS STRUCTURED GROUND TRUTH

BioLitBench turns human reviews into structured references for claim-by-claim evaluation. Its construction, scoring, and entry conditions define what the benchmark compares.

4 HUMAN REVIEWS AS STRUCTURED GROUND TRUTH

  • BioLitBench provides structured human references for evaluating generated reviews claim by claim.The benchmark parses PubMed Central articles into graphs containing section, claim, and paper nodes.Source: §4 · p. 6
  • The benchmark yields 2,042 graphs from 2,063 open-access articles, with 87.4% mean reference resolution and 87.6% of claims classified as entailed by a cited abstract.Claim pairs are labeled with eight relation types, and topically similar unrelated pairs become hard negatives.Source: §4 · p. 6
  • Each evaluation task uses 4–9 same-topic peer reviews to define a reference band, and the dataset is split into 1,842 training and 200 held-out examples.The 50 evaluation tasks are selected from held-out topics, with evidence restricted to papers published before each task’s cutoff.Source: §4 · p. 6
  • Scoring matches output sentences to reference claims, resolves citations, matches headings to sections, and reproduces relations when both claims match and sentence similarity clears a linked-claim threshold.The sentence-match threshold is the larger of 0.70 and the 95th percentile of off-topic similarities.Source: §4 · p. 6
  • The benchmark screens 52 readouts and admits 34 across six axes; fixed-input evaluation covers five post-retrieval axes, while same-pool evaluation adds retrieval.Under same pool, systems search a frozen corpus of 26.6M papers.Source: §4 · p. 6

5 STAGE-LEVEL TRAINING WITH CERTIFIED REWARDS

SCRIBE uses controlled stage-level rewards with KV-Skill training on a frozen Qwen3.8-27B backbone. Its stage rewards are measured at exits under matched inputs, while training data and leaderboard-based reward calibration follow distinct data conditions.

5 STAGE-LEVEL TRAINING WITH CERTIFIED REWARDS

  • SCRIBE combines controlled stage-level rewards with KV-Skill training on Qwen3.8-27B.Planning is trained; synthesis and writing are scored but retain their initial carriers.Source: §5 · pp. 6–7
  • Matched sealed inputs and exit readouts make the score difference between two stage versions equal the controlled contrast, allowing exit scores to serve as stage rewards.Receipts link consecutive artifacts, and the same readout and protocol are used for both versions.Source: §5 · pp. 6–7
  • Training examples and reference graphs come from the training split, but reward weights are calibrated on leaderboard tasks against each readout’s strongest published pipeline.Evaluation tasks, references, and peers are excluded from training examples.Source: §5 · p. 7
  • Planning uses a registered KV-Skill carrier optimized with GRPO on 192 training tasks; KV-Skill operators do not add prompt tokens or change backbone weights.SCRIBE-Luna uses the same harness without stage-level training, while retrieval uses a shared pool service with a task-specific budget cap.Source: §5 · p. 7

6 RESULTS

The certified leaderboard gives SCRIBE a top interval under fixed input and places it among several unresolved systems under same pool. For published pipelines, certification refuses comparisons with mismatched backbones or entries, while the theory checks support the bound on real systems with stated scope limits.

6.1 THE CERTIFIED LEADERBOARD

  • Across the 50 held-out tasks, comparisons use 23 subfield clusters, with matched-length reports as the primary observation and full length as secondary.Within each entry condition, systems share the reference list or frozen corpus; the two conditions are not compared with each other.Source: §6 · p. 7, §6.1 · p. 7
  • SCRIBE’s fixed-input certified rank interval is [1, 2], and it is certified above every reported system except its untrained harness.Its composite is 0.606; the untrained harness scores 0.583, and their planning scores are 0.319 and 0.276, respectively.Source: §6.1 · p. 7, Table 1 · p. 8
  • Under same pool, SCRIBE’s certified interval is [1, 7]; it is certified above four pipelines but not separated from SCRIBE-Luna, SurveyGen-I, or the Claude agents.Its retrieval score is 0.335, compared with SurveyGen-I’s 0.336.Source: §6.1 · p. 7
  • Under fixed input, Claude Science and Elicit are certified above SurveyGen-I, AutoSurvey, and LLM×MR, while Claude Code is certified above LLM×MR.Claude Code and Claude Science share a backbone, so their difference comes from the product harness.Source: §6.1 · p. 7

6.2 CERTIFICATION VERSUS A CONVENTIONAL ANALYSIS

  • The admissibility test refuses 11 of 21 published-pipeline pairs before scores are read, including six with DR-Tulu’s different backbone and five mismatching LiRA entries.Seven of the 14 conventional orderings fall among the refused pairs; the table summarizes conclusions over the pipeline pairs.Source: §6.2 · p. 9, Table 2 · p. 9
  • Among admissible pairs, 7 of 10 self-retrieving comparisons certify, while only 1 of 10 reference-fed comparisons certifies; every undecided admissible pair is sampling-limited.Retrieval certifies 11 of 21 same-pool capability pairs, whereas writing certifies none.Source: §6.2 · p. 9
  • At full length, AutoSurvey is pinned first among reference-fed configurations, unlike the matched-length view, where no system is pinned first.AutoSurvey reports have a median of about 250,000 words, versus 10,000–47,000 for the others; the authors state that the length convention must be declared before comparison.Source: §6.2 · p. 9

6.3 THE THEORY ON REAL SYSTEMS

  • The attribution bound holds in two designs on published systems and decides the five LiRA refusals, but multi-stage composition is checked on synthetic chains.The bound is informative on writing and looser on other axes; non-expansiveness depends on the declared length convention.Source: §6.3 · p. 9

7 CONCLUSION

The conclusion presents compositional controllability as a way to decide which comparisons and orderings an evaluation supports, while SCRIBE provides a stage-reward application. The accompanying theory specifies agent interfaces, comparison windows, nuisance conditions, and finite-sample certification; its claims remain bounded by content readouts and declared conventions.

7 CONCLUSION

  • Compositional controllability tests admissibility before scores are read and certifies an ordering only when the score gap exceeds the radius.On the published pipelines, it refuses 11 of 21 pairs; SCRIBE reaches a fixed-input certified rank interval of [1, 2].Source: §7 · pp. 9–10
  • A certified relation measures agreement with human-review content, not use of supplied evidence, and compares systems up to an exit rather than modules in isolation.The length convention can change the winner and should be fixed in advance.Source: §7 · p. 10

A.1 FORMAL SETTING

  • The paper formalizes agent systems through contexts, histories, actions, observations, policies, and environment-tool kernels, with external task variables placed in the context.For literature review, context may include the question, corpus, requirements, cutoff, budget, and tools.Source: §A.1 · p. 15
  • A semantic interface exposes comparable artifacts without requiring systems to share internal states or actions.For review agents, interface states can include paper sets, evidence ledgers, synthesis graphs, outlines, remaining budget, and verification results.Source: §A.1 · p. 16

A.2 WINDOWS, READOUTS AND THE ATTRIBUTION BOUND

  • A comparison window is bounded by entry and exit frontiers, and its attribution bound limits observed-versus-controlled contrast error using nuisance outside the window.The qualitative assumptions require boundary sufficiency, no hidden bypass, accounted resources, and common exit semantics; these do not require the modules to behave alike.Source: §A.2 · pp. 16–17
  • For non-expansive windows, the bound limits entry-mismatch contributions; estimating a published pipeline’s stage-window sensitivity is unavailable when its black-box input artifact cannot be set.System windows can be entered twice on the same task to estimate that sensitivity.Source: §A.2 · p. 18

A.3 ESTIMATION AND CERTIFICATION

  • Finite-sample certification addresses the fact that evaluations observe only finitely many rollouts and tasks, rather than the population quantities bounded by the theorem.The conclusion combines the attribution bound with sampling uncertainty when certifying comparisons.Source: §A.3 · p. 18

A finite-sample certificate. Let c1, . . . , cN

The section develops finite-sample and nuisance-aware certificates for comparisons, then extends them to stage sequences and behavioral distances. It distinguishes fair comparison from equivalence or substitution, which also constrain behavior inside a window.

A finite-sample certificate. Let c1, . . . , cN

  • The finite-sample certificate assumes bounds on rollout, alignment, and annotation errors; the rollout bound is random and may hold only with high probability.Repeated rollouts of one context tighten the rollout approximation bound, not the sampling term, because they do not increase the number of task contexts.Source: A finite-sample certificate. Let c1, .… · pp. 18–19
  • Fair comparability is certified when the estimated outside-window nuisance radius plus its one-sided uncertainty radius is at most the declared tolerance.The certificate concerns nuisance outside the window, not differences within it.Source: A finite-sample certificate. Let c1, .… · p. 19
  • A direction is certified only when the observed gap exceeds the sampling radius plus the nuisance upper bound; exact cells have zero nuisance-estimation radius.The resulting interval contains the controlled contrast with the stated joint confidence when the sampling and nuisance events both hold.Source: A finite-sample certificate. Let c1, .… · p. 19
  • Certified pairwise relations yield rank intervals rather than a total order, and the relation may be non-transitive when pair-specific radii differ.When the certified relation is acyclic, each interval spans the positions a system occupies across linear extensions of its transitive closure.Source: A finite-sample certificate. Let c1, .… · p. 19

A.4 COMPOSITION ACROSS STAGES

  • Across stages, each replacement error and downstream slack is propagated by later stages’ Lipschitz constants, while the initial mismatch is propagated through all stages.If every stage is non-expansive, the final distance is bounded by the initial mismatch plus the sum of stage errors and slacks.Source: §A.4 · p. 20
  • The propagated window bound is a substitution bound, not a fairness criterion, because it includes errors inside the window.It bounds movement in the terminal law and therefore bounds downstream Lipschitz utilities as well.Source: §A.4 · p. 20

A.5 BEHAVIOURAL GEOMETRY: EQUIVALENCE AND SUBSTITUTION

  • The recursive controllability operator combines current interface distance with the cheapest probabilistic alignment of next branches, and has a unique fixed point under the stated regularity conditions.Different tools or work decompositions can remain close when their labels, effects, and future semantic states can be coupled cheaply.Source: §A.5 · p. 21
  • The fixed point can define one pseudometric across a panel of systems, while zero distance implies behavioral equivalence only under the stated ground-cost conditions.A small controllability distance supports equivalence or substitution; a large controlled module effect can still be compatible with a fair comparison.Source: §A.5 · pp. 21–22

A.6 REFERENCE SPECIFICATIONS FROM HUMAN REVIEW GRAPHS

The section defines the human review as a structured graph and the acceptable reference as a set of possible continuations. It uses the nearest acceptable behavior rather than requiring one exact human trajectory.

A.6 REFERENCE SPECIFICATIONS FROM HUMAN REVIEW GRAPHS

  • An executable reference specification combines boundary states, acceptable continuation kernels, and a reference graph, allowing multiple acceptable continuations.The allowed set can accommodate alternative searches, paper sets, taxonomies, and organizations while enforcing evidence support, scope, contradiction handling, and provenance.Source: §A.6 · p. 23
  • A human review graph represents scientific synthesis as a typed, attributed directed graph of topics, claims, evidence, papers, and conditions linked by typed relations.Node attributes carry semantic and provenance information, and node weights encode importance.Source: §A.6 · p. 23
  • The fused graph discrepancy matches nodes and relations together, but its metric choice must be validated against expert judgment and does not capture all graph semantics.The stated objective is one explicit instantiation, and it satisfies a relaxed triangle inequality in general.Source: §A.6 · p. 23

B CAMPAIGN VALIDATION

The campaign validation describes SCRIBE’s implementation and retrieval harness, then checks analysis reproducibility and identifies report length as a confounder. Full-length scoring reverses the entry-conditioned synthesis conclusion.

B CAMPAIGN VALIDATION

  • SCRIBE uses a frozen Qwen3.8-27B backbone and trained KV-Skill carriers, with sealed entries, hash-linked exits, and synthesis, planning, and writing windows.With the same pipeline and its initial carriers, the composite is 0.595 under fixed input and 0.494 under same pool; the planning carrier’s paired gains are +0.011 and −0.010, neither certified.Source: B CAMPAIGN VALIDATION · p. 24
  • Under same pool, about 40% of SCRIBE’s final bundle is selected by a citation-graph rule rather than the model’s own ranking.Per-task search, row, and document budgets match the largest amount spent by any baseline on that task.Source: B CAMPAIGN VALIDATION · p. 24
  • The entry-disclosure checks report that SurveyGen-I’s citation-route use on 33 of 50 tasks does not produce its position, and SCRIBE’s acquisition guard restricts it to plain search and metadata.Published pipelines’ budget stops are also recorded as normal exits under the stated conditions.Source: B CAMPAIGN VALIDATION · p. 24
  • Full-length scoring reverses the entry-conditioned synthesis conclusion, comparing a pipeline with median reports of about 250,000 words against SCRIBE reports under 5,000 words.The primary view holds each report to the task’s target length.Source: B CAMPAIGN VALIDATION · p. 24
  • Twenty-nine rebuild assertions pass; the reproduced table differs by at most 1.11×10^-16 across 288 cells, and a three-seed replicate null certifies 0 of 114 cells.Entry byte equality holds on 50 of 50 tasks, and independent rescoring differs on 4 of 81,575 cells, all on readouts excluded by the admission rule.Source: B CAMPAIGN VALIDATION · p. 25

C FURTHER LEADERBOARD RESULTS

This section reports a secondary full-length leaderboard view and describes the systems included in the same-pool comparison.

C FURTHER LEADERBOARD RESULTS

  • Table 4 summarizes capability scores under same pool at the primary observation point.The same-pool scores behind Figure 1(b) are also reported at full length as a secondary, descriptive view.Source: Table 4 · p. 25, C FURTHER LEADERBOARD RESULTS · p. 25
  • At full length, AutoSurvey rises to third while the top of the fixed-input table remains unchanged; same-pool certified intervals widen and none is pinned.The authors attribute AutoSurvey’s rise to the full-length composite rewarding material in its roughly 250,000-word reports.Source: C FURTHER LEADERBOARD RESULTS · p. 25, Table 5 · p. 26
  • Claude Science, Claude Code, and the OpenAI tool loop use provided tools to reach the corpus, while Elicit uses its own model and reads full text.Under fixed input, the tools return only the task’s reference list; Elicit receives that list, reads full text, and cites at most 80 papers.Source: C FURTHER LEADERBOARD RESULTS · p. 26

D READOUT VALIDITY UNDER EVIDENCE ABLATION

The evidence-ablation evaluation tests whether the content readouts respond to evidence, rather than only whether scores align with a human review. It also examines whether scores vary with review publication year.

D READOUT VALIDITY UNDER EVIDENCE ABLATION

  • The admission screen tests topical signal by requiring worse scores against an unrelated review than against the system’s own review.The controlled writing-window ablation separately tests whether readouts respond to the evidence provided.Source: D READOUT VALIDITY UNDER EVIDENCE… · pp. 26–27
  • With evidence minus no evidence, the writer scored −0.114 on form, −0.061 on reasoning, and −0.044 on synthesis, but +0.071 on writing.These paired differences are for AutoSurvey’s writer; the reported 95% confidence intervals exclude zero for each measure.Source: D READOUT VALIDITY UNDER EVIDENCE… · p. 27, Table 6 · p. 27
  • Only evidence agreement separated full evidence from both empty and shuffled controls when averaged across six pipelines.The readouts assess agreement with human-review claims, relations, and organization; the authors note that a strong backbone can partly supply this content from its own knowledge.Source: D READOUT VALIDITY UNDER EVIDENCE… · p. 27
  • Scores showed no significant dependence on target-review publication year (pooled slope −0.007 per year, p = 0.15–0.17).Publication year is described as only a proxy for a review’s exposure during pretraining.Source: D READOUT VALIDITY UNDER EVIDENCE… · p. 27

E THE CAPABILITY CAMPAIGN AND THE PUBLISHED PIPELINES

The capability campaign compares systems across axes under fixed-input and same-pool entry conditions, then examines refinement and allocation separately. Theory checks assess attribution bounds at two entries and at a controlled writing window.

E THE CAPABILITY CAMPAIGN AND THE PUBLISHED PIPELINES

  • Under fixed input, SCRIBE was certified above each of four comparable opponents; under same pool, reasoning and writing certified no pair at matched length.The fixed-input campaign certified 18 of 50 cells, and the same-pool campaign certified 32 of 126 cells.Source: E THE CAPABILITY CAMPAIGN AND THE… · p. 27, Table 7 · p. 28, Table 8 · p. 28
  • With the same papers, SCRIBE was first or joint-first on every comparable axis, while its earlier same-pool retrieval harness had one certified deficit that the final harness closed.The final harness reached retrieval 0.335 against SurveyGen-I’s 0.336; the earlier harness recovered fewer papers and produced shorter reports.Source: E THE CAPABILITY CAMPAIGN AND THE… · pp. 27–28, Table 7 · p. 28, Table 8 · p. 28
  • Refinement certified five effects, including three negative synthesis effects and positive writing and form effects; many final-exit relations were already present at the draft exit.At the final exit, 10 of 13 fixed-input and 21 of 24 same-pool pair-by-axis relations were already present at the draft exit.Source: E THE CAPABILITY CAMPAIGN AND THE… · p. 28
  • Allocation is distinct from raw co-location recall: broadcasting evidence to nearly every section yields recall of 1.000 by construction, while chance correction returns systems near zero.Measured by adjusted Rand index, systems were far below human; SCRIBE scored 0.152–0.174 against 0.099 for the best baseline.Source: E THE CAPABILITY CAMPAIGN AND THE… · pp. 28–29
  • For the published-pipeline tiers, the per-pair false-certification rate was 0.0035–0.0047 against a nominal 0.005, with no winner flips when leaving out one subfield.The mean Jaccard overlap under leave-one-subfield-out was 0.994.Source: E THE CAPABILITY CAMPAIGN AND THE… · p. 29, Table 9 · p. 29

F.1 THE ATTRIBUTION BOUND ON TWO ENTRIES

  • Across 118 paired runs from four pipelines, the attribution-bound fit used the 95th percentile of y/x and yielded LM = 0.890.None of the 118 runs expanded; the fitted ratio gives a high-probability envelope rather than a worst-case bound.Source: §F.1 · p. 30, Table 11 · p. 30, Table 12 · p. 31

F.2 THE BOUND AT A CONTROLLED WRITING WINDOW

  • At the controlled writing window, the median bound was 0.955 against a median gap of 0.060, and the bound was informative on writing but vacuous on retrieval.Non-expansiveness held at matched length but failed at full length, where report length dominates exit distance.Source: §F.2 · p. 31, Table 13 · p. 31

F.3 REFUSALS COMPUTED FROM THE BOUND

The bound-based refusal analysis rejects all five tested LiRA comparisons with self-retrieving pipelines under the stated tolerances. Further checks examine multi-stage composition, calibration, comparability versus equivalence, the controllability distance, and scope boundaries.

F.3 REFUSALS COMPUTED FROM THE BOUND

  • All five LiRA comparisons with self-retrieving pipelines are refused: every eligible readout fails the pre-registered tolerance, with the smallest margin at +0.077.The median admissible tolerance is 0.208; verdicts are unchanged under leave-one-pipeline-out constants.Source: §F.3 · p. 31, Table 14 · p. 32

F.4 COMPOSITION ACROSS STAGES

  • On synthetic four-stage chains, the composed bound holds everywhere and remains informative through organisation, but exceeds one at writing.Published pipelines cannot be re-entered at intermediate artifacts, so the multi-stage check uses synthetic chains.Source: §F.4 · p. 32, Table 15 · p. 32

F.5 CALIBRATION OF THE CERTIFICATION RULE

  • Calibration removes only Claude Code over AutoSurvey in the fixed-input table; its gap exceeds its radius by a factor of 1.02.No relation involving the authors’ system is affected.Source: §F.5 · p. 32, Table 16 · p. 32

F.6 COMPARABILITY VERSUS EQUIVALENCE

  • Reference-fed AutoSurvey and LiRA are fair; their retrieval windows are nearly equivalent (CM = 0.070), while their writing windows are comparable but not equivalent (CM ≈0.51).At retrieval they share the entry exactly; substituting writers moves the exit by about half the readout scale.Source: §F.6 · p. 33, Table 17 · p. 33

F.7 THE CONTROLLABILITY DISTANCE

  • The empirical-kernel check confirms the implementation’s contraction modulus, unique fixed point, and pseudometric properties.The check applies the Theorem 4 operator with exact optimal transport to logged-system branch kernels.Source: §F.7 · p. 33, Table 18 · p. 33

G SCOPE

  • A system-window certificate compares systems as built, including their backbones; a scaffold comparison instead orders prompts, tools, and control flow around a fixed model.Cross-backbone pairs are comparable only if that difference is accounted for; comparability is indexed by the declared interface, observation channel, and readout family.Source: G SCOPE · p. 33
  • The evaluation’s 50 biomedical review tasks come from one corpus and distribution, so transfer beyond that distribution remains a separate question.Subfield clustering accounts for dependence within the task distribution.Source: G SCOPE · p. 33
  • Published pipelines cannot be re-entered at chosen intermediate artifacts, limiting direct stage-window tests to pipelines that accept and record intermediate inputs.Their logged intermediate artifacts can be recovered and scored, but only system windows can be perturbed.Source: G SCOPE · p. 33

Related papers

Loading 2609.32318v1…