PaperlayerView exact revision on arXiv

Source-linked AI summary

How Reproducible Are Evaluation Conclusions? A Self-Audit of LLM-Inferred Prompt Structure

Dipankar Sarkar

arXiv:2609.30074v1cs.CLcs.AIcs.LG

TL;DR

Small-prompt evaluations often report model rankings, but the confidence those rankings deserve is uncertain. This paper measures repeated prompt-structure inference and audits the resulting evaluation; node sets vary across calls, and the ranking does not withstand scrutiny. It recommends reporting provenance, executed sensitivity comparisons, rank stability, raw outputs, and measurement dates.

  • Problem

    Evaluations often average over small prompt sets and rank models, raising the question of how much confidence those rankings deserve.

    Source: §1 · p. 2

  • Method

    The paper uses prompt-structure inference as a case study and audits the resulting ranking and analysis choices.

    Source: §1 · pp. 1–2

  • Results

    Mean node-set Jaccard ranges from 0.39 to 0.96, 72% of prompt-model cells are not node-set-perfect, and the full ranking does not survive audit.

    Source: §1 · p. 2, §10 · p. 9

  • Takeaways & Limitations

    The authors recommend reporting per-cell provenance, executed sensitivity comparisons, merge rules and tolerance units, rank stability, raw outputs, and measurement dates.

    Source: §10 · p. 9

    Because the study covers a specialized task without a standard benchmark or downstream evaluation, its broader recommendations are procedural safeguards rather than an estimate of failure prevalence.

    Source: §8 · p. 8

Abstract

from arXiv · show

Evaluations of LLM systems routinely average over small prompt sets and report models as a ranked table. We ask how much confidence such a table deserves, using LLM-based prompt-structure inference as the case study: eight open model variants across five families and 8B to 675B parameters, caching disabled, 293 raw intermediate representations persisted. The measured phenomenon is unstable to begin with. Identical calls do not reliably recover identical structure, with mean node-set Jaccard from 0.39 to 0.96 and 72% of prompt-model cells never node-set-perfect. Auditing the evaluation weakens its conclusions further, and this is our main contribution. Under a joint cluster bootstrap over prompts, only the bottom of the ranking is firm: the two least reproducible models hold rank in 99% and 86% of replicates, the middle four in 27% to 48%, and the top two in 68% each, so the table identifies the worst model reliably but does not reliably identify the best. Two equally defensible rules for merging repeated campaigns change four of eight rows and move the study-wide headline by 7 percentage points. Checking the inferred structure against ground-truth annotations shows reproducibility cannot be read as accuracy. And four of the eight endpoints were withdrawn within ten weeks of measurement, so the study as specified can no longer be run. Small-sample LLM evaluations can therefore look far more definitive than their evidence supports. We recommend reporting rank stability, per-cell provenance, executed sensitivity comparisons, raw per-run outputs, and a measurement date alongside any ranking.

1 Introduction

Prompt-structure inference is a prerequisite for prompt-program systems, but repeated calls may not recover the same structure. This paper measures that instability and audits whether the resulting model ranking and its assumptions withstand scrutiny.

1 Introduction

  • The study measures repeated-run consistency for prompt-structure inference across model families and parameter scales, releasing the raw outputs.It addresses the prerequisite of recovering structure when a prompt was not authored in a system’s own format.Source: §1 · p. 1
  • The full model ranking does not survive the claim-level audit, and four additional findings depend on undocumented analysis or instrumentation choices.Those choices can move a headline without changing the underlying model outputs.Source: §1 · pp. 1–2
  • Rankings from small prompt sets are weakly supported because sorting is what readers retain, while sample size supports it least.The paper notes that evaluations built from tens of prompts and a handful of models are cheap to run and easy to cite.Source: §1 · p. 2
  • The paper frames the audit as a sequence from measurement instability to instability in conclusions, including whether structure stabilizes and whether rankings survive resampling and analysis changes.It also asks whether reproducibility corresponds to validity and whether the study remains runnable over time.Source: §1 · p. 2

2 Background

The paper treats prompt-structure inference as producing an intermediate representation whose repeatability matters to tools built on it. Its methods compare repeated outputs across model variants and prompt corpora, while documenting limits in coverage and measurement.

2 Background

  • The study treats an intermediate representation (IR) as typed nodes with identifiers and source ranges, and asks whether the same prompt yields the same IR across calls.The task includes transcribing explicitly marked structure and recovering structure from ordinary prompt text.Source: §2 · p. 2
  • Across 293 persisted IRs, explicit nodes make up 64% to 100% of nodes by model, so the synthesized-corpus measurement is dominated by transcription.The corpus also includes recovery of unmarked structure, but that regime is measured in a separate campaign.Source: §2 · p. 2
  • Because downstream tools reason over inferred structure, resampled structure prevents them from offering the same-input, same-diagnosis guarantee associated with static analysis.The paper measures this precondition rather than evaluating downstream tools.Source: §2 · p. 2

3 Method

  • Eight model variants met the inclusion threshold of at least three prompts completing at least two runs, spanning five model families and 8B to 675B parameters.Two attempted models did not meet inclusion: one had no successful inference, while another had five runs across two prompts.Source: §3 · p. 3
  • The synthesized corpus contains 83 annotated prompts, but campaigns measured 22 distinct files and persisted raw IRs for 20; a second corpus contains 40 real-world prompts.The reported table and drift analysis rest on 22 and 20 prompts respectively, not 83.Source: §3 · p. 3
  • Node-set Jaccard measures mean pairwise agreement, while ID-stability requires agreement across all runs; node-set-perfect means both metrics equal 1, not identical IRs.The metrics omit metadata and hierarchy differences, which can persist even when node-id sets agree.Source: §3 · p. 3
  • When Jaccard is below 1, disagreements are classified as id-label, count, or metadata drift; the last category concerns per-node metadata, chiefly source ranges.The emitted IR is a flat node list, so calling this category hierarchy drift would overstate what it measures.Source: §3 · p. 3
  • Means use 95% percentile bootstrap intervals, paired within-family tests use Wilcoxon signed-rank tests, and ranking stability uses a joint cluster bootstrap.The joint bootstrap resamples one prompt multiset per replicate and scores every model on that same draw.Source: §3 · p. 3

4 Does the measurement reproduce?

The section tests whether prompt-structure reproducibility supports model comparisons, then examines model dependence, recovery on unannotated prompts, and the kinds of structure that drift. Its findings show substantial variation, but the strength and interpretation of comparisons depend on the panel and measurement regime.

4.1 How reproducible is it?

  • Mean node-set Jaccard ranged from 0.39 to 0.96, but only 35 of 127 prompt-model cells (28%) were node-set-perfect.Perfect reproducibility varied by model: gemma3:12b was perfect on 68% of its prompts, while qwen3-coder-next was perfect on none of 20.Source: §4.1 · p. 4

4.2 Model dependence, and scale within a family

  • The model-mean spread was 0.58 (standard deviation 0.18), but family scaling was mixed: OpenAI-style worsened, Mistral showed a nonsignificant gain, and MiniMax was uninformative.On 18 common prompts, gpt-oss:120b scored 0.386 versus gpt-oss:20b’s 0.766; Mistral’s larger model was higher by 0.061 (p = 0.37), while the MiniMax comparison shared only two prompts.Source: §4.2 · pp. 4–5

4.3 Transcription versus recovery

  • Across 196 recovery cells, mean node-set Jaccard was 0.22 and 30 cells (15%) scored zero, but differences from the annotated regime cannot be isolated to recovery.The regimes also differ in corpus, prompt content, model panel, and completion pattern; only four models appear in both, and the data support a level difference, not transfer.Source: §4.3 · p. 5
  • Recovery’s exact-id mean rose only from 0.22 to 0.25 after normalization, while count agreement was 0.70; the residual disagreement’s meaning remains uncertain.The ASCII-only normalizer can collapse distinct ids, and it does not address synonyms.Source: §4.3 · p. 5

4.4 What drifts

  • Across 98 cells with persisted raw IRs, count disagreements dominated drift at 49%, compared with 19% id-label drift and 31% metadata drift.Because models disagree about node counts, canonicalising ids would address under a fifth of the drift; this panel includes glm-4.7 and excludes gpt-oss:20b.Source: §4.4 · p. 5

5 Do the conclusions reproduce?

Section 5 tests whether the evaluation's conclusions survive resampling and defensible analysis changes. The audit finds that stabilization claims depend on flawed specifications and that the full model ranking is not robust.

5.1 How many reruns a consumer needs

  • The stabilization criterion always passes at r = R, making “did not stabilise” unreachable in 48 of 98 cells.The documented relative tolerance was implemented as an absolute 0.05 nodes; the two readings yield 80 versus 50 stabilising cells, and the data do not distinguish them.Source: §5.1 · p. 6
  • r = 1 is the median among cells stabilising strictly before their endpoint under both readings, but downstream tool outcomes were not measured.The study therefore does not support a recommendation that any fixed number of reruns suffices.Source: §5.1 · p. 6

5.2 Does the ranking survive resampling and analysis choices?

  • 99% and 86% of bootstrap replicates retain gpt-oss:120b last and qwen3-coder-next seventh, while the middle four retain rank 27%–48% and the top two 68% each.The table supports identifying the least reproducible model, but not reliably identifying the most reproducible.Source: §5.2 · p. 6
  • The paper introduces no new rank-uncertainty estimator; its contribution is applying uncertainty analysis within an end-to-end audit of a concrete evaluation.The joint cluster bootstrap draws one prompt multiset per replicate and scores every model on that same draw.Source: §5.2 · p. 6
  • 7.1 percentage points is the study-wide headline shift under the alternative campaign-merging rule; four of eight models change rows.Three models change rank, while qwen3-coder-next remains seventh.Source: §5.2 · p. 7
  • Two rows rely on n = 3 and n = 6, and the 28 implied pairwise comparisons are uncorrected; most models remain undetermined.The authors recommend reading the sorted table as broad groups rather than as support for every pairwise comparison.Source: §5.2 · p. 7

6 Reproducibility is not validity

The section compares reproducibility with agreement against annotations and finds that the two do not provide interchangeable evidence. Stability and correctness order models differently, so reproducibility alone cannot serve as an accuracy ranking.

6 Reproducibility is not validity

  • Mean recall of annotated node ids is 0.86, while mean precision on nodes marked explicit is 0.82.About one in five nodes a model asserted it copied has no corresponding annotation.Source: §6 · p. 7
  • Stability and correctness order models differently; their association ranges from Pearson r = 0.69 under all-node precision to r = 0.90 under explicit precision.None of the four reported correlations survives removal of the single low outlier, and the metrics use different weighting units.Source: §6 · p. 7

7 Reproducibility over time

The authors find that four of eight studied endpoints were withdrawn soon after measurement, making the study as specified no longer executable. They also withdraw an earlier claim of structural inaccessibility because the instrument could not establish its cause.

7 Reproducibility over time

  • Four of the eight studied variants were withdrawn on 2026-07-15, and the study as specified is no longer executable.The second-ranked gemma3:12b endpoint, used in the annotated real-corpus comparison, cannot be queried by the authors or a reviewer.Source: §7 · p. 7
  • A fixed 60,000 ms client timeout logs slower responses as connection failures, while reasoning traces can push models past the deadline.In a probe of 11 currently served models, eight failed with a tight output budget and returned valid JSON with room; reasoning volume did not cleanly divide the groups.Source: §7 · pp. 7–8
  • The authors withdraw the accessibility finding: the cause is unresolved, the historical failure was not reproduced, and the instrument remains the leading candidate.The probe used retired models and a simplified extraction prompt rather than the typed IR request.Source: §7 · p. 8
  • Ten of fifteen named models were gone by the audit, while 293 persisted raw IRs let several analyses regenerate offline.The authors recommend retaining raw per-run outputs and dating measurements when evaluating hosted endpoints.Source: §7 · p. 8

8 Threats to validity

The threats section separates instability in sampled outputs from defects in the instrument and analysis specification. It also limits interpretation to the measured setting, panel, and node-based metrics.

8 Threats to validity

  • Run disagreement and rank movement concern inference from sampled outputs; timeout and stabilization defects concern the instrument or analysis specification.The authors treat these defects as evidence of pipeline failure under audit, not intrinsic LLM nondeterminism.Source: §8 · p. 8
  • All measurements use one commercial endpoint over a bounded period, and the authors cannot verify which weights answered under each model name.A 60-second timeout can itself manufacture an availability finding, while silent revision under a stable name could resemble nondeterminism.Source: §8 · p. 8
  • Only four models appear in both the synthesised and unannotated panels, and annotation status, corpus, prompt, model panel, and completion pattern all change together.The cross-regime ordering therefore rests on those four models and confounded setting differences.Source: §8 · p. 8
  • The study tests neither a standard benchmark nor downstream RAG, agent, or code-generation tasks, so it does not estimate how often these failures occur elsewhere.Its node-id metrics omit metadata disagreement, correctness covers explicit annotations only, and run counts and completion rates vary across models.Source: §8 · pp. 8–9

9 Related work

The paper places its prompt-structure task alongside work on prompt programs and evaluation robustness, while distinguishing its contribution from the established observation that LLM outputs vary. Its conclusion reports unstable structure and recommends more transparent, sensitivity-tested evaluation reporting.

9 Related work

  • Prior work documents non-determinism and proposes consistency measures; this paper’s contribution is the task, cross-family design, and release of raw IRs.The authors do not claim to have discovered LLM output variation.Source: §9 · p. 9
  • Prior prompt-robustness studies examine format or demonstration-order perturbations; this audit examines conclusions changing across resamples when inputs are unperturbed.Source: §9 · p. 9

10 Conclusion

  • Jaccard spans 0.39 to 0.96 and 72% of cells are not node-set-perfect; only one of seven audited claims and assumptions remains strongly supported.The conclusion says the audit found every failure without new data.Source: §10 · p. 9
  • The authors recommend publishing per-cell provenance, executing sensitivity comparisons, stating merge rules and tolerance units, reporting rank stability, and dating leaderboards.They also report releasing the corpus, 293 raw IRs, and scripts.Source: §10 · p. 9

A Model availability audit

The availability audit found that half of the evaluated variants could no longer be queried. Additional checks characterize correctness relative to annotations and document incomplete, non-random recovery-campaign coverage.

A Model availability audit

  • Four of the eight evaluated variants and ten of the fifteen models named in the study could no longer be queried as of 2026-08-17.The audit records its re-query date and the raw output location.Source: Table 3 · p. 9, A Model availability audit · p. 9

B IR correctness against ground-truth annotations

  • Explicit precision scores only nodes marked explicit, while F1 over all inferred nodes also penalizes implicit rules and is not the harmonic mean of adjacent columns.The passage identifies recall and explicit precision as the interpretable columns.Source: B IR correctness against ground-truth… · p. 10
  • No stability–correctness correlation survives removal of the low outlier at α = 0.05, so the authors claim only that the model orderings diverge.The analysis covers seven models and identifies gpt-oss:120b as the low outlier.Source: B IR correctness against ground-truth… · p. 10

C Stabilisation in full, and the recovery-campaign completion counts

  • Of 627 planned campaign outputs, 557 persisted; 196 cells with at least two runs were analyzed, and missing outputs were not missing at random.Models were more likely to fail again on prompts where they had failed before, mildly biasing the analyzed panel toward completing models.Source: C Stabilisation in full, and the… · p. 10

C.1 How many reruns a consumer needs

The rerun analysis exposes endpoint and tolerance-specification problems, leaving limited support for a general rerun recommendation. A separate budget probe compares model outputs under tight and generous output limits.

C.1 How many reruns a consumer needs

  • The measured signal is IR node count, not a downstream linter, coverage, or mutation result; transfer to real tools depends on their output tracking node count.The study does not establish that relationship.Source: §C.1 · p. 10
  • At r = R, the stabilisation criterion always passes by construction; this accounts for 48 of 98 cells when R ranges from 2 to 5.The passage identifies right-censoring r = R as the minimum fix.Source: §C.1 · p. 10
  • The documented relative tolerance was implemented as an absolute 0.05 nodes; cells stabilising before the endpoint number 50 versus 80 under the absolute and relative readings.The relative interpretation changes censored cells from 48 (49%) to 18 (18%).Source: §C.1 · pp. 10–11
  • Among cells stabilising strictly before the endpoint, the median is r = 1 under both tolerance readings; the design does not support a claim that a fixed number of reruns suffices.Source: §C.1 · p. 11

D Failure attribution: harness budget versus model

  • Under the tight 1,500-token condition, eight models returned invalid JSON from length stops, while all returned valid JSON with room; the tight condition was not the historical harness setting.The generous condition allowed 8,000 tokens; the historical harness imposed no token cap.Source: D Failure attribution: harness budget… · p. 11

E Merge-rule sensitivity, full table

The merge-rule table compares node-set-perfect rates across two campaign-precedence rules. The tolerance table contrasts absolute and relative readings, which produce different pre-endpoint stabilisation counts.

E Merge-rule sensitivity, full table

F Stabilisation under both tolerance readings

Related papers

Loading 2609.30074v1…