Source-linked AI summary
What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects
Bowen Xu, Boyu Chen
TL;DR
Harness-search gains around frozen models may reflect newly readable answers, weak baselines, or evaluation defects. This preregistered study tests those possibilities across three small models, three benchmark pools, and six injected defects. Most thinking-off gains came from rows the capped setting left unparsed, while the results also show limits to replication as a defect check.
Problem
Harness-search score gains may reflect newly readable answers rather than better answers, inadequate baselines, or defects in evaluation inputs, labels, splits, or served models.
Source: §1 · p. 1Method
The preregistered study compares harness settings across three small models and three benchmark pools, uses replication and test partitions, runs GEPA search, and injects six evaluation defects one at a time.
Source: §1 · p. 2, §3.1 · p. 4, §3.4 · p. 5Results
Most thinking-off gains over capped thinking came from rows the capped setting left unparsed; thinking-off was non-inferior in 11 of 13 comparisons, while rescue won on GSM8K for two models.
Source: §6 · p. 12, §7 · p. 13Takeaways & Limitations
Reporting truncated and non-committed answers separately, running thinking-off and constant-label baselines, reading inputs, and reconciling prompt tokens per call exposed defects in this study.
Source: §7 · p. 13Findings cover three small models, three pools, and six defects from one line of work, so H4 does not establish how replication behaves across the full space of defects.
Source: §6 · p. 12
Abstract
from arXiv · showhide
Harness search keeps a change to the prompts, reasoning switches, token budgets or parsers around a frozen model if the change raises a score. Such a gain can come from answers the parser could not read before, a weak comparison, or a defect in the evaluation. We preregistered a study of where these gains come from, with three small models, three benchmarks, replication and test partitions, a GEPA search arm and six evaluation defects injected one at a time, and we report all 47 primary endpoints. Turning thinking off raised accuracy over a capped thinking setting in 5 of 9 model-benchmark cells, and in each the gain came mostly from questions where the capped setting gave no readable answer. The thinking-off setting was not meaningfully worse than a rescue configuration or four GEPA-selected harnesses in 11 of 13 comparisons, and lost to the rescue on GSM8K for two models. GEPA repaired its broken starting points, but none of its selected harnesses was more accurate than the thinking-off setting. A thinking budget in the serving engine, which also allows a longer answer, lowered truncation and raised the parse rate in 6 of 9 cells. In 6 of 15 evaluable defect-model pairs, replication through the same pipeline reproduced the defect's distortion instead of revealing it. On the LongevityBench multiple-choice tasks, only the longevity-tuned model beat the strongest constant-label baseline.
1 Introduction
The paper asks whether harness-search gains reflect improved answers, newly readable answers, inadequate baselines, or evaluation defects. It preregisters a study of these questions across small models, benchmarks, and injected defects.
1 Introduction
- Harness gains can reflect visibility rather than answer quality, weak comparisons, or invalid evaluation inputs, labels, splits, or served models.A gain that replicates on untouched rows shows the harness beat its parent by more than selection noise, but does not resolve these questions.Source: §1 · p. 1
- The authors identify a gap in prospective tests that quantify visibility, compare against benchmark-author protocols with a margin, and inject defects individually.The literature search was bounded by a 200-search cap, a 2024-01 to 2026-10-03 window, and rate-limit resumptions.Source: §1 · pp. 1–2
- A prior longevity-model case study found that a defective caller passed reference answers to the model, so replication through that caller could not detect the defect.After correction, the starting configuration parsed none of 556 prospective rows, while an unsearched thinking-off setting was non-inferior to the rescue at less than half the token cost.Source: §1 · p. 2
- The preregistered study uses three small models, three benchmark pools, replication and test partitions, GEPA search, and six individually injected evaluation defects.It reports all 47 primary endpoints and predefines a visibility split, dominance statistic, gain gate, and comparisons with rescue and GEPA harnesses.Source: §1 · p. 2
- The results are limited to three small models, three pools, and six defects, and the authors do not claim harness search finds genuine quality gains.They also do not claim that the possibility of illusory gains, thinking-off wins, or preregistered agent and harness evaluations is new.Source: §1 · p. 2
2 Related work
Related work examines whether harness gains are real through visibility, baseline adequacy, budget controls, evaluation defects, and served-model identity. The paper positions its contribution as prospective tests with fixed decision rules, alongside a controlled comparison of GEPA starts.
Harness search and whether its gains are real
- Prior work reports harness and scaffold gains that disappear against matched-budget scaling or simple baselines, completion-driven gains, defective seeds, and evaluator tampering.This paper narrows the question to where one protocol’s gain comes from and whether known defects are detected, with questions and decision rules fixed in advance.Source: §2 · p. 2
Visibility and quality (H1)
- Prior visibility analyses separate content and format recovery or use completion-rate identities; this study prospectively applies a dominance statistic and gain gate to thinking-off versus capped thinking.The split follows Chen et al. (2026), while the predefined dominance threshold is an added feature.Source: §2 · pp. 2–3
Baseline adequacy (H2)
- Earlier studies find simple baselines can match searched systems; this study fixes a non-inferiority margin for the benchmark authors’ protocol against a derived rescue and four GEPA harnesses.The approach relates to prior preregistered non-inferiority and equivalence tests for agent memory, scaffolds, and compression.Source: §2 · p. 3
Budget forcing (H3)
- Budget forcing has served as an evaluation control; this study tests the serving engine’s thinking-token budget against a capped configuration, rather than claiming the technique as new.Prior work includes forcing an answer or end of thinking and forcing an answer delimiter at a cap.Source: §2 · p. 3
Defects and prospective replication (H4)
- Prior work catalogs pipeline defects and injects other known faults, but the authors found no study injecting pipeline defects individually and testing whether same-pipeline replication reproduces distortion.Examples include answer-key access, label defaults, parser fallbacks, stop sequences, denominators, and gold errors.Source: §2 · p. 3
Served-model identity and determinism
- Serving stacks can alter outputs at fixed weights, and served routes need not match model identifiers; this study adds a measured instance of that mismatch.The vLLM version used loads a stored output head after a fix to an earlier configuration-based tying decision.Source: §2 · p. 3
GEPA from a broken start
- A prior study found GEPA failed to repair an official GSM8K seed with reversed answer-and-reasoning order; this paper compares clean and broken starts.Why a search repairs one defect but not another is not tested here.Source: §2 · p. 3
Preregistration and prospective design
- The paper combines a preregistration freeze, pilot, replication then test, adaptive search before replication, complete endpoint reporting, and an exposure log.Thomas et al. (2026) confirm on the first eligible model released after the plan, a stronger standard because this study’s freeze guards rules, not model-specific tuning.Source: §2 · p. 3
3 Design
The design freezes its rules in advance and evaluates harness configurations, visibility, baseline adequacy, and injected defects across models, pools, and partitions. It pairs comparisons on rows and uses family-adjusted bootstrap intervals for confirmatory verdicts.
3 Design
- The study follows a frozen preregistration, with its hash and time recorded in Appendix E.The freeze occurred before replication or test items were called; the pilot called 700 items.Source: §3 · p. 4, §3.5 · p. 6
3.1 Setting
- The study covers three models and three benchmark pools, split into search, replication, and test partitions, with LongevityBench split by participant component.The test partition contains 3,942 LongevityBench letter rows, 799 MMLU-Pro items, and 527 GSM8K items; GEPA searches only on the search partition.Source: §3.1 · p. 4
3.2 Configurations and scoring
- Table 1 lists six grid configurations and GEPA harnesses, with max_tokens and thinking on/off set in every request; calls are unparsed when the scorer cannot read an answer, truncated at finish reason length, or non-committed when stop-ended and unparsed.RESCUE-1500 is derived from the two grid runs and verified against the real two-call sequence on 300 items per model.Source: §3.2 · p. 4, Table 1 · p. 5
3.3 Hypotheses and endpoints
- The confirmatory analysis has 47 primary endpoints, with hypothesis verdicts based on family-adjusted intervals and 0.05 minimal-difference and non-inferiority margins.Table 2 states the hypotheses, estimands, and verdict rules.Source: §3.3 · p. 4, Table 2 · p. 5
3.4 Defects, claims and checks (H4)
- Six defects are injected individually; for each, distortion is measured on replication and test samples, with a predefined rule for whether replication confirms it.Defects include reference-answer exposure, gold fallback, unparsed-as-A scoring, a skewed split, a broken start, and a tied output layer; six audit checks examine signatures and clean-instance false alarms.Source: §3.4 · pp. 5–6
3.5 Statistics and what was fixed in advance
- Contrasts are paired on rows and evaluated with 50,000 unit-level bootstrap resamples; verdict intervals are Bonferroni-adjusted within endpoint families.Units are participant components for LongevityBench and items for MMLU-Pro and GSM8K; percentile-bootstrap coverage is approximate, not guaranteed.Source: §3.5 · p. 6
4 Results
Across the 47 primary endpoints, the results show when harness changes improve readable-answer visibility, how the thinking-off baseline compares with rescue and GEPA harnesses, and whether defects persist under replication. They also report audit-check behavior, baseline comparisons, and sensitivity to analysis choices.
4.1 The 47 primary endpoints at a glance
- 30 of 31 H1–H3 endpoints replicated, one replication was inconclusive, and none was labelled not replicated.The endpoints are the study’s confirmatory results; H4 adds 16 defect–model pairs.Source: §4.1 · p. 6, §4 · p. 6
4.2 H1: visibility dominates where the thinking cap truncates
- In 5 of 9 cells, AUTHORS-1500 gained over THINK-512 through visibility, with ∆ from +0.060 to +0.403.In those cells, ∆V was at least ∆; rows parsed by both configurations contributed at most ∆B ≤ +0.001.Source: §4.2 · pp. 6–7, Table 3 · p. 7, Figure 1 · p. 7
4.3 H2: the thinking-off baseline is non-inferior in most cells and fails on GSM8K for two models
- AUTHORS-1500 was non-inferior to RESCUE-1500 in 7 of 9 cells, but lost on GSM8K for SmolLM3-3B and Intern-S1-mini.The three non-seed GEPA selections scored 0.620 to 0.631 against 0.632 for AUTHORS-1500, and their adjusted intervals contained zero.Source: §4.3 · pp. 7–8, Table 3 · p. 7
4.4 H3: BUDGET-512-1500 lowers truncation where THINK-512 truncates
- In the six cells where THINK-512 often truncated, BUDGET-512-1500 lowered truncation and raised parsing, with DT from +0.260 to +0.979 and DP from +0.151 to +0.662.The comparison changes both the thinking limit and total allowance, so H3 supports the combined change, not forced termination alone.Source: Figure 1 · p. 7, §4.4 · p. 8
4.5 H4: prospective replication confirms the distortion in six of fifteen pairs
- Replication reproduced large distortion in 6 of 15 evaluable defect–model pairs; four claims failed, five were inconclusive, and one pair was not evaluable.The six holds included the broken start in all three models; in the other pairs, distortion could be too small, too uncertain, or not evaluable.Source: §4.5 · pp. 8–10, Table 4 · p. 9, Figure 2 · p. 9
- On clean instances, IR4 fired on 7 of 7, SCF1 and SCF2 on 4 each, TB2 on 3, SCF3 on 2, and PR’s gain rule on 1.The detection matrix was descriptive and largely true by construction; these clean-instance false alarms were the empirical check result.Source: §4.5 · p. 10, Table 5 · p. 10
4.6 Descriptive results
- On LongevityBench letters, Qwen3-1.7B-Longevity beat the 0.539 strongest constant in every parsing configuration (AUTHORS-1500: 0.632, +0.093 [+0.074, +0.112]; THINK-512: +0.096 [+0.077, +0.115]); Intern-S1-mini also beat it on LB-0034 with AUTHORS-1500 (+0.060 [+0.003, +0.117]).Source: §4.6 · p. 10
4.7 Sensitivity analyses
- Across the ten descriptive sensitivity analyses, the strict scorer changed 2 of 31 endpoint verdicts and restricting to never-called rows changed 3.Putting the left-out rows back changed no H1–H3 verdicts and changed two H4 claims.Source: §4.7 · p. 11
5 Deviations and addenda
After the preregistration freeze, the primary criteria and scoring stayed unchanged, with two non-mechanical exceptions; the audit recomputed every verdict and agreed with the report.
5 Deviations and addenda
- Two non-mechanical exceptions followed the freeze: one stopping rule was waived for a single stage, and one descriptive label was settled after outputs were seen.The general rule governed the check-sample claim for the skewed split in SmolLM3-3B; the paper also reports the narrower reading’s alternative H4 total.Source: §5 · p. 12
- No replication or test item was called before the freeze, and no main-study accuracy, score, or verdict was seen until all stages had run.The paper lists result-bearing deviations and exceptions with dates, reasons, rules, and outputs seen at the time.Source: §5 · p. 12
- The audit recomputed every verdict from its interval and agreed with the report.Tables and figures format aggregates committed in the authors’ repository.Source: §5 · p. 12
6 Discussion and limitations
The study finds that thinking-off gains mostly reflect previously unparsed answers, while baseline adequacy varies by cell; its conclusions are bounded by the models, pools, defects, and evaluation settings studied.
6 Discussion and limitations
- In 6 of 15 evaluable defect-model pairs, replication reproduced a large distortion within five points of the selection-sample distortion; some defects therefore persisted across samples.The checks that exposed defects read inputs and scoring, but their detections were largely true by construction and several checks produced false alarms.Source: §6 · p. 12
- Thinking-off gains over THINK-512 mostly came from rows THINK-512 left unparsed, and none of GEPA’s selected harnesses was more accurate than the thinking-off baseline.The rescue beat thinking-off by more than the margin on GSM8K for two models, so the baseline should be run but not assumed adequate.Source: §6 · p. 12
- The study covers three small models, three pools, and six defects from one line of work, so its H4 result does not represent the full space of defects.Accuracy counts unparsed answers as wrong, mixing parsing and correctness; the study measures configurations as served, not what models know.Source: §6 · p. 12
- The paper’s small-model research-task accuracies do not support medical advice or clinical decisions.It also cautions that thinking-off is not always an adequate baseline, as two cells show it is not.Source: §6 · p. 13
7 Conclusion
The study concludes that thinking-off is a useful baseline to test, not one to assume adequate: its gains mostly reflect parsing recovery, and rescue configurations outperform it on GSM8K for two models.
7 Conclusion
- In 6 of 15 evaluable defect-model pairs, a large selection-sample distortion reappeared within five points on the check sample.Replication through the same pipeline can reproduce a distortion rather than reveal it.Source: §7 · p. 13
- Most thinking-off gains over THINK-512 came from rows THINK-512 left unparsed, and the thinking-off setting was non-inferior in 11 of 13 comparisons with the rescue and four GEPA-selected harnesses.The rescue beat it on GSM8K for two models.Source: §7 · p. 13
- Thinking-off and constant-label baselines, separate reporting of truncated and non-committed answers, input reading, and per-call prompt-token reconciliation exposed defects in this study.The paper describes these checks as low-cost.Source: §7 · p. 13
Code and data availability
The paper’s repository is not currently public; ancillary files include the frozen preregistration and aggregate analysis, while a filtered export is planned.
Code and data availability
- The study repository is not public at present, and run logs containing benchmark item text and model outputs will not be released.The paper retains paths, commits, tags, and hashes so a later public export can be checked against them.Source: Code and data availability · p. 13
- The frozen preregistration and aggregate analysis report are attached byte for byte, but the freeze time is not independently timestamped.The attached SHA256 can verify the ancillary copy against the frozen file, not establish when it was frozen.Source: Code and data availability · p. 13
- A filtered public export of code, tests, and hash manifests is planned; benchmark and model use follows their stated licences and the study redistributes neither item text nor weights.The export is planned for a later paper version and will exclude item-level and row-level content derived from held-out rows.Source: Code and data availability · p. 13
A Background: the case study this study follows
The earlier harness search found a replicated parse-rate gain, but later audits identified both an inadequate comparator and a caller defect. This study re-evaluates those lessons with fixed rules and tests across new configurations and partitions.
A Background: the case study this study follows
- The earlier search’s B1 rescue improved parse rate on a second shard, but a later audit found the caller had supplied reference answers and thinking-off was an equally accurate simpler comparator.Prompt-token reconciliation revealed the reference-answer turn; after correction, the parent parsed none of 556 holdout rows, making the reliability endpoint nearly guaranteed.Source: A Background: the case study this… · p. 17
- The corrected re-evaluation concluded that replication controlled selection noise but detected neither the inadequate comparator nor the evaluation defect; trivial baselines and input audits did..Source: A Background: the case study this… · p. 17
B.3 Configurations
- Table 7 connects lessons from the case study to the tests in this study, while the configuration tables distinguish primary endpoints from sanity checks and descriptive comparisons.The study uses new stack, model, and pool combinations and fixes its rules in advance, though its pilot had already shown where the parent truncates.Source: Table 7 · p. 17, A Background: the case study this… · pp. 17–18, Table 9 · p. 18
B.1 H1 to H3
- The supplied captions identify H1 primary endpoints (Table 8), H2 primary and descriptive comparisons (Tables 10–11), H3 primary endpoints (Table 12), H4 adjusted pairs (Table 13), and a descriptive detection matrix (Table 14).Source: Table 8 · p. 18, Table 10 · p. 18, Table 11 · p. 19
B.4 Trivial baselines, computed after the outputs existed
This appendix defines the trivial baselines and reports descriptive comparisons, sensitivity checks, deviations, and decisions made before the freeze. The reported baseline for the letter-task pool is stronger than a constant letter alone.
B.4 Trivial baselines, computed after the outputs existed
- For each task, the strongest baseline is the larger of its two oracle constants; reported pool values are row-weighted means, and table cells show accuracy and paired difference to that baseline.Chance is the mean of 1/k over rows, where k is the number of options.Source: §B.4 · pp. 21–22, Table 17 · p. 22
B.5 Other descriptive items
- The descriptive tables report seeded parsing and truncation, tied-versus-untied output layers, GEPA runs, age regression, sensitivity verdict changes, and never-called-row composition.The table captions identify these analyses but do not give their outcome values.Source: Table 21 · p. 24, Table 23 · p. 24, Table 24 · p. 25
C Deviations, exceptions and addenda in full
- The deviations table separates entries bearing on results from other exceptions, deviations, pre-freeze decisions, and post-freeze addenda.The “Seen” field records what study outputs had been seen when an entry was made.Source: C Deviations, exceptions and addenda… · pp. 25–26, Table 28 · p. 26
C.1 Exceptions accepted before the freeze (X1–X9)
- Accepted exceptions included a permitted FreeToken listener and a restart with no server running; the listed passages do not indicate that either affected calls.The listener carried no model traffic, and the restart left the GPU identifier and SSH settings unchanged.Source: §C.1 · pp. 26–27
C.2 Known deviations from the plans (K1–K9)
- Known deviations included three models instead of four and a local GEPA proposer replacing Meta-Harness; the provided entries also note changes to answer allowance and server setup.The passages identify deviations from the plan, not their effect on results.Source: §C.2 · p. 27
C.3 Decisions between the first draft and the freeze
- Before the freeze, H1’s primary set changed twice after pilot parse-rate and parsed-answer results and a review; the stated changes did not use accuracy.The scoring defect was split, the tied-output-layer defect redesigned, and the H4 family fixed at 16 with a 0.05 margin and within-family Bonferroni adjustment.Source: §C.3 · pp. 27–28
- Seven replication rows whose reference answers could be recovered from mistakenly uploaded render-hash lists were excluded from every analysis.The exclusion followed the final Codex review and was not reviewed there.Source: §C.3 · p. 28
C.4 Addenda after the freeze (A1–A12)
Post-freeze addenda document execution and analysis decisions, including an instance-only analysis run and descriptive calculations made after outputs existed. A later rule clarification changed one H4 verdict, and the appendix contrasts preregistered expectations with observed outcomes.
C.4 Addenda after the freeze (A1–A12)
- The post-freeze analysis ran once on the instance after pinned files matched their frozen hashes; three descriptive items were computed after outputs existed.Selections matched the locked selections on all 24 comparisons.Source: §C.4 · p. 28
C.5 The rule for DR (addendum A11)
- For defect C in SmolLM3-3B, the general DR boundary rule gives a descriptive fail rather than an inconclusive result, changing H4 counts to 6 holds, 4 fails, and 5 inconclusive.All other claims retain the same verdict under either reading of the rule.Source: §C.5 · p. 29
C.6 What this paper adds
- The paper presents aggregates committed in the authors’ repository and recomputes each verdict from its interval, agreeing with the report on every endpoint that has a verdict.The appendix says the paper adds no computation beyond table counts and this independent verdict check.Source: §C.6 · p. 29
D Expected in advance against observed
- The expectations table compares preregistered expectations with observed outcomes, noting that H1 and H3 directions were visible in the pilot and that “inconclusive” expectations were informal.Those expectations were based on planned power under assumed effects, not a prediction with a decision rule.Source: D Expected in advance against observed · p. 29, Table 29 · p. 30
E Reproducibility
The reproducibility appendix records the frozen analysis and code hashes, but the repository is not currently public and run logs will not be released. It also documents held-out-material contacts, serving procedures, and limits on the related-work search.
E Reproducibility
- The repository is not public, and run logs containing benchmark item text and model outputs will not be released.The paper gives hashes and paths for checking a later public export, but the named files cannot currently be opened by readers.Source: E Reproducibility · pp. 30–31
F Exposure and disclosure
- Seven replication rows were excluded after a mistaken upload of render-hash lists made their reference answers recoverable; the authors report no other replication or test partition disclosure this way.The reader’s statement that the lists were not opened or used for reconstruction was relayed by an author and not independently checked.Source: F Exposure and disclosure · p. 32
G Serving details
- The serving appendix reports per-run calls, reasoning-split disagreements, role breaks, and mean tokens and seconds; the main study used pinned vLLM settings and checked rendered prompts against local renders.The server table includes all called rows, including flagged rows and LB-0038.Source: G Serving details · p. 33, Table 31 · p. 34
H How the related work was searched
- The related-work search used topic queries across six slices, but a 200-search cap and rate limits stopped three slices before searches resumed only for write-up.“We found no study that” means none was found, not that none exists; most recent harness-evolution work was unreviewed preprint material.Source: H How the related work was searched · p. 34
I H4 in detail
This appendix details H4’s defect comparisons, claim rules, and limits, alongside a prompt-only GEPA search documenting four selected runs. It also describes a seed-one harness that directs age-related survival judgments toward selected biomarkers and metabolic profiles, requires a binary answer, and restricts reasoning to supplied data and thresholds.
I H4 in detail
- H4 defines claim status using selection- and check-sample distortions and their difference, with a 0.05 margin and adjusted intervals for a family of 16.A pair is evaluable only when its mechanism-based bound exceeds 0.05; otherwise it makes no claim and is not counted as failing.Source: I H4 in detail · pp. 35, 37
- C’s validation halves draw from both partitions, so they are neither the other pairs’ replication and test rows nor temporally prospective.The C split is computed after the test calls.Source: I H4 in detail · p. 36
- Rules PR and every interval following selection on S are descriptive labels, not tests; selection makes accS(c*) and the gain optimistic.Source: I H4 in detail · p. 37
J The GEPA arm in detail
- GEPA edited only system prompts and appended instructions, leaving request parameters fixed; four runs used clean or broken starts and seeds zero or one.The 300-row search used three letter tasks, and selected harnesses were recorded before replication calls.Source: J The GEPA arm in detail · p. 38
GEPA-CLEAN-s0 (clean start, GEPA seed zero)
- 0.680 was GEPA-CLEAN-s0’s mean validation score; no proposal was accepted, so its selected harness remained the thinking-off seed.The seed was candidate index 0 among 31 candidates.Source: J The GEPA arm in detail · p. 38, GEPA-CLEAN-s0 (clean start, GEPA seed… · p. 39
GEPA-CLEAN-s1 (clean start, GEPA seed one)
- 0.700 was GEPA-CLEAN-s1’s mean validation score, and candidate 12 of 31 was selected.Selection used the highest mean validation score, with ties resolved by lowest candidate index.Source: J The GEPA arm in detail · p. 38, GEPA-CLEAN-s1 (clean start, GEPA seed… · p. 39
GEPA-BROKEN-s0 (broken start, GEPA seed zero)
- 0.660 was GEPA-BROKEN-s0’s mean validation score, and candidate 1 of 24 was selected.The selected harness was chosen by mean validation score, with ties resolved by lowest candidate index.Source: J The GEPA arm in detail · p. 38, GEPA-BROKEN-s0 (broken start, GEPA… · p. 40
GEPA-BROKEN-s1 (broken start, GEPA seed one)
- 0.667 was GEPA-BROKEN-s1’s mean validation score, and candidate 1 of 29 was selected.The selected harness was chosen by mean validation score, with ties resolved by lowest candidate index.Source: J The GEPA arm in detail · p. 38, GEPA-BROKEN-s1 (broken start, GEPA… · p. 41
- The prompt prioritizes red blood cell indices, hemoglobin trends, Cystatin C, Vitamin D, and age-specific metabolic norms for age questions.It specifically names mean corpuscular volume, glucose, and cholesterol.Source: GEPA-BROKEN-s1 (broken start, GEPA… · p. 42
- It requires answers to be only “A.” or “B.”, with no explanation or partial answer, aligned with calculated risk scores and biomarker thresholds.Source: GEPA-BROKEN-s1 (broken start, GEPA… · p. 42
- Examples map higher-risk profiles to “B. No” and lower-risk profiles to “A. Yes” for 10-year survival.The prompt also says to use only provided data and thresholds, not external knowledge or specific study results.Source: GEPA-BROKEN-s1 (broken start, GEPA… · p. 42
K Planned power
Planned power was estimated from preregistration assumptions for H1 and H2, with no power planned for H4 or the GEPA endpoints.
K Planned power
- Five planned-power figures are rough preregistration estimates from pilot parse shares and assumed effects, not recomputed after the outputs.The paper’s “expected to be inconclusive” label is an informal reading of those figures, not a prediction with a decision rule; observed verdicts do not test it.Source: K Planned power · p. 42
- For H2, Intern-S1-mini on GSM8K against RESCUE-1500 had planned power of 0.19 adjusted and 0.52 unadjusted at θ = 0.The assumed truth was that thinking-off accuracy equaled rescue accuracy; the estimate used n = 527 test items and a non-inferiority margin of 0.05.Source: K Planned power · p. 42
- H1 power used pilot joint parse shares and two assumed accuracy scenarios, without using pilot gold answers or accuracy.Its planned test was whether the adjusted interval’s lower bound for T exceeded zero, using a normal approximation on 527 GSM8K test items.Source: K Planned power · p. 42
- No power was planned for H4’s distortion measurements or the four GEPA endpoints, whose sizes depended on harness answers unavailable before search.Source: K Planned power · p. 43
Related papers
This paper cites
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- Large Language Models as Optimizers2023
- Rethinking the Evaluation of Harness Evolution for Agents2026
- ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement2026
- HarnessOpt-Bench: Evaluating LLMs at Harness Optimization2026
- Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention2026