PaperlayerView exact revision on arXiv

Source-linked AI summary

Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations

Toby D. Pilditch

arXiv:2608.14425v1cs.AI

TL;DR

LLM evaluations repeatedly test many items, creating substantial computational costs even after estimates become precise. optstop adaptively stops sampling when uncertainty is sufficiently reduced, eliminating most planned trials while preserving score fidelity across validation settings.

  • Problem

    Repeated testing across large evaluation suites creates substantial computational burden for obtaining stable language-model performance estimates.

  • Method

    optstop uses hierarchical Bayesian credible-interval widths and stabilization criteria to stop sampling individual items and model-task groups at user-specified precision.

  • Results

    Across a 3 × 3 validation matrix, optstop eliminated the majority of planned trials while preserving score fidelity and statistical equivalence, with an 81.1% mean efficiency gain.

  • Takeaways & Limitations

    For computationally constrained evaluations, optstop can reclaim resources from well-characterized model-task combinations and redirect them toward regions of genuine uncertainty.

  • Takeaways & Limitations

    Controlled simulation found approximately 80% empirical coverage at 94% nominal for the hierarchical binary model under moderate heterogeneity.

Abstract

from arXiv · show

LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem: keep sampling where uncertainty remains high, and stop where estimates are precise or stable enough. The framework builds on hierarchical Bayesian inference, supports binary, ordinal, and continuous outcomes, and keeps every benchmark item eligible for sampling, without requiring a calibrated item bank. It runs live or retrospectively, and includes a safeguard that samples more cautiously as measured performance approaches zero, where rare successes matter most. In an illustrative 200-item, 10-epoch evaluation, it removes 57%-97% of planned trials across nine validation settings, with overall conclusions equivalent to the full run. These results show that LLM evaluation compute can be allocated by uncertainty rather than by fixed repetition counts, with the magnitude of savings depending on evaluation design.

1 Introduction

LLM evaluations face escalating computational costs and uneven inferential uncertainty, while prevailing fixed-sampling practices often lack systematic uncertainty quantification. optstop addresses this mismatch with hierarchical Bayesian, precision-based adaptive stopping that reallocates sampling toward uncertain evaluation items without requiring pre-calibrated item parameters.

  • Evaluation burden: 1,000,000 inference calls are required for a suite of 200 tasks with 1,000 items and 5 repetitions per item, before model or condition comparisons.Repeated sampling is used because stochastic generation requires stable performance estimates.
  • Methodological problem: Current evaluation practices predominantly report aggregate point estimates without systematic uncertainty quantification, hierarchical-data treatment, or formal statistical validation.This creates concern that evaluations may either expend resources beyond statistical requirements or stop before reliable conclusions are possible.
  • Uneven uncertainty: Uncertainty varies across model-task combinations, yet conventional evaluation applies uniform sampling regimes regardless of inferential goal or intrinsic variability.Performance also varies across tasks, difficulty levels, prompt formulations, and their interactions.
  • Proposed framework: optstop monitors Bayesian credible-interval widths during evaluation and stops collection when precision is adequate while continuing where uncertainty persists.It applies hierarchical Bayesian principles to the prior question of when data collection can safely stop, complementing post-collection analysis frameworks such as HiBayES.
  • Distinct contribution: The framework makes real-time stopping decisions at item and grouping levels without requiring pre-calibrated item parameters or modifying evaluation content.This distinguishes it from adaptive item-selection methods and inference-time sampling approaches that optimize test-item choice or generation quality.

2 The optstop Framework

optstop is a hierarchical Bayesian framework for adaptive LLM evaluation that stops sampling when posterior estimates become sufficiently precise or stable. It supports multiple score types and uses more cautious stopping for low-performing evaluations where rare successes matter.

  • Implementation: The Python package runs as a live early-stopping protocol integrated with inspect_ai or as a standalone tool for retrospective analysis of completed datasets.The repository is publicly available on GitHub.
  • Stopping criteria: Stopping occurs when posterior credible-interval width W = θU − θL falls below a user-specified precision threshold δ at both item and grouping levels.A secondary criterion stops when recent credible-interval widths have a near-zero slope without renewed descent.
  • Stopping criteria: A stabilisation fallback detects diminishing inferential returns by monitoring the slope of recent credible-interval widths within a sliding window.It is intended for settings where the width threshold would require impractically many observations.
  • Rare-success safeguard: 5× conservatism is applied by default below 1% estimated performance, widening the effective interval and tightening the stabilisation threshold to delay stopping.This safeguard addresses the risk that premature termination misses rare but important successes, especially in capability detection and pass@k-style evaluations.
  • Hierarchical structure: Hierarchical Bayesian inference models repeated responses within items and partially pools information across items within user-defined groupings.Each grouping maintains its own inference state and stopping criteria, with factorial designs recommended when manipulated factors may systematically affect performance.
  • Inference pathways: Three tailored inference pathways support binary, ordinal, and continuous scores, with routing controlled by user configuration.Continuous bounded scores use item-level summary statistics and provide an approximately 20× theoretical computational speedup relative to observation-level inference.

3 Empirical Validation

Across nine validation cells spanning binary, ordinal, and continuous outcomes at low, medium, and high performance, optstop triggered early stopping while preserving accurate estimates. Efficiency gains averaged 81.1%, with a mean absolute score deviation of 0.006 on the normalized [0, 1] scale.

  • Core validation results: 97.3% maximum efficiency gain was achieved in the low-continuous cell, while the minimum was 57.2% in mid-ordinal.All nine validation cells triggered early stopping.
  • Pathway effects: 95.1% mean efficiency was achieved by the continuous pathway, compared with 75.0% for ordinal and 73.3% for binary.The passage attributes the continuous pathway’s advantage to the greater information density of real-valued observations.
  • Score fidelity: 0.001 was the ordinal pathway’s mean absolute deviation, compared with 0.006 for binary and 0.011 for continuous.The largest single-cell deviation was 0.022 in high-continuous, below the precision threshold δ = 0.05.
  • Ranking preservation: The ranking max_tokens = 5,000 > 500 > 50 was preserved with non-overlapping 94% HDIs in both full-run and truncated conditions.This ranking analysis used the ordinal and continuous WritingBench cells.
  • Equivalence testing: Hierarchical meta-analysis resolved three individually undecided cells and established statistical equivalence at the pooled level.The per-cell equivalence test was deliberately more stringent than the framework’s precision target, so undecided results did not indicate detected differences.

4 Conclusions … A.5 Worked Illustration

optstop is a Bayesian, precision-based stopping framework that adaptively allocates LLM evaluation sampling, retaining score fidelity while reducing unnecessary trials. It combines hierarchical inference, flexible score pathways and groupings, stabilisation checks, and low-performance conservatism, but its validity depends on exchangeability and broader validation.

  • 4 Conclusions: 57%-97% of planned trials were eliminated across nine validation settings while preserving score fidelity and statistical equivalence.Hierarchical meta-analysis confirmed truncation bias was negligible; savings were largest for continuous scores, followed by ordinal and binary.
  • A.1.1 Credible Interval Widths / A.1.3 Conservatism and pass@K / A.5 Worked Illustration: 100% empirical coverage was observed across binary and continuous cells in 90/90 permutation comparisons, but controlled simulation found approximately 80% coverage at 94% nominal.A conservatism mechanism extends sampling near zero performance, where rare successes matter most; broader replication remains needed.
  • A.1 Precision-based Stopping: optstop stops sampling when posterior credible intervals are sufficiently narrow, or when interval-width improvement stabilises and further data collection becomes uninformative.The primary rule uses W < δ; stabilisation requires a near-zero slope and a non-accelerating trajectory.
  • A.2 Hierarchical Inference: The framework performs coordinated stopping at item and grouping levels, using hierarchical Bayesian partial pooling to estimate model-task performance despite uneven item sample sizes.Every benchmark item remains eligible for evaluation, and no pre-calibrated item bank is required.
  • A.3 Leveraging Groupings: Flexible user-defined groupings let well-characterised configurations stop independently while computational effort shifts toward uncertain regions, trading inferential resolution against statistical power.Finer groupings provide more granular capability characterisation but require more observations per grouping to meet precision thresholds.
  • A.4 Inference Pathways: Binary, ordinal, and bounded continuous pathways share Bayesian credible-interval stopping but require explicit configuration for non-binary scoring schemes.Binary inference is the default; routing persists at grouping level and depends on user-specified task patterns or aggregation settings.
  • A.4.1 Binary: The binary pathway uses an adaptive prior whose strength decays with sample size while maintaining a 0.5 floor corresponding to the Jeffreys baseline.The prior is meaningfully regularising for small samples and increasingly dominated by the likelihood as evidence accumulates.
  • A.4.2 Ordinal (Discrete) / A.4.3 Bounded Continuous: The ordinal Dirichlet model is available when ordered-logistic assumptions are inappropriate or sampling convergence fails, while bounded continuous inference models sample means with a Normal likelihood.The continuous pathway uses configurable population priors and item-level precision heterogeneity.

A.5.1 Evaluation Setup · A.5.2 Model A: Strong Performer

The evaluation compares two models across binary and ordinal tasks using 200 items and up to 10 epochs, with early stopping reducing unnecessary trials. For strong Model A, Bayesian posteriors typically become precise quickly, while ordinal stopping additionally verifies genuine distributional concentration.

  • A.5.1 Evaluation Setup: The setup evaluates strong Model A and weak Model B on binary factual question answering and ordinal reasoning-quality tasks.Model B performs below 1% accuracy on Task 1.
  • A.5.1 Evaluation Setup: Ordinal analyses use hybrid inference, combining modal credible intervals with entropy convergence across four independently analyzed groupings.Hybrid inference is the default through inspect_ai, whereas standalone mode defaults to modal inference.
  • A.5.1 Evaluation Setup: Each task contains 200 items with up to 10 epochs, making the complete fixed-budget evaluation 8,000 trials.The total assumes 2 models × 2 tasks × 200 items × 10 epochs.
  • A.5.2 Model A: Strong Performer: For Model A’s binary task, consistently successful items quickly obtain narrow credible intervals as item-level Beta posteriors concentrate near the upper bound.Items with mixed outcomes take longer because their posteriors span a wider range of plausible success rates.
  • A.5.2 Model A: Strong Performer: At the binary grouping level, hierarchical evidence rapidly concentrates around a high mean success probability and narrows the credible interval.Grouping precision is reassessed every user-defined reanalysis_interval, after which G1 terminates and remaining items need not be evaluated.
  • A.5.2 Model A: Strong Performer: For Model A’s ordinal task, scores predominantly cluster at 7–9, producing peaked distributions and quickly narrowing modal credible intervals for consistently scored items.The Bayesian bootstrap concentrates on a clear modal category.
  • A.5.2 Model A: Strong Performer: Ordinal G2 stopping requires both a narrow hierarchical modal CI below delta_cap and posterior median entropy below the peakedness threshold.The default entropy threshold is 80% of log2 K bits, helping distinguish genuine concentration from a spurious peak caused by limited data.

A.5.3 Model B: Weak Performer … B.2 Precision Threshold Sensitivity

The paper shows that optstop adapts stopping behavior to performance, score type, and posterior precision: weak or diffuse results require extensive characterization, while tighter precision thresholds trade more trials for narrower intervals. Integration validation across live LLM evaluations confirms pathway routing and stopping behavior across diverse benchmarks and configurations.

  • A.5.3 Model B: Weak Performer: Model B’s low binary accuracy activates conservatism, inflating effective CI widths by c = 5 and slowing adaptive prior decay five-fold.These mechanisms preserve sensitivity to rare successes despite strings of failures.
  • A.5.3 Model B: Weak Performer: Item-level stopping is rare for weak binary performance, while grouping stabilization is deliberately difficult because conservatism inflates effective uncertainty and tightens the slope criterion.Most items proceed through all available epochs, and the slope threshold becomes |β̂| < ϵ/c.
  • A.5.3 Model B: Weak Performer: For Model B’s low, variable ordinal scores, wide modal credible intervals keep most items running through all available epochs.The diffuse lower-rubric distribution lacks a clear modal concentration.
  • A.5.3 Model B: Weak Performer: Entropy validation blocks modal-CI stopping for diffuse ordinal results, while Pathway 2 stops when entropy credible-interval width falls below the default 0.10 threshold.This permits stopping once uncertainty about distributional shape is resolved, even when the distribution remains diffuse.
  • A.5.4 Summary of Expected Behaviour: Weak or variable performance requires more extensive characterization than strong, consistent performance, with binary tasks estimating small probabilities and ordinal tasks resolving diffuse distributions.Peaked distributions use Pathway 1, whereas diffuse distributions use Pathway 2.
  • B.1 Inference Pathway Coverage: Twenty-nine validation runs across four benchmarks covered all three inference pathways; every run met its stopping criteria and produced benchmark-consistent scores.Runs used live LLM inference rather than mocked outputs.
  • B. Package Validation: Configurations spanning 10–1,500 planned trials and 2–5 epochs per item were automatically routed to the correct inference pathway without manual override.The validation suite included single-model and multi-model runs.
  • B.2 Precision Threshold Sensitivity: A 4× tighter precision threshold required approximately 5× more trials overall, while additional tests confirmed that looser thresholds stop earlier with wider intervals and tighter thresholds stop later with narrower intervals.All final credible-interval widths met their respective thresholds, with discrete checkpoints sometimes yielding substantially tighter precision.

B.3 Ordinal Stopping Pathway Validation · B.4 Shadow Mode Counterfactual Validation

Ordinal stopping was validated across two differently sized ordinal scales, with entropy-aware Pathway 1 stopping on peaked distributions and scale-aware thresholds enabling convergence for larger scales. Shadow mode supported counterfactual validation by comparing stopping behavior across runs and recording complete stopping diagnostics.

  • B.3 Ordinal Stopping Pathway Validation: Two benchmarks validated the ordinal pathway: WritingBench used an 11-category, 0–10 rubric, while SciKnowEval used six modelled categories scored 1–5, including zero.The pairing exercised modal-CI stopping with entropy validation under distinct distributional conditions and scale sizes.
  • B.3 Ordinal Stopping Pathway Validation: 247 trials was the identical SciKnowEval stopping point under 250- and 500-trial plans.SciKnowEval’s near-ceiling mean of ≈4.86/5.0 produced 0.43 bits entropy versus the 2.07-bit default threshold, triggering Pathway 1 at the first grouping-level reanalysis.
  • B.3 Ordinal Stopping Pathway Validation: 365 trials yielded 27% potential efficiency for WritingBench in 500-trial shadow-mode runs.WritingBench’s mean was ≈7.6/10 with intermediate entropy, requiring the scale-aware threshold because a fixed threshold blocked convergence above seven categories.
  • B.3 Ordinal Stopping Pathway Validation: Pathway 1 governed initial stopping for both ordinal benchmarks because their distributions were sufficiently peaked, while Pathway 2 was not triggered in these tests.Pathway 2 was later triggered in the matrix validation experiment’s mid-ordinal cell.
  • B.4 Shadow Mode Counterfactual Validation: Shadow mode executes all planned trials while tracking when stopping criteria are met, enabling validation against complete-run ground truth.It was used to assess cross-run consistency and diagnostic completeness without actual early termination.
  • B.4 Shadow Mode Counterfactual Validation: 87.6% live efficiency at trial 31 contrasted with 92% shadow potential efficiency at trial 20 for the same continuous-pathway configuration.The stopping points were not directly comparable because LLM non-determinism changes scores across runs; the difference reflects input variability rather than algorithmic inconsistency.
  • B.4 Shadow Mode Counterfactual Validation: Shadow mode recorded each grouping’s first qualifying trial, invoked pathway, and full stabilisation history for post-hoc stopping analysis.The pathways distinguished CI-width precision from CI-width/entropy stability, without the confound of actual early termination.

B.5 Reproducibility

optstop’s internal MCMC procedure is deterministic given identical inputs, seeds, and software/hardware configurations, while full stopping-decision reproducibility additionally depends on deterministic LLM outputs. Reinstallation from the same repository commit and dependency versions reproduced benchmark convergence and consistent score estimates.

  • MCMC determinism: Identical input scores, random seeds, and software/hardware configurations yielded identical posterior distributions, credible intervals, and stopping decisions.Paired runs with different LLM responses diverged only where input scores differed, confirming controlled internal randomness.
  • MCMC determinism: Full end-to-end reproducibility additionally requires deterministic LLM outputs, such as temperature = 0; trial-count differences otherwise reflect response variability rather than algorithmic instability.Score estimates and credible-interval widths remained consistent across runs despite LLM variability.
  • Deployment reproducibility: 521 vs. 608 of 1,500 trials were observed between fresh-install and prior validation runs, while all tasks met final credible-interval-width thresholds and scores remained consistent.The package was reinstalled from the same repository commit with identical dependency versions before rerunning the three-benchmark validation suite.

B.6 Logit-Normal vs. Beta-Binomial Model Comparison · B.7 Validation Scope · B.8 Downstream Analysis Equivalence

The logit-normal model matched the Beta-Binomial on binary estimation metrics while providing substantially more reliable sampling, preserving equivalent stopping behaviour. Validation covered all inference and ordinal stopping pathways, while the matrix experiment used shadow-mode within-shadow comparisons to isolate truncation effects.

  • B.6 Logit-Normal vs. Beta-Binomial Model Comparison: The controlled simulation deliberately generated data from a Beta-Binomial process to conservatively test logit-normal adequacy.The design crossed 9 true population probabilities with 3 item-level sample sizes, yielding 27 conditions and 675 total runs.
  • B.6 Logit-Normal vs. Beta-Binomial Model Comparison: Exact-boundary conditions represented a qualitatively different regime because the Beta distribution was degenerate and all items shared identical success probability.Mid-range conditions instead involved genuine item-level variation around the population mean.
  • B.6 Logit-Normal vs. Beta-Binomial Model Comparison: In the mid-range, absolute bias, CI width, and coverage were statistically indistinguishable between models; near boundaries, performance remained closely matched.Both models showed approximately 80% coverage in non-boundary conditions, below the nominal 94% level.
  • B.6 Logit-Normal vs. Beta-Binomial Model Comparison: 120× more divergences occurred for the Beta-Binomial than the logit-normal in the mid-range, increasing to 192× near boundaries and 9,206× at exact boundaries.At exact boundaries, totals were 73,646 versus 8 divergences, respectively.
  • B.6 Logit-Normal vs. Beta-Binomial Model Comparison: Equivalent CI widths imply equivalent stopping times because optstop triggers grouping-level stopping when CI width falls below delta_cap.The logit-normal also had approximately 3× higher ESS in the mid-range and over 37× higher ESS at boundaries.
  • B.7 Validation Scope: The appendix tests and matrix experiment exercised all three inference pathways and both ordinal stopping pathways.Additional validation covered conservatism sensitivity, CI stabilisation, presentation-order robustness, and a fixed-n baseline.
  • B.8 Downstream Analysis Equivalence: Each cell of the 3 × 3 matrix experiment was executed in shadow mode with within-shadow comparison to isolate the pure truncation effect.The section provides the full experimental protocol and supplementary analyses for the experiment summarized in Section 3.

B.8.1 Experimental Design · B.8.2 HiBayES Validation Framework

The validation uses a 3 × 3 experimental design spanning binary, ordinal, and continuous pathways, then compares full and adaptively truncated runs through complementary hierarchical Bayesian analyses. The framework tests item-level equivalence, pooled truncation effects, ranking preservation, and posterior consistency.

  • B.8.1 Experimental Design: B.8.1 Experimental Design: The 3 × 3 matrix varies pathway, benchmark, model, and max_tokens to assess within-pathway rankings and cross-pathway generalisability.Binary cells use distinct benchmarks and models, while ordinal and continuous cells use WritingBench with Claude Sonnet 4.5 and vary max_tokens.
  • B.8.1 Experimental Design: B.8.1 Experimental Design: 200 items and 10 epochs define 2,000 planned trials per cell, with seed 42, δ = 0.05, and 97% credible intervals.The mid-binary cell has 1,980 planned trials because two items failed to load.
  • B.8.1 Experimental Design: B.8.1 Experimental Design: The ordinal pathway applies an 11-category ordered-logistic rubric with dual stopping pathways based on modal-interval precision and entropy thresholds.Pathway 1 combines a narrow modal credible interval with normalized entropy below 0.8 × log2 K; Pathway 2 requires scaled entropy CI width below 0.10.
  • B.8.1 Experimental Design: B.8.1 Experimental Design: The continuous pathway uses a hierarchical logit-normal model with δ = 0.05 as its precision threshold.The supplied passage also identifies a hierarchical logit-normal model for the binary pathway, but its description is truncated.
  • B.8.2 HiBayES Validation Framework: B.8.2 HiBayES Validation Framework: Five complementary hierarchical Bayesian analyses compare full runs using all 2,000 trials with truncated runs stopping at the would-have-stopped point.All MCMC fits use four chains, 2,000 posterior draws per chain, 1,000 warmup iterations, seed 42, and 94% HDIs.
  • B.8.2 HiBayES Validation Framework: B.8.2 HiBayES Validation Framework: The primary item-matched paired comparison fits a Normal model to per-item full-versus-truncated score differences to estimate truncation bias.Pairing by item identity controls for item-level variance.
  • B.8.2 HiBayES Validation Framework: B.8.2 HiBayES Validation Framework: Hierarchical meta-analysis pools cell-level primary results with multiplicity adjustment, producing a joint truncation-effect estimate and resolving undecided verdicts through shrinkage.Supplementary ordered-logistic and continuous hierarchical models separately test ranking preservation across ordinal and continuous cells.
  • B.8.2 HiBayES Validation Framework: B.8.2 HiBayES Validation Framework: Per-cell posterior comparisons use independent full and truncated fits with ROPE testing as a lower-power consistency check.This auxiliary analysis does not control for item-level pairing.

B.8.3 Item-Matched Paired Comparison (Analysis A)

Analysis A pairs full and truncated scores by item and tests their mean difference against ROPE-based equivalence. All nine cells converged, six accepted equivalence, three were undecided, and none rejected it.

  • Method: Item-matched pairing fits a Normal model to per-item score differences and compares the 94% HDI for µdiff with ROPEs of ±0.02 or ±0.10.The binary and continuous ROPE is ±0.02; the ordinal ROPE is ±0.10.
  • Results: All nine models converged with R̂ = 1.0, 0 divergences, and bulk ESS > 4,600 for µdiff and > 2,500 for σdiff.Six of nine cells accepted equivalence; low-binary, mid-binary, and high-ordinal were undecided, while none rejected equivalence.
  • Results: 93–97% efficiency continuous cells matched 54–142 of 200 items, yet narrow within-item variance produced tight posteriors and clear equivalence acceptance.Most ordinal and binary cells matched 200 items, except mid-binary, which matched 198 after 2 items failed to load.
  • Undecided cells: Low-binary and mid-binary remained undecided despite µdiff ≈+0.007 and −0.011, respectively, because per-item standard deviations were 0.135 and 0.136.The passage attributes these verdicts to binary scoring’s limited information per observation rather than a truncation problem.
  • Undecided cells: Mid-binary’s posterior centered near −0.011, but its 94% HDI [−0.029, +0.007] extended beyond the ±0.02 ROPE.The extended tails were attributed to high per-item Bernoulli variance in the near-50% binary benchmark.

B.8.4 ROPE Sensitivity Analysis

The ROPE sensitivity analysis evaluated Analysis A across six researcher-specified practical-equivalence widths. Narrow ROPEs produced mostly undecided verdicts, while wider ROPEs yielded progressively more equivalence accepts in a staircase pattern linked to per-item variance.

  • Method: Analysis A was evaluated across six ROPE widths: ±0.005, ±0.01, ±0.02, ±0.05, ±0.10, and ±0.20.ROPE width represents the magnitude of difference deemed practically negligible.
  • Results: At ±0.005, most cells were undecided; widening the ROPE produced staircase-like equivalence accepts across cells.Mid-continuous accepted at ±0.005; low-continuous and high-continuous at ±0.01; high-binary and low-ordinal at ±0.02; and low-binary, mid-binary, and mid-ordinal at ±0.05.
  • Interpretation: Cells accepting at narrower ROPE widths had lower per-item variance and tighter posteriors, while binary cells had high variance from binary observations’ low information content.Low-ordinal and mid-ordinal accepted at intermediate widths, whereas high-ordinal showed a larger truncation effect driven by the modal-vs-mean estimate.

B.8.5 Hierarchical Meta-Analysis (Multiplicity-Adjusted) … B.8.8 Information Gain

Across hierarchical, ranking, posterior-comparison, and information-gain analyses, truncation produced negligible overall effects, preserved known condition rankings, and reduced uncertainty rapidly early in sampling. Hierarchical pooling resolved previously undecided cells while maintaining equivalence conclusions.

  • B.8.5 Hierarchical Meta-Analysis (Multiplicity-Adjusted): At primary ROPE widths, 6 of 9 cells accepted; at ±0.05, 8 of 9 accepted, while high-ordinal accepted only at ±0.20.No cell rejected at any tested ROPE width.
  • B.8.5 Hierarchical Meta-Analysis (Multiplicity-Adjusted): The hierarchical meta-analysis estimated an overall truncation effect of +0.0003 with a 97% HDI of [−0.002, +0.003], entirely within the ±0.02 ROPE.Estimated between-cell heterogeneity was ˆτ = 0.001, indicating highly homogeneous cell-level effects.
  • B.8.5 Hierarchical Meta-Analysis (Multiplicity-Adjusted): All nine cells received accept-null verdicts after partial pooling, compared with 6/9 under independent Analysis A.Pooling resolved the low-binary, mid-binary, and high-ordinal undecided cells by borrowing strength across cells.
  • B.8.6 Ranking Preservation (Analyses B and C): The ranking max_tokens = 5,000 > 500 > 50 was preserved in both full-run and truncated ordinal models, with non-overlapping 94% HDIs between adjacent levels.Posterior means for max_tokens = 50 were −6.24 full run and −5.89 truncated, versus +5.28 and +4.85 for max_tokens = 5,000.
  • B.8.6 Ranking Preservation (Analyses B and C): Continuous models likewise preserved the ranking, with posterior mean differences between full-run and truncated conditions of |∆| ≤0.011 at every max_tokens level.Both models converged cleanly with 0 divergences, ˆR = 1.0, and ESS > 3,200 for all parameters.
  • B.8.7 Per-Cell Posterior Comparison (Analysis D): Independent per-cell posterior comparisons found no cell rejecting equivalence, although reduced truncated-condition sample size lowered statistical power.Because posteriors were fitted independently rather than to item-paired differences, undecided verdicts reflected lower precision.
  • B.8.8 Information Gain: Posterior variance collapsed within the first 200–400 trials across pathways, with later trials producing progressively smaller uncertainty reductions.The continuous cells showed the steepest initial declines.

B.8.9 Known Limitations · B.9 Fixed-n Baseline

The validation design has limitations involving pseudoreplication, truncation-related selection bias, multiple comparisons, and Normal approximations. Fixed-n estimates nevertheless converge toward full-run values, with deviations at optstop stopping points within the stated precision threshold.

  • B.8.9 Known Limitations: Epoch pseudoreplication treats 10 within-item epochs as exchangeable, inflating effective sample size and affecting convergence speed without biasing point estimates.Within-item observations may be correlated, so they are not fully independent.
  • B.8.9 Known Limitations: Posterior variance collapses within the first 200–400 trials across all pathways, showing diminishing returns during convergence.Figure 8 normalizes each cell’s group-level posterior variance by its full-run between-item variance.
  • B.8.9 Known Limitations: Selection bias can arise when early-stopped items or epochs differ systematically from later observations because item ordering correlates with difficulty.The fixed-ordering within-shadow comparison captures this bias in the truncation effect, matching deployment ordering.
  • B.8.9 Known Limitations: Nine independently tested cells increase the chance of misleading verdicts, while hierarchical meta-analysis with partial pooling yields an overall accept-null verdict.The joint model reports µ = +0.0003, 97% HDI [−0.002, +0.003].
  • B.8.9 Known Limitations: Analysis A’s Normal approximation is adequate for mean inference with n ≥54 in every cell but may mischaracterize binary difference-distribution tails.Binary epoch-averaged differences are discrete multiples of 0.1 within [−1, +1].
  • B.9 Fixed-n Baseline: Fixed-n estimates at 100–2,000 trial checkpoints across 100 random item orderings establish non-adaptive convergence trajectories and precision references.These checkpoints contextualize the efficiency gains reported for optstop.
  • B.9 Fixed-n Baseline: At optstop stopping points, fixed-n means deviate from full-run values by at most 0.017 for binary and 0.003 for continuous estimates.Cross-ordering standard deviations remain below 0.005 and 0.008, respectively, within the δ = 0.05 precision threshold.
  • B.9 Fixed-n Baseline: Ordinal deviations are smaller still, below 0.002, but exhibit additional behavior described in the fixed-n convergence analysis.Binary and continuous estimates converge smoothly, with confidence-interval widths narrowing monotonically.

B.10 Presentation-Order Robustness · B.11 Conservatism Sensitivity · B.12 CI Stabilisation Validation

Across presentation-order, conservatism, and CI-stabilisation tests, adaptive stopping was generally stable and efficient, while ordinal coverage exposed an estimand mismatch and default stabilisation rarely activates. Conservatism protected very-low-performance estimates, with c = 5 adopted as the package default.

  • B.10 Presentation-Order Robustness: 100% of 135 presentation-order replications stopped early, with theta standard deviation below 0.006 for binary and continuous cells and below 0.033 for ordinal cells.Replications replayed shadow datasets with epoch-interleaved processing and different item-shuffle seeds, using full-run estimates as reference.
  • B.10 Presentation-Order Robustness: 100% empirical CI coverage held across binary and continuous cells, while ordinal coverage was 0/15 for low, 12/15 for mid, and 1/15 for high.The ordinal caveat reflects that the pathway estimates modal category rather than mean, whereas binary and continuous intervals absorb observed truncation biases of approximately 0.01.
  • B.10 Presentation-Order Robustness: Subset selection bias was at most 0.002 for continuous, 0.007 for ordinal, and 0.012 for binary pathways, remaining within δ.Binary bias was larger because item-level variance makes early-epoch item composition more consequential.
  • B.11 Conservatism Sensitivity: At 1% performance, c = 1 stopped at 120 trials with an overestimate of 0.027 versus true 0.0065, whereas c = 5 stopped at 910 trials with an estimate of 0.008.At performance levels of at least 10%, conservatism had negligible effect regardless of c.
  • B.11 Conservatism Sensitivity: Higher conservatism values c = 10 and c = 50 consumed 1,590–1,920 trials for marginal accuracy gains, while high-c behaviour became non-monotonic and MCMC-dependent.At c = 50, repeated MCMC runs showed coefficient of variation up to 13%; only 2/5 data seeds had monotonic trial counts across c at 1% performance.
  • B.11 Conservatism Sensitivity: The package defaults were changed to c = 5 and low_performance_threshold = 0.01, preserving accurate 1% estimates while keeping δ/c = 0.01 achievable with moderate budgets.In the experiment, c = 5 produced ˆθ = 0.008 versus true 0.0065, while c = 1 permitted premature stopping.
  • B.12 CI Stabilisation Validation: All 9 CI-stabilisation conditions stopped through stabilisation, achieving 75–81% efficiency and maximum absolute theta deviation of 0.017, independently of performance level or δ.Relaxed stabilisation parameters were used to activate this fallback pathway; under production defaults, no matrix cell triggered it.
Loading 2608.14425v1…