PaperlayerView exact revision on arXiv

Source-linked AI summary

Instability Floors: Separating Bias from Noise in Fairness Audits of Clinical LLM Agents with FairMedAgent

Rohith Reddy Bellibatlu, Manpreet Singh, Deepak Parashar, Rahul Joshi

arXiv:2609.03221v3cs.CLcs.AIcs.CYcs.LGstat.AP

TL;DR

Counterfactual fairness audits of clinical agents need to distinguish demographic effects from an agent’s own run-to-run action changes. The paper measures that instability floor with FairMedAgent and relates it to flip rates. Floors vary across models and decoding configurations, so the audit’s floor must be measured for the configuration being used.

  • Problem

    Without measuring how often a stochastic clinical agent changes its action on unchanged input, a counterfactual flip rate cannot be separated from the agent’s own disagreement.

    Source: §I · p. 2

  • Method

    The authors use FairMedAgent to rerun fixed clinical vignettes and measure action disagreement while controlling the clinical content and tool-result fixture across conditions.

    Source: §I · p. 2

  • Results

    8.7% of replicate pairs changed the primary model’s action, and pooled floors across six models ranged from 2.5% to 23.7%; the floor also varied with decoding configuration.

    Source: §I · p. 2

  • Takeaways & Limitations

    Measure the instability floor on the audited system at its actual decoding and serving configuration before interpreting demographic flip rates.

    Source: §I · p. 2

    Flip rates are a weak instrument for small effects because a small effect raises them only by its square.

    Source: B. LIMITATIONS · p. 10

Abstract

from arXiv · show

Counterfactual fairness audits of clinical language-model agents report a flip rate: how often an action changes when only the patient's demographic descriptor changes. Part of that rate is not demographic. A stochastic agent also changes its own action when nothing changes, and a flip rate cannot be interpreted without knowing how often. We measured it. Re-running one condition ten times over sixteen synthetic vignettes at default sampling changed a clinical agent's action in 8.7 percent of replicate pairs, from 2.2 percent for intensive-care escalation to 17.9 percent for controlled-substance caution, an output given no operational criteria. Across six models from five vendors, pooled floors ranged from 2.5 to 23.7 percent; in this panel neither disclosed size, vendor, nor hosting ordered them. The floor depends on the decoding configuration: majority voting over five draws removed 39 percent of it (95 percent confidence interval (CI) 18 to 64); at temperature 0 three of four locally served models showed no disagreement, but a hosted model still did. We also show that, for a binary action, the flip rate expected under no demographic effect equals the floor and a real effect adds only its square, so a flip rate inside the floor is not evidence of fairness, and direction must be tested with a signed paired test. We give a four-step reporting procedure and release FairMedAgent, the harness, with its protocol, vignettes and analysis scripts, so any team can measure the floor for its own agent.

I. INTRODUCTION

Clinical-agent fairness audits need to separate demographic effects from actions that change on unchanged inputs. The paper measures this instability floor, explains its relation to flip rates, and positions the released harness against prior fairness, capability, and repeatability work.

I. INTRODUCTION

  • FairMedAgent measures action changes in a multi-step clinical agent while holding clinical content and the tool-result fixture fixed across conditions.The harness and its protocol, vignettes, and analysis scripts are released; the broader within-range, signed, LEP, and intersectional estimands are specified but not computed here.Source: §I · p. 2
  • 8.7% of replicate pairs changed the primary model’s action; pooled floors across six models ranged from 2.5% to 23.7%, with no ordering by disclosed size, vendor, or hosting in this panel.Per-action floors ranged from 2.2% for intensive-care escalation to 17.9% for controlled-substance caution.Source: §I · p. 2
  • For a binary action, the expected flip rate under no demographic effect equals the measured floor, and an effect adds only its square.A flip rate within the floor is not evidence of no effect; case-level effects require repeated draws, while an aggregate directional test can use one draw per case.Source: §I · p. 2
  • At temperature 0, three of four locally served models showed no disagreement, while majority voting over five draws removed 39% of the primary model’s floor.A hosted model still disagreed at temperature 0, so the floor depends on decoding and serving configuration.Source: §I · p. 2

II. RELATED WORK

  • Prior counterfactual audits examine demographic differences in recommendations or decisions, while clinical-agent benchmarks measure capability rather than demographic disparity in actions.The related-work comparison identifies a measured per-action floor beside flip rates as absent from the listed audits.Source: §II · p. 2, Table 1 · p. 3
  • Run-to-run variability was already studied for accuracy, text, and semantic consistency; this paper focuses on same-action changes per case and their role in interpreting fairness flip rates.The reported measurements are bare-model, component-level audits, not assurance results for a fielded system.Source: §II · pp. 2–3

III. METHODS

The methods measure action-level instability in controlled descriptor-swap audits of multi-step clinical agents. They distinguish the reference-condition instability floor from demographic effects and specify when repeated draws or directional tests are needed.

III. METHODS

  • The paper reports per-action instability floors and their consequences, but claims no disparity estimate; implemented within-range and signed-disparity measures are not computed.The harness refuses to compute the within-range flip rate without clinician-adjudicated acceptable-action bands.Source: §III · pp. 3–4, B. OUTCOMES, FLOOR, AND WHAT THE FLOOR… · p. 5

A. TRAJECTORY AND CONDITIONS

  • Each vignette follows six fixed stages, with five model calls; downstream instability can accumulate from earlier sampled actions.Fixed phase order controls against demographic conditions triggering different tool paths.Source: A. TRAJECTORY AND CONDITIONS · p. 4
  • Clinical narratives are frozen across descriptor conditions, and condition-independent tool fixtures make returned results identical when order sets coincide.Different results after an ordering flip are a propagation pathway, not a confound; repeated draws are needed for case-level attribution.Source: A. TRAJECTORY AND CONDITIONS · p. 4
  • The metamorphic descriptor-swap audit varies fixed-form demographic descriptors while freezing clinical content; attributes include race/ethnicity, sex, age, insurance, LEP, and two intersectional cells.The intersectional cells are reported as disaggregated estimates, not a full intersectional analysis; three control arms bound surface-form sensitivity.Source: A. TRAJECTORY AND CONDITIONS · p. 4

B. OUTCOMES, FLOOR, AND WHAT THE FLOOR IS A REFERENCE FOR

  • Reported counterfactual flip rates compare thresholded binary actions, while the instability floor averages disagreements across reference-condition replicate pairs.The floor is configuration- and case-dependent, not an irreducible minimum, fairness threshold, or quality score; all six outcomes are defined for every vignette, though applicability varies.Source: B. OUTCOMES, FLOOR, AND WHAT THE FLOOR… · p. 4, Table 2 · p. 4
  • Under no demographic effect, expected discordance equals the floor; an effect adds its square, so a flip rate within the floor is not evidence of no effect.Estimating case-level effects requires repeated draws under each condition, while an aggregate directional difference can be tested across vignettes with one draw per condition using exact McNemar testing.Source: B. OUTCOMES, FLOOR, AND WHAT THE FLOOR… · p. 5

C. VIGNETTES, MODELS, AND PROVENANCE

The study uses a selected subset of synthetic clinical vignettes and six model endpoints to measure instability. Its estimates are conditional on the fixed cases, and the acceptable-action bands needed for a within-range estimand remain incomplete.

C. VIGNETTES, MODELS, AND PROVENANCE

  • The floor study uses 16 of 300 synthetic vignettes, selected to span ambiguity rather than screened for instability; inference is conditional on these fixed cases.The 300 vignettes span triage/escalation, diagnostic and laboratory ordering, medication management, and documentation/disposition; only the 16 floor-study vignettes are released now.Source: C. VIGNETTES, MODELS, AND PROVENANCE · p. 5
  • Acceptable-action bands remain author-derived because clinician adjudication to protocol specification is incomplete, and no reported number depends on them.A licensed clinician reviewed four draft bands, but two were outside her specialty and her signed attestation had not been returned.Source: C. VIGNETTES, MODELS, AND PROVENANCE · p. 5
  • Six models were measured: two hosted endpoints from one vendor family and four locally served open-weight models from distinct vendors.The hosted endpoints’ identities and decoding settings were not independently attested; all models ran at provider-default sampling, and one local model was dropped for unusable completions.Source: C. VIGNETTES, MODELS, AND PROVENANCE · p. 5

D. STATISTICAL ANALYSIS

The primary model’s measured floor varied substantially across actions, while model attributes did not monotonically order pooled floors in this panel. The estimates are sensitive to case count and the interpretation of controlled-substance caution is limited by its underspecified criteria.

D. STATISTICAL ANALYSIS

  • Floor confidence intervals use vignette-cluster bootstrap resampling; majority vote aggregates replicate outcomes, while the harness’s signed-disparity tests are implemented but unused here.The analysis labels intervals with G ≲ 12 clusters approximate; R=5 majority voting is pre-specified, with tied votes left undetermined.Source: D. STATISTICAL ANALYSIS · p. 6

A. THE FLOOR ON THE PRIMARY MODEL

  • The primary model’s action-specific floor ranged from 2.2% for intensive-care escalation to 17.9% for controlled-substance caution, an eightfold ratio of point estimates.The paired difference was 0.157 (95% CI [0.051, 0.269]); its interval remains positive when any one vignette is dropped, while the pooled floor describes neither action well.Source: A. THE FLOOR ON THE PRIMARY MODEL · p. 6
  • After dividing out action prevalence, disagreement ratios still ranged from 0.05 for any-opioid to 0.42 for controlled-substance caution, an eightfold spread.Seventy-five of 96 vignette–action cells were decided the same way every time; this normalization is descriptive and does not remove case heterogeneity, action applicability, or construct ambiguity.Source: A. THE FLOOR ON THE PRIMARY MODEL · p. 6
  • Controlled-substance caution lacks operational criteria, so its high floor may partly reflect construct ambiguity rather than model variability alone.Excluding it, the primary model’s pooled floor is 0.068 ([0.036, 0.101]), and the ordering of the six models’ pooled floors is unchanged.Source: A. THE FLOOR ON THE PRIMARY MODEL · p. 6
  • The earlier four-vignette estimate was volatile, and the 16-vignette estimate was less volatile but had not converged.The earlier estimate was 0.136 with an approximate interval [0.063, 0.209]; subsampling checked K ∈ {4, 8, 12, 16} vignettes.Source: A. THE FLOOR ON THE PRIMARY MODEL · pp. 6–7
  • 8.7% of replicate pairs disagreed on the primary model’s pooled floor (374/4320; 95% CI [0.056, 0.117]), and Table 3 reports pooled floors across models without a monotone relationship to vendor, host, or disclosed size.Model B ran six replicates rather than ten; Table 3’s pooled floors weight every comparison equally.Source: A. THE FLOOR ON THE PRIMARY MODEL · p. 6, Table 3 · p. 7

B. THE SIX-MODEL PANEL

Across six models, pooled instability floors varied widely without ordering by disclosed size, vendor, or hosting, and per-action orderings also differed. The floor depended on decoding configuration: temperature-zero decoding reduced it in some settings, while five-draw voting reduced but did not eliminate it.

B. THE SIX-MODEL PANEL

  • 0.025 to 0.237: pooled floors spanned a factor of 9.4 across the six-model panel, with no listed model attribute ordering them.The authors caution that this is an observation about this panel, and that model differences are confounded with unmatched default decoding configurations.Source: B. THE SIX-MODEL PANEL · p. 7, Figure 2 · p. 7

C. PER-ACTION ORDERING ACROSS MODELS

  • Per-action ordering differed across models; only the same-vendor hosted pair agreed at nominal significance, and the earlier two-cluster pattern did not survive ten replicates.The sole nominally significant pairwise result was unadjusted for 15 comparisons and did not survive Benjamini–Hochberg adjustment.Source: C. PER-ACTION ORDERING ACROSS MODELS · p. 7, Figure 3 · p. 8
  • With sixteen vignettes, per-action intervals were wide: only two of six actions on the primary model and one of six on the second excluded zero.The pooled magnitudes per model and broad comparisons survived, but precise middle-action ranks did not.Source: C. PER-ACTION ORDERING ACROSS MODELS · p. 8

D. HOW MUCH OF THE FLOOR IS REMOVABLE

  • At temperature 0, three of four locally served models had zero disagreements, phi3:mini’s floor fell 89%, but a hosted endpoint still disagreed.For the hosted endpoint, the temperature-zero floor was 0.023 (95% CI 0.000–0.052); the authors note that its interval reaches zero but any disagreement shows the endpoint is not deterministic.Source: D. HOW MUCH OF THE FLOOR IS REMOVABLE · p. 8
  • 39% of the primary model’s floor was removed by five-draw majority voting (95% CI 18–64%); the R=5 observed floor of 0.053 fell within the simulated null band.The null comparison is a model check, not independent validation.Source: D. HOW MUCH OF THE FLOOR IS REMOVABLE · p. 9, Figure 4 · p. 9
  • At ten replicates, 75 of 96 cells were unanimous; three exact 5–5 splits leave near-boundary and balanced cells unresolved, so R=5 does not establish convergence.The authors report no estimate of the primary model’s floor under deterministic decoding because its runtime did not allow them to set it.Source: D. HOW MUCH OF THE FLOOR IS REMOVABLE · p. 9

E. DEMOGRAPHIC CONTRASTS, READ DESCRIPTIVELY

The four-vignette pilot supports only descriptive comparisons: its null arm flipped in 5 of 24 cells, and no disparity is claimed. Two apparent signals cannot be attributed to demographics because the reference prompt was reworded.

E. DEMOGRAPHIC CONTRASTS, READ DESCRIPTIVELY

  • 5 of 24 cells flipped in the identical-descriptor null contrast; with four vignettes, no test could detect or rule out an effect.The smallest two-sided exact McNemar p-value with four discordant pairs is 0.125.Source: E. DEMOGRAPHIC CONTRASTS, READ… · p. 9
  • Four of five demographic contrasts flipped less often than the null arm, and the largest exceeded it by one cell; no disparity is claimed.The pilot used one draw per cell over four vignettes, so its comparisons remain descriptive.Source: E. DEMOGRAPHIC CONTRASTS, READ… · p. 9
  • Both apparent three-of-four controlled-substance signals are uninterpretable because the prescribing-step reference prompt was reworded.The change also affects comparisons against the reference at that step, including the null arm; its single-draw floor is excluded from the sizing comparison.Source: E. DEMOGRAPHIC CONTRASTS, READ… · p. 9

V. DISCUSSION

The discussion treats the per-action instability floor as the reference for interpreting flip rates and sizing audits. It also bounds the findings to in-silico, component-level measurements with important limits on models, inputs, and clinical validity.

A. WHAT THE FLOOR CHANGES FOR AN AUDIT

  • A flip rate within the per-action floor is uninformative, not evidence of no effect; test aggregate direction with a signed paired test.Case-level effects require repeated draws, and a pooled floor cannot substitute for action-specific measurement.Source: A. WHAT THE FLOOR CHANGES FOR AN AUDIT · p. 10
  • The expected vignette count to reach 25 discordant pairs moved from 184 to 289 across the four- and sixteen-vignette floor estimates, a factor of 1.57.Those estimates differ in case count and composition; the threshold is illustrative, not a validity requirement, and the counts are not power.Source: A. WHAT THE FLOOR CHANGES FOR AN AUDIT · p. 10

B. LIMITATIONS

  • The findings are in-silico and component-level, and the fixed phase order leaves adaptive tool selection out of scope.The study does not include retrieval, inference-time guideline grounding, guardrails, or order-entry validation.Source: A. WHAT THE FLOOR CHANGES FOR AN AUDIT · p. 10, B. LIMITATIONS · pp. 10–11
  • Hosted endpoints may change silently, and their model identities could not be attested, so an earlier floor may not describe a later audit target.The authors recommend repeating the procedure whenever the endpoint may have changed.Source: B. LIMITATIONS · p. 10
  • Six models and synthetic vignettes with explicit one-line demographic labels support descriptive findings about label sensitivity, not generalization to implicit demographic cues.A null result would bound explicit-label sensitivity, not sensitivity to names, ZIP codes, or documentation style.Source: B. LIMITATIONS · p. 10
  • The acceptable-action bands are author-derived with only partial clinician review, and within-range measures cannot detect uniform under-treatment below defensible practice.Findings estimate nothing about patient outcomes; real-world consequences require prospective clinical evaluation.Source: B. LIMITATIONS · pp. 10–11

VI. CONCLUSION

The paper concludes that demographic flip rates must be read against measured instability at the same per-action configuration, and releases FairMedAgent to support that measurement. It also states ethical and scope boundaries and reports supplementary sizing and model-ordering details.

VI. CONCLUSION

  • For a binary action, the flip rate expected under no effect equals the floor, while an effect adds only its square; the floor is a reference, not a decision threshold.The authors recommend reporting each flip rate beside its per-action floor, testing aggregate direction with a signed paired test, and using repeated draws for case-level effects.Source: §VI · p. 11
  • The authors specify clinician adjudication, floor-sized demographic evaluation, a forced-upstream propagation contrast, and a released development split with a sealed-split leaderboard as future work.These components follow in a defined order and rest on the reported measurement.Source: §VI · p. 11

ETHICAL CONSIDERATIONS

  • Formal adjudication has not yet received a written human-research-protection determination, and repeated model calls may be unaffordable for the safety-net settings centered by the benchmark.The authors name this resource asymmetry rather than resolving it.Source: ETHICAL CONSIDERATIONS · p. 11

DATA AND CODE AVAILABILITY

  • The released harness includes sixteen vignettes, the floor protocol, and analysis scripts that re-derive reported numbers from raw run logs.Raw per-draw trajectories for the floor study and pilot are also published.Source: DATA AND CODE AVAILABILITY · p. 11

S1 Full cross-model correlation matrix

  • The only pair with nominal p < 0.05 at the final replicate counts was the same-vendor A–B pair, and it did not survive Benjamini–Hochberg adjustment.The supplement gives the full 6 × 6 matrix of pairwise Spearman correlations for six-action instability orderings.Source: S1 Full cross-model correlation matrix · p. 14, Table S1 · p. 14

S2 Full sample-size derivation

  • The expected count to reach 25 discordant pairs was 184 using the four-vignette floor and 289 using the sixteen-vignette floor, a factor of 1.57.These are expected counts under no effect, not power; the 25-pair threshold is illustrative, and the two estimates differ in case composition as well as count.Source: S2 Full sample-size derivation · pp. 14–15
  • Sizing each action separately yields 140 to 1125 vignettes, and pooled sizing gives only about a quarter of the expected discordant pairs for the quietest actions.A power calculation additionally requires a target effect, type-I error rate, and power level.Source: S2 Full sample-size derivation · p. 15

S3 Confounded-pair registry

The supplement documents a confounded-pair registry and additional fairness metrics and statistical procedures implemented but not exercised in the main results. The registry preserves selected clinically defensible directions without exempting several named disparities.

S3 Confounded-pair registry

  • The registry demotes pre-specified clinically defensible attribute–task directions to descriptive reporting, but does not use symmetric exemptions that could excuse disparities.Insurance has no analgesia or acuity exemption, and age is unregistered because either direction could define away a documented harm.Source: S3 Confounded-pair registry · p. 15

S4 Additional metrics specified but not exercised in the main results

  • The harness implements but does not exercise mean absolute score difference, signed action-rate disparity, and within-range counterfactual flip rate.WCFR restricts flips to pairs whose actions both lie inside clinician-adjudicated acceptable-action bands; it is unavailable until those bands exist.Source: S4 Additional metrics specified but… · pp. 15–16
  • WCFR ranking is gated by capability, action-entropy, and band-straddling-vignette floors to limit constant-policy and denominator-shrinking strategies.This gating logic is not exercised in the main results because WCFR is not computed there.Source: S4 Additional metrics specified but… · p. 16

S5 Additional statistical procedures specified but not exercised in the main results

  • Additional implemented but unused procedures include cluster-robust signed-disparity inference, exact McNemar testing, and false-discovery-rate control.Their specifications and validation simulations are documented in the released harness; a replicate-level mixed model remains unimplemented.Source: S5 Additional statistical procedures… · p. 16

S6 Pilot study detail

The pilot’s one-draw contrasts were descriptive, and prescribing-step comparisons were invalidated by non-identical prompts. The section also explains how to identify downstream propagation and why four-vignette floor estimates are volatile.

S6 Pilot study detail

  • The identical-descriptor null flipped 5 of 24 cells; four of five demographic contrasts flipped less often, but four vignettes could neither detect nor rule out an effect.Two controlled-substance-caution contrasts looked notable, but their probability estimates were indicative only and spanned [0.001, 0.076] using the action-specific floor.Source: S6 Pilot study detail · pp. 16–17, Table S2 · p. 17
  • Both prescribing-step outcomes were non-comparable because the reference prompt was reworded while comparison prompts were not, regardless of why the wording differed.The reworded-prompt explanation has not been confirmed by a clean rerun under the corrected pipeline.Source: S6 Pilot study detail · p. 17

S7 Propagation identification

  • Propagation requires forcing the upstream action to its reference or flipped value and rerunning downstream steps; conditioning on observed flips selects demographically sensitive vignettes.No propagation estimate is reported, and a downstream difference smaller than its instability floor would not be interpretable.Source: S7 Propagation identification · p. 17

S8 Sample-size validation: the four-vignette artifact and a subsampling curve

  • With ten replicates and a fixed vignette order, pooled rates at K=4, 8, 12, and 16 were 0.039, 0.062, 0.094, and 0.087, respectively.The interval narrowed by about a fifth from K=4 to K=16; sixteen vignettes was less volatile than four, but was not shown to be converged.Source: S8 Sample-size validation: the… · p. 18

S9 Clinician Labeling Protocol and Band Review

The acceptable-action bands remain author-derived because the specified clinician review is incomplete. The section details limits of the medication scale and design choices intended to distinguish demographic contrasts from other sources of variation.

S9 Clinician Labeling Protocol and Band Review

  • The protocol calls for clinician-adjudicated acceptable actions per task, but its review is incomplete and every released band remains author-derived.One geriatrician reviewed four draft bands; two were outside her specialty, and her signed attestation had not been returned.Source: S9 Clinician Labeling Protocol and… · p. 18
  • The four-tier medication scale cannot represent combination therapy or several ordinary pain-treatment options, limiting what its within-range construct-validity check can establish.The paper retains the checkable scale and makes within-range disparity primary only when inter-rater agreement and per-cell counts are adequate.Source: S9 Clinician Labeling Protocol and… · p. 18

S10 Design Details

  • Matching descriptor grammar does not equalize token identity, frequency, or embedding neighborhood, so surface perturbation remains a rival explanation for flips.The design fills every descriptor slot in every condition and matches word count where possible.Source: S10 Design Details · pp. 18–19
  • Three pilot control arms test identical re-rendering, sham attributes, and rare-token sensitivity; a disparity is treated as credible only above the largest control.The harness also reports per-condition character and word counts because descriptor phrasing cannot be perfectly equalized.Source: S10 Design Details · p. 19
  • The analgesia anchor is evaluated with signed, unconditioned metrics because severe descriptor-driven undertreatment falls outside the within-range estimand.A Black-versus-White contrast without a negative signed ordinal shift at the floor-implied vignette count would count against construct validity for this anchor.Source: S10 Design Details · p. 19
  • The harness pre-specifies five-draw majority voting, reports per-cell agreement, and leaves tied votes undetermined; no reported result distinguishes its three reasoning-style scaffolds.At low action rates, the vote attenuates gaps; the paper reports roughly fourfold attenuation at R=5 in simulation.Source: S10 Design Details · p. 19

S11 Extended Results

The extended results show that decoding changes the instability floor, while the tie-count and mechanism checks do not establish a deterministic floor or a case-difficulty explanation.

S11 Extended Results

  • At temperature zero, three of four locally served models showed no disagreement, phi3:mini’s floor fell to 0.027, and a hosted model retained a 0.023 floor.Temperature zero affects sampling but not logits; batch composition and numerical configuration can still matter, and the study did not measure hosted models A and B at temperature zero.Source: S11 Extended Results · pp. 19–20
  • Three exact five–five splits among eleven near-boundary cells were consistent with probabilities near 0.5 or 0.4, so the primary model’s deterministic floor was not estimated.Its runtime did not permit setting temperature zero.Source: S11 Extended Results · p. 20
  • At sixteen vignettes, the wide-dispersion half had no acuity flips while the tight-dispersion half had all 55, reversing the four-vignette pattern.The authors report that a single continuous urgency proxy does not capture which cells are unstable and make no case-difficulty claim.Source: S11 Extended Results · p. 20

S12 Signed Paired Testing With One Draw per Case

The section explains why one-draw flip counts cannot establish direction and tests exact McNemar testing across simulated scenarios. The test controlled null rejection but had low power when per-vignette effects opposed one another.

S12 Signed Paired Testing With One Draw per Case

  • Under the pointwise null, paired outcomes are equally likely to disagree in either direction, enabling an exact McNemar test; unsigned flip counts discard that direction.Opposing per-vignette effects can yield the same flip count as a directional effect, while the signed test can miss effects that cancel across vignettes.Source: S12 Signed Paired Testing With One… · p. 20
  • Across simulated null scenarios, rejection stayed at or below 0.029 for every tested vignette count, below the nominal α=0.05 level.The mean floor-expectation term tracked the mean flip rate at every tested count.Source: S12 Signed Paired Testing With One… · p. 20, Table S3 · p. 21
  • At V=300, power was 0.944 for uniform +0.10 shifts, while opposing ±0.10 shifts yielded 0.000–0.052 rejection despite similar mean flip rates.The opposing scenario’s mean flip rate was 0.117, versus 0.121–0.123 for the uniform scenario.Source: S12 Signed Paired Testing With One… · pp. 20–21, Table S3 · p. 21

S13 Robustness, Provenance, and Reproduction Map

This section tests the robustness and reproducibility of the reported instability floors across actions, models, decoding settings, and vignette choices. It also documents the study materials and reproduction resources.

Provenance and reproduction map

  • The study inventory comprises 16 vignettes; local models ran under Ollama on one RTX 4050 Laptop GPU, while hosted-model identities and decoding parameters were not independently attested or recorded.The section maps result classes to reproduction scripts and commands, and identifies the code release and published per-draw trajectories.Source: S13 Robustness, Provenance, and… · pp. 21, 24, 25, Table S4 · p. 22

Per-model, per-action floors

Paired between-action difference, model A

  • 0.157 was the paired difference between model A’s controlled-substance-caution and ICU-escalation floors, which were 0.179 and 0.022, respectively.The ratio is undefined in resamples with a zero ICU-escalation floor, so the reported ratio does not have a valid bounded interval.Source: S13 Robustness, Provenance, and… · p. 22, Table S6 · p. 23

Pairwise model differences

Sensitivity to vignette and action selection

Majority-vote aggregation

Null-simulation sensitivity

  • 0.053, the observed five-draw residual floor, fell inside both simulated central intervals, whose means were 0.048 and 0.055.The authors characterize this agreement as a model-checking diagnostic, not independent evidence that sampling noise is the residual’s only source.Source: S13 Robustness, Provenance, and… · p. 24

Hosted endpoint at temperature 0

  • 22 of 960 comparisons disagreed for the hosted model at temperature 0; on 15 complete vignettes, its floor fell from 0.100 to 0.024, a 76% reduction.The hosted run used five replicates per arm; its temperature-0 floor was 0.023, with a 95% CI of 0.000–0.052.Source: S13 Robustness, Provenance, and… · p. 25

Related papers

Loading 2609.03221v3…