PaperlayerView exact revision on arXiv

Source-linked AI summary

When the AI Leaves the Tailorshop: Measuring What an LLM Advisor Leaves Behind in Complex Problem Solving

Robin Welsch

arXiv:2610.00163v1cs.HCcs.AI

TL;DR

The paper examines whether AI advice in complex problem solving supports users’ own understanding and independent capability, as well as their performance with assistance. Across two preregistered studies using the Tailorshop, assistance improved aided performance, but much of the performance benefit disappeared after withdrawal and the knowledge advantage was limited and tied to solvency.

  • Problem

    It remains unclear whether AI assistance in dynamic complex systems improves users’ own understanding and later unaided control, beyond improving performance while assistance is present.

    Source: §1 · p. 2

  • Method

    Two preregistered studies compared people managing the Tailorshop with and without an LLM advisor, measuring system knowledge and, in Study 1, control after advisor withdrawal.

    Source: §1 · p. 2

  • Results

    Across the studies, most of the aided performance advantage disappeared after withdrawal, while Study 2’s small registered knowledge advantage was closely tied to solvency.

    Source: §8 · p. 25

  • Takeaways & Limitations

    Evaluation of human–AI interaction should assess users’ understanding and independent capability alongside performance achieved with AI support.

    Source: §8 · p. 25

    Without pretests, the studies cannot quantify individual knowledge gains, and Study 2’s knowledge advantage was closely tied to solvency.

    Source: §7.7 · p. 24

Abstract

from arXiv · show

Complex problem solving depends on acting effectively and understanding how a system works. AI advice may support these outcomes unequally. Two preregistered experiments compared participants managing a simulated clothing factory with and without an LLM advisor. Across studies, AI-supported participants reported greater confidence and understanding with less effort. In the first study (N=200), assistance increased company value but produced no detectable prediction-accuracy difference. After withdrawal, previously supported participants outperformed controls when decisions were scored against repeating previous choices, but not default settings. Within the AI-supported group, more frequent recommendation alterations predicted better unaided performance. In the second study (N=198), AI-supported participants went bankrupt less often and showed a small knowledge advantage in the registered analysis, largely associated with remaining solvent. More frequent recommendation alterations predicted higher knowledge within the AI-supported group. Applied HAI evaluation should assess users' understanding and independent capability alongside the performance achieved with AI support.

1 Introduction

The paper asks whether LLM assistance in complex, dynamic systems improves users’ understanding and independent capability, not only performance while assistance is present. Across two preregistered studies, it finds that assistance can raise confidence and aided performance while leaving knowledge and unaided-control benefits limited or conditional.

1 Introduction

  • Prior HAI work has focused on bounded tasks, isolated AI benchmarks, or short static tasks, leaving effects on structural knowledge and unaided control in dynamic systems unresolved.Complex problem solving involves dynamic, opaque, multi-goal systems, where users need a mental model to test hypotheses before acting.Source: §1 · p. 2
  • The two studies use the Tailorshop microworld to compare unaided participants, participants advised by an LLM, and an autonomous LLM, measuring systems knowledge and, in Study 1, later unaided control.Study 1 tests prediction accuracy and withdrawal performance; Study 2 replaces the prediction format with a structural-knowledge battery.Source: §1 · p. 2
  • After withdrawal, Study 1’s aided performance advantage largely disappeared against do-nothing defaults, with a small advantage remaining against repeating prior decisions (g=0.32).Study 1’s prediction test could not detect knowledge because both groups scored at chance; Study 2 found a small registered knowledge advantage that largely tracked solvency.Source: §1 · pp. 2–3
  • Among AI-supported participants, more frequent recommendation alterations predicted better later unaided performance and higher structural-knowledge scores.The association does not establish that altering recommendations caused these outcomes.Source: §1 · p. 3
  • Assistance increased users’ reported sense of understanding without a corresponding difference on the accuracy scale; 64% estimated their unaided firm value at or above their assisted outcome, despite a 38k decline.The authors distinguish what users believe they know from what they can answer correctly.Source: §1 · p. 3

2 Related Work

Complex problem solving depends on building a structural model of a dynamic system and using it to guide interventions. Prior HAI research leaves open whether AI-supported performance translates into knowledge and control users retain after assistance ends.

2.1 Complex Problem Solving and Systems Thinking

  • Complex systems combine many coupled variables, autonomous change, hidden structure, and conflicting goals; microworlds provide controlled but sufficiently rich settings for studying them.The Tailorshop sits between highly elaborate simulations and minimal systems designed primarily for psychometric tractability.Source: §2.1 · p. 3
  • In complex problem solving, structural knowledge supports control, while disciplined probing—especially varying one factor at a time—helps identify causal relations.Confounded probes can leave system structure unidentified, and acting too early without a model can lead to flight into action.Source: §2.1 · p. 3

2.2 Augmentation, Offloading, and What the Human Retains

  • HAI studies have largely examined structured tasks, while prior evidence on unaided performance after AI use comes from short, static tasks, leaving dynamic-system retention unresolved.The present setting adds an extended horizon, conflicting goals, and autonomous dynamics with feedback to model.Source: §2.2 · p. 4
  • Assistance can either offload knowledge or support learning through explained reasoning; performance with an assistant cannot distinguish these possibilities, making post-withdrawal knowledge and control key outcomes.The relevant assessment is what users know and can do independently after support is removed.Source: §2.2 · p. 4

3 Study 1

Study 1 tests whether LLM advice changes users’ system knowledge and control, including after the advisor is withdrawn. It uses a preregistered two-group, two-phase Tailorshop experiment, with outcomes based on prediction accuracy and state-conditional performance.

3 Study 1

  • The design compares Control and AI-supported participants across an aided Phase A and an unaided Phase B after the firm is reset; both phases last 12 months.The advisor is available throughout Phase A only, and Study 1 tests prediction accuracy after Phase A and decision quality across both phases.Source: §3 · pp. 4–5
  • Study 1’s verdict table leaves the joint confirmatory result inconclusive because the H1 prediction battery was answered at chance and could not be judged equivalent.The table classifies each dependent variable as worse, equivalent, or better for AI-supported participants; Study 1’s H1 axis is void rather than equivalent.Source: Table 1 · p. 5

3.1 Participants and power

  • The analytic sample comprised 200 participants, 100 per group, recruited to a target set using simulations of the two confirmatory models.The reported simulated power at N=200 was .84 for an 8-point H1 accuracy gap and .91 for a conservatively reduced H2 log-odds effect of −0.45.Source: §3.1 · p. 5

3.3.1 Tailorshop microworld.

  • Both groups used the Tailorshop, a shirt-manufacturing microworld with 12 monthly decision controls and a 12-variable state vector under nonlinear, coupled dynamics.The system’s delayed or indirect relationships include demand responding to customer interest and retail price, and machine capacity decaying without maintenance.Source: §3.3.1 · pp. 7–8

3.3.2 Prediction task.

  • The post-Phase-A mental-model task contains 24 engine-scored items asking whether a decision would raise or lower company value after three months versus no change.Items span marketing, production, and human-resources subsystems and are keyed by replaying each intervention against the do-nothing default from the same state.Source: §3.3.2 · p. 8

3.3.3 LLM advisor.

  • The autonomous LLM benchmark used the advisor’s model and task description through a structured action interface, with 50 sessions on the same fixed market and reset conditions.Its firm value increased, with a rising-months rate of 0.91 and end value of 305k, unlike typically declining human trajectories reported in prior Tailorshop samples.Source: §3.3.4 · p. 8

3.5 Measures

  • The state-conditional performance measure scores each month against a do-nothing branch from the participant’s recorded state, while prediction accuracy measures correct answers across 24 items.Self-reports included confidence, effort, understanding, perceived control, and AI reliance; the prediction task was administered once after Phase A.Source: §3.1 · p. 6, §3.5 · pp. 8–9

3.6 Analysis Plan

  • Confirmatory hypotheses were tested with mixed-effects logistic regressions, and practical equivalence required both a non-significant test and a Bayesian 95% interval within the ±0.40 ROPE.Missing outcomes were not imputed, and sensitivity analyses examined per-test inclusion and exclusion departures.Source: §3.6 · p. 10

4 Study 1: Results

Study 1 found that LLM assistance improved performance while available, but its advantage largely disappeared after withdrawal; the study’s prediction test could not establish whether users learned the system. Recommendation alterations were associated with better unaided performance, while participants’ confidence and reported understanding did not reliably track what they knew or achieved.

4.1 Phase A performance: the pair beat the human and lost to the AI

  • 88.4% versus 73.4% of months, AI-supported participants beat the do-nothing counterfactual compared with Control, and their mean Phase-A end value was 232k versus 138k.The AI-supported group also had fewer bankruptcies during assistance, but the human–AI pair ended below the autonomous model’s value.Source: §4.1 · p. 10

4.2 The user’s model (H1).

  • Prediction accuracy did not differ detectably: 0.495 for Control versus 0.512 for AI-supported, and both groups scored around chance on a battery that could not establish equivalent knowledge.The registered contrast was +0.08 log-odds, with a 95% CI of [−0.10, 0.27].Source: §4.2 · p. 10

4.4 Exploration

  • Systematic single-variable probing was lower during aided months but did not predict later unaided control; similar settings after withdrawal were consistent with retained configurations, not proof of structural learning.Aided VOTAT scores were 0.089 for AI-supported participants versus 0.148 for Control, while Phase-A probing did not predict unaided Phase-B control.Source: §4.4.1 · p. 11, §4.4.2 · p. 11

4.4.3 Recommendation alteration predicted unaided performance.

  • Participants altered at least one recommended control in 51.3% of consultations, and alteration frequency was associated with better unaided performance, though an unmeasured disposition to engage could explain the association.During assistance, altered months were less likely to beat the do-nothing counterfactual, and more altered controls were associated with lower success; baseline ability and regression to the mean also qualify withdrawal comparisons.Source: §4.4.3 · pp. 11–12, §4.4.4 · p. 12

4.4.4 Who was lifted, and who paid at withdrawal.

  • With the advisor, AI-supported participants reported greater confidence and understanding and lower effort, but those advantages disappeared at withdrawal and self-assessments were weakly related to accuracy.Participants also substantially overestimated company value; at withdrawal, AI-supported participants’ estimates did not reflect the size of their firms’ losses.Source: §4.4.5 · p. 12, §4.4.6 · pp. 12–13

4.3 Unaided performance (H2).

  • On the persistence reference, AI-supported participants retained a small unaided advantage (b = +0.24), while their advantage over Control disappeared against the do-nothing reference.The confirmatory withdrawal interaction was −1.23 log-odds; the persistence-reference comparison was b = +0.24, p = .022, g = 0.32.Source: §4.3 · pp. 10–11, Figure 5 · p. 13

4.5 How participants described using the advisor

  • Participants described both delegating decisions to the advisor and selectively combining or overriding its recommendations; advice also drew attention to overlooked conditions, with persistence of that recognition left open.Setbacks prompted reassessment in either direction: participants sometimes returned to the advisor and sometimes rejected its advice.Source: §4.5 · pp. 14–15

5 Study 2: Testing for Structural Knowledge

Study 1’s prediction task did not measure the intended knowledge construct, so Study 2 replaced it with a structural-knowledge battery. The study tested whether advisor availability affected users’ mental models after a longer aided phase, without testing withdrawal.

5 Study 2: Testing for Structural Knowledge

  • Study 1’s prediction task did not measure its intended construct; Study 2 therefore introduced a battery intended to detect structural knowledge.The paper distinguishes forecasting system outputs from knowing which variables are linked and in what direction.Source: §5 · pp. 15–16
  • Across the 199 complete reflection sequences, Control confidence rebounded at the Phase-B reset, while AI-supported participants’ confidence declined after the advisor’s withdrawal despite equal performance.Source: Figure 8 · p. 16

5.1 Design, participants, and power

  • Study 2 tested H1 alone after a 36-month aided phase, replacing the prediction task with a structural-knowledge battery and ending before any withdrawal phase.The phase was three times as long as Study 1’s; Study 2 did not test unaided control after withdrawal.Source: §5.1 · p. 16

5.2 Structural-knowledge battery

  • The 41-item battery tested recognition of direct relations, mediated chains, and connectedness among the Tailorshop’s variables.Overall reliability was KR-20 = .66 and ωt = .76; facet KR-20 values were .15, .67, and .37.Source: §5.2 · pp. 17–18

5.3 Hypothesis and scoring

  • The registered outcome was latent ability θ, with chance performance set at .209 and the AI-alone benchmark at .670 on the same battery.Study 2 retained the 2PL specification and ±0.40 region of practical equivalence.Source: §5.3 · p. 18

6 Study 2: Results

With the advisor, Study 2 participants performed substantially better and reported greater confidence and understanding with less effort. Knowledge favored AI-supported participants in the registered analysis, but the difference was closely associated with solvency; within the aided group, recommendation alterations were associated with higher knowledge.

6.1 A small registered knowledge advantage for AI-supported

  • AI-supported participants had a registered latent-knowledge advantage of +0.42, but the difference was closely associated with solvency.The pooled proportion-correct estimate across studies was g= +0.14 (95% CI [−0.05, +0.34], p= .15), using different instruments and not resolving Study 2’s result.Source: §6.1 · p. 18, Figure 10 · p. 19

6.2 Performance separated enormously

  • The rising-months rate was .342 in Control versus .495 in AI-supported, and insolvency was 44% versus 7%.The registered knowledge contrast favored AI-supported by +0.42, while a solvency-adjusted contrast was +0.04; the decomposition is not causal.Source: §6.2 · p. 18, §6.1 · p. 18, Figure 10 · p. 19

6.3 Knowledge predicted performance mainly in the unaided group

  • Structural knowledge was more strongly associated with the rising-months rate in Control than in AI-supported participants.The month-level interaction was b= −0.73, z= −2.61, p= .009; the participant-level interaction was not significant, and the association does not establish causality.Source: §6.3 · p. 19, Figure 11 · p. 20

6.4 What participants authored predicts what they knew

  • Within the AI-supported group, more frequent recommendation alterations were associated with higher structural knowledge, but not with performance.Alteration frequency correlated with latent structural knowledge at r= +.330, p= .001, and predicted θ with performance held constant; prior knowledge or engagement could explain the association.Source: §6.4 · p. 19

6.5 And it felt like understanding

  • AI-supported participants reported higher confidence, understanding, and perceived control, alongside lower effort, across both studies.In Study 2, confidence was 55.3 versus 47.2 and understanding was 4.64 versus 4.16; measured-knowledge associations with mean understanding and confidence were weak or unresolved.Source: §6.5 · pp. 20–21, Figure 12 · p. 21

7 General Discussion

Across the studies, assistance improved in-use performance and users’ self-assessments, while much of the performance benefit did not persist after withdrawal. The discussion distinguishes assisted success from retained capability and emphasizes limits on claims about learning and independence.

7 General Discussion

  • With the advisor available, participants ran more functional firms with substantially less insolvency, but the large aided performance advantage largely disappeared after withdrawal.The persistence reference showed a small unaided advantage, so some benefits remained while most depended on the advisor.Source: §7 · p. 22, §7.1 · p. 22

7.1 Performance was lent, the knowledge benefit was less clear

  • Study 1’s chance-level knowledge test could not establish whether groups learned equally, while Study 2’s small registered knowledge advantage was closely tied to solvency.Within solvency strata, no group difference was detected; because solvency is itself an outcome, the comparisons do not distinguish learning enabled by survival from selection.Source: §7.1 · p. 22

7.2 Capability lending

  • Capability lending names a pattern in which improved outcomes and felt understanding exceed gains in independent capability; it is not a distinct mechanism or claim of synergy.The discussion connects the pattern to how work is allocated between user and advisor and to what users retain after following beneficial advice.Source: §7.2 · p. 22

7.3 A bypass, not a deficit

  • Participants altered recommendations more often for monthly flow controls than for structural controls, but alteration rates alone do not establish what they understood.The discussion notes that structural variables work through delays and couplings that may require tracking changes over months.Source: §7.3 · p. 23

7.4 Authorship predicted what remained

  • More frequent recommendation alterations predicted unaided performance in Study 1 and structural knowledge in Study 2, but authorship was self-selected.The observational associations are consistent with a generation account but do not establish that selective authorship produced lasting understanding.Source: §7.4 · p. 23

7.5 Assistance is felt as understanding

  • Confidence and felt understanding rose during assistance, but subjective changes lagged behind the loss of the performance advantage.Higher ratings and weak knowledge correlations do not establish individual miscalibration, given the knowledge measures’ limitations.Source: §7.5 · pp. 23–24

7.6 Implications

  • Evaluation should distinguish assisted performance, the ability to challenge advice, and the ability to continue without assistance.The discussion also proposes unaided outcomes and microworlds with recorded decisions, states, and ground truth for assessing retained capability.Source: §7.6 · p. 24

7.7 Limitations and Future Work

  • Learning evidence is limited, and the withdrawal test establishes immediate transfer after a reset rather than lasting independence.Neither study had pretests to quantify individual knowledge gains; the primary score’s ceiling also limits sensitivity.Source: §7.7 · pp. 24–25

8 Conclusion

Across two preregistered studies, AI advice improved performance and raised confidence and felt understanding while reducing effort. Much of the performance benefit depended on continued assistance, underscoring the need to assess users’ independent capability.

8 Conclusion

  • Across two preregistered studies, AI advice improved performance and raised confidence and felt understanding while reducing effort.Source: §8 · p. 25
  • After assistance was withdrawn in Study 1, most of the performance advantage disappeared, though a small benefit remained against the persistence reference.Source: §8 · p. 25
  • More frequent recommendation alterations predicted better unaided performance in Study 1 and higher structural knowledge in Study 2.Source: §8 · p. 25
  • HAI evaluations should assess assisted performance, measured knowledge, and independent control separately; better outcomes with AI do not by themselves establish greater user capability.Source: §8 · p. 25

A.1 Deviations from preregistration

The appendix records preregistration deviations and analytic specifications, including exclusions, allocation and sampling choices, and an unused telemetry measure.

A.1 Deviations from preregistration

  • Five Study 1 deviations and one Study 2 deviation are recorded; all but D5 were decisions made blind to outcomes, while D5 was a post-freeze implementation omission.Source: §A.1 · p. 30
  • The Study 1 manipulation check excluded 12 Control participants and no AI-supported participants; selection bias remains possible, although H1 and H2 interpretations held across exclusion sets.The reported H1 estimates ranged from +0.08 to +0.11 and H2 from −1.17 to −1.23; this does not establish robustness of every exploratory result.Source: §A.1 · p. 30
  • Study 1 also used a 44:2 Control-heavy final batch, an unregistered verdict label, a shared 200-person confirmatory sample, and omitted one registered exclusion rung.The allocation had no detectable effect; the omitted rung would exclude one Control participant and leave H1 and H2 verdicts, ROPE decision, and direction unchanged.Source: §A.1 · p. 30
  • Study 2 recruitment stopped at 101 Control and 97 AI-supported participants, three short of the registered quota, because fielding closed for manuscript preparation.Source: §A.1 · p. 31
  • Study 1 screen-inspection telemetry was collected but not analyzed because its recorded events proved a poor index of information search.Source: §A.1 · p. 31
  • Study 2 analytic specifications included Laplace estimation, per-chain sign alignment for the 2PL model, and an added convergence gate that changed no reported verdict.These choices were recorded as specifications rather than preregistration deviations.Source: §A.1 · p. 31

A.2 Power analysis details

Both studies’ sample sizes were planned with Monte Carlo simulations of their confirmatory mixed models, rather than a two-sample summary.

A.2 Power analysis details

  • Both studies used Monte Carlo simulations of confirmatory mixed models across analytic group sizes of 60, 80, 100, and 120, with 400 samples per cell.Source: §A.2 · p. 31
  • At 100 participants per group, Study 1’s H1 power was 0.84 for an 8-point accuracy gap but 0.49 for a 5-point gap.Source: §A.2 · p. 31
  • At 100 participants per group, Study 1’s H2 reached 0.91 power for the registered withdrawal interaction; no pilot data informed its power parameters.Source: §A.2 · p. 31

A.3 Study 2 power analysis details

Study 2’s target of 100 completers per group was set by Monte Carlo simulation of the registered item-level model, accounting for guessing floors and uncertain item discrimination.

A.3 Study 2 power analysis details

  • The Study 2 target was 100 completers per group, based on Monte Carlo simulation of the registered item-level model.Source: §A.3 · p. 31
  • Guessing floors reduced the observable accuracy shift at d=0.40 from 7.5 to 3.3 points, motivating the power simulation’s forced-choice battery design.The floors were 1/6 on 20 direct-relation items and 1/4 on the other 21 items.Source: §A.3 · p. 31
  • Power simulations generated data with a three-parameter model carrying the guessing floors and analyzed it with the registered two-parameter model; power was reported as a band because item discrimination was least certain.Source: §A.3 · p. 31

B Method details

The methods specify the Tailorshop environment and participant procedures, the measures and exclusions, and the statistical models used to evaluate knowledge and performance. They also document diagnostic and design qualifications relevant to interpreting the analyses.

B Method details

  • Recruitment required English as a first language, UK residence, and no prior Tailorshop exposure, with participants outside the age ceiling retained after self-report.The stated age ceiling was 65, although one 67-year-old in Study 1 and one 74-year-old in Study 2 were retained.Source: B Method details · p. 32
  • The 2PL estimates item difficulty and discrimination while inferring ability from response patterns; the registered one-parameter model fixes discriminations equal.Study 1’s reanalysis and Study 2’s primary item-response model used the specifications described in the passage.Source: B Method details · pp. 32–33
  • The Tailorshop tracked a changing business state and let participants adjust twelve monthly controls, including staffing, pricing, advertising, production, and maintenance.Demand depended on customer interest and price; production capacity depended on workers, machinery, and its condition.Source: B Method details · p. 32
  • Participants were screened for attention, manipulation, engagement, and session duration, with additional data-quality rules for extreme forecasts and unusually fast decisions.Forecasts above 1,000,000 were set missing without excluding the participant; phase medians below 15 seconds per decision triggered exclusion.Source: B Method details · p. 32
  • Study 2 used a single 36-month aided phase followed by a 41-item structural-knowledge battery, with no withdrawal phase.The AI-alone benchmark and engine seeds matched Study 1.Source: Figure 13 · p. 33
  • Bayesian models were fit in brms over Stan, with convergence targets and posterior summaries specified for the analyses.Diagnostics noted that the population discrimination intercept missed the bulk-ESS target under the unconstrained-sign model, but no conclusion rested on it.Source: B Method details · p. 33
  • The reported design remained registered, while Table 3’s stratified values were exploratory and uncorrected.The reference-level detail was moved out of the main text for concision.Source: B Method details · p. 31, Table 3 · p. 34
  • Participants completed post-task trust, manipulation, and strategy instruments, including prompts about their understanding and use of advice.Prediction items covered marketing, production, and human resources, with intuition-aligned and intuition-misaligned subsets.Source: B Method details · p. 34

B.1 Supporting robustness checks

Supporting checks examine the robustness of prediction and withdrawal findings, solvency-related knowledge patterns, and the sensitivity of the Study 2 knowledge contrast. They also describe the advisor’s role in the decision loop.

B.1 Supporting robustness checks

  • The advisor could recommend decisions but had no engine actuation rights; participants alone submitted each month’s choices.Advice used observable dashboard information and the running Phase-A transcript, while the no-advisor loop reduced to participant submission and engine response.Source: Figure 14 · p. 35
  • Restricting Study 1 H1 to stronger-effect items did not reveal a group difference (p = .53); the eight items with engine effects below 5k had accuracy .447 (95% CI [.421, .473]).The other sixteen items yielded accuracy .532.Source: §B.1 · pp. 34–35
  • Alternative Study 1 withdrawal estimates remained negative across scoring variants and solvent-only samples, with all reported p-values below .001.Estimates ranged from −1.35 for rising-months scoring to −0.53 for persistence scoring; solvent-only estimates were −1.18 and −1.07 under the stated thresholds.Source: §B.1 · p. 35
  • Among 147 solvent Study 2 participants, the knowledge–performance interaction was not detected; correcting for restricted performance spread left both group correlations at +.13.The solvent-only interaction was +0.14 (z = 0.50, p = .62).Source: §B.1 · p. 35
  • A fixed semantic-option-code response pattern scored 16/41 = .390, and 103 of 198 participants exceeded it.This was specified as a strategy benchmark, not a general ceiling on knowledge-free responding.Source: §B.1 · p. 35
  • Study 2 accuracy and condition-centered 2PL scores gave similar performance correlations, and excluding participants with zero rising-months rates left a positive correlation.After excluding 28 participants, r = +.182 (p = .018), with rank correlation ρ = +.235.Source: §B.1 · p. 36
  • Study 2’s estimated knowledge contrast stayed near +0.42 under several prior specifications, but the checks did not establish measurement validity.The symmetric-prior and positive-prior fits yielded +0.43 and +0.42, respectively; the tighter unrestricted fit mixed poorly and was not counted as robustness evidence.Source: §B.1 · p. 36

C Bayesian evidence for the null claims

The section distinguishes evidence for a point null from evidence that an effect is practically small, and records corrections to earlier analyses. It also reports a qualified exploratory item split and specifies what the deployed prompts contained.

C Bayesian evidence for the null claims

  • The pooled calibration row was removed because it treated correlations from the same participants as independent, and an unreproducible dropout claim was withdrawn.Separate calibration correlations remain reported.Source: C Bayesian evidence for the null claims · p. 36
  • The registered discrimination-sign split classified items by posterior-mean sign, but 19 of 41 intervals spanned zero.A selected-item analysis sharpened the interaction but was exploratory because the same responses determined inclusion and estimated the contrast; it rested on only two negatively discriminating items.Source: C Bayesian evidence for the null claims · p. 36
  • Bayes factors assess the point null, while the ±0.40 ROPE describes practical magnitude; a positive interval alone does not settle either question.BF01 above 3 marks moderate evidence for the null and above 10 strong evidence; the ROPE indicates posterior mass within the registered practical-equivalence region.Source: C Bayesian evidence for the null claims · p. 36, Table 4 · p. 37
  • One reported contrast was direction-positive with practical magnitude unresolved and no Bayes factor reported.This note does not identify the contrast beyond the information supplied in the table entry.Source: C Bayesian evidence for the null claims · p. 37

D Prompts used in the study

  • The deployed prompts specified the advisor’s objective, observable state, controls, response structure, and a prohibition on claiming access to engine internals.Prompts contained no simulation formulas or coefficients, so the advisor and AI-alone benchmark had only interface-observable information.Source: D Prompts used in the study · p. 40

Related papers

Loading 2610.00163v1…