PaperlayerView exact revision on arXiv

Source-linked AI summary

Calculation and rejection diverge in language models tested at known mechanical limits

Shaoliang Yang, Jun Wang

arXiv:2610.38096v1cs.CEcs.AIcs.CL

TL;DR

Models may report engineering designs beyond physical limits as solved, while prior work had not systematically separated limit-calculation precision from rejection. This study tested models on beam and disk problems around certified limits and found that precise calculation did not ensure full-answer rejection, even when feasibility-only questions prompted rejection.

  • Problem

    Whether models that can calculate physical limits reject specifications beyond them had not been systematically tested while separating calculation precision from feasibility judgment, despite software trusting solved labels.

    Source: Introduction · pp. 1–2

  • Method

    The authors tested beam-collapse loads and disk-burst speeds at known margins around certified limits, measuring each model’s limit-calculation precision separately from rejection.

    Source: Introduction · pp. 1–2

  • Results

    Across two problem families, precise calculation did not ensure rejection; models could accept impossible full-answer requests yet reject the same flaws when asked only about feasibility, and format gains did not transfer consistently.

    Source: Introduction · p. 2, Asked only about feasibility, Opus 5… · p. 7

  • Takeaways & Limitations

    Evaluations and software that act on model status should separate feasibility judgment from the decision to reject a full-answer request.

    Source: Introduction · p. 2

    Small-margin cases treat printed inputs as exact; rounding can make some flaws feasible, and corner checks do not establish impossibility throughout the rounding intervals.

    Source: Capability and rejection diverge · p. 4

Abstract

from arXiv · show

When a language model returns engineering quantities with the status "solved", software that trusts it passes them on, even when the requested design is physically impossible. Such failures could reflect poor calculation, how the answer is requested, or a failure to judge feasibility. We set beam loads and disk speeds at known percentages above or below plastic collapse or burst, measured each model's limit calculation separately and tested 12 models in 4,860 calls, including follow-up experiments on new designs. We find that a model can recognize an impossible design when asked only whether it is feasible, yet accept the same design, reporting "solved", when asked for a full answer. Three models accepted beams 10 to 30% beyond collapse despite precise limit calculations in separate questions, and many accepted replies stated the violation. Asked only about feasibility, Opus 5 and Gemini 3.8 Flash rejected every impossible new design they were given, including those accepted in full answers, without false alarms. Rewording when to decline or changing the answer format raised rejection on the original beams, but not consistently on new designs. Software and benchmarks using a model's status should test feasibility judgment and full-answer rejection separately, and report the exact prompt.

Introduction

The study examines whether models judge feasibility and reject impossible full-answer requests across certified beam-collapse and disk-burst limits, including comparisons with direct feasibility questions and prompt-format changes. It also outlines the planned stated-limit comparison, experimental safeguards, design and reproducibility practices, and release of data and code.

Each problem sits at a certified distance from its limit

  • The study tested 12 served models in 4,860 calls using independently certified beam-collapse loads and disk-burst speeds at controlled distances above and below each limit.The main-set comparison used 20 flawed and 20 valid beams, plus 18 flawed and 18 valid disks; the limits were measured separately from rejection trials.Source: Introduction · p. 1, Each problem sits at a certified… · p. 2, Figure 1 · p. 3

Capability and rejection diverge

  • Opus 5, Opus 4.8 and Gemini 3.8 Flash each rejected at most 2/20 main-set beam flaws, including violations 10% or 30% beyond collapse, despite passing at least 9/10 limit probes.The largest median relative limit-probe error was 0.014%, about 700 times smaller than a 10% violation; separate probes do not show which calculation a model used in a missed trial.Source: Capability and rejection diverge · pp. 2, 4, Table 1 · p. 4
  • Stated loads or speeds alone showed near-chance discrimination of flawed items (AUC 0.48–0.57), and classifiers on all stated inputs found no consistent shortcut cue.The classification tests had limited sensitivity; treating printed values as rounded can also make small flaws feasible, unlike larger flaws that remain impossible at tested rounding-interval corners.Source: Capability and rejection diverge · p. 4

Under one status, replies state the violation and report the problem solved

  • 27 of 74 answered, accepted main-set flaws from Opus 5, Opus 4.8 and Gemini replies stated or printed the violation yet still reported “solved”.Another 29 replies followed a changed model, while 18 showed no sign of a limit check; six Gemini calls without an answer were excluded.Source: Under one status, replies state the… · p. 5, Figure 3 · p. 6

Asked only about feasibility, Opus 5 and Gemini reject the flaws they accept in full answers

  • Opus 5 rejected 40/40 flaws and Gemini 3.8 Flash 20/20 in feasibility-only questions, with no false alarms, including flaws each accepted in full answers.The new designs declared inputs exact and stated the beam support rule; the feasibility question was a separate task and does not establish what models detected during full-answer calls.Source: Figure 4 · p. 7, Asked only about feasibility, Opus 5… · p. 7

A confidence target raises disk rejection; a Python tool shows no detectable change

  • On disks, the confidence-target prompt raised Opus 4.8’s rejection from 5/18 to 18/18 and Opus 5’s from 9/18 to 18/18, with no false alarms.Only Opus 4.8’s gain survived Holm adjustment; Haiku flagged 16/20 valid beams, while the Python tool produced no detectable rejection change for Opus 5 or Fable 5.1.Source: Figure 5 · p. 8, A confidence target raises disk… · p. 8

Rewording the decline rule raises rejection on the original beams

  • On the original beams, Opus 5 rejected 3/20 flaws under the ordinary single status, 10/20 with reworded decline rules, and 14/20 with a status per answer, with no false alarms.The rewording increase was significant; the further four rejections under per-answer status were not. On disks, Opus 5 rejected 13/18 under ordinary status and 17/18 under either alternative.Source: Figure 6 · p. 9, Rewording the decline rule raises… · pp. 9–10

The predicted fall in rejection with more valid answers fails for Opus 5

On the new items, the answer format has no consistent effect

  • On new items, the answer-format effect was inconsistent: the sheet rejected 79/80 flawed designs and per-answer status 80/80 (P = 1), while the neutral note had mixed effects.Opus 5 and Gemini raised no false alarms with the sheet; the 1/80 format difference was below the prespecified decision rule.Source: On the new items, the answer format… · p. 10

Discussion

  • Models accept beam answers whose stated loads exceed collapse by 10–30%; Opus 5 and Gemini reject every tested flaw when asked only whether load or speed is reachable, without rejecting valid items.Both accept some new flaws in full answers, so a rejection rate under one request cannot stand in for feasibility judgment under another.Source: Discussion · p. 10
  • On original beams, rewording the decline rule raised Opus 5 rejection from 3 to 10 of 20, and per-answer status raised it to 14, but gains did not consistently transfer.On new designs, the reworded rule’s gain over the per-answer rule recurred on beams without significance, reversed on Opus 5’s and GPT-6-Luna’s disks, and was absent on Gemini’s beams.Source: Discussion · p. 10
  • Clarifying beam assumptions raised rejection in Opus 5 and Gemini but neither rejected most flaws; the study cautions that new-design results also changed geometry and margins.For Opus 5, two predictions of the account that models preserve valid answers failed; adding Python did not detectably change rejection in the two tested models.Source: Discussion · pp. 10–11
  • Benchmark and software evaluations should separately measure calculation, feasibility judgment, full-answer rejection, and false alarms at known margins, reporting the exact prompt and assumptions.A feasibility question before a full answer should be tested on both flaws accepted in full answers and solvable items that might be falsely declined.Source: Discussion · p. 11

Methods

  • Two selected mechanical families—beams and disks—had scalar collapse or burst limits certified by two solvers, and flaws were generated around those limits.A disk flaw had to exceed burst speed and lack an admissible elastic–plastic boundary in both references; a failed numerical search alone was insufficient certification.Source: Methods · pp. 11–12
  • The ordinary prompt requested numerical answers and one JSON status, while deterministic grading counted declines or missing required answers as flags and separately coded infeasibility statements.Calls without parsed replies remained in the denominators as misses or unsolved valid items, not rejections or false alarms.Source: Methods · p. 13
  • Reply-evidence coding was partly corroborated: an independent model coder agreed on stated infeasibility in 32/38 sampled replies, short of the prespecified criterion.The initial rule was refined using 26 replies by an AI coding assistant, with no human check; the authors describe this as neither held-out validation nor human review.Source: Methods · pp. 14–15
  • The primary contrast tested whether adding a limit-and-validity reference sheet to PA rather than S1 improved rejection of flawed designs without excess false alarms or nonresponses.The contrast used 80 paired model–item observations across 40 distinct items; the decision rule required a positive difference in each model, at most one false alarm per 20 valid items per cell, and at most two unanswered calls per 40.Source: Methods · p. 15
  • The experiment followed staged design choices rather than preregistration, and the authors retained responsibility for study decisions and conclusions.The authors report that AI tools assisted with implementation, checking, and editing, but did not change reported outcomes or classes; authors reviewed and edited adopted work.Source: Methods · pp. 15–16

Data availability

  • The planned public repository release includes item sets, answer keys, model replies with grades, outcome tables, plans, and provenance records, while excluding an unrun private hold-out.The release is described as CC BY 4.0, with an archived copy carrying a DOI.Source: Data availability · p. 16

Code availability

  • The repository will release the generators, reference solvers, evaluation harness, grader, and scripts under the MIT license; the scripts regenerate outputs from stored replies without model calls.The release covers scripts that compute every reported number, table, and figure.Source: Code availability · p. 16

Funding

The authors report research support from Santa Clara University and declare no competing interests.

Funding

  • The authors report research support from Santa Clara University.Source: Funding · p. 16

Competing interests

Supplementary Information · 1. Fixed texts per harness · 2. Twins (main set, unwarned prompt Bu)

The supplement records the exact harness prompts and answer formats, and provides valid and flawed beam and disk examples for the unwarned prompt Bu. Its “twins” are independent valid and flawed draws from the same generator, not matched pairs.

Supplementary Information

  • The supplement includes full beam and disk prompts, reply-rule definitions, operational methods, Supplementary Figs. 1–10, and Supplementary Tables S1–S35.It defines the first tier as replies that state infeasibility or print the limit below the datum; source-data and table CSV folders accompany the PDFs.Source: Supplementary Information · p. 19

1. Fixed texts per harness

  • The fixed system prompt instructs the model to solve engineering mechanics problems and finish with one JSON object in a fenced block matching the user’s specification.Closed-book prompts also prohibit tools and internet access, and require calculations in the reply.Source: §1 · p. 22

2. Twins (main set, unwarned prompt Bu) — T02, valid twin: T02-main-000 (signed margin -1.011 %)

  • 1.737 is below the beam’s 1.75475 collapse-factor certificate, making valid twin T02-main-000 solvable.The prompt asks for first-hinge load factor and location, shakedown limit, and residual moment at B after unloading.Source: T02, valid twin: T02-main-000 (signed… · pp. 22–23

2. Twins (main set, unwarned prompt Bu) — T02, flawed twin: T02-main-006 (signed margin +0.991 %)

  • The stated load-factor datum 2.273 exceeds the two-reference collapse-factor certificate 2.2507, so the specified continuous two-span beam problem has no solution under the stated settlement and proportional loading.Source: T02, flawed twin: T02-main-006 (signed… · p. 23

2. Twins (main set, unwarned prompt Bu) — T11, valid twin: T11-main-006 (signed margin -1.000 %)

  • The stated 31,060 rpm datum is below the disk’s 31,373.6 rpm burst-speed certificate, so the annular titanium-alloy disk problem is solvable.The requested outputs are first-yield speed during the spin test, elastic-plastic boundary radius at overspeed, residual bore hoop stress at rest, permanent bore-radius change, and bore hoop stress during later operation.Source: T11, valid twin: T11-main-006 (signed… · p. 24

2. Twins (main set, unwarned prompt Bu) — T11, flawed twin: T11-main-005 (signed margin +1.001 %)

3. Prompt conditions (shown as differences from Bu) · 4. Reply excerpts (visible reply text; Claude CLI replies carry no reasoning text)

This section documents how prompts, status rules, and reply text were evaluated, including stated-limit and feasibility-only tests, follow-up controls, and a new-design experiment. It also describes the operational rules, audits, and supplementary analyses used to examine model responses and design feasibility.

3. Prompt conditions (shown as differences from Bu)

  • Prompt variants tested confidence-based abstention, Python access, a requested collapse certificate, and single-status versus per-answer status formats.The concurrent format experiment compared unchanged Bu (S0), single-status (S1), and per-answer-status (PA); the collection comprised 4,860 final calls.Source: §3 · p. 26, Models, harnesses and final calls · p. 32

Stated-limit experiment (new beams and disks; Supplementary Note 3)

  • The stated-limit experiment compared reference-sheet prompts with feasibility-only yes/no questions on new beams and disks.Reference sheets stated certified collapse or burst limits and whether the datum exceeded them; all five arms retained the closed-book instruction.Source: Stated-limit experiment (new beams and… · pp. 27–28

Ceiling: Opus 5.5 rejects a T02 flaw at the smallest fine margin

  • At a +0.0100% margin, Opus 5.5 reported “cannot solve” for a beam whose stated load factor exceeded the collapse certificate.Because the requested JSON used one status for all quantities, the unanswerable collapse question led it to set answers to null despite working the other parts.Source: Ceiling: Opus 5.5 rejects a T02 flaw… · p. 28

Under one status: Opus 5 with the Python tool states the problem is unattainable and reports "solved"

  • Opus 5 with Python reported “solved” for an unattainable beam, despite giving a stated load factor above the collapse certificate.Its reply supplied the requested quantities, while the record classified the datum as infeasible.Source: Under one status: Opus 5 with the… · p. 28

Certificate asked (C4): Gemini 3.8 Flash reports λc below the stated λ1 and still answers “solved”

Supplementary Note 2. Reply rule and curve fits

  • The reply rule counts a certificate statement only when its number meets proximity, context, and below-datum criteria, and counts infeasibility when an impossibility phrase or datum-to-certificate comparison appears.Python output has a separate rule: a printed number within 0.1% of the certificate and below the datum counts as tool output when the line names the limit; two earlier problem families were dropped before calls.Source: Supplementary Note 2. Reply rule and… · p. 30, Supplementary Note 3. Detailed… · p. 31

Generation, precision and item sets

  • Oracle dry runs solved every valid item and flagged every flaw, while build validation confirmed all 130 main, fine, and grid flaws and all 20 new-beam flaws exceeded both reference certificates.Disk speed margins correspond to roughly twice that load margin, so margins are not on one physical scale across beam and disk families.Source: Generation, precision and item sets · p. 31

Sensitivity to printed precision

  • Under the wider input-rounding box, median certificate shifts were 4.8% for beams and 2.2% for disks; the cue analysis’s Fable disk-grid random forest had median AUC 0.72 and Holm-adjusted P = 0.11.The authors caution that the corner check does not certify impossibility throughout the box unless monotonicity holds; the classifier test covers specified observable cues, not every possible cue.Source: Sensitivity to printed precision · p. 31, Surface-cue checks · p. 32

Prompts and interventions

  • The confidence-target prompt awards one point for a correct answer, loses nine for a wrong answer and gives zero for withholding, with answering favored only when p > 0.9.Because a correct rejection of a flawed twin also earns zero, the scoring rule rewards withholding but not detection; the intervention also makes declining salient.Source: Prompts and interventions · p. 33

Grading, missing answers and capability

  • A flaw is rejected by a flag or withheld required answer, while a completed unflagged flawed reply is accepted; no-answer calls count as missed flaws, not rejections.Capability is assessed separately through limit probes and valid-twin solving, and probe accuracy does not establish that a model uses the calculation in each trial.Source: Grading, missing answers and capability · p. 34

Regimes

  • The three descriptive regimes distinguish near-universal rejection without false alarms, step-like transitions, and large beam misses despite capable probes; the latter models rejected 9/18, 5/18 and 17/18 disk flaws.The regimes were assigned after seeing counts, describe observed groups under one status per reply, and are not confidence intervals; brackets use tested rejection thresholds.Source: Regimes · p. 34

Evidence within replies

  • Among accepted main-set disk replies under Bu, 4/9 of Opus 5’s and 4/13 of Opus 4.8’s stated infeasibility, while four more Opus 4.8 replies printed the limit below the datum.The rule classes visible reply text; a decoy check found true-certificate statements without decoy hits in 17/75 disk and 52/186 beam replies, while an independent coder showed partial agreement with the rule.Source: Evidence within replies · p. 36, Checks of the rule · p. 36

Support-rule control; Per-answer-status control

  • The support-rule control added an assumption that a plastic hinge forms in the weaker section, while the per-answer-status control let each quantity have its own status and asked models to decline only quantities without valid values.Both reran the 40 main-set beams for Opus 5 and Gemini; the per-answer outcome applies to the changed format and closing instruction together, not status count alone.Source: Support-rule control · p. 37, Per-answer-status control · p. 37

Concurrent answer-format experiments

  • Opus 5 rejected 0/20 original beam flaws and 5/20 clarified flaws; Gemini rejected 1/20 and 3/20, respectively, with no false alarms.The clarity test changed two assumption lines while keeping parameters, questions and keys unchanged; neither model met the preregistered recovery threshold.Source: Concurrent answer-format experiments · p. 38
  • Stage 2 and Stage 3 predictions were contradicted for Opus 5, while Gemini’s Stage 3 result was inconclusive; single-status disk rejection rose between collections for Opus 5 from 9 to 13 of 18 and Haiku from 12 to 17.Because served models may change across collections, conclusions about answer format rely on concurrent arms.Source: Concurrent answer-format experiments · pp. 37–38

Stated-limit and feasibility-only experiment

  • The stated-limit experiment used 40 new designs per family, with five designs at each target margin, and compared per-answer formats with a feasibility-only yes/no judgment.The datum alone weakly separated flawed from valid items (AUC 0.507 for T02 and 0.450 for T11); the supplied passages describe the comparison rule but do not provide its outcome.Source: Stated-limit and feasibility-only… · p. 38
  • Observed rejection was 80/80 under PA and 79/80 under S1 (difference 0.0125; P = 1).There was one discordant item; model differences were 0.025, 0 and 0, and six cells failed their gates: Gemini’s two note arms and GPT-6-Luna’s four full-answer arms. GPT-6.1-Sol recounted the stored records with its own code and found no mismatch.Source: Stated-limit and feasibility-only… · p. 39
  • Gemini’s 13 calls without an answer and GPT-6-Luna’s 48 full-answer false alarms did not change the primary contrast.Usable earlier or excluded JSON replies changed some post hoc note-arm counts, but not the limit arms or decision; the 48 false alarms included declines citing difficulty or a non-unique solution and declines asserting a wrong limit, including a burst speed of about 25,420 rpm for a valid disk whose sheet gives 22,555.55 rpm.Source: Stated-limit and feasibility-only… · p. 39

Staged design and reproducibility

  • Follow-up analyses were conditioned on earlier observations, and the authors do not describe the staged decisions as preregistration.Displayed results use frozen item keys and first final records recomputed from stored replies; a private hold-out from separate seeds is excluded. A provenance note dates plan documents and lists later changes.Source: Staged design and reproducibility · p. 39

Supplementary Figures

  • Supplementary Fig. 8 depicts beam and disk certificates, while Supplementary Fig. 9 reports surface-cue checks and classifier discrimination.For the single-datum range computed on the earlier study’s 30 item pairs, the AUC range was 0.94–1.00; one of twelve classifier tests had unadjusted P < 0.05 and none did after Holm adjustment.Source: Figure 8 · p. 40, Figure 9 · p. 41
  • Supplementary Figs. 10–11 show per-model rejection against signed margin and descriptive fit parameters.Fig. 10 covers pooled fine, main and grid sets under Bu and Fable 5.1’s dense grid; Fig. 11 presents lapse rate and false-alarm floor estimates or marks parameters not estimable from the data.Source: Figure 10 · p. 42, Figure 11 · p. 43
  • Supplementary Figs. 12–13 examine beam outcomes under the inadmissible stronger-section reading and identify six beams that remain impossible under that substitution.Those six flaws have target margins of 10% and 30%; the other six models reject all six. Fig. 13 also categorizes answers by section-reading and reports first-hinge load-factor and location accuracy on accepted flawed and solved valid twins.Source: Figure 12 · p. 44, Figure 13 · p. 45
  • Supplementary Figs. 14–15 compare confidence-target rejection with Bu and describe output-token counts and call times.Fig. 14 covers the main set and eight models run with Bt; Fig. 15 reports median tokens by rejection outcome and median wall-clock seconds with interquartile ranges.Source: Figure 14 · p. 46, Figure 15 · p. 47
  • Supplementary Fig. 16 reports per-item first-run and retest outcomes for Opus 5 and Gemini 3.8 Flash under Bu.The concurrent single-status arms from 19–20 October provide a third measurement.Source: Figure 16 · p. 48

Supplementary Tables

The supplementary tables reproduce CSV analysis tables and define the experiment-set labels, prompt conditions, symbols, and statistical conventions needed to interpret them.

Supplementary Tables

  • Supplementary tables reproduce the supplied CSV analysis tables, with exact P values in Table S32 and stated-limit results in Tables S33–S35.P values rounded to two significant figures use ties-to-even, so exact P = 0.125 may appear as 0.12. Tables S4, S5, S6, S11, and S13 give Holm-adjusted P values within each table.Source: Supplementary Tables · p. 49
  • Main, fine, and grid denote independent item sets; labels distinguish certificate, beam, disk, new-design, clarity, and prompt-condition variants.C4 asks for the certificate; mpb states the support-B plastic moment, and pas uses one status per answer. Concurrent answer-format sets are brg, dsk, and kdr, with arms S0, S1, and PA; clr compares original and clarified main beams. Bu, Bt, and Bx denote the unwarned prompt, confidence target, and Python condition, respectively.Source: Supplementary Tables · p. 49
  • Rates retain unanswered calls in their denominators, and intervals are 95% unless marked otherwise.Dates use Coordinated Universal Time; CI means confidence interval, and m is the signed target margin. The passage also defines Φ, λsd, MB,res, σθ, σr, Y, fy, Wpl,1, and Wpl,2.Source: Supplementary Tables · p. 49

Supplementary Table S1. Run inventory

Supplementary Table S1 inventories model collections, prompt sets, call records, and run metadata. Its notes clarify how final counts were selected and identify limits affecting particular runs.

Supplementary Table S1. Run inventory

  • Final counts retain the first final record per item and prompt, while the inventory also includes non-final attempts.The table records model, harness, prompt set, call timestamps, output caps, and time limits.Source: Supplementary Table S1. Run inventory · p. 51
  • The x10 stated-limit experiment includes GPT-6-Luna attempts that failed with rate-limit errors.The x10 rows are identified as the stated-limit experiment in Supplementary Note 3.Source: Supplementary Table S1. Run inventory · p. 51
  • Gemini 4 Argon ran only within its usage quota and completed its main set and probes on 24 October 2026, meeting its inclusion rule.Its listed main-set row reports 76 prompts and 76 final calls.Source: Supplementary Table S1. Run inventory · pp. 49, 51

Supplementary Table S2. Outcomes per model, set, prompt and family (first runs) · Supplementary Table S3. Descriptive fits on the log margin, with identifiability labels

The supplementary tables report model outcomes by model, set, prompt, and family alongside descriptive log-margin fits with identifiability labels; many fitted transitions are not estimable. The fits describe the data, while model-free counts and brackets carry the claims.

Supplementary Table S2. Outcomes per model, set, prompt and family (first runs)

Supplementary Table S3. Descriptive fits on the log margin, with identifiability labels

Supplementary Table S5. Paired prompt contrasts on the main set

On the main set, Bt prompt contrasts often increased rejection of flawed designs, but effects varied across models and some prompts also triggered false alarms on valid designs.

Supplementary Table S5. Paired prompt contrasts on the main set

  • Haiku 5.5 T02 raised flawed-design rejections from 7/20 to 18/20 (+0.55), but also raised valid-design false alarms from 0/20 to 16/20 (+0.80).The planned Holm-adjusted P values were 0.085 for flawed designs and 0.00085 for valid designs.Source: Supplementary Table S5. Paired prompt… · p. 57
  • Gemini 3.8 Flash T02 increased flawed-design rejections from 2/20 to 14/20 (+0.60 paired difference; planned Holm P=0.048), while valid-design false alarms rose from 0/20 to 2/20.For T11, flawed-design rejections changed from 17/18 to 18/18, while valid-design false alarms rose from 0/18 to 8/18.Source: Supplementary Table S5. Paired prompt… · p. 57
  • Opus 5 T11 increased flawed-design rejections from 9/18 to 18/18 (+0.50 paired difference; planned Holm P=0.094), with valid-design false alarms remaining at 0/18.For T02, flawed-design rejections rose from 1/20 to 7/20, while valid-design false alarms remained at 0/20.Source: Supplementary Table S5. Paired prompt… · p. 57
  • For Opus 4.8, T11 increased flawed-design rejections from 5/18 to 18/18 (+0.72 paired difference; planned Holm P=0.0066), with no valid-design false alarms.Valid twins remained at 0/20 for T02 and 0/18 for T11 under both prompts.Source: Supplementary Table S5. Paired prompt… · p. 57

Supplementary Table S7. Capability probes (certificate only), per family

Certificate-only probes found that most listed model-family entries were capable on all five first runs, with a few missed cases and larger relative errors. Capability meant relative error below half the source item’s margin.

  • Most listed entries were capable on 5/5 first runs for both T02 collapse factor and T11 burst speed.The table reports results by model, set, and task.Source: Supplementary Table S7. Capability… · p. 59
  • GPT-6.1-Sol main T02 was capable on 4/5 runs with a 33.0097 % maximum |relative error|; GPT-6-Luna main T11 was capable on 4/5 with 10.4590 %.GPT-6-Luna main T02 was also capable on 4/5 runs, with a 3.9765 % maximum |relative error|.Source: Supplementary Table S7. Capability… · p. 59
  • Capability required relative error below half the source item’s margin, and the reported results were from first runs.The probe requested the T02 collapse factor or T11 burst speed for a near-boundary instance without a datum to judge.Source: Supplementary Table S7. Capability… · p. 59

Supplementary Table S8. Evidence in accepted flawed replies (strict rule; one class per reply) … Supplementary Table S11. Surface cue: can the stated data separate flawed from valid twins?

Accepted flawed replies sometimes included numerical limit evidence, but held-out classifiers generally failed to distinguish flawed from valid twins; interpretation of T02 answers also depends on the plastic moment used at support B.

Supplementary Table S8. Evidence in accepted flawed replies (strict rule; one class per reply)

  • 15/20 accepted beam replies and 7/8 accepted disk replies from Opus 5 with Python included labeled tool-output values below the datum, regardless of reply class.The strict visible-text rule defined certificate statements by proximity to a certificate term and infeasibility statements by an impossibility phrase or comparison in the same or next sentence.Source: Supplementary Table S8. Evidence in… · p. 60

Supplementary Table S9. Which plastic moment at support B do T02 answers use?

  • 15/15 valid twins in Opus 5.5 fine Bt matched the weaker-section reading, versus 0/15 matching the stronger reading.Across the table, valid twins are counted where the two readings give different answers; matches use λsd and MB,res within grader tolerance.Source: Supplementary Table S9. Which plastic… · p. 61
  • 6 of 66 flawed T02 items remained impossible under the stronger-section reading, and all were at m ≥ 10%; references use the weaker section at B.The references' weaker-section choice follows plastic theory, while the problem statement says only that the cross-section changes at B.Source: Supplementary Table S9. Which plastic… · p. 61

Supplementary Table S10. Certificate asked (C4; T02 main items with the collapse factor as question 3)

  • 2 of Haiku's 3 accepted main-beam replies stated infeasibility under the final rule, in a C4 follow-up on two models whose accepted T02 replies had omitted the collapse factor under the earlier rule.Source: Supplementary Table S10. Certificate… · p. 62

Supplementary Table S11. Surface cue: can the stated data separate flawed from valid twins?

  • 0.72 AUC for the grid T11 random forest was the highest held-out score, but its Holm P was 0.11; all other classifier tests had Holm P 1.The analysis used repeated stratified 5-fold cross-validation on independently drawn valid and flawed instances.Source: Supplementary Table S11. Surface cue:… · p. 63
  • 0.94 single-datum AUC for the beam and 1.00 for its four other families contrasted with the weaker held-out classifier results.These single-datum AUCs were computed on the earlier study's 30 twin pairs, with 6 pairs per family.Source: Supplementary Table S11. Surface cue:… · p. 63

Supplementary Table S12. Time and output tokens per call · Supplementary Table S14. The six flawed beams that stay impossible under both readings of support B · Supplementary Table S15. Answers that do not depend on the stated datum

Supplementary results show that rejection of six persistently impossible beams varied across models and prompts, while some accepted flawed answers still gave correct quantities independent of the stated datum. The time and token measures are operational call-level observations, not intrinsic speed estimates.

Supplementary Table S12. Time and output tokens per call

  • Call time was CLI start-to-exit wall-clock, and median output tokens covered answered calls; shared-subscription parallel waves made time an observed cost, not intrinsic speed.For answered flawed twins, medians were compared using two-sided permutation tests with Holm adjustment; the token and time tests were exploratory.Source: Supplementary Table S12. Time and… · pp. 64–65

Supplementary Table S14. The six flawed beams that stay impossible under both readings of support B

  • 0/6 were rejected by Opus 4.8 on main Bu and Opus 5 on main Bx, while Gemini 3.8 Flash rejected 1/6 on main Bu.Haiku 5.5 rejected 4/6 on main Bu; these results contrast with the complete rejection under main Bu by the six models above.Source: Supplementary Table S14. The six… · p. 66
  • 6/6 impossible beams were rejected under main Bu by Opus 5.5, GPT-6.1-Sol, GPT-6-Astra, GPT-6-Sol, Sonnet 5.5, Fable 5.1, and GPT-6-Luna.The six beams remain impossible when the stronger section governs at B.Source: Supplementary Table S14. The six… · p. 66

Supplementary Table S15. Answers that do not depend on the stated datum

  • 11/11 accepted flawed Fable 5.1 T02 fine Bu answers were correct for λy, location, and λsd under both readings; its T02 grid Bu count was 7/7 for each.These counts are correct answers among accepted flawed twins.Source: Supplementary Table S15. Answers that… · p. 67
  • 13 Gemini 3.8 Flash T02 main Bu flawed answers included 12 correct λy, 12 correct locations, 11 correct reference λsd, and 13 correct λsd under either reading.For T11 main Bu, Opus 5 had 9 accepted flawed answers and all 9 ny answers were correct; Fable 5.1 had 1 accepted flawed answer and its ny answer was correct.Source: Supplementary Table S15. Answers that… · pp. 67–68
  • 19 Opus 5 T02 main Bu flawed answers included 17 correct λy, 17 correct locations, 8 correct reference λsd, and 19 correct λsd under either reading.Opus 4.8 T02 main Bu had 20 accepted flawed answers, with 19 correct λy, 19 correct locations, 11 correct reference λsd, and 20 correct λsd under either reading.Source: Supplementary Table S15. Answers that… · p. 68

Supplementary Table S16. Flawed disks under alternative disk models · Supplementary Table S17. Decoy test: hits when the certificate is replaced by a false value · Supplementary Table S18. The qualifying sentence of every first-tier reply

The supplementary tables show that alternative disk assumptions shift calculated limits, final certificate and infeasibility rules nearly eliminate decoy hits, and some accepted beam replies explicitly state that the design is infeasible.

Supplementary Table S16. Flawed disks under alternative disk models

  • Alternative disk readings raise the limit speed by 1.5–22.7%, with median increases of 2.9–6.7%, making some flawed overspeeds attainable under assumptions that contradict the item.The reported increases are 1.6–22.7% for constant rim traction, 1.5–5.1% for von Mises, and 3.2–14.2% for in-plane Tresca.Source: Supplementary Table S16. Flawed disks… · p. 69

Supplementary Table S17. Decoy test: hits when the certificate is replaced by a false value

  • Earlier T02 implementation hits included 9/186 at Decoy −5%, 10/186 at Decoy −2%, and 4/92 at Decoy +5%.The earlier T11 rule recorded 0/75 at each of the −5%, −2%, and +2% decoy values, and 0/24 at +5%.Source: Supplementary Table S17. Decoy test:… · p. 70
  • Final certificate-statement and infeasibility rules produced 0–1 decoy hits, compared with multiple hits under T02’s earlier implementation.The decoy test replaced the certificate with c(1 + δ); hits were numbers the reply had no reason to print.Source: Supplementary Table S17. Decoy test:… · p. 70

Supplementary Table S18. The qualifying sentence of every first-tier reply

  • Opus 5’s accepted T02-main-009 reply at +10% states infeasibility while reporting that the load cannot be exceeded beyond collapse.The table marks the full answer accepted, although the same item under Bt was rejected.Source: Supplementary Table S18. The… · p. 71
  • Opus 4.8’s T02-main-009 reply at +10% is marked yes and accepted; its reference notes the beam’s plastic-collapse factor is approximately 1.79.Source: Supplementary Table S18. The… · p. 72
  • Table S18 covers 27 accepted flawed replies from Opus 5 and Opus 4.8 on the main set under Bu; Gemini 3.8 Flash had no first-tier replies.The table gives each reply’s certificate statement and its first-run outcome under the confidence-target prompt for the same item.Source: Supplementary Table S18. The… · p. 73

Supplementary Table S19. Beams with the plastic moment at support B stated · Supplementary Table S20. Which state the load-dependent beam answer describes · Supplementary Table S21. Item sets, conditions and the models run under each

The supplements specify how stating the support rule, classifying load-dependent beam answers, and varying beam sets and prompt conditions shaped the experiments. They also report how often two models returned no answer in the support-rule test.

Supplementary Table S19. Beams with the plastic moment at support B stated

  • Gemini 3.8 Flash returned no answer on 9/20 flawed and 0/20 valid calls; Opus 5 returned no answer on 0/20 flawed and 0/20 valid calls.These were the 40 main-set beams tested with an added assumption line stating that a plastic hinge at B forms in the weaker section.Source: Supplementary Table S19. Beams with… · p. 74
  • The support-rule test divided 20 flawed beams into 14 reading-dependent flaws and 6 robust flaws, and compared the stated-rule items with the same items without that line.A call without an answer counted as a miss; the decision rule treated at most 4/14 reading-dependent rejections as persistent misses and at least 10/14 as rule-dependent.Source: Supplementary Table S19. Beams with… · p. 74
  • The decision rule was fixed before the first call, and an exact McNemar test paired the 14 reading-dependent flaws with the main set.The stated thresholds distinguished misses that persisted when the rule was stated from those dependent on the unstated rule.Source: Supplementary Table S19. Beams with… · p. 74

Supplementary Table S20. Which state the load-dependent beam answer describes

  • For flawed beams, the load-dependent residual moment at B after loading to λ1 and unloading has no physically existing state to describe.Replies were compared with reference-solver collapse, beyond-collapse, and fully elastic states; collapse and beyond-collapse values could be indistinguishable within tolerance at small margins or coincide.Source: Supplementary Table S20. Which state… · p. 75

Supplementary Table S21. Item sets, conditions and the models run under each

  • The pilot T11 set and fine T02 and T11 sets each included 16 flawed and 16 valid items, with target margins varying by set.The pilot T11 set used 0.3%, 1%, 3%, and 6%; the fine T02 and T11 sets used 0.01%, 0.03%, 0.1%, and 0.3%.Source: Supplementary Table S21. Item sets,… · p. 76
  • Additional T02 controls stated the support rule or assigned each answer its own status, while concurrent-format tests reused main-set items or added 40 new beam designs.The clarity test asked the main T02 designs as written and with the support rule stated and data declared exact.Source: Supplementary Table S21. Item sets,… · p. 76
  • The new-beams-by-valid-answers set tested 20 T02 designs at k = 0, 1, and 3 with 10% and 30% target margins.The stated-limit experiments used 20 T02 and 20 T11 items each, with the support rule stated and data declared exact.Source: Supplementary Table S21. Item sets,… · pp. 76–77

Supplementary Table S22. Which disk model the accepted flawed disk replies answer (main set) … Supplementary Table S27. Sensitivity of certification to the printed precision of the inputs

These supplementary analyses examine how accepted disk replies map to alternative models, how confidence-target replies are scored and classified, and how beam decisions and feasibility certificates respond to answer format and input precision.

Supplementary Table S22. Which disk model the accepted flawed disk replies answer (main set)

  • Accepted flawed disk replies reporting an inside-rim boundary were matched to constant-rim-traction Tresca applied in-plane or von Mises; no reply matched von Mises.The stated model has no equilibrium boundary above burst, and the rim column counts replies reporting the limit state, not an admissible overspeed state.Source: Supplementary Table S22. Which disk… · p. 78

Supplementary Table S23. Main-set score under the confidence-target prompt’s stated rule

  • The confidence-target scoring rule assigns +1 for a correct answer, -9 for a wrong answer, and 0 for “cannot solve”; flagging valid twins earns zero.The table scores a whole reply as one answer; valid-twin flags count as false alarms only in the paper’s feasibility measure.Source: Supplementary Table S23. Main-set… · p. 79

Supplementary Table S24. Non-final attempts by reason (flawed / valid twins)

  • Across collections, non-final attempts totaled 57/62 sign-in failures, 14/12 CLI exits, 2/0 usage limits, and 66/1 status errors for flawed/valid twins.Both ERROR attempts are counted when the second ERROR is the final no-answer outcome; the private hold-out is excluded.Source: Supplementary Table S24. Non-final… · p. 80

Supplementary Table S25. Reasons given for flags, main set

  • Flags were classified first by whether the reason named a violation, then by uncertainty or penalty without a violation, and otherwise as other.The fixed term rule used each JSON reason or, if missing, the reply’s last 1,500 characters; the violation class is a lower bound.Source: Supplementary Table S25. Reasons given… · p. 81

Supplementary Table S26. Beams with one status per answer

  • Opus 5’s beam misses depended on the single-status format, but status count alone cannot explain the comparison because format and closing instruction changed together.The single-status support-rule control was a separate, nonconcurrent run; the per-answer format let a reply decline one quantity while answering the others.Source: Supplementary Table S26. Beams with… · p. 82

Supplementary Table S27. Sensitivity of certification to the printed precision of the inputs

  • For T02 main, flaws were impossible at every corner in 3/4 cases at a 3% margin under last-digit rounding, versus 0/4 under generator rounding; both conventions yielded 4/4 at 10% and 30%.Corner checks bound the certificate over the full input box only where monotonicity is established; otherwise they are sensitivity checks, not box-wide certificates.Source: Supplementary Table S27. Sensitivity… · p. 83

Supplementary Table S28. Main-set beams under three answer formats, run concurrently · Supplementary Table S29. Main-set disks under the three answer formats, run concurrently · Supplementary Table S30. New beams by the number of valid answers requested

On the original beams, rejection increased with more explicit or per-answer status formats for Opus 5 and Gemini 3.8 Flash, but did not consistently improve on new designs. On new beams, Opus 5 rejected nearly all flaws across answer-count conditions, while Gemini 3.8 Flash’s rejection varied with the number requested.

Supplementary Table S28. Main-set beams under three answer formats, run concurrently

  • On the original beams, Opus 5 rejected 3/20 flaws with ordinary single status, 10/20 with an explicit rule, and 14/20 with per-answer status; each arm had 0/20 false alarms.The reported paired test for per-answer status versus ordinary status was 7:0, P = 0.016.Source: Supplementary Table S28. Main-set… · p. 84

Supplementary Table S29. Main-set disks under the three answer formats, run concurrently

  • On the original disks, Opus 5 rejected 13/18 flaws with ordinary status and 17/18 with either explicit-rule or per-answer status; Haiku 5.5 rejected 17/18, 18/18, and 17/18, respectively.All six arms had 0/18 false alarms; the paired tests did not show significant differences between formats.Source: Supplementary Table S29. Main-set… · p. 85
  • The original-disk results were marked as shown for both models, while the main-beam outcomes varied by format and model.The disk table’s verdict bands required at least 13/18 rejections and at most 2/18 false alarms; its main-collection column showed increased single-status rejection for both models between collections.Source: Supplementary Table S28. Main-set… · p. 84, Supplementary Table S29. Main-set… · p. 85

Supplementary Table S30. New beams by the number of valid answers requested

  • On new beams, Opus 5 rejected 19/20 flaws at k=0 and 20/20 at k=1 and k=3 with per-answer status, with 0/20 false alarms in every condition.With an explicit single-status rule, rejection rose from 14/20 at k=0 to 18/20 at k=3, also with 0/20 false alarms.Source: Supplementary Table S30. New beams by… · p. 86
  • Across the new-beam answer-count conditions, Opus 5’s explicit-rule rejection rose from 14/20 at k=0 to 18/20 at k=3, whereas Gemini 3.8 Flash’s fell from 19/20 to 16/20.The reported k-effect predictions were contradicted for Opus 5 and inconclusive for Gemini 3.8 Flash.Source: Supplementary Table S30. New beams by… · p. 86
  • On new beams, Gemini 3.8 Flash rejected 20/20 flaws at k=0 with per-answer status, but 17/20 at k=3; false alarms rose from 0/20 to 3/20.With an explicit single-status rule, rejection was 19/20 at k=0 and 16/20 at k=3.Source: Supplementary Table S30. New beams by… · p. 86

Supplementary Table S31. Independent coding of sampled replies

Independent coding compared sampled accepted replies with the fixed rule and found substantial agreement, but the rule was only partly corroborated. The table also reports how coder classifications and the corroboration decision were defined.

Supplementary Table S31. Independent coding of sampled replies

  • Coder labels matched the rule’s tiers in 32/38 infeasibility cases, 24/28 limit-only cases, and 27/28 neither cases in the main collection.The coder said the violation in words in 38/38, 4/28, and 1/28 cases, respectively; coder-tier agreement was 27/28 for neither.Source: Supplementary Table S31. Independent… · p. 87
  • κ was 0.82 (sample) and 0.85 (population); population-weighted agreement was 0.91, and the fixed rule was judged partly corroborated.With a half-unit precision test applied to coder values, agreement was 32/38 for replies classified as stating infeasibility, 24/28 for printing the limit only, and 28/28 for neither.Source: Supplementary Table S31. Independent… · pp. 87–88
  • A blind coder from another model family labeled 100 accepted flawed replies, using rule-defined tiers and a fixed agreement threshold.The sample was fixed before coding; the tiers were states infeasibility, prints the limit only, or neither. The rule was corroborated if at least 0.85 of rule-explicit and rule-neither replies agreed, and not corroborated below 0.70 on rule-explicit replies.Source: Supplementary Table S31. Independent… · p. 88

Supplementary Table S32. Same-item clarity test · Supplementary Table S33. Stated-limit experiment: outcomes per model, family and arm

In the stated-limit experiment, feasibility-only and certified-limit prompts often elicited rejection of impossible designs where note-only arms did not; changing assumptions in the same-item clarity test improved rejection only slightly. Outcomes varied by model and prompt arm, and some stronger rejection came with false alarms.

Supplementary Table S32. Same-item clarity test

  • 5/20 Opus 5 flaws were rejected with clarified assumptions versus 0/20 on the original text; Gemini 3.8 Flash rejected 3/20 versus 1/20, respectively.The paired test changed only two assumption lines; parameters, questions, and keys were unchanged.Source: Supplementary Table S32. Same-item… · p. 89
  • The same-item clarity test used 20 flawed and 20 valid main-set beams per model and predefined recovery criteria of at least 14/20 flaw rejections with at most 2/20 false alarms.The original text had to reject at most 6/20 flaws for clarity to count as recovery.Source: Supplementary Table S32. Same-item… · p. 89

Supplementary Table S33. Stated-limit experiment: outcomes per model, family and arm

Supplementary Table S34. Stated-limit experiment: paired contrasts on flawed items · Supplementary Table S35. Stated-limit experiment: visible evidence in accepted flawed replies (post hoc)

The pooled primary paired contrast showed nearly identical rejection rates for PA + limit and S1 + limit. A post hoc rule categorized visible evidence in accepted flawed replies and compared those items with feasibility-only rejection.

Supplementary Table S34. Stated-limit experiment: paired contrasts on flawed items

  • Rejection rates were 1.0 for PA + limit and 0.9875 for S1 + limit across 80 model–item observations; the exact item-cluster sign-flip test gave P = 1.The pooled primary contrast covered 40 distinct flawed items across three models.Source: Supplementary Table S34. Stated-limit… · p. 92
  • The primary contrast was fixed before the first call; secondary contrasts used two-sided, unadjusted exact McNemar tests.Each flawed item was paired with itself across two arms of the same model.Source: Supplementary Table S34. Stated-limit… · p. 92

Supplementary Table S35. Stated-limit experiment: visible evidence in accepted flawed replies (post hoc)

  • The post hoc rule categorized accepted flawed replies by stated-limit evidence and compared each flawed item with its feasibility-only outcome.A limit within 2% of the certificate and below the datum counted as stating the violation if paired with an impossibility phrase or datum comparison; without that phrase, it counted as printing the limit only. For beams without a limit, the rule used the reading implied by shakedown factor and residual moment; the final column recorded whether feasibility-only rejected the same item.Source: Supplementary Table S35. Stated-limit… · p. 93

Related papers

Loading 2610.38096v1…