Source-linked AI summary
Adversarial Cues in Decision Models Used as Judges: The Role of Request Presentation
Hongliang Liu
arXiv:2610.11436v1cs.CLcs.CR
TL;DR
The paper asks whether judges follow explicit final-answer grading rules when candidates contain competing values. It pairs a colon edit with a request-presentation intervention, finding a presentation-dependent increase in false acceptance for Jev in the tested construction. The result is bounded by the reference-aware task and compound sorting intervention.
Problem
A judge must apply the grading rule to the specified part of a response, yet accepting a wrong final answer despite an earlier matching value violates that rule.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
Evaluations should record the executed request representation and test candidate perturbations under the presentations used by the integration.
Source: §6 · p. 9The finding is limited to a reference-aware numeric grammar and compound sorting change, which alters several request objects and leaves the responsible component unidentified.
Source: §6 · p. 9
Abstract
from arXiv · showhide
An answer judge instructed to grade the final commitment should reject an explicitly wrong final value even when an earlier value matches the reference. We show that adding one colon to a candidate can violate this requirement depending on the presentation of the structured judging request. Numeric references certify the error, and paired interventions distinguish the candidate edit from the integration's presentation choices. On 200 previously unused DROP and GSM8K source clusters, the edit increased Jev's false acceptance from 1.0% to 26.0% with three output labels and from 3.0% to 26.5% with the published four-label grading instruction under sorted JSON keys. Both candidate variants were rejected under insertion presentation. These interactions passed the prespecified statistical correction even though Jev met the control thresholds under both grading configurations and presentations. Most excess acceptances occurred among candidates assigned larger numerical errors. GPT-6 Sol produced no observed cue-condition false acceptances, with missing responses unresolved. The result shows that basic judging competence can coexist with sharply different vulnerability to a fixed candidate edit across logically equivalent request presentations. The reference-aware grammar and compound ordering change limit the finding's operational scope and leave its internal cause unmeasured.
1 Introduction
The paper tests whether judges follow explicit final-answer rules when candidates contain conflicting values. It reports a certified colon-edit failure whose outcome depends on request presentation.
1 Introduction
- A judge must evaluate the response part specified by its grading rule; accepting a wrong final value despite an earlier matching value violates that rule.A numeric reference makes the error checkable.Source: §1 · p. 1
- Under sorted presentation, adding a colon changed Jev’s verdict from incorrect to correct although the final value remained 54 against reference 14; insertion presentation rejected both variants.The independent confirmation found excess false acceptance under sorting while Jev passed basic controls in both presentations.Source: §1 · pp. 1–2
- The study tests how candidate edits and request presentation interact while named rubric bindings and decoded task content remain fixed.The request objects change order, and the candidate edit is tested against an explicit final-answer correctness requirement.Source: §1 · p. 2
- The reported rates apply to a balanced DROP/GSM8K and near/far construction, with effects concentrated among larger mutations; natural prevalence and the responsible component remain unmeasured.Sorting changes several request objects together.Source: §1 · p. 2
2 Related work
Prior work studies candidate-text attacks, representation sensitivity, decision-model failures, and invariance testing. This paper combines a candidate edit with a request-order intervention to test their interaction in a specified judging task.
2 Related work
- Candidate attacks include optimized response text, shared phrases and surrogate transfer [16], punctuation and generic openers [27], and next-token proposals with beam exploration [11].Related model-evaluation work also studies response position, verbosity, and self-enhancement.Source: §2 · p. 2
- Representation-sensitivity work measures performance across meaning-preserving prompt formats [18], compares plain text, Markdown, JSON, and YAML, and tests validation-equivalent schema reordering.These precedents motivate testing presentation alongside candidate edits.Source: §2 · p. 2
- Decision-model precedents study context additions, input edits against clean decisions [7], rubric-binding swaps, and correct/incorrect revisions under Jev-as-a-Judge [12].The cited work reports semantic failures despite schema-valid outputs.Source: §2 · p. 3
- This study combines a candidate-side colon edit with a request-order intervention, preserving named rubric bindings and an explicit wrong final commitment.A numeric reference certifies the error, and a separate source bank prospectively confirmed the interaction after an initial null and exploratory diagnosis.Source: §2 · p. 3
3 Task and threat model
The task evaluates whether a structured judge assigns the required label to a candidate’s final answer against a trusted reference. The construction keeps an earlier matching value and a wrong final value fixed while testing candidate edits and request presentation.
3 Task and threat model
- The judge returns a native label for a question, trusted reference set, candidate, and grading configuration under a specified request presentation.Displayed scores are measured separately; the GPT-6 Sol comparator returns a label without an elicited probability vector.Source: §3 · p. 3
- When a candidate contains competing values, an explicit final-answer rule determines which commitment is compared with the trusted reference.A numeric mismatch makes the required incorrect label checkable even when an earlier value matches the reference.Source: §3 · p. 3
- The restricted candidate places an earlier value before a short bridge and an explicit final value; the earlier value matches a reference while the final value differs from every accepted reference.A deterministic parser can grade this construction, and its certificate is relative to the supplied references and rule.Source: §3 · p. 3
- Recursive sorting changes several request-object orders while preserving decoded field values and named criteria, making presentation a compound intervention.The constructor changes candidate text while keeping the question, reference, instruction, criterion meanings, and wrong final value fixed.Source: §3 · pp. 3–4
- A false-positive flip occurs when the control is rejected but the edited candidate is accepted despite an unchanged incorrect required label.The oracle certifies the error, and the matched control establishes the change; native false acceptance and displayed-score movement are measured separately.Source: §3 · p. 4
4 Study design
The study moves from candidate-construction and formatting exploration to a prespecified confirmation study. Its design fixes source and mutation groups, evaluates controls and paired requests, and uses workload-stratified inference with correction.
4 Study design
- The study tests whether request presentation changes the effect of a colon edit while the required correctness label stays fixed.The design introduces the construction and hypothesis, then specifies the new source bank, controls, schedule, and inference rules for prospective confirmation.Source: §4 · p. 4
4.1 From observed failures to independent confirmation
- The hypothesis followed exploratory sorting comparisons on reused identities; a separate 200-cluster comparator had null insertion contrasts, and the later diagnosis generated the presentation-interaction hypothesis.Candidate scaffolds and bridge strings were developed through manual ablations and Qwen proposals; the independent source bank was reserved for confirmation.Source: §4.1 · p. 4, Table 1 · p. 5
4.2 Sources, constructions and configurations
- The confirmation bank contains 100 DROP training and 100 GSM8K test questions, with one question per declared cluster and audit exclusions against prior inventories.Each workload uses fifty near and fifty far mutations, with the same source-seeded letters with and without a colon under both presentations.Source: §4.2 · p. 5
4.3 Controls, call roles and scheduling
- Each source/configuration receives sixteen calls across presentations, cue conditions, controls, and duplicate requests; missing outputs count against control qualification.Primary and duplicate calls were designated before execution, interleaved, and analyzed without filtering sources by control outcomes.Source: §4.3 · p. 5
4.4 Endpoints and paired inference
- The primary estimand averages DROP and GSM8K source interactions equally, tested with a workload-stratified cluster bootstrap and Holm correction.Native categorical outcomes are primary; displayed probabilities and duplicate variability are descriptive, and missing outputs remain unresolved.Source: §4.4 · pp. 5–6
5 Results
The independent confirmation found that the colon edit increased Jev’s false acceptance under sorted presentation, despite passing control thresholds in both presentations. The effect was concentrated in larger-mutation groups; duplicate calls and earlier studies provide descriptive context rather than independent confirmation.
5.1 Independent confirmation of the presentation interaction
- GPT-6 Sol had zero observed false acceptances in its four cue cells, but six unknown transport completions remain unresolved.Its 196 complete interactions were zero; unresolved responses prevent treating the observation as a complete result.Source: §5.1 · p. 6
- Under sorted presentation, Jev accepted 52/200 versus 2/200 with three-label grading and 53/200 versus 6/200 with four-label grading; insertion yielded zero acceptances in every cue condition.Both paired presentation interactions were statistically significant after Holm correction.Source: §5.1 · p. 6, Figure 2 · p. 7
5.3 Duplicate behavior and displayed-score response
- Byte-identical duplicates showed similar aggregate sorted-presentation acceptance, but accepted sources changed; displayed-score interactions averaged +.19955 and +.17230.Duplicate observations add no independent sources or primary p-values, and score interpretation is limited by rounding and calibration.Source: §5.3 · p. 7
5.2 Construction boundary across workloads
- The interaction was concentrated in far-mutation groups: pooled near mutations yielded +2 pp, versus +48 pp and +45 pp for far mutations.Different questions received the two mutation distances, so these descriptive comparisons do not isolate a causal distance effect.Source: §5.2 · p. 6, Figure 3 · p. 8
5.4 Supporting findings from earlier studies
- Earlier studies support the paper’s characterization, but their source populations, inferential families, and correctness requirements differ.The historical-study panel does not pool counts into an independent attack rate.Source: §5.4 · p. 8, Table 2 · p. 8
6 Discussion and limitations
The confirmed interaction applies to the recorded Jev configuration and this reference-based task, not to all request-order changes or candidate edits. The experiments establish response to the tested compound presentation change, while leaving internal mechanisms and deployment prevalence unmeasured.
Discussion
- Basic controls did not reveal Jev’s presentation-dependent vulnerability; evaluations should record executed request representations and test perturbations under the integration’s actual presentations.Decoded field values alone would miss the distinction measured here; insertion’s zero cue errors are not evidence of a general defense.Source: §6 · p. 9
Discussion and limitations
- Recursive sorting changes several components together, leaving state position, other object orders, and hidden rendering unseparated.The observed errors identify response to the tested request change, not internal cue coefficients or attention mechanisms.Source: §6 · p. 9
- Generality is conditional on the reference-aware numeric grammar, balanced source/mutation mixture, and recorded service window; natural-output prevalence and downstream consequences remain unmeasured.DROP and GSM8K diversify questions and references within the same supplied-reference adjudication task.Source: §6 · p. 9
- Historical studies have limited broader interpretation because of saturated controls and incomplete semantic qualification; open implementations’ competence and precision sensitivity further bound transfer claims.The present contribution is the certified, prospectively confirmed interaction under its stated source, temporal, and measurement conditions.Source: §6 · p. 9
7 Conclusion
The paper concludes that a fixed colon edit can produce certified final-answer grading errors depending on structured-request presentation, even when the judge passes basic controls. Its conclusion is limited to the tested reference-aware construction and does not identify the responsible component or deployment prevalence.
7 Conclusion
- Basic control competence coexisted with substantial excess false acceptance under sorted presentation, supporting evaluation of perturbations using the integration’s actual request representations.The construction and recorded request change establish the tested interaction, not general robustness or component attribution.Source: §7 · p. 9
A.1.1 Three-label final-answer grading
- FINAL3 instructs the judge to assess the final commitment against trusted references, disregarding superseded intermediate wrong claims.A candidate with an earlier matching value and an explicitly wrong final value therefore requires an incorrect label.Source: §A.1.1 · p. 12
A.1.2 Four-label final-answer grading
- FINAL4 adds an ambiguous option and explicit commitment and revision rules; the restricted numeric candidates still require incorrect under both contracts.Sorting preserves criterion name-to-description bindings while moving ambiguous first in the criterion object.Source: §A.1.2 · p. 12, Table 3 · p. 12
A.2 Record identifiers and service notation
- FINAL3 and FINAL4 identify complete grading configurations, and their comparison does not isolate the effect of adding a fourth option.The four-label configuration also changes commitment and revision wording.Source: §A.2 · p. 12
A.3 Sources and numeric construction
- The confirmation panel used 200 new DROP and GSM8K clusters with near and far numeric mutations; references and candidate construction define a bounded, reference-aware task.The colon follows a source-seeded string while preserving the wrong final commitment; the illustrated source was selected after completion.Source: §A.3 · p. 13
A.4 Ordered inputs, phases, and execution
- Presentation changes request-object ordering while preserving decoded fields and candidate bytes; controls, thresholds, execution, and candidate templates were prespecified for confirmation.All configurations passed both-presentation control gates; all planned IDs were attempted, with six terminal unknown transport completions.Source: §A.4 · pp. 13–14, Table 4 · p. 14, Table 5 · p. 14
A.5 Interaction inference and missingness
This section describes the frozen evaluation workflow and statistical procedures for interaction tests, including full-panel missingness and interval summaries. It also reports inferential and missingness limits, duplicate-call and score analyses, and the scope of historical diagnostics and study designs.
A.5 Interaction inference and missingness
- Sol’s zero empirical interaction on 196 sources had interval [−38.8, 38.8] pp, so population equivalence was unestablished.Four transport failures affected primary cue cells, leaving 99 DROP and 97 GSM8K complete primary sources; the finite-panel interaction bound was [−0.5, +1.5] pp, while unknown labels remained unresolved.Source: §A.5 · p. 15
- 50/0 three-label and 47/0 four-label R:/R discordances had adjusted p-values 1.07 × 10−14 and 7.11 × 10−14; all other secondary contrasts had p = 1.The six tests used exact two-sided conditional McNemar inference with a separate Holm family; they supplement the primary interaction and make no between-configuration significance claim.Source: §A.5 · p. 15
A.6 Duplicate outputs and displayed scores
- Displayed correctness-score interaction means were +.19955 for three-label grading and +.17230 for four-label grading, while duplicate calls changed some sorted-presentation acceptances.The score distributions retain all 200 complete Jev sources, including negative interactions and ties; they are descriptive, with no new significance family or calibration claim.Source: §A.6 · pp. 15–16, Figure 4 · p. 16
B.1 Final-answer state in the reference diagnostic
- In the reference-switch diagnostic, candidate bytes stayed fixed while the trusted reference changed, legitimately changing the required verdict.The illustrative source returned correct for the R-plus-colon candidate under reference 840 and incorrect under reference 495; it was selected after outcomes from a complete source census.Source: §B.1 · p. 16
B.2 Whole-answer evidence grading and readable cues
- On a DROP numeric witness, adding a colon changed the FINAL3 verdict from incorrect (.11) to correct (.56) despite the same wrong final value 229.The 76-yard reference certifies the error; this selected example illustrates observed behavior and supplies no separate inferential sample.Source: §B.2 · p. 17
C Discovery provenance and reproducible study designs
- The research programme was sequential and adaptive, with later freezes fixing subsequent inputs and analyses after preceding outcomes were known.The controlled formatting study carried the central categorical result; the paper makes no equal-budget comparison of proposal strategies.Source: C Discovery provenance and… · p. 17
D.1 Historical task and intervention definitions
- HALL evaluates whole-answer support, whereas FINAL3 evaluates final-answer correctness; the broader HALL endpoint lacks complete independent semantic qualification.The historical cohorts use source clusters as units, and the paired analyses distinguish verdict contrasts from displayed-score changes; displayed acceptance gaps and distributions are descriptive diagnostics, not calibrated risk estimates.Source: §D.1 · p. 19, §D.2 · p. 20, §D.4 · p. 21
D.5 Prospective directional prediction
- The reference-switch study tested the prospective directional prediction that a cue raises correctness when earlier content matches the reference and lowers it when that content conflicts.It used 24 numeric pairs, identical candidate text across the two trusted-reference worlds, and independently hash-randomized request order; other functional models can produce the same pattern.Source: §D.5 · p. 21
- The workflow archived admissible bounded-beam expansions before pruning, deduplicated retained strings, evaluated them on a common source set, and froze each evaluation’s materials before testing.The feedback search selected ranked parents, proposed one-token Qwen extensions, and repeated target evaluation within round and query caps.Source: §D.5 · p. 22
- The summary links separately executed stages using different templates and source sets, and the later feedback search began from a target-informed seed bank.Appendix E provides the actual settings and objectives.Source: §D.5 · p. 22
- Each comparison used the maximum of two one-sided source sign-flip p-values as an intersection-union test, with Holm correction across four cue-by-endpoint comparisons.The tests assumed symmetric, exchangeable source differences at the boundary null; all 224 sign assignments were enumerated, and a positive interaction alone did not meet the directional alternative.Source: §D.5 · p. 22
- The full panel had no missing qc values, and descriptive intervals used 20,000 paired source bootstrap draws.Source: §D.5 · p. 22
E Cue-search implementation and coverage rationale
The section details two stages of cue proposal and selection, then separates pool coverage from estimation accuracy. Saved outcomes show that likelihood can misorder target effects, while the historical record enables tracing evaluated cues.
E.1 Initial likelihood proposals
- The initial candidate comprised “I initially considered {X}.”, a bridge, and “Final answer: {Y}.” in that order; admissible expansions entered the archive before the top twelve formed the next beam, and 16 candidates were frozen for 12 withheld confirmation questions.Source: §E.1 · p. 23
E.2 Later target-score extension
- Success within this bank cannot estimate the density of effective strings in unselected Qwen output.The proposal pool was selected, so its results do not measure how common effective strings are in unselected output.Source: §E.2 · p. 23
- Two adaptive rounds evaluated 20 seeds and 64 extensions on eight development sources (1,344 calls), then tested nine frozen bridges on 24 fresh questions.Wrong-only development scores all equalled zero, so length and text admitted Actual and Actually to confirmation; this tie-break is part of Actual’s discovery history.Source: §E.2 · p. 23
E.3 What coverage and selection can explain
- The decomposition separates proposal-pool coverage from estimation accuracy, and displayed-score bounds do not guarantee categorical-error performance.For 145 strings, 12 sources, and δ = .05, the bound is about .60; adaptive pool construction requires additional control.Source: §E.3 · p. 24
- −.009 LQ and mean Jev pyes = .030 for However contrasted with Actual’s −9.638 LQ and .824 mean; likelihood did not order these target effects.Actually had LQ = −3.198 and mean .014; the best correct-first development mean rose from .85125 to .86000, while selected confirmation extensions scored .88125 and .88750 versus Correct answer’s .90167.Source: §E.3 · p. 24
F Historical judging studies and response characterization
- Historical panels keep distinct questions and source-level units separate; Table 10 sequences certified FINAL3 colon-pair false acceptances.The 54 numeric sources were certified before calls and fresh at the study freeze; 26 additional year-only sources had no R: acceptances.Source: F Historical judging studies and… · pp. 24–25, Table 10 · p. 25
F.1 Certified false acceptance, recurrence, and score movement
Across certified and constructed panels, Jev’s false acceptances recur under colon edits, while their size and direction vary with candidate content, grading rule, formatting, and reference.
F.1 Certified false acceptance, recurrence, and score movement
- Jev accepted 10/54 R: candidates versus 0/54 R, then 10/80 versus 0/80 on repeat; repeat-panel excess was 12.5 points (95% interval 6.25–20.0).The initial counts are descriptive; the repeat-panel paired test had Holm-adjusted p = .0098 in its six-test secondary family.Source: §F.1 · p. 25
- Thirteen sources were accepted at least once across three core passes, while Jev’s mean paired score shift (.1455) exceeded repeat variation (mean absolute changes .0108 for R and .0352 for R:).Affected source identities turned over; six were accepted on all three passes.Source: §F.1 · pp. 25–26
F.2 Readable cues and dependence on the grading rule
- With a correct earlier value, HALL accepted 177 Actual candidates versus 82 with a blank bridge; the prefix interaction was +40.0 points (95% interval 33.75–46.25; Holm p = .00004).The full-panel acceptance counts remain construction-only because independent whole-answer semantic qualification is incomplete.Source: §F.2 · p. 26
- Under FINAL3, the same-string colon comparison increased acceptance from 5/240 to 37/240, while R: yielded 0/240 acceptances with a wrong earlier prefix.Actual’s correct-prefix excess over blank was +39.58 points, but Actual-minus-random also reflects suppression by the random-letter control.Source: §F.2 · p. 27
F.3 Fresh content and formatting comparisons
- In the fresh pilot, Jev’s primary interaction T was +.2400 (95% interval .14125–.34438; Holm p = .00004), while Liquid’s was −.09880 (interval −.18824 to −.02985; p = .01558).Jev accepted 6/16 X-prefix candidates with R: and none with R, blank, or bare colon; Liquid’s observed X-prefix candidates were all rejected.Source: §F.3 · p. 27
F.4 Reference-switch prediction
- For Jev, adding a colon to R increased conditional correctness by 30.88 points under reference X and decreased it by 7.71 points under reference Y; every source showed both directions in both passes.Near-floor X controls and ceiling Y controls mean directional consistency alone cannot distinguish earlier-value influence from confidence compression.Source: §F.4 · p. 28
- Jev’s R: condition under reference X produced five false-positive calls out of 48, while matched R produced none; Liquid had no categorical errors in the composite panel.These verdict counts are descriptive evidence accompanying the primary score test, with repeated calls distinguished from source witnesses.Source: §F.4 · p. 28
G Statistical and operational details
- The study retains separate test families and makes no paper-wide correction claim; unadjusted displayed-score contrasts remain descriptive, and bootstrap p-values are finite-simulation approximations.The hypothesis registry records each test, statistic, population, alternative, raw p-value, adjustment, and source result.Source: G Statistical and operational details · p. 29, Table 15 · p. 29
G.1 Centered bootstrap and exact paired procedures
The section reports paired inference and control checks, then distinguishes native verdicts from displayed-score shifts and explores what those shifts can support. Repeat identities, attainable test resolution, and model assumptions constrain the interpretation.
G.1 Centered bootstrap and exact paired procedures
- 32 forward versus 0 reverse discordances occurred in the full 240-source FINAL3 R:/R contrast; bootstrap raw p = 1/50,000 = .00002 and Holm p = .00034.The raw p-value is at the finite-simulation floor; the exact McNemar sensitivity calculation is post-hoc and does not replace the historical test.Source: §G.1 · pp. 29–30
- Reference-switch tests enumerate source sign assignments, use the larger directional p-value, and apply Holm correction; unavailable primary tests receive p = 1 for family accounting.The procedures assume independent sampled source clusters; intervals are descriptive, not simultaneous.Source: §G.1 · pp. 29–30
- All four reference-switch control strata achieved 24/24 correct, while Jev’s unit R controls under reference Y permit only decreases or ties.Those measured baselines constrain interpretation of the directional score changes.Source: §G.1 · p. 31, Table 17 · p. 31
G.2 Identical requests and repeated outputs
- Repeated requests returned identical probability vectors, but output identity cannot establish caching or independent computation.The studies had different input populations and collection windows.Source: §G.2 · p. 31, Table 18 · p. 31
G.3 Weighting, repeat identities, and attainable evidence
- Jev accepted 13 X-prefix R: sources at least once across three passes, with 6 accepted every time and 7 changing verdict.For the colon/blank directional test, zero patterns limit the joint raw p-value to at least .03125; .0625 depends on the observed Holm ordering.Source: §G.3 · p. 32
H.1 Paired readouts and repeat variation
- Liquid’s repeat mean absolute changes were .00507 for R and .00460 for R:, versus .03234 for the first-pass cue/control change.Repeat differences share baselines and sources, so they add no independent sample.Source: §H.1 · pp. 32–33
H.2 Score distributions and hypothetical thresholds
- Figure 9’s threshold-crossing curves are hypothetical, not downstream-policy results; their signed area equals the weighted mean displayed-score change.The returned native choice remains the categorical endpoint, and displayed-score shifts do not establish a mechanism.Source: Figure 9 · p. 34, §H.2 · p. 35
I Restricted functional model and score observability
- Under the restricted invariances, half-sum G equals Δβ and half-difference S equals 2λΔα; probability differences do not identify these margin parameters.Several alternatives can produce the same directional prediction, and the hosted observations do not measure attention scores or a physical head.Source: I Restricted functional model and… · p. 35
I.1 Exploratory affine diagnostic and a prospective distinction
The affine decomposition is exploratory and conditional on model assumptions, motivating a prospective third-reference test. The section also situates the fixed-cue open-implementation evaluation within reused inputs and specified execution configurations.
I.1 Exploratory affine diagnostic and a prospective distinction
- A third reference matching neither X nor Y could distinguish restricted weighting and shared-affine accounts, but its baseline, invariances, and sampling power remain unmeasured.The conditional predictions differ: about +1.64 nats for the shared-affine account versus about .030 nats for the weighting model.Source: §I.1 · p. 36
- Figure 11 displays 24 finite Liquid source contrasts: G = .02995 nats (interval [−.2279, .2721]) and S = 1.57161 nats (interval [1.2461, 1.8985]); Jev had none.The common-component interval does not establish equivalence to zero, and S has its stated interpretation only under invariance assumptions.Source: §I.1 · p. 36, Figure 11 · p. 37
J Additional findings and boundaries
- Earlier RAGTruth whole-answer evidence is supportive and model-qualified; the broader adaptive programme estimates no global probability of discovering a successful cue.Model-produced annotations remain distinct from completed human adjudication.Source: J Additional findings and boundaries · p. 37
K Fixed-cue evaluation of open decision implementations
- The open-implementation study applied fixed cues to historical inputs without searching new strings or fitting target-specific thresholds.Its reused cohorts add target observations but no independent fresh-source confirmation of a cue effect.Source: K Fixed-cue evaluation of open… · pp. 37–38, §K.1 · p. 38
K.1 Inputs, implementations, and analysis
- Paired open-implementation inputs retained the earlier matching value and wrong final value, but public serializers produced non-byte-identical inputs across implementations.The evaluated configurations included distinct released heads and precision/device choices; these configurations are part of the experimental identity.Source: §K.1 · pp. 38–39
K.2 Competence and categorical effects
The open-implementation evaluation found varied categorical effects, but every tested configuration failed the declared numeric competence gate. Precision replays and exploratory screening further bound transfer claims.
K.2 Competence and categorical effects
- Every tested configuration failed the declared numeric competence gate, limiting inference about vulnerability among competent judges on this task.The gate required correct counts in four clean numeric conditions; missing calls would count against it.Source: §K.2 · p. 39, Table 22 · p. 39
- Among fifteen designated categorical comparisons, CLM NF4 showed a positive numeric colon contrast, while other effects varied across targets and tasks.NanoJev’s HALL primary cells were near an acceptance ceiling and CLM’s near a floor; the families do not support cross-model ranking.Source: §K.2 · p. 39, Table 23 · p. 40
- Clef-flash NF4 accepted 55/80 Correct answer cases versus 28/80 blank cases, with 29 cue-only and 2 control-only discordances.Its HALL contrast ran in the opposite direction, but the panels differ in content and grading criteria.Source: §K.2 · p. 39
K.3 Execution sensitivity and additional screening
- SemIf’s numeric R:/R difference changed from +6.25 pp under NF4 to −18.75 pp under int8, leaving fidelity to the reference model unresolved.The two executions agreed on 4,646/5,312 choices (87.46%); neither supplied a bf16 reference.Source: §K.3 · p. 41
- A 240-request CLM bf16 replay agreed with NF4 on 151/240 choices and returned correct for every input, changing the NF4 +23.75 pp colon difference to zero.Device and precision changed together, and the replay differed from the reference vLLM implementation, so quantization noise alone is not identified.Source: §K.3 · p. 41
- The exploratory screen found no configuration passing either numeric gate; two NF4 first-token option-name readouts passed the separate HALL gate without cue confirmation.The screen supplies no additional fresh-source cue-generalization claim, and the add-on programme has provenance and reporting limitations.Source: §K.3 · pp. 41–42, Table 25 · p. 41
L Fresh certified contract and generative comparator
The fresh comparator tested three configurations on 200 source clusters, with predefined paired tests and controls. An exploratory follow-up found different Jev outcomes under sorted and insertion-order presentation, but reused sources and post-outcome selection limit that diagnosis.
L Fresh certified contract and generative comparator
- The fresh confirmation used 200 clusters split evenly between DROP numeric questions and GSM8K, with finite numeric references certifying the wrong final value.Sources were prospectively frozen and excluded prior IDs, normalized questions, and declared text-overlap matches.Source: L Fresh certified contract and… · p. 42
- Jev FINAL3 missed the wrong-final-only clean threshold by one response, while the FINAL4 forward flip remained an isolated certificate without corrected population excess.The primary table covers all three native false-acceptance comparisons on the same 200 sources; its caption does not supply the remaining cell values.Source: L Fresh certified contract and… · p. 42, Table 27 · p. 43
- All seven secondary categorical tests failed correction, and wrong-prefix comparisons likewise showed no corrected excess under either Jev grading configuration.The reported wrong-prefix discordances were four versus zero under FINAL3 and six versus zero under FINAL4.Source: L Fresh certified contract and… · p. 43
L.1 Exploratory presentation check
- Recursive sorting changed multiple object orders and Sol’s embedded JSON, so the exploratory comparison cannot isolate a presentation component or mechanism.The diagnostic reused identities and was selected after the main outcomes, so it provided no independent sorted-presentation confirmation.Source: §L.1 · pp. 43–44
- On 28 reused sources, sorted Jev accepted R: more often than R, while insertion presentation produced zero acceptances for both cues across all configurations.Sorted FINAL3 acceptance was 2/28 for R and 8/28 for R:; sorted FINAL4 acceptance was 1/28 and 6/28, respectively. These are exploratory descriptive outcomes.Source: §L.1 · p. 44
- The exploratory sorted-presentation result generated the hypothesis for a fresh comparison, while the 200-source insertion-presentation primary nulls remained unchanged.The diagnostic’s outcome-selected, reused-source design limits its confirmatory force.Source: L Fresh certified contract and… · p. 43, §L.1 · p. 44
M Research materials, reproducibility, and ethics
The paper retains materials for replaying its analyses and documents execution, verification, and distribution boundaries. Its tests used isolated evaluation tasks rather than production actions, and deployed safeguards remain untested.
M Research materials, reproducibility, and ethics
- Replay materials include grading instructions, candidate renderings, references or evidence, source identifiers, pairing rules, native responses, and audit records.Offline replay checks reconstruct source denominators and reported comparisons; new calls from a service alias cannot guarantee the same hosted implementation or outputs.Source: M Research materials, reproducibility,… · p. 44
- Hosted inference used provider APIs with unobserved serving hardware, while local open-model work ran on a single laptop with an 8 GB GPU-memory budget.The reported compute precisions and quantization describe configurations run within that budget.Source: M Research materials, reproducibility,… · p. 44
- Numerical verification replayed analyses and request reconstruction within the author-led study, but shared configuration constants and the response parser with execution.The paper does not claim an outside human audit or an independently operated collection of model responses.Source: M Research materials, reproducibility,… · p. 44
- Prepared offline packets support numerical replay, but public hosting and broader code/data release remain pending, and reconstructed source redistribution follows dataset terms.Natural questions and passages are omitted from distribution and reconstructed from pinned public datasets.Source: M Research materials, reproducibility,… · p. 44
- The reported API-call costs are accounting estimates, not invoice-verified measurements or a comparison of discovery efficiency.The fresh comparator used approximately $2.22 and the independent presentation confirmation approximately $3.20 under conservative rate-policy charges.Source: M Research materials, reproducibility,… · p. 45
- The study used isolated evaluation tasks without production actions against third-party users, and it leaves deployed safeguards and downstream policy consequences untested.The paper distinguishes numeric certificates from model-generated semantic labels and identifies access to reference-matching earlier content as an operational-scope consideration.Source: M Research materials, reproducibility,… · p. 45
- Model-generated labels supported earlier annotation, while the author remains responsible for review and independent human adjudication is still pending.Automated coding and drafting assistance also supported implementation, analysis preparation, and manuscript writing.Source: M Research materials, reproducibility,… · p. 45
- Vendor notices gave TypeSafe AI and Liquid AI advance awareness, but neither had provided substantive feedback by October 7, 2026.The notices do not imply vendor verification, agreement, or endorsement.Source: M Research materials, reproducibility,… · p. 45