PaperlayerView exact revision on arXiv

Source-linked AI summary

Type-Safe Is Not Error-Free: A Constrained Decision Head Follows the Option Name, Not the Rubric Bound to It

Yu Sun, Junhao Xu, Jiajia Shi, Zijin Yang

arXiv:2609.26758v2cs.AI

TL;DR

Type safety ensures a decision stays within the declared options, but does not ensure that a model interprets them according to their definitions. The paper tests this by changing which names are bound to definitions, finding that polar names can reverse rankings while neutral and random names have much smaller effects.

  • Problem

    The paper asks whether typed models choose valid options according to their explicit definitions, since a 0% type-error guarantee says nothing about decision quality.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    The study changes which option name is bound to each definition while holding the question, state, names, and definition texts fixed.

    Source: §8 · p. 6

  • Results

    70.4 pp higher flip rate than the neutral 0/1 control accompanies an AUC fall from 93.8% to 23.2% under yes/no reassignment.

    Source: §8 · p. 7

  • Takeaways & Limitations

    A type-error check cannot detect name-driven decision inversions because type safety constrains the available options, not probability distribution among them.

    Source: §2 · p. 2, §8 · p. 7

    The open-weight evaluation covers two models on English questions, and whether the scoring explanation holds for other architectures is untested.

    Source: §7 · p. 6

Abstract

from arXiv · show

Typed decision models are built for settings where model outputs are consumed directly by software. Instead of generating free-form text, they return a decision over a predefined set of options. By construction, every output conforms to the required schema. Yet this guarantee does not tell us whether the model interprets the options as intended. We study Jev and two Jev-like models with open weights by changing how option names are assigned to rubrics. Each option consists of an option name and a textual rubric that defines what the option means. We change only which option name is assigned to each rubric; the question, state, rubric wording, and set of option names remain exactly the same. On 1200 workflow decisions with task-specific rubrics, renaming the two options from 0/1 to no/yes changes 70.4 more answers per hundred (95% CI: [67.6, 73.1]) and shifts AUC from .94 to .23, revealing a systematic reversal in the decision ranking rather than simple uncertainty. The same operation has little effect with neutral option names. This pattern holds across all 4 predicates, where the effect is at least 7.4x larger than under the neutral control, and becomes stronger as the number of options increases. The effect also depends on the read-out geometry: a second model family that mean-pools over the full option span flips 4.1x less often. The hosted model exhibits the same behavior: the swap changes AUC from .8146 to .5806 and produces 24x as many answer flips as its test-retest floor. In contrast, replacing the option names with random character strings returns all model families to the neutral-control regime without reducing accuracy. The failure therefore depends on the semantic polarity of the option names rather than on the renaming operation itself. Across all conditions, the type-error rate remains 0%, even when decision accuracy degrades substantially.

1 Introduction

The introduction asks whether typed decision models follow explicit option definitions or the competing cues of option names. It describes a reassignment test and reports that polar names can reverse decision rankings, with effects varying by name type and model family.

1 Introduction

  • The study keeps the question, state, option names, and definition texts fixed while changing which names are paired with which definitions.It tracks the selected definition rather than the returned name.Source: §1 · p. 1
  • For yes/no names, reassignment lowers one model’s AUC below chance, indicating a ranking reversal rather than merely changed individual answers.The introduction notes that threshold adjustment cannot correct this below-chance ranking.Source: §1 · p. 2
  • Polar-name swaps affect decisions across every tested predicate and can reverse score rankings, not merely change individual answers.Source: §1 · p. 2
  • Familiar polar names, including interface defaults, are most affected, while random strings behave like neutral identifiers with comparable accuracy.Source: §1 · p. 2
  • Name dependence extends beyond binary tasks and one model family; neutral renaming changes non-binary decisions, and effects vary with readout geometry.Source: §1 · p. 2

2 The scope of the type-safety guarantee

Typed decision models return probabilities over a declared option set, and their type-safety guarantee excludes options outside that set. The guarantee does not establish whether probability is assigned according to the options’ explicit definitions.

2 The scope of the type-safety guarantee

  • A typed decision model scores each declared option, returns a softmax distribution, and answers with the most probable option.Each option pairs a name with a definition; the model set includes Jev, Laya, and Open-Jev.Source: §2 · p. 2
  • Laya scores an option using a contextual [MASK] embedding, while Open-Jev averages tokens across the option text, including its name and definition.The authors discuss this readout difference as one possible reason for differing name effects.Source: §2 · p. 2
  • Type safety guarantees that answers belong to the declared option set, but does not constrain how probability is distributed among those options.The paper says this guarantee holds by construction for the open-weight models and is documented behavior for Jev.Source: §2 · p. 2
  • A 0% type-error rate says nothing about decision quality and cannot detect the studied failure, because the guarantee applies even when decisions are inverted.The open-weight models mask candidates outside the declared set before normalization.Source: §2 · p. 2

3 Reassigning names and definitions

The section operationalizes reassignment by changing the binding between option names and definitions while leaving the other input content unchanged. It specifies the name controls, decision data, and metrics used to measure the effect.

3 Reassigning names and definitions

  • On ordinary questions, names and definitions agree, so accuracy alone cannot reveal which cue the model follows.Source: §3 · p. 2
  • The binary reassigned arm binds each name to the other definition, while the gold answer continues to follow the definition.A model following definitions should preserve its definition-level choice; one following names inverts its answer.Source: §3 · pp. 2–3
  • Five polar name pairs are compared with neutral 0/1 and A/B controls, with effects reported as differences against the neutral controls.The design also extends to random-string names and questions with more than two options.Source: §3 · p. 3
  • The main dataset contains 1,200 binary questions across four tasks, with task-specific definitions and 57.5% yes answers.Another 600 questions use generic definitions beginning with “no” and “yes”; the authors exclude these confounded cases from main results.Source: §3 · p. 3
  • The evaluation measures definition-level flip rates, balanced accuracy, and AUC; Jev also receives a test-retest flip-rate floor because it is nondeterministic.The flip-rate intervals use 2,000 bootstrap resamples over states.Source: §3 · p. 3

4 Results

Across binary, non-binary, and model-family tests, reassignment of option names often changes the selected definitions, especially for polar names, while type errors remain absent. The effect includes a below-chance ranking reversal for yes/no and is smaller for random strings and some readout designs.

4.1 Polar names override the definitions

  • 70.4 pp more decisions flip for yes/no than 0/1 (95% CI [67.6, 73.1]); yes/no balanced accuracy falls from 87.2% to 28.4%, while type-error rate remains 0%.For 0/1, balanced accuracy stays at 83.7% and 83.2%.Source: §4.1 · p. 3, §4 · p. 3, Table 1 · p. 4

4.2 The ranking is reversed

  • AUC for yes/no falls from 93.8% to 23.2%, below chance; negative/positive flips 48.3% while its AUC declines only from 82.6% to 75.8%.The below-chance yes/no ranking is reversed, whereas negative/positive changes many decisions but leaves most of its ranking intact.Source: §4.2 · p. 3, Figure 1 · p. 4

4.1 Polar names override the definitions

  • Yes/no flips at least 56.7% in each of four tasks, at least 7.4 times the 0/1 rate, which never exceeds 11.3%.Other polar pairs vary substantially across tasks, from 12.3% to 92.0%.Source: §4.1 · p. 3, Table 2 · p. 4

4.3 Polarity, amplified by familiarity

  • Polarity adds 41.0 pp over neutral names; familiarity adds another 16.0 pp for pairs Laya emitted during training.The larger contribution comes from polar names that Laya had not emitted during training.Source: §4.3 · pp. 3–4, Table 3 · p. 4
  • Random strings differ from neutral controls by at most 3.2 pp in flip rate across models, without reducing balanced accuracy by more than 3.3 pp.The familiar polar pairs flip 32.2% to 70.7% of decisions on the same questions.Source: §4.3 · p. 4, Table 4 · p. 5

4.4 Beyond binary decisions

  • With 3–16 options, neutral letters flip 52.4% and original-name reassignment flips 79.7%, lowering accuracy to 15.5%.The definitions stay unchanged, and the gold answer follows the definition.Source: Table 5 · p. 5, §4.4 · p. 5

4.5 Other model families

  • Across models the name-class ordering holds, but Laya flips 4.1 times as often as Open-Jev, which averages the full option text; Jev’s 32.5% flip rate is 24 times its retest floor.Laya scores a marker token at the option, while Open-Jev averages tokens across the option text.Source: Table 6 · p. 5, §4.5 · p. 5

5 Related work

Prior work examines sensitivity to label words, prompt format, and semantic cues in constrained outputs. This paper distinguishes its explicit name–definition reassignment from work that varies demonstrations, rubrics, or schema keys.

5 Related work

  • Krishna Kumar (2025) reports zero semantic override under inverted demonstrations, while this study writes reassignment explicitly and observes a reversed ranking at 23.2% AUC.Le (2026) studies schema keys under constrained decoding and reports accuracy deltas; this paper uses neutral names as a control.Source: §5 · p. 5
  • Prior work on label words and prompt format predicts the direction of the effect, while Lim et al. (2026) vary rubrics with names fixed.This study instead holds definitions fixed and varies which name each is bound to; the cited curriculum fix is not evaluated here.Source: §5 · p. 6

6 Discussion

The discussion proposes reporting name sensitivity alongside type-error rates and identifies neutral naming and name randomization as practical mitigations.

6 Discussion

  • On Laya, yes/no reassignment yields a 76.9% flip rate, motivating a name-invariance measure alongside type-error rates.The proposed flip rate needs no gold labels and costs two extra forward passes per question.Source: §6 · p. 6
  • On Laya, neutral names reduce the flip rate to 6.5% instead of 76.9%.The paper also proposes training models to follow definitions by randomizing option names.Source: §6 · p. 6

7 Limitations

The evaluation is limited by its model, language, and task coverage, and some cross-model or non-binary findings remain qualified by accuracy and measurement constraints.

7 Limitations

  • The study uses two open-weight models on English questions; whether the scoring-based explanation generalizes to other architectures is untested.Open-Jev’s balanced accuracy before reassignment is 59.8%–63.9%, versus 79.4%–89.6% for Laya, which may also explain its lower flip rate.Source: §7 · p. 6
  • Jev is closed-source, accessed by API at one point in time, and nondeterministic, so its served version may change and its test-retest floor matters.The study observes only Jev’s returned distribution.Source: §7 · p. 6
  • Name-pair meaning preservation varies, so the paper reports minimum rather than mean effects; human annotation could sharpen Table 3’s middle rows.For questions with more than two options, original-name accuracy is 56.4%, so that result is treated as directional.Source: §7 · p. 6

8 Conclusion

The conclusion shows that changing only the name–definition pairing can substantially alter typed decisions. Type safety guarantees membership in the declared option set, not interpretation according to definitions.

8 Conclusion

  • Type safety guarantees that a model returns a declared option, but not that it interprets that option according to its explicit definition.A type check therefore does not establish that the model selected the intended meaning.Source: §8 · pp. 6–7
  • The comparison changes only which definition is bound to each name while holding the question, state, names, and definition texts fixed.The reassignment can substantially alter model decisions.Source: §8 · pp. 6–7
  • 70.4 pp higher flip rate than the neutral 0/1 control, with AUC falling from 93.8% to 23.2% for yes/no reassignment.These results show that structurally valid decisions can remain dependent on semantic cues in option names.Source: §8 · p. 7

B Flip rate by decision task

The supplied visuals describe Laya’s flip rates across decision tasks and name pairs, and by name class, with task ranges shown for the latter.

C Questions with generic definitions

With generic definitions beginning “no” and “yes,” reassignment flips most decisions, but that comparison is only an upper bound because names and definitions are confounded. The random-string section lists 15 generated name pairs.

C Questions with generic definitions

D Random-string names

  • Table 8 lists 15 random-string name pairs from a seeded generator; Jev was run on the first five pairs.The generator’s acceptance rule depends only on names already accepted, so adding pairs does not change earlier ones.Source: D Random-string names · p. 8, Table 8 · p. 8

E An example with more than two options

In a four-option program example, Laya selects the definition named 0 after reassignment, although the correct definition is then named 5.

E An example with more than two options

Loading 2609.26758v2…