PaperlayerView exact revision on arXiv

Source-linked AI summary

JevOut: Natural Context Can Flip Decision Models

Zixiang Xu

arXiv:2609.30243v1cs.CL

TL;DR

Dedicated decision models expose probability distributions over choices that downstream systems use, but surrounding context is part of their normal inputs. This paper fixes a wrong target and optimizes short, answer-preserving additions using option probabilities. Across datasets and systems, these additions redirect many initially correct decisions, including with high probability on the fixed wrong option.

  • Problem

    Dedicated decision models turn language into finite choices for downstream use, but it is unclear whether natural context additions can redirect them when the task and correct answer remain unchanged.

    Source: §1 · p. 1

  • Method

    For initially correct items, the study fixes a wrong target and uses option probabilities to refine fluent additions while preserving the source, question, choices, and gold answer.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Across seven datasets and four systems, short answer-preserving contexts redirected initially correct decisions; Jev's TFR was 61.4% within 64 accepted evaluations.

    Source: §1 · p. 3, §5 · p. 11

  • Takeaways & Limitations

    A finite option set and probability distribution do not ensure that a downstream choice follows the task rather than a persuasive contextual detail.

    Source: §5 · p. 11

Abstract

from arXiv · show

Dedicated decision models such as Jev map unstructured language to probability distributions over finite choices, allowing their outputs to directly route requests, select tools, and trigger actions. Yet real-world inputs rarely arrive in isolation: they come with background details and surrounding context. We find that short additions that fit naturally into this context can nevertheless redirect an otherwise correct decision, even when the correct answer remains unchanged. To study this behavior, we fix a wrong target option for each initially correct item and use the model's option probabilities to refine fluent context additions while preserving the source, question, choices, and gold answer. Within 64 accepted target evaluations, the optimizer identifies contexts that redirect Jev on 312 of 508 initially correct decisions (61.4%); in 229 cases, Jev assigns at least 0.7 probability to the fixed wrong option. Across seven datasets, three additional decision systems show targeted flip rates of 64.9%-73.2% on decisions they initially answer correctly. Taken together, these results expose a pronounced fragility in current decision models: short, ordinary-looking context can shift a correct choice to a high-confidence wrong one. Because these models turn language directly into downstream choices, this sensitivity raises concerns about treating their probability outputs as reliable decision interfaces.

1. Introduction

Dedicated decision models turn language into option probabilities that downstream systems can use, but their inputs normally include surrounding context. The paper studies whether short, natural additions that preserve the task and answer can redirect an initially correct choice, and reports frequent targeted flips under probability-guided optimization.

1. Introduction

  • Natural context is a normal part of decision-model inputs, yet short background or procedural additions can redirect a correct choice without changing the question, options, or gold answer.The additions do not explicitly name the wrong option or ask the model to change its answer.Source: §1 · p. 1
  • Earlier work examined sensitivity to prompt presentation and fluent context, while leaving open how dedicated decision models respond when the task, options, and correct answer stay fixed.Prior studies addressed prompt formatting, option order, evidence placement, and controlled changes to question-answering inputs (Ishibashi et al., 2023; Pezeshkpour and Hruschka, 2024; Liu et al., 2024; Cao et al., 2022; Ye et al., 2025).Source: §1 · pp. 1–2
  • Targeted Flip Rate measures the share of initially correct decisions redirected to a fixed wrong option within a fixed target-evaluation budget.The study fixes an incorrect option for each item and preserves the source, question, choices, and gold answer while constructing fluent additions.Source: §1 · p. 2
  • Probability-guided optimization ranks candidate additions by movement toward the fixed target, concentrating evaluations on contexts approaching the decision boundary.Unlike label-only feedback, option probabilities provide a graded progress signal before the selected answer changes.Source: §1 · p. 2
  • Within 64 accepted target evaluations, Jev flipped 312/508 initially correct decisions (61.4% TFR), with 229 reaching at least 0.7 probability on the fixed wrong option.Successful Jev contexts contained a median of 31 added words.Source: §1 · p. 3
  • OpenSourceJev, Von, and plain Qwen reached TFRs of 64.9%–73.2% on their respective initially correct populations under the same budget.The results also include a 7.1 percentage-point TFR improvement from probability rather than label feedback on a matched subset of 140 Jev decisions.Source: §1 · p. 3

2. Related Work

Prior work documents that language-model predictions and model-based judgments can shift with input presentation and controlled features. This paper studies a distinct intervention for dedicated decision models: adding context while keeping the original decision problem and a fixed wrong target unchanged.

2. Related Work

  • Prompt templates, demonstration order, option order, and relevant-evidence placement can alter language-model behavior (Lu et al., 2022; Ishibashi et al., 2023; Pezeshkpour and Hruschka, 2024; Liu et al., 2024).Natural adversarial question answering also shows fluent contexts steering a model toward a distractor while preserving the gold answer (Cao et al., 2022).Source: §2 · p. 3
  • This study isolates answer-preserving context addition with the source, question, options, and correct answer fixed, and fixes the wrong target before optimization.The resulting outcome is directed redirection within an unchanged decision problem, rather than arbitrary disagreement.Source: §2 · p. 3
  • Model-based evaluators show sensitivity to response position, style, and other controlled features (Zheng et al., 2023; Ye et al., 2025; Shi et al., 2025), and optimized phrases can manipulate judgments across evaluators (Raina et al., 2024).These works motivate studying sensitivity where a decision model exposes probabilities over a finite set of options.Source: §2 · p. 3

3. Natural-Context Redirection

Section 3 formalizes redirection as a fixed-target flip under answer-preserving context constraints, then uses option probabilities to guide construction and learns direct generation from development trajectories.

3.1. Setting and Context Additions

  • The eligible population contains decisions the model initially answers correctly; each item's most probable wrong option is fixed as the target before context construction.Multi-answer tasks use binary option-membership decisions with one eligible absent option selected per item.Source: §3.1 · pp. 3–4
  • Answer-preserving additions fit naturally while keeping the original characters, question, options, answerability, and complete correct-answer set intact, without explicit answer-selection cues.New information and changes of emphasis are allowed if the answer remains unchanged.Source: §3.1 · p. 4
  • A targeted flip requires selecting the fixed target in an accepted, evaluated context within budget; selecting another wrong option does not count.Rejected or unsuccessful proposals do not remove an initially correct decision from the denominator.Source: §3.1 · p. 4

3.2. Probability-Guided Context Optimization

  • The optimizer maximizes the target's log-probability margin over its strongest competitor, with a positive margin making the target uniquely most probable.A language-model proposer adds a sentence and insertion boundary; mechanical checks and a separate base-model check enforce placement, coherence, and answer preservation.Source: §3.2 · p. 4
  • Probability feedback allocates construction toward archived contexts with stronger margins while retaining exploration, and iterations extend sampled parents with some restarts from the original input.Construction stops after a completed iteration yields a targeted flip above the specified threshold or reaches the iteration limit; rejected attempts consume generation resources but not target calls.Source: §3.2 · pp. 4–5

3.3. Learning to Generate Contexts

  • Development trajectories train a fixed-decision-model proposer: V1 predicts the next addition using margin-weighted accepted transitions, while V2 generates selected complete successful contexts with item-balanced weights.Length filtering is applied to the retained training examples; V1 weights emphasize larger-margin outcomes, including useful contexts that have not yet flipped the decision.Source: §3.3 · p. 5
  • At held-out evaluation, the proposer generates a complete context from an unseen item and fixed target without intermediate target feedback; selecting among independent generations uses target evaluations.The setting tests whether development-time optimization improves direct context generation on held-out decisions.Source: §3.3 · pp. 5–6

4. Experiments

The experiments test answer-preserving context optimization across seven datasets and four decision systems, then examine feedback, transfer, and learned direct generation. They find frequent targeted flips and gains from probability feedback, cross-model transfer, and development-time training.

4.1. Experimental Setup

  • The evaluation covered seven datasets and four target systems, with held-out primary results and initially correct populations of 508 Jev, 328 OpenSourceJev, 328 Von, and 285 plain Qwen decisions.The optimizer used at most 64 accepted target evaluations; rejected proposals remained in the original denominator.Source: §4.1 · p. 6

4.2. Natural Context Redirects Decisions Across Models and Tasks

  • 61.4% of initially correct Jev decisions were redirected within 64 accepted evaluations, versus 16.9% with one target-aware addition and 2.2% with one neutral addition.OpenSourceJev, Von, and plain Qwen reached 72.6%, 73.2%, and 64.9% TFR, respectively.Source: §4.2 · pp. 6–7, Table 1 · p. 7
  • 229/508 Jev decisions reached target probability at least 0.7, while redirections appeared across all target–dataset combinations and Jev TFR ranged from 40.0% on legal reasoning to 85.7% on tool routing.Among successful Jev contexts, 51.9% used one addition, 81.1% used at most two, and the median added length was 31 words.Source: §4.2 · pp. 7–8

4.3. Probability Feedback Improves Context Optimization

  • At 64 evaluations on the same 140 Jev decisions, probability-only feedback reached 63.6% TFR versus 56.4% for label-only feedback, a 7.1-point gain.Across the fixed 508-decision Jev population, TFR rose from 40.4% at 16 calls to 61.4% at 64.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4. Context Additions Transfer Across Models

  • Frozen contexts transferred to every directed model pair, with targeted transfer rates from 21.1% to 47.8%; transfer was strongest between OpenSourceJev and plain Qwen.Their transfer rates were 44.2% and 47.8% in the two directions.Source: §4.4 · p. 9

4.5. Learning Improves Direct Context Generation

  • Development-time training improved held-out direct generation: V2 raised one-generation TFR from 16.1% to 21.9% and best-of-four TFR from 29.1% to 34.3%.The paired comparisons do not establish V2 superiority over V1.Source: Table 2 · p. 10, §4.5 · p. 10

5. Conclusion

The conclusion argues that finite-choice interfaces do not ensure choices follow the task rather than persuasive context. The paper distinguishes its intervention from prior input-sensitivity work and formalizes how fixed-target redirection is evaluated.

5. Conclusion

  • Finite option sets and probability distributions make decisions convenient for downstream software, but do not ensure choices follow the task rather than persuasive contextual details.The conclusion calls for models used in downstream actions to distinguish context that changes what should be decided from context that merely changes favored options.Source: §5 · p. 11

A. Expanded Related Work

  • Unlike prompt-representation studies, this work inserts new background detail while preserving the original source spans, choice schema, and correct answer.It also differs from item-specific counterfactuals: contexts target a preselected wrong option without changing the task’s correct answer, and are not claimed to be minimal explanations.Source: A. Expanded Related Work · pp. 13–14
  • The intervention is not an instruction override or jailbreak: ordinary background context can redirect a bounded choice without requesting task disregard or policy bypass.When such choices control routing or tool use, the paper identifies a potential security surface, while noting that policy bypass is not required.Source: A. Expanded Related Work · p. 13
  • Concurrent typed-schema analyses vary option bindings or order, whereas this study holds those fixed and changes surrounding context to target a fixed wrong option.Dedicated decision models expose full option distributions, so probability movement can guide construction before the selected option changes.Source: A. Expanded Related Work · pp. 13–14

B. Formalization and Evaluation Metrics

  • Evaluation uses initially correct eligible units, fixes the most probable wrong option as target, and counts success when an accepted context selects that target within budget.The thresholded measure additionally requires target probability at least τ; reported thresholds are not assumptions of common calibration across models.Source: B. Formalization and Evaluation Metrics · pp. 14–15
  • The evaluation distinguishes pooled item-weighted rates from equal-weight dataset macro estimates and uses matched eligible populations for own-target comparisons and exact-unit matching for transfer.Wilson intervals summarize the fixed evaluation sample, while the probability-versus-label comparison uses paired, dataset-stratified source-item bootstrap resamples.Source: B. Formalization and Evaluation Metrics · p. 15

C. Dataset Construction and Task Adaptation

The paper adapts seven datasets into bounded decisions, standardizes model outputs for optimization, and iteratively proposes contexts under acceptance constraints. Probability-based archive allocation and feedback guide proposals while preserving the original decision problem.

C. Dataset Construction and Task Adaptation

  • MMLU-Pro uses validation development and test evaluation items, while SuperGPQA is field-balanced after duplicate-question removal; both become question-only Choice decisions preserving candidate order, with insertion before the question.MuSR uses narrative state with native questions and choices, ToMBench retains unique stories, and LAR-ECHR combines case facts with prior legal arguments to ask for a plausible next argument; SATA-Bench creates one binary unit per option, and BFCL V4 represents function schemas as Choice branches.Source: C. Dataset Construction and Task… · pp. 15–16

D. Decision Interfaces and Probability Extraction

E.1. Construction and Budget Accounting

  • The optimizer archives accepted evaluated contexts, extends selected parents, and permits at most four additions and 64 accepted target evaluations under the primary configuration.Runs can use fewer calls because of early stopping or rejected slots; success is counted over the evaluated archive, and representative contexts follow a compactness and target-probability rule.Source: §E.1 · pp. 17–18

E.2. Archive Allocation

  • Archive allocation weights contexts by target margin using a temperature-controlled Gibbs distribution mixed with uniform probability, with systematic resampling assigning parents to slots.This allocation characterizes recorded candidates, not a global optimum over possible texts.Source: §E.2 · p. 18

E.3. Feedback Conditions

  • The feedback comparison changes archive allocation from margin-based probabilities to a binary target-selected indicator while keeping proposal settings, acceptance checks, and budgets shared.Before a targeted flip, label-only allocation is uniform over archive entries apart from scheduled restarts; probability feedback can also provide numerical history to the proposer.Source: §E.3 · pp. 18–19

F. Prompts and Automatic Constraint Enforcement

  • Proposals add natural, locally coherent context, while mechanical and semantic checks reject direct answer cues or changes to the complete correct-answer set or required next action.The checker evaluates the completed or growing context without seeing the target option or probability feedback; one-shot neutral and target-aware controls differ in access to the fixed target.Source: F. Prompts and Automatic Constraint… · pp. 19–20

G. Experimental Protocol

The experimental protocol applies the optimizer and controls to initially correct decisions, with matched subsets for feedback and learned-generation comparisons. Execution accounting separates proposals from accepted target evaluations and distinguishes reused outcomes from new calls.

G. Experimental Protocol

  • Primary optimization runs on each target’s initially correct population with a fixed wrong option selected before proposals, while neutral and target-aware one-shot controls each receive one proposal.The matched Jev feedback subset contains 140 decisions; its conditions vary probability history and allocation feedback while holding other settings and budgets fixed.Source: G. Experimental Protocol · p. 21

G.1. Execution Scale and Accounting

  • Primary optimization records 68,912 proposals and 60,685 evaluations across 1,449 cases, averaging 47.6 proposals and 41.9 evaluations per case.The 64-call maximum, early stopping, and rejected slots explain the lower mean evaluation count; reused transfer outcomes and initial clean evaluations are excluded from new-call accounting.Source: §G.1 · pp. 21–22, Table 5 · p. 22

H. Task-Level Experimental Results

Task-level optimization redirects initially correct decisions across Jev and three additional systems, though rates and eligible populations vary by task. The results also compare one-shot controls and matched own-target populations.

Jev

  • Jev targeted flips occur on all seven datasets, including 32/80 legal decisions and 60/70 tool-routing decisions; BFCL V4 optimization reaches 85.7% versus 40.0% for target-aware one-shot.MMLU-Pro and LAR-ECHR have lower rates than other tasks, but successful cases there are not limited to close ties.Source: H. Task-Level Experimental Results · p. 22, Table 6 · p. 22

OpenSourceJev

Von

  • Von reaches 100% TFR on 16 eligible SuperGPQA decisions, while larger MuSR and SATA populations also show frequent redirection and contribute to its pooled 73.2% result.The small SuperGPQA population should not be read as perfect success on that dataset overall.Source: H. Task-Level Experimental Results · p. 23, Table 8 · p. 23

Plain Qwen

  • Plain Qwen is most susceptible on SATA and least on LAR-ECHR; its pooled TFR exceeds both one-shot controls, with many successful contexts reaching the 0.7 threshold.Comparison with OpenSourceJev involves different eligible populations and interfaces.Source: H. Task-Level Experimental Results · p. 23

Matched own-target comparisons

  • The other system has higher own-target TFR in all three matched aggregate comparisons with Jev, using populations eligible and initially correct for both systems.For SATA, systems may select different absent-option units, so these comparisons do not replace exact-unit transfer matching.Source: H. Task-Level Experimental Results · p. 23

I. Budget Dynamics and Probability Feedback

Jev’s targeted flip rate continues to rise with more accepted-call budget, while probability-based proposal allocation outperforms label-based allocation on matched decisions. The feedback comparison concerns allocation: both conditions hide numerical history from the proposer.

I. Budget Dynamics and Probability Feedback

  • 21.1 percentage points: Jev’s TFR rises from the first to the last reported accepted-call prefix, and all systems continue gaining through 64 calls.These are cumulative prefixes of recorded trajectories, not separately rerun budget conditions.Source: Table 11 · p. 24, §I · p. 24
  • Both feedback conditions hide numerical history from the proposer, so the comparison supports probability-based parent allocation rather than a separate benefit from numerical history in the prompt.The shared stopping condition may still check whether target probability reaches 0.7.Source: §I · pp. 24–25
  • On the same 140 Jev decisions, probability-only achieved a 7.1-percentage-point TFR advantage over label-only at 64 calls, with a paired 95% interval excluding zero based on 5,000 bootstrap resamples.Source: §I · p. 24, Table 12 · p. 25

J. Cross-Model Transfer Details

The section documents transfer across decision systems and the data, training, and generation procedures behind proposer-based context construction. Its historical single-addition evaluation reports improved Jev targeted flip rates for V1.

J. Cross-Model Transfer Details

  • 21.1%–31.2% targeted transfer appears in the other directed system pairs, alongside the strongest directions between OpenSourceJev and the shared-backbone plain Qwen scorer.The pattern supports a component of cross-system susceptibility alongside source-specific effects.Source: J. Cross-Model Transfer Details · p. 25

K.1. Development Data and Supervision

  • V1 learns to predict the next addition from construction trajectories, whereas V2 learns complete successful contexts selected from development runs.V1 uses accepted transitions; V2 selects distinct successful contexts and ranks them partly by whether target probability reaches 0.7.Source: §K.1 · p. 27

K.2. Training Objective and Length Filtering

  • Training weights give each item equal total mass before length filtering, and the weighted mean token loss masks input tokens while supervising completion tokens.Examples over 7,168 tokens are removed in full; the retained examples keep weights assigned before filtering.Source: §K.2 · p. 28

K.3. Adapter Training

  • Both recipes adapt the same Gemma4-12B proposer with LoRA while keeping base weights, the target model, and the checker fixed.Training and validation are disjoint by item and exclude held-out evaluation items.Source: §K.3 · p. 28

K.4. Generation Protocols

  • Under the historical single-addition protocol, V1 raises aggregate one-proposal TFR from 16.3% to 19.5% and best-of-four TFR from 26.8% to 31.7%.The historical Base differs from both the formal target-aware one-shot control and the Base in the complete-context comparison.Source: Table 18 · p. 29, §K.4 · p. 29

K.5. Held-Out Generation Results

Held-out proposer training improves targeted flips over Base in paired comparisons for both one-generation and best-of-four evaluation. The gains vary by task and recipe, and remain in a sensitivity analysis excluding evaluation items sharing underlying legal cases.

K.5. Held-Out Generation Results

  • V2 improves over Base on six of seven tasks for one generation and gains on six tasks with best-of-four, while BFCL V4 falls by one success in the latter comparison.V1 and V2 distribute gains differently: V1 is stronger on MuSR, while V2 is stronger on SuperGPQA and LAR-ECHR.Source: Table 19 · p. 29, §K.5 · pp. 29–30, Table 20 · p. 30
  • 29 additional one-generation successes and 26 additional best-of-four successes: paired comparisons support V2 improvements over Base, but do not establish V2 superiority over V1.For one generation, 53 items succeed only under V2 and 24 only under Base; for four generations, the counts are 51 and 25.Source: Table 21 · p. 30, §K.5 · p. 30
  • V2 has fewer accepted contexts than Base or V1 yet higher TFR than Base on the full 508-decision denominator; V1 has the largest best-of-four count at probability 0.7.Aggregate TFR and the high-probability threshold therefore need not rank the recipes identically.Source: Table 22 · p. 30, §K.5 · p. 30
  • After excluding 15 eligible evaluation items sharing underlying LAR-ECHR cases, V2 still exceeds Base by 5.7 percentage points for one generation and 5.1 points for best-of-four.The remaining population is 493 decisions; this post hoc exclusion does not make the original split case-disjoint.Source: §K.5 · p. 31

L. Context Compactness and Operational Outcomes

Selected successful contexts are often short, while the study’s scope is bounded to finite typed decision distributions and initially correct decisions. The paper connects these findings to safeguards for systems whose choices can directly drive downstream software.

L. Context Compactness and Operational Outcomes

  • Jev’s 312 selected contexts include 162 with one addition and 91 with two; across the other systems, more than half of selected successes use a single addition, with a median of 28 added words.The reported lengths describe selected successful contexts, not every attempted context or every initially correct decision.Source: Table 23 · p. 31, L. Context Compactness and Operational… · p. 31

M. Scope and Generalization

  • The study is limited to systems exposing finite, typed distributions and decisions that begin correct; broader model scales, multilingual contexts, continuous actions, and deployment streams remain settings for further examination.The implementations span hosted, open, autoregressive, and non-autoregressive systems.Source: M. Scope and Generalization · p. 32

N. Ethical Considerations

  • Because downstream software can execute these choices without another natural-language interpretation step, the authors identify provenance tracking, invariance testing, abstention for unstable probabilities, and separation of untrusted context from decision criteria as safeguards.The experiments use public benchmark tasks and bounded option sets and focus on decision redirection rather than harmful-content generation.Source: N. Ethical Considerations · p. 32

O. Reproducibility Record

The reproducibility record freezes the optimization configuration and records proposal-level execution traces, software versions, and artifact-generation checks.

O. Reproducibility Record

  • The frozen configuration specifies the seed, proposal and iteration counts, optimization settings, stopping threshold, retry policy, and hashes and revisions stored in the formal lock.The lock includes source-data and clean-evaluation checksums, target revisions, prompt and schema hashes, cache revisions, and the complete optimization configuration.Source: O. Reproducibility Record · p. 32
  • Execution traces record every proposal and its lineage, filtering and checking outcomes, target distribution, margin, and target-call index.The formal Jev components contain 37,277 logical evaluations and 53.7 million logical input tokens, reduced by caching to 29,532 unique requests and 42.3 million unique input tokens.Source: O. Reproducibility Record · p. 32
  • The artifact pipeline locks software and local Qwen target versions and regenerates manuscript macros and figures from frozen summaries and unit-level records.The asset builder verifies headline counts, clean coverage, adaptation totals, compactness statistics, and the Figure 1 source-item hash.Source: O. Reproducibility Record · p. 32
Loading 2609.30243v1…