Source-linked AI summary
Consider, Then Choose: Order-Invariant Decisions in System One Models
Jianguo Huang, Yao Long Teng, Yanchen Deng, Bo An
TL;DR
Choice answers from System One models can depend on option order, even though the same input is meant to yield the same answer. Runoff scores options individually, keeps a few candidates, and compares them across positions; on open models it changes no answers under shuffling, while retaining accuracy close to cyclic averaging with many options. Its invariance assumes identical requests produce identical probabilities, and it addresses option order rather than other decision biases.
Problem
For choice questions, System One answers can change with option order, while cyclic averaging reduces but does not eliminate that dependence.
Source: §1 · p. 1Method
Results
On two open models and five datasets, Runoff changes no answers under shuffling versus 9.3% for cyclic averaging, and with 20 options its accuracy is close to cyclic averaging.
Source: §1 · p. 2Takeaways & Limitations
Runoff’s answer is independent of option order by construction, and it can match cyclic averaging’s accuracy with many options at a fraction of its cost.
Source: §8 · p. 8Runoff’s invariance assumes identical requests yield identical model probabilities, and it addresses option order but not other biases such as preferences for certain option names.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
System One models answer typed questions in a single forward pass and promise the same answer for the same input. For choice questions, this promise fails: the answer depends on the order of the options, increasingly so as they grow in number, and averaging over their cyclic shifts reduces this dependence without removing it. Scoring each option alone, which the order cannot affect, shows that the order changes the answer mostly when the model is unsure among its top few options, even though the listwise answer looks confident. We therefore propose $\textbf{Runoff}$, which considers, then chooses: it scores each option alone to keep a few candidates, then compares them in listings where each candidate takes every position. It needs no training and can run in two forward passes, and its answer does not depend on the option order by construction. On two open models and five datasets, it changes no answer under shuffling, against 9.3% for cyclic averaging, and with 20 options it is about as accurate and three to four times faster. Wrapped around eight System One models, it removes the order dependence of the five benchmarks, while cyclic averaging changes up to 15.4%.
1 Introduction
System One models return answer distributions for typed questions, but choice answers can change when options are reordered, and existing approaches reduce rather than remove this dependence. The paper traces the changes to uncertainty among a few options and introduces Runoff, which narrows candidates before comparing them without favoring their positions.
1 Introduction
- With 20-way intent questions, shuffling changes 17% of Qwen3-8B answers, versus 5% on two-way questions, making arbitrary option order a practical source of answer changes.The order may come from a tool registry or page layout rather than a deliberate choice.Source: §1 · p. 1
- Cyclic averaging reduces but does not remove order dependence, while text reasoning gives up the single forward pass and does not reliably ensure order-independent choice answers.Cyclic averaging’s cost grows quadratically with the number of options.Source: §1 · p. 1
- Flips occur among a few options: at least one answer before or after a flip almost always ranks among the pointwise top options, and flips are more frequent when that top option is uncertain.Listwise answers look confident in both more- and less-certain groups.Source: §1 · pp. 1–2
- Runoff first retains the highest-ranked pointwise options, then compares them across cyclic shifts so each candidate takes every position, making the answer order-independent by construction.It needs no training and can run in two forward passes.Source: §1 · p. 2
- On two open models and five datasets, Runoff changes no answers under shuffling, versus 9.3% for cyclic averaging, and with 20 options is about as accurate and several times faster.Across eight System One models, it removes order dependence for models that return the same probabilities for repeated requests, with only a small residue on others.Source: §1 · p. 2
- Runoff removes order dependence across every deterministic model tested and is close in accuracy to cyclic averaging, falling behind it only on reasoning questions.On reasoning questions, the consideration set more often misses the correct answer.Source: §1 · p. 2
2 Preliminaries
The preliminaries define the option listings and decision rules used to compare order dependence. They specify order invariance as equality of the output distribution across all option orders.
2 Preliminaries
- A choice question has K options in a set O, and a listing of a subset is a sequence containing each option once, with position i showing λ(i).The question’s option order σ is a listing of all options.Source: §2 · p. 2
- Listwise renormalizes next-token probabilities for option letters A, B, … under the question’s presented order.The standard answer on order σ is denoted Pσ.Source: §2 · p. 2
- Cyclic averaging makes Listwise decisions on K rotations, placing every option at every position once, and averages their log-probabilities.Its rotations are determined by the original order σ.Source: §2 · p. 2
- A decision rule is order-invariant when its output distribution is identical for every option order σ and σ′.The Figure 1 caption defines flip rate as the share of answers that change when options are shuffled; Figure 2 describes pointwise certainty and its relation to Listwise flips.Source: §2 · p. 2, Figure 1 · p. 3, Figure 2 · p. 3
3 Motivation and Analysis
Option order changes choice answers more often as the number of options grows, and the effect is strongest when pointwise predictions are uncertain. Flips mostly occur among options ranked highest by the model.
Flip rate and the number of options
- Listwise flip rates rise from 4.7% to 17.0% on Banking20 and 5.3% to 18.7% on Newsgroups; AnyJev rates also rise, from 0.3% to 10.3% and 0.7% to 11.3%.The results cover K=2 to K=20 options; accuracy falls with Listwise.Source: §3 · p. 3
Pointwise decision
- At K=20, Qwen3-8B’s top three options hold 99.9% of pointwise probability for 83% of Banking20 and 93% of Newsgroups questions.These top options are called candidates.Source: §3 · p. 4
Where the answer moves
- In over 90% of flips, at least one answer is among the three candidates, and neither answer is a candidate in at most 10% of flips.The table reports the shares with both, one, or neither answer among the candidates for K=20.Source: §3 · p. 4, Table 1 · p. 4
Flip rate and pointwise certainty
- Flip rate exceeds a third below 0.8 pointwise top-1 probability, versus 7% at 0.999 and above, while Listwise top-1 probability barely moves from 93% to 99.5%.Source: §3 · p. 4
Certainty and the number of options
- As K grows from 2 to 20, uncertain questions rise from 7% to 24% and certain questions fall from 82% to 43%, accompanying the increase in order sensitivity.Source: §3 · p. 4
4 Method
Runoff first narrows options using order-invariant pointwise scores, then compares the candidates in cyclic shifts. Its requests to the model, and thus its output distribution, are the same for every option order.
Method
- Runoff uses two steps: Consider keeps options ranked highest by pointwise decisions, then Choose compares only those candidates.The ruled-out options cannot interfere with the choice, and the method needs no training.Source: §4 · p. 4
Consider
- Consider selects the smallest top-option set holding at least α probability and at least kmin options, or all options if no such set exists.The consideration set grows when the model’s prediction is uncertain.Source: §4 · p. 4
Choose
- Choose ranks candidates by decreasing π and evaluates every cyclic shift, so each candidate takes every position once before their log-probabilities are averaged.This avoids giving a position advantage to the top option in a single canonical listing.Source: §4 · p. 4
Method
- Runoff is order-invariant: it sends the same requests to the model and produces the same distribution for every order of the options.The stated invariance holds whatever position bias the model has; on a black-box model, remaining flips can only come from different answers to identical requests.Source: §4 · p. 5
5 Experiments
Across five decision benchmarks, Runoff removes option-order flips while keeping accuracy comparable to baselines. Results also cover closed-source and fine-tuned models, with latency advantages when option sets are large.
5.1 Setup
- The evaluation compares ten decision models on five choice-question benchmarks, reporting original-order accuracy and answer flips under one shuffle per question.Runoff uses α = 0.99 and kmin = 3.Source: §5.1 · p. 5
5.2 Main results
- Runoff changes no answers under shuffling on the open models, with 20-option accuracy of 73.8% versus 74.5% for AnyJev and 73.0% for Listwise.On Qwen3-8B and MMLU-Pro, the flip rates are 44.4% for Listwise, 24.7% for AnyJev, and 0% for Runoff.Source: §5.2 · p. 5, Table 2 · p. 6
- Runoff reduces option-order dependence more than AnyJev on all three closed-source models and is at least as accurate on average.For GPT-6 Luna, Runoff has 0% flips and 77.0% accuracy, versus 6.6% flips and 77.2% accuracy for AnyJev.Source: §5.2 · pp. 5–6, Table 2 · p. 6
- Runoff removes order dependence up to model nondeterminism and is at least as accurate as native answers on all five fine-tuned models.On Torchcast 12B, Runoff has 0% flips and 72.2% accuracy, versus 15.6% flips and 70.2% accuracy natively.Source: §5.2 · p. 6, Table 3 · p. 6
- Runoff averages 2.1 times faster than AnyJev across the five datasets, but is slower than Listwise.On Banking20 with Qwen3-8B, it takes 77 ms per record versus 308 ms for AnyJev; on JevBench, it takes 77 ms versus 67 ms.Source: §5.2 · pp. 6–7, Table 4 · p. 6
6 Discussion
Runoff also improves task completion when used in a computer-use agent. Ablations and sensitivity tests examine how its stages and consideration-set settings affect invariance and accuracy.
6 Discussion
- On 316 runnable OSWorld tasks, Runoff solves 27 versus 21 tasks with Qwen3.5-35B-A3B and 14 versus 9 with Qwen3.5-9B, compared with native decisions.The agent uses a classifier to choose actions, including on-screen items to click.Source: §6 · p. 7
- The ablation reaches 69.0% accuracy with no flips when Choose averages cyclic shifts, versus 63.3% for pointwise decisions alone and 68.3% accuracy with 15.8% flips for one input-order listing.A single canonical ranking has 67.3% accuracy with no flips; the authors say both stages are needed for invariance and accuracy.Source: §6 · p. 7, Table 5 · p. 7
- Accuracy stays between 65.5% and 65.8% for α ≥ 0.9, while lowering kmin to 1 or 2 costs 1.5 or 1.2 points as gold-answer inclusion falls to 77% or 83%, from 90%.Raising kmin to 4 or 5 gains about 0.4 points at more time; the authors describe kmin = 3 and α = 0.99 as a reasonable trade-off.Source: §6 · p. 7, Table 6 · p. 7
- Across option counts from 2 to 20, Runoff changes no answers and stays within 1.3 accuracy points of AnyJev; at 20 options, accuracy is 73.8% versus 74.5%.At 20 options, Listwise and AnyJev flip 14.4% and 8.0% of answers, up from 4.1% and 0.3% at two options.Source: §6 · pp. 7–8, Table 7 · p. 8
7 Related Work
Related work studies System One decision models and several approaches to option-order sensitivity. Runoff combines pointwise ranking with listwise comparisons across cyclic shifts of a shortlist.
System One decision models
- System One models return answer distributions in one pass, and existing implementations either train decision models or read them out from frozen language models [Sun et al., 2026].Prior systems may score options independently or read labels from one listing; Runoff makes a two-stage, order-invariant procedure explicit on frozen models.Source: §7 · p. 8
Option-order sensitivity and invariance
- Cyclic-shift and random-shuffle averaging leave some order dependence, while structural methods either remove option comparison or retain order-varying accuracy [Izacard et al., 2023; Zheng et al., 2023; Zhang et al., 2026].Averaging all K! orders removes dependence but requires K! listings [Guda et al., 2025].Source: §7 · p. 8
- Runoff ranks options with order-blind pointwise decisions, then averages listwise decisions over cyclic shifts of the top m ≪ K options.Unlike elimination methods, it uses an order-independent shortlist; unlike RoToR, its ranking comes from the model rather than option text or an external reranker [Yoon et al., 2025].Source: §7 · p. 8
8 Conclusion
The conclusion presents Runoff as an order-invariant approach for System One choice decisions. It reports comparable accuracy to cyclic averaging with many options at a fraction of its cost.
8 Conclusion
- Runoff uses pointwise decisions to shortlist options and listwise comparisons to choose among them, yielding order invariance by construction without training.The conclusion states that it matches cyclic averaging in accuracy with many options at a fraction of its cost.Source: §8 · p. 8
Limitations
Runoff’s invariance is limited to option order and assumes deterministic responses. The prompt templates implement listwise and pointwise decisions using next-token probabilities.
Limitations
- Runoff addresses option-order bias, not other decision biases, and assumes identical requests receive identical responses.Some variable-output models can therefore retain a few flips.Source: Limitations · p. 8
A Prompt templates
- Listwise decisions renormalize the next-token probabilities of the option letters after the answer prefix.The Qwen3-8B prompt lists options in the supplied order.Source: A Prompt templates · p. 10, Figure 5 · p. 11
- Pointwise decisions compare the next-token probabilities of “yes” and “no” for one option, without including other options or letters.The option-specific prompts share a common prefix, which is encoded once in the packed forward pass.Source: A Prompt templates · pp. 10–11
B Controlled shuffles
Controlled shuffles show that rearranging options outside the pointwise top candidates can change the answer even when those candidates stay in place. This supports excluding ruled-out options from Runoff’s comparison.
B Controlled shuffles
- The experiment separately permuted the top three candidates among their positions and the remaining options while holding candidates fixed.A full shuffle provided the comparison condition.Source: B Controlled shuffles · p. 11
- 14.0% and 15.3% of Qwen3-8B answers changed when only candidates or only other options were shuffled, versus 17.8% under a full shuffle.For Gemma 4 12B, the corresponding rates were 7.2%, 6.0%, and 11.0%.Source: B Controlled shuffles · p. 11
- Shuffling only the other options changed answers despite leaving each candidate’s position and letter unchanged.The rates for both controlled shuffles were higher when the pointwise top-1 probability was below 0.9.Source: B Controlled shuffles · p. 11
C Gemma 4 12B
Gemma 4 12B shows growing order dependence as options increase, with flips concentrated among high-ranked candidates and uncertain questions. The supplied sections also report fine-tuned-model results, an invariance proof, and accuracy comparisons across option counts.
C Gemma 4 12B
- At K=20, Gemma 4 12B’s Listwise flip rate reached 7.7% on Banking20 and 14.3% on Newsgroups, while AnyJev reached 3.0% and 7.3%.Its accuracy fell from about 97% at K=2 to 84% on Banking20 and 68% on Newsgroups at K=20.Source: C Gemma 4 12B · p. 12
- At K=20, the pointwise top three held 99.9% of probability for 82% of Banking20 questions and 85% of Newsgroups questions.At least one answer in a flip was among those candidates in all Banking20 flips and 93% of Newsgroups flips.Source: C Gemma 4 12B · p. 12
- Gemma’s shuffle flip rate was 26% when pointwise top-1 probability was below 0.6, versus 4% when it was at least 0.999, despite mean Listwise probability 0.95 in the uncertain group.The share of uncertain questions rose from 7% at K=2 to 29% at K=20.Source: C Gemma 4 12B · p. 12
D Fine-tuned decision models
- On MMLU-Pro, Runoff’s largest residual flip rate for Tev1 4B was 7.3%, below native decisions at 32.9% and AnyJev at 17.4%.Tev1 4B’s probabilities vary slightly between identical requests.Source: D Fine-tuned decision models · p. 13
- On MMLU-Pro, Runoff trailed AnyJev on every fine-tuned model but Laya by up to 3.6 points, while classification results were within 1.7 points.On Typed Decisions, Runoff matched or exceeded AnyJev on every model but Laya.Source: D Fine-tuned decision models · p. 13
E Proof of Proposition 1
- Under the stated deterministic-model and unique-ranking assumptions, the proof shows Runoff’s output depends on the option set, not its ordering.The proof traces this from pointwise scores and the consideration set through cyclic candidate listings.Source: E Proof of Proposition 1 · p. 13
F Number of options
- Across option counts, Runoff remains invariant, but its accuracy relative to Listwise and AnyJev varies by model and dataset.For Gemma 4 12B on Banking20, it falls behind both baselines from K=8 onward; on Newsgroups it stays within 1.0 point of AnyJev.Source: F Number of options · p. 13
G Computer-use agent
The computer-use evaluation compares a native classifier with Runoff in an otherwise shared agent setup. It uses a subset of OSWorld tasks and scores completed runs with the benchmark evaluator.
G Computer-use agent
- The evaluation runs 316 OSWorld tasks; 53 are excluded because they require unavailable proxy or Google Drive access.Thirty included tasks are defined as infeasible and count as unsolved in every setting.Source: G Computer-use agent · p. 13
- The agent uses a classifier for each action and asks up to four choice questions, including action type and target selection.Runs stop on completion, lack of useful screen content, low action confidence, no screen change, or 50 steps.Source: G Computer-use agent · pp. 13–14
- The native and Runoff settings share tasks, agent, prompts, and model weights; Runoff handles choices with at least three options using α=0.99 and kmin=3.Each setting runs once, and a task is solved when the OSWorld evaluator score is at least 0.5.Source: G Computer-use agent · p. 14