Position, Role or Label? A Controlled Study of Instruction Conflicts in LLM Agents

Approach

• WHOWINS holds instruction text fixed while varying delivery across five seats and seven experiments covering conflict size, ordering, roles, labels, message boundaries, context, agreement, load, and priority tiers.The benchmark comprises 1,080 scenarios across five task types and uses code-scored replies.Source: §1 · pp. 1–2

Results

• 72.42 to 94.66% of decided replies follow the first conflicting line within one message; across messages, outcomes depend on model family and policy placement.The study tests 14 open-weight models and checks main findings on held-out scenarios.Source: §1 · pp. 1–2
• A system-first operator policy loses most conflicts with later lines in Qwen, wins most when acting on gpt-oss, and underperforms the same labelled-user text in tool and file-write tasks.The latter comparison holds across all 26 model-modes.Source: §1 · p. 2
• Pairwise seat rankings alone do not determine larger conflicts; forecasts improve in every model-mode once they account for where the task line appears.By the authors’ keyword patterns, at most 2.92% of decided pair replies flag or hedge the conflict.Source: §1 · p. 2

Instruction hierarchies

• Most hierarchy benchmarks treat each source as one unit and test one pair of levels, while WHOWINS separates position, role, and label and tests conflicts among two, three, or five seats.The cited work includes IHEval, Control Illusion, IH-Benchmark, and ManyIH.Source: §2 · p. 2, Table 1 · p. 3

Role, label and position cues

• Studies of role, label, and position cues often move those cues together; WHOWINS sends the same instruction in its real role or as labelled user text, first or last.Related work includes “CEO requires” framing (Geng et al., 2026), role probes (Ye et al., 2026; Leong, 2026), and prompt-position studies (Wang et al., 2025).Source: §2 · p. 3

Position, tools and memory; pairs and larger sets

• Prior benchmarks place conflicting constraints in one user prompt, while WHOWINS also tests order across messages, tool outputs, and saved memory notes.The paper additionally tests whether pairwise results forecast three- and five-way conflicts.Source: §2 · p. 3

3.1 Scenarios and Scoring

• 1,080 scenarios cover five task types, with up to five mutually exclusive instructions; code checkers and format classifiers score replies without an LLM judge.The scenarios are screened for exclusivity and option-revealing wording, and each is checked for a working scorer.Source: §3.1 · pp. 3–4

3.2 Five Seats

• Five seats represent operator, repository configuration, user, tool result, and saved memory, each delivered through its specified message and label arrangement.The task arrives as a user message; some experiments move it among the other lines.Source: §3.2 · p. 4

3.3 Seven Experiments

• Seven experiments vary conflict size and order, role versus label, message boundaries, realistic context, agreement, instruction load, and declared priority tiers while keeping text fixed.The Operator test compares the real system/developer role with the same labelled text in a user message; Qwen 3.5 cannot use a late system message.Source: §3.3 · p. 4, Table 2 · p. 5

3.4 Models and Measures

• 14 open-weight models generate 1,887,356 scored replies; measurements use decided replies, scenario-resampled bootstrap intervals, and held-out checks of identified patterns.Pair-trained choice models forecast triples and five-way conflicts using position, task-line location, or added seat strengths; main contrasts held on both scenario halves in at least 25 of 26 model-modes.Source: §3.4 · pp. 4–5

4.1 Position and Message Boundaries

• 72.42 to 94.66% of decided replies follow the first line within one message, while its share across two messages falls by 19.96 to 60.68%.Across two messages, the first line keeps 22.95 to 62.65%; the later message wins in 16 of 26 model-modes.Source: Figure 2 · p. 5, §4.1 · p. 5

4.2 Order Against the Operator Line

• The later line wins 69.36 to 89.05% on Qwen 3, 59.77 to 80.52% on Qwen 3.5, and 51.37 to 57.86% on gpt-oss; operator-first wins are much higher on gpt-oss.The operator line wins 6.07 to 17.35% of conflicts on Qwen 3, 11.66 to 41.50% on Qwen 3.5, and 62.94% and 75.73% on gpt-oss.Source: §4.2 · pp. 5–6, Table 3 · p. 6

4.3 Chat Role Against Text Label

• In tool and file-write tasks, sending the operator line first in the real system message lowers its wins versus the same labelled user text by 6.41 to 48.75% in every model-mode.Across all task types, the real role lowers operator wins in every Qwen mode, while it helps gpt-oss in format, ordering, and refusal tasks.Source: §4.3 · pp. 6–7
• In format tasks, the role-related drop appears in 0 of 26 model-modes; sending policy last raises operator wins on Qwen 3 by 19.79 to 48.22%.In the realistic Context set, first-position operator wins fall by 15.12 to 29.77% on gpt-oss, while Qwen changes go both ways.Source: §4.3 · p. 7

4.4 Forecasting Larger Conflicts from Pairs

• Task-aware pair fits beat position fits in all 26 modes by 0.012 to 0.113 nats per reply; seat strengths help 10/14 Qwen 3.5 modes and both gpt-oss models, but 3/10 Qwen 3.On Qwen 3, a later line wins 16.80 to 29.29% less when both lines precede the task line than when the task comes first or between them.Source: §4.4 · p. 7, Figure 5 · p. 8

4.5 Thinking, Tiers, Flagging and Load

• Thinking raises declared-tier following across Qwen sizes, by up to +25.83% on Qwen 3.5; it raises operator wins at 4/5 Qwen 3 sizes but lowers them at 5/7 Qwen 3.5 sizes.For Qwen3.5-27B, thinking-on tier following is 98.51% of decided replies but 87.08% of usable replies.Source: §4.5 · pp. 7–8
• At most 2.92% of decided pair replies flag or hedge a conflict, and instruction count shows no consistent effect across families.The flagging result uses the paper's keyword patterns.Source: §4.5 · p. 8

4.6 Implications

• The authors recommend testing policy placement across messages and tasks, locating the task line in forecasts, and testing conflicts across multiple scenarios and replies.They also caution against relying on declared tiers or conflict warnings.Source: §4.6 · p. 8

5 Conclusion

• Across 14 open-weight models and 1,887,356 replies, the winner depends on position and message boundaries as well as chat role.In every model, the first of two lines in one message wins most decided replies, while on Qwen the later line often wins across messages.Source: §5 · p. 8
• A policy sent first in the system message wins most conflicts with later lines only on gpt-oss, and loses more often than the same text as a labelled user message in tool and file-write tasks.The latter comparison holds in every model-mode.Source: §5 · p. 9
• Pair forecasts of larger conflicts improve in all model-modes once they model where the task line is in the prompt.The paper concludes that hierarchy benchmarks should vary position and message boundaries as well as role.Source: §5 · p. 9

Limitations

• The conclusions cover three open-weight model families, not hosted models, whose system prompts and input handling are not visible to users.The benchmark can be adapted to other chat APIs.Source: Limitations · p. 9
• Most instructions are short and synthetic; all text is English, each seat has one label wording, and each trial lasts at most six tool turns.Only 50 scenarios embed instructions in realistic text.Source: Limitations · p. 9
• Contrasts between gpt-oss and Qwen do not isolate one cause because training, chat format, serving settings, and family size differ.The gpt-oss family has two models.Source: Limitations · p. 9
• Rates are shares of decided replies, and in tool tasks the role contrast also shifts policy position relative to tool definitions.Thinking-on and thinking-off comparisons use different sampling settings, while conflict flagging relies on fixed keyword patterns.Source: Limitations · p. 9

Ethical Considerations

• The study involves no human subjects or personal data; scenarios use ordinary software tasks, and the refusal task concerns a harmless gated action in a sandbox.Models are used under their published licences, and authors reviewed every batch.Source: Ethical Considerations · p. 9
• Knowing where instructions are most likely to be followed could help attackers place injected lines, while also showing defenders where policies hold best.The release includes no attack or injection payloads beyond the benchmark’s plain conflicting instructions.Source: Ethical Considerations · p. 9

Appendix

• Appendices A–G describe the benchmark and its execution, H–O provide Section 4 results, and P–Q cover release details and related work.Source: Appendix · p. 13

A The Benchmark in Detail

• The benchmark covers five task types—tool, file-write, format, ordering, and refusal—with options distinguished by the first relevant tool call or write, the written format, or the refusal action.Tool, file-write, format, and ordering scenarios have five options; refusal has two. Scoring reads tool-call traces for tool, file-write, and ordering tasks, the trace or refusal text for refusal, and model-written text for format.Source: A The Benchmark in Detail · p. 13, Table 4 · p. 14
• Replies are classified into five outcome behaviors and separately marked as flagging, hedging, or silent; truncated or faulty replies are excluded from rates, and tool traces are read in order.Conflict marks use fixed keyword patterns over answer text, not reasoning; format tasks are scored on the last turn that is more than talk or a closing note.Source: A The Benchmark in Detail · p. 13
• Scenarios were authored in batches and reviewed against seven screens, then checked by two automatic validators, exact reference-trial rebuild tests, and format-classifier tests on 1,382 replies.The tests also checked adversarial near-misses, and all 1,200 format values were reached by at least one written reply.Source: A The Benchmark in Detail · pp. 13–14

B Seats and How Each Family Renders Them

• Seats contribute a label line plus unchanged scenario option text, while Qwen and gpt-oss render operator messages differently and reject requests their chat templates would alter or drop.Qwen 3 can place the system operator message first or last, Qwen 3.5 only accepts it first, and gpt-oss uses a developer operator message.Source: B Seats and How Each Family Renders Them · p. 14

C Prompts as Sent

• The prompts reproduce benchmark messages verbatim: seats, tasks, tools, and tool results are assembled in specified layouts, with no custom system prompt beyond the Tiers rule and no judge prompt.Ladder and Agreement trials place the task in a separate user message except in the Boundary test’s merged layout; Load trials put all instruction lines in one operator message.Source: C Prompts as Sent · p. 15, §C.1 · pp. 15, 18

C.1 Messages the Models Receive

• Qwen thinking-on runs used temperature 0.6 and top_p 0.95 with thinking enabled; gpt-oss used temperature 0.6, top_p 0.95, and high reasoning effort.The passage lists these as the fields that differ for the respective model configurations.Source: §C.1 · p. 18

C.2 Prompts Given to the Authoring Agents

• Each base scenario pairs a task with mutually exclusive instructions and a deterministic checker; source, position, and other delivery features are assigned later per trial.Refusal scenarios use two choices; the other families use five.Source: §C.2 · p. 19
• The seven authoring screens require exclusive checker values, unique value sets, balanced option length and reasonableness, honest sourcing, varied domains and difficulty, and unique seed families.For tool and path scenarios, the guide emphasizes preventing task merits from favoring one option; tool stubs should be equivalent and destinations remain within one sandbox.Source: §C.2 · pp. 20–21
• The realism set wraps 50 tool or path scenarios in four source-specific contexts, then tests each item both bare and in context.Each context has a single instruction slot, positioned away from the beginning and end, and must read naturally with any option inserted.Source: §C.2 · p. 22
• Context guidance forbids source-ranking cues, endorsements, option-steering facts, and hints about measured choices so context does not decide the conflict.The instructions specifically prohibit language about who outranks whom and facts that make one option more fitting than others.Source: §C.2 · pp. 22–23
• Realism batches were checked with a validator, and warning sentences that might steer an option choice were to be reviewed and reworded.Authors were instructed to fix every failure and rerun validation until none remained.Source: §C.2 · p. 23
• Load bundles vary conflict count while holding the source constant: every instruction is in one system message, and nonconflicting fillers keep message length fixed.The guide frames this design as testing whether accuracy falls because instructions conflict or simply because there are many of them.Source: §C.2 · p. 24
• Each bundle combines one tool scenario, one path scenario, and four compatible format scenarios, with conflict lines kept verbatim and distributed across positions.The task must request the tool step before writing an artifact, without favoring any option; format classifiers should read distinct features.Source: §C.2 · p. 24

D The Seven Experiments in Detail

• Seven deterministic experiments vary conflicting seats, order, role versus label, message boundaries, realistic context, agreement, instruction load, and declared priority tiers.The designs include free pairs and larger sets, paired layouts, and scenarios in which instructions agree or conflict.Source: D The Seven Experiments in Detail · p. 26

E Models, Serving and Decoding

• Qwen models use recommended sampling settings by thinking mode, while gpt-oss uses high reasoning effort; only answers and tool calls are scored.Models run at fixed published-weight revisions, and each draw has a reproducible seed, though another serving stack may return different tokens.Source: E Models, Serving and Decoding · p. 27, Table 6 · p. 28, Table 7 · p. 28

F Measures and Statistics

• A reply counts as decided only when it chooses an option uniquely requested by a seat in play; rates otherwise use decided replies.The study uses scenario-clustered bootstrap intervals and corrects 2,103 tests together with Benjamini–Hochberg and Holm procedures.Source: F Measures and Statistics · p. 28
• Pair forecasts include position, task-aware, seat-strength, and empirical-reference models, with log score and top-1 accuracy used for evaluation.The task-aware model weights line rank before and after the task differently and includes a term for coming after the task; empirical-reference predictions use the scored replies themselves.Source: F Measures and Statistics · pp. 28–29

G Why Pairs Cannot Settle Larger Conflicts

• Pairwise win rates do not encode how preferences are correlated, although an assumption such as Luce’s choice axiom lets pairs determine larger-set choices.The winner of a larger set depends on the correlations among preferences.Source: G Why Pairs Cannot Settle Larger… · p. 29
• The same pairwise data can leave triple outcomes uncertain: the best pair-only forecast remains 0.066 nats per reply short of the truth in both constructed worlds.A Luce model fitted to the pairs is 0.112 nats short in both, and observing more pairs does not close either gap.Source: G Why Pairs Cannot Settle Larger… · p. 29, Table 8 · p. 29
• WHOWINS collects triples and five-way sets to score pair-only forecasts; task-aware forecasts recover different shares of the gap across model families.From pairs alone, the task-aware fit recovers 60.08 to 75.81% on Qwen 3 and 5.67 to 31.51% on gpt-oss.Source: G Why Pairs Cannot Settle Larger… · p. 29

H Full Results

• The full-results section defines Table 3’s measures for later wins, seat comparisons, operator layouts, task types, and seat-strength gains.Its difference columns use paired scenarios, and the table’s shading scales with the magnitude of each difference.Source: H Full Results · pp. 29–30

H.1 Findings at a Glance

• The first-listed line wins inside one message and loses part of its lead across messages in all 26 model-modes on both scenario halves.The task-aware fit also beats the position fit in every model-mode on both halves.Source: §H.1 · p. 30, Table 9 · p. 31

H.2 The Ladder

• The user line beats tool results by point estimate in every model-mode, with the interval above one half in 20 of 26.The user and memory-note comparison is near even, while a memory note beats a tool result on gpt-oss and several Qwen 3.5 modes.Source: §H.2 · pp. 30, 33, Figure 7 · p. 30
• Across messages, later lines win in nearly all Qwen modes, while the second line within one system message loses in every model-mode.The recency advantage weakens when both lines precede the task line; the operator wins more than half of pinned pairs only in two gpt-oss modes.Source: §H.2 · pp. 30, 33, Table 10 · p. 31, Table 11 · p. 32

H.3 The Operator Test

• In tool-call and file-write tasks, the real system or developer role loses to the same policy text as a labelled user message in all 26 model-modes.Sending the real operator last improves its wins on Qwen 3 and gpt-oss, but Qwen 3.5 has no late-system layout.Source: §H.3 · p. 33, Table 12 · p. 34, Table 13 · p. 35

H.4 The Boundary Test

• The first-listed user or memory line wins 72.42 to 94.66% inside one message, but only 22.95 to 62.65% across separate messages.The paired drop is above zero in all 26 model-modes, and the first line loses in most Qwen 3 modes while gpt-oss stays near even.Source: Table 15 · p. 37, §H.4 · p. 38

I Realistic Context

• The context comparison changes text length and surrounding sentences as well as realism, so its results describe the written texts rather than realism alone.Each item is paired across bare and contextual versions.Source: I Realistic Context · p. 38, Table 16 · p. 39
• In written context, the operator sent first loses 15.12 to 29.77% on gpt-oss, while Qwen 3.5 results vary by size and thinking mode.On Qwen3.5-122B-A10B, the operator rate falls from 47.39% bare to 15.38% in context.Source: I Realistic Context · p. 38
• The later line wins above 50% in context in 25 of 26 model-modes, while context changes its rate by −10.05 to +24.56%.The direction varies overall, rising in 10 and falling in 6 of 26 model-modes.Source: I Realistic Context · p. 38

• The task-aware fit beats the position fit in 26 of 26 model-modes by 0.012 to 0.113 nats per reply, with intervals above zero in each.The gain holds on both held-out scenario halves.Source: J Forecasting Larger Conflicts in Full · p. 40
• Adding seat strengths beats the position fit in 10 of 14 Qwen 3.5 and 2 of 2 gpt-oss model-modes, by up to +0.139 nats per reply.On Qwen 3, the gain ranges from −0.020 to +0.051 nats; held-out results on Qwen 3.5 range from −0.007 to +0.142 and −0.019 to +0.136 nats in the two directions.Source: J Forecasting Larger Conflicts in Full · p. 40, Table 18 · p. 42
• On Qwen 3, the position fit recovers 49.86 to 61.77% of the gap, versus 4.69 to 49.88% on Qwen 3.5 and 0.90 to 26.71% on gpt-oss.Its winner-ranking accuracy on Qwen 3 is 47.60 to 69.70%, close to the reference’s 50.57 to 69.70%.Source: J Forecasting Larger Conflicts in Full · p. 40
• The reference identifies the winner in only 44.44 to 69.70% of replies, and outcomes still vary across scenarios, task-line places, and draws.No forecast based only on the set and order can remove that variation.Source: J Forecasting Larger Conflicts in Full · p. 40

• In Agreement trials, the seats fit predicts the shared option’s win rate from the forecast chances of seats in the larger camp; two-against-two sets are excluded.The fit is learned on Ladder pairs, and no Agreement replies enter fitting.Source: K When Seats Agree · p. 43
• The shared option wins more often than predicted in 9 of 14 Qwen 3.5 model-modes, by at most +5.41%; the gap ranges from −2.68 to +6.85% on Qwen 3.On gpt-oss, the observed share is below prediction by −1.09 to −0.56%.Source: K When Seats Agree · p. 43, Table 19 · p. 44
• Across model-modes, the observed-minus-predicted shared-option share ranges from −2.68 to +6.85%, above zero in 13 and below zero in 2.Across usable replies, the observed share falls below prediction in most model-modes, so Agreement trials end with no single winner more often than pairs predict.Source: K When Seats Agree · p. 43, Table 19 · p. 44

• With thinking on, the declared tier rule is followed more often at 9 of 12 Qwen sizes, by up to +25.83%.The gain covers every Qwen 3 size from 4B up and every Qwen 3.5 size from 2B up; Qwen3.5-122B-A10B rises from 75.67% to 95.44%.Source: L Thinking On and Off · p. 45
• Counting every reply and treating cut-offs or unusable replies as failures, thinking still raises tier following at 6 of 7 Qwen 3.5 sizes, by +1.48 to +15.43%.Qwen 3.5 cut-offs rise at all seven sizes; at Qwen3-14B, strict counting turns the gain to −4.85% because thinking adds replies choosing no option.Source: L Thinking On and Off · p. 45
• Thinking raises operator wins in real system-message conflicts at 4 of 5 Qwen 3 sizes but lowers them at 5 of 7 Qwen 3.5 sizes.The all-reply count widens the Qwen 3.5 drop to 6 of 7 sizes.Source: L Thinking On and Off · p. 45
• Thinking shrinks the first-line advantage inside one message at 10 of 12 Qwen sizes.For Qwen3.5-0.8B, off-menu replies rise by +5.01%; free Ladder-pair recency falls at 4 of 12 sizes and rises at none.Source: L Thinking On and Off · p. 45

M Declared Tiers and Capacity

• The lower declared tier wins in 24 of 26 model-modes, with rates of 50.78 to 98.51% on decided replies; all-usable rates fall to 24.29 to 87.08%.The two decided-reply exceptions are the smallest model of each Qwen family with thinking off.Source: M Declared Tiers and Capacity · p. 45, Table 21 · p. 46
• With the winning tier held fixed, much of the apparent increase with tier gap disappears, so the rise mostly reflects which tiers are paired.In this design, the only pair five steps apart is tier 1 against tier 6.Source: M Declared Tiers and Capacity · p. 45
• The upper bound on R(0.95) is one tier in all 26 model-modes, so no model is shown to keep even two tiers in order at 0.95 reliability.Replies that do not choose one of the two options count as failures; Qwen3.5-27B with thinking on follows the rule in 98.51% of decided replies but 87.08% of usable ones.Source: M Declared Tiers and Capacity · pp. 45, 47, Table 21 · p. 46

N Reply Outcomes and Conflict Flagging

• gpt-oss takes no action in 17.64 to 23.40% of usable replies across experiments except Load, and such replies never make the scored call.Thus, decided-only rates describe gpt-oss when it acts, while all-usable rates describe its behavior per request.Source: §N.1 · p. 47
• Off-menu answers comprise 1.56 to 7.32% of usable Ladder conflict replies, while cut-offs comprise 0.03 to 9.03% of all replies.Unresolved replies are the largest undecided outcome in every model-mode except gpt-oss-20b, where no action is larger.Source: §N.1 · p. 47

N.2 Flagging the Conflict

• 0.14 to 2.92% of decided Ladder pair replies flag or hedge the conflict, with gpt-oss flagging most.Figure 8 separates no action, unresolved, off-menu, and multiple outcomes; the rest of replies choose one seat’s option.Source: §N.2 · p. 47, Figure 8 · p. 47
• +20.03% [8.62, 33.35] on Qwen 3 and +15.65% [5.47, 26.43] on Qwen 3.5: flagged replies more often choose the earlier line than silent replies.The unpaired comparison shows an association, not causation; gpt-oss’s +7.77% [−3.92, 18.76] difference includes zero.Source: §N.2 · p. 47

N.3 Three Replies

• Three examples show a no-action reply asking for missing content, a flagged five-way conflict where the user option wins, and a thinking trace applying a declared tier rule.The first reply is not decided; the other examples choose retrain_scheduled and snake_case, respectively.Source: §N.3 · pp. 47, 50

O Load: More Instructions and More Conflicts

• At fixed length, adding conflict groups has no steady pooled cost, while adding instructions at one conflict has no common direction across model families.For eight instructions, one to three conflict groups changes chosen share by −3.98% [−12.80, 5.00] on Qwen 3, −1.03% [−6.90, 4.71] on Qwen 3.5, and −8.77% [−18.32, 0.57] on gpt-oss; all intervals span zero. At one conflict, twelve versus two instructions changes it by +9.59% [−0.37, 20.55] on Qwen 3, −17.64% [−29.15, −7.08] on Qwen 3.5, and +1.40% [−15.04, 15.52] on gpt-oss; the experiment is descriptive with wide bundle-clustered intervals.Source: O Load: More Instructions and More… · p. 50

P.1 Code, Data and Reproducibility

• The public repository contains 1,080 scenarios, trial-building code, checkers, run configurations, frozen splits, tests, and authoring guides; scored replies will be released separately.Pinned model revisions, chat templates, sampling settings, and per-request seeds are recorded to support rerunning requests.Source: §P.1 · p. 50
• Deterministic code checkers score tool-call traces and answer text without an LLM judge, and stored replies allow scores to be rebuilt without querying models again.The public code handles individual runs, but the aggregation scripts producing paper tables, figures, and pooled estimates are not in the repository.Source: §P.1 · pp. 50, 52

P.2 Installation and Usage

• The package supports scenario checks and reply scoring without a GPU; running models requires a GPU machine with the family-specific vLLM release.It includes scenarios and run configurations, while optional extras support tests, downloading weights, and counting Load-bundle tokens.Source: §P.2 · pp. 52–53

Q More Related Work

• Unlike prior studies that vary only some source features, the Operator test sends the same instruction text in its real role and as a labelled line while holding message order fixed.The related-work comparison describes prior hierarchy work as installing or enforcing a ranking, whereas WhoWins measures the ranking models already apply and how much comes from position.Source: Q More Related Work · pp. 53–55