PaperlayerView exact revision on arXiv

Source-linked AI summary

Asking What a Director Sees Can Change What Vision-Language Models Choose

Jiaxuan Guo, Xingyu Ren, Kejia Zhang, Jingxin Yang, Youran Sun, Haizhao Yang

arXiv:2610.29318v1cs.CLcs.CV

TL;DR

Report-before-action evaluations often treat a correct perspective report beside a wrong action as evidence that a model failed to use the perspective, without checking whether asking for the report changes the action. In a rendered Director Task, the paper finds that the visibility question raises hidden-block choices and proposes paired prompts as a baseline for measuring that effect.

  • Problem

    Report-before-action tests do not show what the action would have been without the report question, leaving a correct report beside a wrong action unable to distinguish model failure from a test effect.

    Source: §1 · pp. 1–2

  • Method

    The paper compares byte-matched prompts with and without a visibility question in a rendered Director Task, then examines reasoning traces and prompt rewordings.

    Source: §1 · p. 2, §5 · p. 7, §7 · p. 10

  • Results

    Asking which blocks the director can see raises hidden-block choices by 12 to 88 points on six GPT models and 17 points on Qwen3.8-27B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Paired prompts that differ only in the report question provide a baseline for tests that score an action after eliciting a report.

    Source: §7 · p. 10

    The tests use one rendered scene family with digits as block labels, one referring expression, and English prompts.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Vision-language models are now routinely tested on theory of mind: whether they can work out what another person sees, knows, and means. A common design asks the model to report that perspective and then to act for the person, and reads a correct report beside a wrong action as a failure to use the perspective. That reading rests on a quiet assumption: that asking for the report leaves the action unchanged. We show that it does not. In a rendered Director Task, a partition hides the largest block from a director who asks for "the largest block". Asking first which blocks the director can see raises the share of replies that choose this hidden block by 12 to 88 percentage points on six GPT models and by 17 on Qwen3.8-27B, even though the models mostly report correctly which blocks the director sees. Reasoning traces and prompt rewordings suggest why: the question shifts how the request is read, from the director's view toward the model's own. When the prompt states what the director knows and refers to, fewer than 1% of replies pair a correct report with a hidden-block choice. Any evaluation that elicits a perspective report before the action it scores needs a baseline without the question, and we deliver one: paired prompts that differ only in that question, a design any perspective-taking benchmark can adopt.

1 Introduction

The paper questions whether a correct perspective report beside a wrong action proves a model failed to use another person’s perspective. In a rendered Director Task, it tests whether asking for that report changes the action and proposes paired prompts as a baseline.

1 Introduction

  • The standard report-before-action verdict assumes that asking for a perspective report leaves the scored action unchanged, an assumption the protocol does not check.A test that asks for both report and action does not reveal what the action would have been without the report question.Source: §1 · pp. 1–2
  • In the rendered task, a partition hides the model-view largest block while the director requests “the largest block”; paired prompts differ in only one part.The second question asks either which block the director means (MEAN) or which block the model will move (MOVE).Source: §1 · p. 2
  • Asking which blocks the director can see raises hidden-block choices by 12 to 88 points on six GPT models and 17 points on Qwen3.8-27B.Visibility reports remain mostly correct, so the action can change without a corresponding loss in reported perspective accuracy.Source: §1 · p. 2
  • Reasoning traces and request rewordings suggest the visibility question shifts how models read “the largest block” toward their own view.Qwen3.8-27B’s MOVE traces take the largest block in its own view in 39% rather than 12% of traces when the question is asked.Source: §1 · p. 2
  • Stating what the director knows and refers to leaves fewer than 1% of replies pairing a correct report with a hidden-block choice.The paper argues that a correct report beside a wrong action alone does not establish failure to use the perspective; it proposes paired prompts with and without the report question as a baseline.Source: §1 · pp. 2–3

2 Related Work

Related benchmarks test perspective reports, beliefs, and actions in varied formats, but often elicit the report and action together. Prior work also shows that wording can move scores; this paper isolates the effect of asking for the report.

2 Related Work

  • Theory-of-mind benchmarks often score behavior predictions, while related tests separately assess awareness, belief, perception, or action (Riemer et al., 2025; Zhou et al., 2023).Some agent benchmarks elicit a belief or mapping alongside action (Juneja et al., 2026; Goel et al., 2026; Wang et al., 2026a).Source: §2 · p. 3
  • In the Director Task, what a speaker can see does not settle what their words refer to.The paper’s later constraint therefore states both what the director knows and what the director refers to.Source: §2 · p. 3
  • Prompt wording and format can move theory-of-mind scores, but prior studies leave aside the report question shared by report-before-action tests.A prior Director Task design varied that question and found asking reduced errors.Source: §2 · p. 3

3 Task and Design

The study uses rendered scenes and byte-matched prompt comparisons to test how a visibility question affects block choices. Its measures distinguish visibility-report accuracy from choosing the block hidden from the director.

3 Task and Design

  • Each scene has three or four numbered cubes, and the partition hides the largest cube from the model’s viewpoint in the 160-scene main set.Digits are assigned independently of size, and a rotated set varies whether the hidden block is largest, middle, or smallest.Source: §3 · p. 3
  • The base prompt asks which blocks the director can see, then asks either what “the largest block” means (MEAN) or which block the model will move (MOVE).Prompt variants change one part while leaving the rest identical byte for byte.Source: §3 · pp. 3–4, Table 1 · p. 4
  • Each comparison uses conditions sampled together in one batch, with the base prompt included in every batch.Base-prompt hidden-block choice rates vary by up to 8.1 points across MOVE batches, so comparisons are made within a batch.Source: §3 · p. 4
  • Visibility accuracy measures correct reports of exactly which blocks the director sees, while hidden-block choice rate counts replies choosing the hidden block.The latter can compare conditions with and without a visibility question; a joint rate captures a hidden-block choice alongside a correct report.Source: §3 · p. 4
  • The study tests ten models, with each condition asked five times per scene and differences reported in percentage points with 95% intervals.A change is called an increase or decrease only when its interval excludes zero and survives Benjamini–Hochberg correction.Source: §3 · pp. 4–5

4 The Visibility Question Raises Hidden-Block Choices under MOVE

Under MOVE, asking the visibility question raises hidden-block choices, while visibility accuracy changes little between MEAN and MOVE. Controls suggest the effect is not explained simply by adding a question or stating the visible blocks.

4 The Visibility Question Raises Hidden-Block Choices under MOVE

  • Asking the visibility question raises MOVE hidden-block choices by 12 to 88 points on six GPT models and 17 points on Qwen3.8-27B.The increase persists in scenes where the hidden block does not carry the largest digit: 12 to 87 points on six GPT models and 16 on Qwen3.8-27B.Source: §4 · pp. 5–6
  • A block-count question changes MOVE hidden-block choices by at most 5.4 points on five GPT models, versus 12 to 88 points for the visibility question.Neither question changes the rate on grok-4.6; the block-count question lowers it by 15.3 points on Qwen3.8-27B.Source: §4 · p. 6
  • The visibility question changes hidden-block choices while visibility accuracy remains within ±4.4 points on the main set and ±5.8 on the rotated set.The section treats changes within ±10 points as unchanged for visibility accuracy.Source: §4 · p. 6
  • Stating which blocks the director sees raises hidden-block choices by 28 to 87 points less than asking the visibility question across four tested models.The statement itself raises choices only on gpt-6-sol, by 20.8 points, and lowers them on Qwen3.8-27B.Source: §4 · p. 6
  • The visibility question also raises hidden-block choices when asked after the request or in a first turn, so the effect does not require a report before the choice in the same reply.For three GPT models, the increases are 19 to 63 points after the request and 28 to 95 points when the question comes in a first turn.Source: §4 · p. 6

5 How the Request to Move Is Read

The hidden-block choices vary with how the request is phrased, and Qwen3.8-27B’s traces suggest the visibility question can shift its reading from the director’s view to its own.

5 How the Request to Move Is Read

  • An automatic coder assigns traces a block-reading rule, whether the director is taken to see the chosen block, and whether the trace determines which blocks the director can see.Two people unaware of the hypotheses independently coded 36 sampled traces; their reading codes agreed with the automatic coder on 36 and 34 traces.Source: §5 · p. 7
  • With the visibility question, 39% of Qwen3.8-27B’s MOVE traces use the own-view reading, versus 12% without it and one trace in 800 under MEAN.Figure 5 plots trace-reading categories and whether the trace takes the director to see the chosen block.Source: §5 · p. 7, Figure 5 · p. 8
  • 318 of 334 Qwen3.8-27B hidden-block choices with the question came with traces taking the director not to see the chosen block.Every MOVE trace with the question worked out which blocks the director could see; a second coding pass gave the same code for those choices.Source: §5 · pp. 7–8
  • For the six GPT models, rewording MOVE’s second line as “Which number does the director want you to move?” lowers hidden-block choices by 12 to 76 points.Source: §5 · p. 8
  • Under the visibility question, “the block that you will move” raises hidden-block choices by 59 to 80 points, while “want you to move” raises them by at most 40.1 points.Under base MOVE in the same batch, the increase is 49 to 88 points.Source: §5 · p. 8, Figure 6 · p. 8
  • Asking what the director knows reduces hidden-block choices on all six GPT models, but three still choose the hidden block in 36 to 47% of replies whose reports leave it out of the director’s knowledge.For the other three models, those choices occur in at most 5% of such replies, so the test does not separate a knowledge-belief account from the proposed explanation.Source: §5 · pp. 8–9

6 Stating What the Director Knows and Refers To

Stating what the director knows and what the director’s words refer to sharply reduces hidden-block choices and correct-report/hidden-choice pairs, though each constraint clause alone leaves some such pairs.

6 Stating What the Director Knows and Refers To

  • Below 1%: with the combined constraint, the joint rate is below 1% on every model in every batch, versus 11 to 83% for the six GPT models under the base prompt in the same batch.The constraint also keeps the joint rate below 1% when scene digits are rearranged; the largest rate there is 0.8% on gpt-6-luna.Source: §6 · p. 9
  • At most 7.9%: the six GPT models and Qwen3.8-27B choose the hidden block under the constraint, whatever their reports say.Muse Glimmer-30B chooses it in 18.1% of replies, all with reports listing the hidden block as visible.Source: §6 · p. 9
  • 53 to 96% of the six GPT models’ constrained MOVE replies choose the director-view block; 77 to 93% of their other choices are the largest-digit visible block.These other choices remain among blocks the director can see.Source: §6 · p. 9
  • −0.9 to +3.0 points: under the constraint, the visibility question barely changes hidden-block choice rates for the three tested GPT models, versus +38 to +81 points under the base prompt.No constrained-prompt interval reaches above +6.2 points.Source: §6 · p. 9
  • Either constraint clause alone leaves joint correct-report/hidden-choice rates as high as 16.3% for the knowledge clause and 10.9% for the reference clause.Padding alone lowers gpt-6-luna’s joint rate by 13.3 points, while the interval for that change includes zero on gpt-5.6-luna.Source: §6 · p. 9

7 Discussion

The discussion argues that perspective reports can reshape the action a test scores, and recommends measuring that effect with paired prompts. It presents explicit constraints and reasoning-trace checks as ways to interpret choices and distinguish prompt effects from unresolved perspective-taking failures.

7 Discussion

  • The visibility question can make MOVE read from the model’s own view, while MEAN asks again about the director and shows little of this reading.Qwen3.8-27B’s traces state the own-view reading, and the six GPT models’ choices under two rewordings fit it.Source: §7 · pp. 9–10
  • The prior Director Task design found more correct moves after asking which blocks the director saw, whereas this design found fewer.The studies differ in labels and answers, scenes, and prompt wording, so the data do not identify which difference sets the direction.Source: §7 · p. 10
  • Paired prompts with and without the report question directly measure whether asking changes the scored action.The discussion argues that this effect can raise or lower measured behavior, so it cannot be dismissed as noise.Source: §7 · p. 10
  • The proposed toolkit pairs a baseline design with a constraint recipe and a reading check based on coding the rule behind a choice.The constraint recipe states what the other person knows and refers to; the reading check is for settings where reasoning traces are available.Source: §7 · p. 10
  • Without the missing baseline, a correct report beside a wrong action leaves open whether the failure reflects the model or the question.With the comparison, remaining failures cannot be attributed to asking, while failures that disappear were effects of asking.Source: §7 · p. 10
  • Perspective-taking evaluation can separate what a model knows about another person’s view from what the prompt made it do.Source: §7 · p. 10

8 Conclusion

The conclusion shows that asking which blocks the director sees can change a model’s subsequent choice; the limitations bound the findings to the tested models, prompts, and settings.

8 Conclusion

  • Asking which blocks the director sees raises hidden-block choices by 12 to 88 points on six GPT models and by 17 on Qwen3.8-27B, despite mostly correct reports.When the prompt states what the director knows and refers to, fewer than 1% of replies pair a correct report with a hidden-block choice.Source: §8 · p. 10

Limitations

  • The study uses one rendered paradigm, one referring expression, and English prompts.The prompt-variant materials include conditions differing in whether the visibility question is asked and how the director’s knowledge or references are stated.Source: Limitations · p. 10
  • Measurements cover ten models at medium reasoning effort; the increase appears on six GPT models and Qwen3.8-27B, but is not detected on grok-4.6 or Muse Glimmer-30B.Providers may update the closed models.Source: Limitations · p. 10
  • Reasoning-trace analysis applies only to two open-weight models, and a returned trace need not be the process that produced the reply.The passage cites Turpin et al. (2023), Lanham et al. (2023), and Chen et al. (2025) on this caveat.Source: Limitations · p. 10
  • The condition that states the director’s visible blocks instead of asking for them covers four models and provides no report in the reply.Source: Limitations · p. 10
  • The study tests models; whether people change their choices when asked the visibility question remains open.Source: Limitations · p. 10

A Prompts, Conditions and Models

  • Prompt variants adjust the second question and add knowledge or reference constraints; the two-turn version carries the first turn’s message and reply into the second.The visibility report comes from the first turn and the choice from the second.Source: A Prompts, Conditions and Models · p. 14
  • Closed-model calls used provider-default temperature and medium reasoning effort, while the supplement includes stimuli, answer keys, replies, coded traces, and coder sheets.Replies were scored from the last matching JSON object after removing Markdown code fences.Source: A Prompts, Conditions and Models · p. 14

B Stimuli and Answer Keys

The appendix describes the rendered scenes and answer keys, including checks on apparent size and digit arrangements.

B Stimuli and Answer Keys

  • Each scene contains three or four differently sized cubes, and an independent depth-buffer verifier checks the director’s visibility against the scene generator.The main set has 160 scenes; digits are drawn without replacement and independently of size.Source: B Stimuli and Answer Keys · p. 14
  • In 40 of 160 main-set scenes the hidden block has the largest digit, while in 95 a visible block smaller than the director-view block has a larger digit.Source: B Stimuli and Answer Keys · p. 14
  • The rotated set has 160 three-block scenes in which the hidden block is largest, middle, or smallest in 54, 53, and 53 scenes, respectively.Source: B Stimuli and Answer Keys · p. 14
  • 19 of 160 main-set scenes made a visible block other than the director-view block look largest from the director’s side; excluding these scenes left the reported change intervals excluding zero on the same models, with one addition.For gpt-5.6-luna under MEAN, the change became −1.8 [−3.6, −0.1] points.Source: B Stimuli and Answer Keys · p. 14
  • Below 1% of replies in every batch pair a correct report with a hidden-block choice under the combined constraint.Source: B Stimuli and Answer Keys · p. 15
  • The rearranged-digit copy changes both whether a visible smaller block carries a larger digit and whether the hidden block carries the largest digit; redrawn with original digits, scenes match pixel for pixel.Source: B Stimuli and Answer Keys · p. 15

C Statistics

The appendix specifies how paired condition differences and uncertainty intervals are estimated, and how comparison families are corrected.

C Statistics

  • Comparisons use answered replies, with scenes as clusters, and generalized estimating equations estimate paired condition shares and their difference.The model uses a logit link; the difference receives a delta-method t interval on G − 2 degrees of freedom.Source: C Statistics · p. 15
  • When a condition has no event or clusters show identical differences, the procedure uses cluster-based t intervals or bounded intervals that account for within-cluster correlation.A single share instead receives a percentile interval over 2,000 scene resamples.Source: C Statistics · pp. 15–16
  • The 10-point margin is compared with changes reported as 30 to 33 points in SimpleToM and 47 points in the closest prior Director Task design.The accuracy differences tested against the margin are outside the listed comparison families.Source: C Statistics · p. 16
  • Benjamini–Hochberg correction uses a 0.05 false discovery rate across 72, 42, and 39 comparisons in Sections 4, 5, and 6, respectively.Two Table 2 estimates have intervals excluding zero but adjusted p = 0.058, above 0.05.Source: C Statistics · p. 16

D Estimates and Counts

The appendix maps figures and sections to tables containing estimates or counts.

D Estimates and Counts

E Coding the Reasoning Traces

The codebook classifies reasoning traces by the reading behind the chosen block, the director’s perceived visibility of that block, and whether the trace works out what the director can see. The authors checked the coding against known-code traces and independent human coding.

E Coding the Reasoning Traces

  • The codebook distinguishes five readings behind a block choice and separately records the director’s perceived visibility of the choice and whether the trace works out visibility.Readings include director-view, own-view, prior-knowledge, largest-digit, and no identifiable rule; each code is paired with a short trace quote.Source: E Coding the Reasoning Traces · p. 16
  • The coder gave intended codes on all 30 known-code test traces, and the two codes matched on all 30 real items coded again.The coder saw only neutral identifiers, replies, and reasoning—not the model, condition, or scene.Source: E Coding the Reasoning Traces · pp. 16–17
  • On a second coding pass without the report, all 334 choices with the visibility question kept their codes, as did 195 of 199 choices without it (Cohen’s κ = 0.96).The coder gave the intended code on all 32 test traces written to a known code.Source: E Coding the Reasoning Traces · p. 17
  • Two independent human coders matched the automatic reading codes on 36 of 36 and 34 of 36 traces, respectively; they agreed with each other on 34 of 36.Whether the trace took the director to see the chosen block matched on all 36 traces in every coder pair.Source: E Coding the Reasoning Traces · pp. 17–18
Loading 2610.29318v1…