PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Is Talking to the Agent? LLMs in Multi-User 3D Virtual Environments

Mohammad Al-Ratrout, Shayla Sharmin, Roghayeh Leila Barmaki

arXiv:2610.07732v1cs.HC

TL;DR

Shared LLM agents must decide whether a speaker is addressing them and what information about co-present users to use. LOOKAWAY tests these decisions with controlled orientation and profile access, finding strong reliance on orientation during cue conflict and greater use of unrevealed profile information when both profiles are available.

  • Problem

    Shared agents must identify whether an utterance is addressed to them and whose information to use during ongoing multiparty conversation, while evidence about conflicting spatial cues remains limited.

    Source: §I · p. 1

  • Method

    LOOKAWAY varies assigned addressee, speaker orientation, and profile availability in a controlled multi-user corpus, complemented by a browser-based demonstrator using runtime head orientation.

    Source: §I · p. 1, C. Spatial-Cue Manipulation · p. 3

  • Results

    More than 85% of incongruent turns saw tested agents follow speaker orientation, while access to both profiles increased use of information not revealed in the shared conversation.

    Source: §I · p. 1, §IX · p. 8

  • Takeaways & Limitations

    Shared agents need to weigh spatial cues and control when user-specific information is appropriate to use, because more profile context improves person grounding but also increases unrevealed information in responses.

    Source: §IX · p. 8

    LOOKAWAY uses synthetic orientation, model-generated conversations, and two-person interactions; tests of other models, larger groups, natural conversations, and noisy tracking remain future work.

    Source: §VIII · p. 8

Abstract

from arXiv · show

When several people share a 3D virtual room with an LLM agent, the agent must decide not only what to say, but whether an utterance was addressed to it and, if accessible, what profile information about the others present it may use. To study both problems, we construct LookAway, a controlled corpus of 40 sessions involving 80 distinct personas and an LLM agent (1,200 turns), including ambiguous-addressee turns in which speaker orientation agrees or conflicts with the intended addressee. Across three open-weight large language models and five conditions varying which profiles the agent sees and whether it is told where each person faces (18,000 decisions), adding speaker orientation increased addressee accuracy from 56% to 99.5% when orientation was congruent, but when the speaker faced someone other than the addressee, two of the models went by where the speaker faced on more than 85% of those turns. Warning one model that orientation could be misleading reduced this only modestly. A browser-based 3D demonstrator shows the effect live: the same sentence gets an answer when the speaker faces the agent and silence when they face the other person. Providing both personas' profiles improved responses about the person being asked about, but also increased the use of profile attributes not revealed in the shared conversation, reaching 45.3% of answers for one model. More context thus improves multi-user interaction but also leads to oversharing, so shared LLM agents need mechanisms for weighing spatial cues and controlling when user-specific information enters a response.

I. INTRODUCTION

Shared LLM agents must decide whether a speaker is addressing them and what information about co-present users to use. LOOKAWAY tests these decisions with controlled orientation and profile access, revealing strong reliance on orientation during cue conflict and greater use of unrevealed profile information when both profiles are available.

I. INTRODUCTION

  • Shared agents must make addressee and information-use decisions within ongoing multiparty conversation, unlike conventional voice assistants that rely on wake words or explicit invocation.The paper frames these as questions of whether an utterance addresses the agent and whose information to use in responding.Source: §I · p. 1
  • Natural conversations rarely separate cue integration from cue-following because speaker orientation usually agrees with the intended addressee.Prior video-based work could not separate model use of gaze from errors in estimated gaze.Source: §I · p. 1
  • Providing both users’ profiles improves responses about the person being asked about but increases use of profile attributes they did not reveal in conversation.A strict privacy instruction reduces such use but also makes the agent refuse questions whose answers were already public.Source: §I · p. 1
  • LOOKAWAY independently varies assigned addressee, speaker orientation, and profile availability in a controlled multi-user conversational corpus.The study also includes a browser-based 3D demonstrator using users’ runtime head direction.Source: §I · pp. 1–2
  • On incongruent turns, two models followed speaker orientation on 85.0–94.2% of turns, and warning qwen3.6 that it could mislead reduced this behavior only modestly.The same sentence prompts an answer when the speaker faces the agent and silence when the speaker faces the other person.Source: §I · p. 1, Figure 1 · p. 2

II. RELATED WORK

Prior work studies spatial cues for addressee detection and how shared agents manage user-specific information. This paper focuses on the less-studied case of one person asking an agent about another co-present person, including whether profile information appears in the response.

A. Addressee Detection in Multi-Party Interaction

  • Prior work combines gaze or head orientation with speech and conversational context for addressee detection, but does not establish what happens when spatial cues conflict with the assigned addressee.Video-based studies also make it difficult to separate model behavior from errors in gaze estimation; LOOKAWAY controls orientation directly and includes congruent and conflicting turns.Source: A. Addressee Detection in Multi-Party… · p. 2

B. Shared Agents and Multi-User Profiles

  • Prior shared-agent systems address speaker identity, information ownership, or conflicting preferences; this work instead studies one person asking about another co-present person.It examines whether access to both profiles helps the response and changes which profile information appears.Source: B. Shared Agents and Multi-User Profiles · p. 2

C. Information Sharing Across Users

  • The study separates identity confusion from use of profile information that was not revealed in the shared conversation.Its profile-based measures use automatic extraction and scoring, which the authors compare with human judgments.Source: C. Information Sharing Across Users · p. 3

III. LOOKAWAY: CORPUS AND 3D DEMONSTRATOR

LOOKAWAY pairs a controlled conversational corpus with a separate browser-based 3D demonstrator. Its synthetic sessions vary persona information, addressee ambiguity, and speaker orientation so the study can examine both response decisions and cue conflict.

III. LOOKAWAY: CORPUS AND 3D DEMONSTRATOR

  • The corpus contains 40 synthetically constructed sessions, 80 personas, and 1,200 turns, while a separate browser-based demonstrator instantiates the setting interactively.The demonstrator uses runtime head orientation.Source: §III · p. 3

A. Persona and Profile Construction

  • Each persona has eight attributes: four revealed in conversation and four latent profile-only attributes selected to have cosine similarity below 0.45 to every revealed attribute.The corpus contains 640 attributes, including 205 authored opposing traits designed to make wrong-person attribution observable.Source: A. Persona and Profile Construction · p. 3

B. Conversation and Addressee Design

  • Each session has 30 turns, including eight ambiguous-addressee traps; across the corpus, 320 traps split evenly between agent-assigned and other-person-assigned turns.The trap wording avoids persona names and other cues that uniquely identify the listener, so always responding or always staying silent yields 50% accuracy.Source: B. Conversation and Addressee Design · p. 3

C. Spatial-Cue Manipulation

  • Speaker orientation is set per turn: the speaker faces the assigned addressee on congruent turns and the other co-present entity on incongruent turns.Of 320 traps, 200 are congruent and 120 incongruent; the direction of conflict is balanced across 60 turns each way.Source: C. Spatial-Cue Manipulation · p. 3

D. Experimental Conditions

The experimental conditions vary which identity, profile, and spatial details appear in each prompt. Each decision is stateless, and the prompt does not reveal the assigned addressee or whether a turn is congruent or incongruent.

D. Experimental Conditions

  • The same 1,200 turns are evaluated under five conditions, with each decision sent as a separate request containing only the conversation up to that turn.No model memory carries across conditions, and prompts omit the assigned addressee and the turn’s congruence status.Source: D. Experimental Conditions · pp. 3–4
  • P0 withholds names, profiles, and spatial information; P1a adds names and the current speaker’s profile, while P2a adds both personas’ profiles.P0 provides a no-profile baseline for addressee decisions and profile-use measures.Source: D. Experimental Conditions · p. 3
  • P1b adds spatial information to P1a, and P2b adds it to P2a; the spatial line names people, so P0 has no spatial counterpart.The spatial prompt line includes each persona’s distance from the agent and whom they are looking at.Source: D. Experimental Conditions · pp. 3–4, Figure 2 · p. 4

E. Conversation Generation

LOOKAWAY uses seeded plans to generate controlled conversations, evaluates three open-weight models, and demonstrates the same addressee setting in a browser-based 3D room.

E. Conversation Generation

  • Sixteen of 1,200 turns required regeneration after constraint checks; all were traps, and three required all three attempts.The seeded rule-based program sets turn type, assigned addressee, and orientation; gemma4:31b generates persona utterances without receiving latent attributes, partner profiles, orientation, or turn-type labels.Source: E. Conversation Generation · p. 4

F. Models and Inference Setup

  • Three open-weight models were selected to run with 4-bit quantization on one 24 GB GPU; the authors do not intend them to represent frontier models.The evaluation produced 18,000 decisions across five conditions; inference used temperature 0, disabled reasoning mode, and an 8,192-token context window.Source: F. Models and Inference Setup · p. 4

G. 3D Virtual-Room Demonstrator

  • The demonstrator responds to the same ambiguous utterance when the speaker faces the agent and stays silent when the speaker faces the other person.It derives orientation from runtime head direction; a target falls within a 22° cone, and the addressee is never supplied to the model.Source: G. 3D Virtual-Room Demonstrator · pp. 4–5, Figure 3 · p. 5

IV. EXPERIMENT 1: ADDRESSEE DETECTION UNDER CONFLICTING CUES

Experiment 1 tests whether spatial information helps agents identify their addressee, including when speaker orientation conflicts with the assigned addressee.

IV. EXPERIMENT 1: ADDRESSEE DETECTION UNDER CONFLICTING CUES

  • The experiment tested three models on clear turns and ambiguous turns, using paired conditions that differed only in whether spatial information was included.The eight main spatial comparisons remained significant after Holm correction (p < 0.001).Source: §IV · p. 5, B. Effect of Spatial Context · p. 5

A. Baseline Addressee Detection

  • On 880 clear turns, qwen3.6 classified 99.5% correctly and mistral-small3.2 98.6%; nemotron3 answered 61.9% of turns addressed to the other person.The remaining analyses focus on qwen3.6 and mistral-small3.2 because nemotron3 did not reliably remain silent when another person was addressed.Source: A. Baseline Addressee Detection · p. 5

B. Effect of Spatial Context

  • 56.0%→99.5% for qwen3.6 and 70.0%→96.5% for mistral in P1 when orientation agrees with the assigned addressee.P2 shows the same pattern; both P1 comparisons have p < 0.001.Source: B. Effect of Spatial Context · p. 5
  • 55.0%→9.2% and 55.8%→5.8% for qwen3.6, and 62.5%→11.7% and 60.8%→15.0% for mistral, as orientation conflicts in P1 and P2.All four spatial-condition accuracies are below the 50% baseline.Source: B. Effect of Spatial Context · p. 5

C. Cue-Following Behavior

Spatial cues strongly influence addressee decisions, but this comes with a precision–recall trade-off.

C. Cue-Following Behavior

D. Precision–Recall Trade-off

  • 64.5%→84.8% recall and 97.7%→86.5% precision for qwen3.6 in P1 after adding spatial information.Precision measures how often responses occur on turns assigned to the agent; recall measures how often it responds when the agent is the assigned addressee.Source: D. Precision–Recall Trade-off · p. 6

E. Robustness to Prompt Warnings

Warnings modestly reduce qwen3.6’s reliance on orientation; the profile-use experiment examines responses about co-present users and use of unrevealed profile attributes.

E. Robustness to Prompt Warnings

  • The warning check tested qwen3.6 on all 320 P1b traps; one prompt called orientation an imperfect cue, while another advised relying on conversation when cues conflict.The analysis was not run with mistral-small3.2.Source: E. Robustness to Prompt Warnings · p. 6
  • 90.8%→85.0% and 81.7% spatial-cue following after warnings, while incongruent accuracy rose from 9.2% to 15.0% and 18.3%.Congruent accuracy fell from 99.5% to 96.0% and 91.5%, respectively.Source: E. Robustness to Prompt Warnings · p. 6

V. EXPERIMENT 2: PROFILE USE IN RESPONSES ABOUT CO-PRESENT USERS

  • Experiment 2 analyzes 120 cross-reference turns with qwen3.6 and mistral-small3.2; nemotron3 was excluded after failing Experiment 1’s baseline addressee check.The study asks whether profile access helps responses about the target person and whether replies use profile information that person has not revealed.Source: §V · p. 6
  • A different LLM judge, deepseek-r1:32b, extracts which listed persona attributes appear in each reply using a closed-list matching task.Its output supports the right-person, wrong-person, disclosure, and mix-up measures.Source: §V · p. 6

A. Response Measures

  • The extractor failed on 177 replies, but counting every missing reply in the least favourable direction did not reverse any Table IV comparison.Response quality was rated from 1 to 5; refusals were treated as evasive.Source: §V · p. 6, A. Response Measures · p. 7

V. EXPERIMENT 2: PROFILE USE IN RESPONSES ABOUT CO-PRESENT USERS

  • Spatial information showed no response-content pattern that repeated across models and profile conditions, so P1a/P1b and P2a/P2b were combined for the main results.Separate spatial comparisons are reported at the end of the experiment.Source: §V · p. 6

A. Response Measures

  • Disclosure means stating a latent attribute of the person discussed; right-person, wrong-person, and mix-up responses distinguish whose attributes a reply uses.Revealed attributes appear in conversation; latent attributes are available only through the profile. Disclosure may overlap with a right-person response.Source: A. Response Measures · pp. 6–7
  • P0 disclosure rates were 5.4% for qwen3.6 and 3.9% for mistral despite no profile access; these replies happened to match latent attributes.The P0 rates therefore do not represent disclosure resulting from profile access.Source: A. Response Measures · p. 7

B. Identity Separation

Identity mix-ups were rare, but access to both profiles substantially increased disclosure of attributes not revealed in conversation. The examples show that this can happen even when the agent discusses the intended person.

B. Identity Separation

  • Across 1,939 replies about the speaker, neither model attributed the other person’s facts to the speaker; identity mix-ups were rare.qwen3.6 had no identity mix-ups; mistral had five mixed-up attributes in four cross-reference replies, all with both profiles available.Source: B. Identity Separation · p. 7

C. Effect of Profile Access

  • 3.3%→45.3% for qwen3.6 and 1.4%→25.9% for mistral: disclosure increased when both profiles were available.Paired tests confirmed increases across profile-access conditions for both models.Source: C. Effect of Profile Access · p. 7
  • In both profile-access conditions, qwen3.6 answered a question about Astrid with two latent attributes that had never appeared in conversation.The attributes were that Astrid worked long hours on a farm and watched television shows.Source: C. Effect of Profile Access · p. 7

D. Effect of Profile Access on the Person Discussed

Both profiles improved responses about the person being discussed, and response quality increased, but spatial information did not consistently change response content. Human validation supported key annotation judgments, while orientation conflicts revealed that models often followed the spatial cue.

D. Effect of Profile Access on the Person Discussed

  • With both profiles, right-person responses rose to 56.2% for qwen3.6 and 39.1% for mistral, while wrong-person responses fell to 6.5% and 8.6%, respectively.With only the speaker’s profile, right-person responses were 14.5% and 14.0%, while wrong-person responses were 29.8% and 28.8%, respectively.Source: D. Effect of Profile Access on the… · p. 7

E. Response Quality

  • Both profiles increased response quality: qwen3.6’s mean rose from 2.26 to 2.88 and mistral’s from 2.37 to 2.78, with both effects small.Both comparisons had p < 0.001.Source: E. Response Quality · p. 7

F. Effect of Spatial Information on Response Content

  • No consistent evidence showed that adding spatial information changed response content across models and conditions.No wrong-person or disclosure comparison was significant; quality’s only significant comparison did not repeat across conditions or models.Source: F. Effect of Spatial Information on… · pp. 7–8

VI. HUMAN VALIDATION

  • Human annotators confirmed 95.0% of authored orientation contrasts as real conflicts and agreed with the attribute extractor on 91.0% of judgments.On congruent traps, the spatial line raised annotator accuracy to 71.6%, while agreement remained low.Source: §VI · p. 8

A. RQ1: Spatial Cues Can Dominate Addressee Decisions

  • The models followed orientation on more than 85% of incongruent turns, and warning qwen3.6 that orientation might mislead reduced this only modestly.The authors suggest the spatial line names a target outright, making it the clearest signal when trap wording is deliberately ambiguous.Source: A. RQ1: Spatial Cues Can Dominate… · p. 8

B. RQ2: Profile Access Changes What the Agent Says

Providing both profiles improved person grounding but increased use of attributes unrevealed in conversation, even though identity mix-ups were rare. The authors conclude that shared agents need to weigh spatial cues and control when user-specific information enters a response.

B. RQ2: Profile Access Changes What the Agent Says

  • Both profiles made models more likely to answer about the person being discussed and less likely to answer about the questioner, but also increased use of unrevealed profile attributes.Identity mix-ups were rare, so keeping identities separate did not by itself limit profile use; the main prompts did not label latent attributes as private.Source: B. RQ2: Profile Access Changes What… · p. 8

C. Implications for Shared Multi-User Agents

  • More context alone is not enough: orientation can override intended addressee information, while profiles can bring unrevealed facts into responses.The authors argue that shared agents must decide not only what information is available, but when and how strongly to use it.Source: C. Implications for Shared Multi-User… · p. 8

VIII. LIMITATIONS AND FUTURE DIRECTIONS

  • LOOKAWAY uses synthetic, predefined orientation, model-generated conversations, and two-person interactions; future work should test privacy rules across models and conditions.The authors also call for testing interactions with real participants and whether an explicit record of what each person said aloud can reduce disclosure without over-refusal.Source: §VIII · p. 8

IX. CONCLUSION

  • The conclusion is that shared agents must weigh how strongly cues influence decisions and when user-specific information is appropriate to use.In LOOKAWAY, orientation reliance and profile access with unrevealed attributes were the two observed challenges.Source: §IX · p. 8
Loading 2610.07732v1…