Source-linked AI summary
Challenges in Human-Agent Communication
Gagan Bansal, Jennifer Wortman Vaughan, Saleema Amershi, Eric Horvitz, Adam Fourney, Hussein Mozannar, Victor Dibia, Daniel S. Weld
TL;DR
Complex, increasingly capable agents create communication challenges that matter for users’ ability to understand and guide them. The paper draws on prior research and experience with complex agents to identify challenges in establishing common ground. It organizes these challenges across agent-to-user, user-to-agent, and overarching communication, and calls for methods and tools to support shared understanding aligned with human needs and expectations.
Problem
As agents’ capabilities and potential failure modes grow, users need effective two-way communication to establish common ground about goals and how agents will pursue them.
Source: §1 · p. 2Method
Drawing on experience building complex agents and prior human-AI interaction and collaboration research, the paper identifies challenges in establishing common ground and groups them by communication direction and overarching concerns.
Source: §1 · p. 3Results
The paper presents communication challenges involving information from agents to users, information from users to agents, and concerns that span both directions.
Source: §1 · p. 3Takeaways & Limitations
The authors call for methods and tools to establish and maintain shared understanding so agents operate in ways aligned with human needs and expectations.
Source: §6 · p. 21The paper focuses on human-agent communication and places technical, ethical, safety, and fairness challenges outside its scope.
Source: §1.2 · p. 4
Abstract
from arXiv · showhide
Remarkable advancements in modern generative foundation models have enabled the development of sophisticated and highly capable autonomous agents that can observe their environment, invoke tools, and communicate with other agents to solve problems. Although such agents can communicate with users through natural language, their complexity and wide-ranging failure modes present novel challenges for human-AI interaction. Building on prior research and informed by a communication grounding perspective, we contribute to the study of \emph{human-agent communication} by identifying and analyzing twelve key communication challenges that these systems pose. These include challenges in conveying information from the agent to the user, challenges in enabling the user to convey information to the agent, and overarching challenges that need to be considered across all human-agent communication. We illustrate each challenge through concrete examples and identify open directions of research. Our findings provide insights into critical gaps in human-agent communication research and serve as an urgent call for new design patterns, principles, and guidelines to support transparency and control in these systems.
1 Introduction
Modern tool-using agents make effective human-agent communication important but difficult: users need to understand and guide systems whose capabilities, workflows, and failure modes are complex. The paper frames communication as establishing common ground and presents challenges for future research.
1 Introduction
- The paper identifies communication challenges in establishing common ground, grouped around conveying information to users, enabling users to guide agents, and concerns spanning both directions.The authors draw on their experience with complex agents and prior human-AI interaction and collaboration research; they describe the list as a starting point, not an exhaustive account.Source: §1 · p. 3
1.1 What is different about modern AI agents?
- Modern agents combine generative foundation models with tools that let them act in the world, expanding their capabilities and creating new communication challenges.These agents may also execute code, while their behavior can be difficult to anticipate because of planning limitations and hallucinations.Source: §1.1 · pp. 3–4
1.2 Outline
- The paper focuses on human-agent communication, leaving technical, ethical, safety, and fairness challenges outside its scope.It reviews prior work, presents four overarching challenges, then discusses communication from agents to users and from users to agents, with examples and research directions.Source: §1.2 · p. 4
2 Previous Work
The paper draws on theories of human communication and teamwork, computational grounding, and human-AI interaction guidelines. It argues that tool use, generalist capabilities, and multi-agent workflows warrant revisiting communication challenges and design guidance.
Grounding and communication
- Human communication research treats grounding as the coordination of conversation content and process toward common ground, while media constraints can increase the effort of achieving it.Grounding can involve establishing a channel, exchanging recognized signals, interpreting intentions, and coordinating conversational turns.Source: §2 · pp. 4–5
Computational grounding
- Computational work has modeled grounding as joint activity, coordinated levels of inference, expected-utility-guided action, multimodal signals, and shared memories.These approaches explore how systems can communicate uncertainty and make past interactions referable.Source: §2 · p. 5
Human teamwork and AI interaction
- Human-AI interaction guidance builds on teamwork research, but differences in machine understanding and communication styles create new challenges for human-AI teams.Research on groups links performance differences across tasks to how demonstrable proposed solutions are, while shared mental models, closed-loop communication, and mutual trust are identified as teamwork mechanisms.Source: §2 · p. 6
Human-AI interaction guidance
- Tool use, generalist capabilities, and task delegation among agents add complexity to grounding, motivating a renewed examination of human-AI interaction guidelines.Shavit et al. address a subset of related concerns; this paper also considers communicating capabilities and limitations, managing information flow, and multi-agent issues.Source: §2 · p. 6
3 Overarching Challenges for Human-Agent Communication
The section presents four overarching communication challenges: verifying shared understanding, avoiding confusing inconsistencies, calibrating detail, and selecting relevant context. Together, they concern making agent behavior understandable without making interaction burdensome.
3 Overarching Challenges for Human-Agent Communication
- The section introduces four challenges that apply across human-agent communication and recur when communicating specific information between users and agents.The four concern verification, consistency, level of detail, and context relevance.Source: §3 · p. 6
X1: How should the agent help the user verify its behavior?
- Users need ways to verify both that agents understand their goals and that they understand agents’ plans and actions, since verification supports steering behavior and avoiding costly mistakes.Without verification, users may also fail to develop an accurate mental model of agent capabilities or provide feedback for future performance.Source: X1: How should the agent help the user… · p. 7
X2: How should the agent convey consistent behavior?
- Agent behavior can appear inconsistent because model outputs are stochastic, workflows interact with changing environments, or users and agents have different information about the world.Perceived inconsistency can confuse users about goal completion and future behavior, and may diminish trust over time.Source: X2: How should the agent convey… · pp. 7–8
X3: How should the agent choose an appropriate level of detail?
- Communication must provide enough detail for verification without burdening users, overwhelming them, or overconstraining agents’ ability to explore alternatives or recover from failures.For recurring email requests, using previously expressed preferences can avoid repeated clarification, while novel or ambiguous cases may need more detail.Source: X3: How should the agent choose an… · p. 8
X4: Which past interactions should the agent consider when communicating?
- As interaction history and retrieved information grow, agents must focus on context relevant to the current request while letting users restrict background information when they choose.The authors point to browser controls such as incognito mode and editable or purgeable history as analogous patterns, and cite workflow memory as useful for repetitive tasks.Source: X4: Which past interactions should the… · pp. 8–9
4 Information Flow from the User to the Agent
The section examines how users communicate goals, preferences, and feedback to agents. These inputs can overlap, but their timing and context differ.
4 Information Flow from the User to the Agent
- The section groups user-to-agent communication into goals, preferences, and feedback, whose content may overlap but whose timing and context differ.It discusses existing research, how modern agents change these challenges, examples, and possible solutions.Source: §4 · p. 9
U1: What should the agent achieve?
- Modern agents must interpret higher-level, multi-step goals expressed in natural language, where ambiguity and incomplete dialog make intent harder to understand.Misunderstood goals can waste resources, create side effects, or frustrate users.Source: U1: What should the agent achieve? · pp. 9–10
U2: What preferences should the agent respect?
- Complex goals permit many plans, but users differ in their preferences and notions of harm, which may be difficult to specify in open-world settings.Tools expand the possible scope of harm; making inferred preferences and constraints visible is one proposed approach.Source: U2: What preferences should the agent… · p. 11
U3: What should the agent do differently next time?
- Even when agents understand goals and preferences, imperfect planning and changing user tastes can leave them making mistakes; free-text feedback adds an interpretation challenge.Users may need agents to integrate feedback across multiple aspects of complex tasks.Source: U3: What should the agent do… · pp. 12–13
5 Information Flow from the Agent to the User
The section identifies five agent-to-user communication challenges: explaining capabilities, planned actions, ongoing actions, side effects, and goal completion.
A1: What can the agent do?
- Users need to understand an agent’s capabilities and limitations to make informed choices, including what information it can access and under what conditions.Capabilities can change over time, and stochastic behavior may produce different outcomes across attempts.Source: A1: What can the agent do? · pp. 13–15
A2: What is the agent about to do?
- Before execution, agents should communicate plans at a detail level suited to the user, highlighting surprising or consequential steps while avoiding overload.Permission is especially relevant for actions that consume resources, are irreversible, or may violate preferences.Source: A2: What is the agent about to do? · pp. 15–16
A3: What is the agent currently doing?
- During execution, users need progress and impact updates that let them monitor actions and intervene, a challenge intensified by many rapid or parallel steps.Real-time updates and summaries are possible approaches in domains with high-speed system actions.Source: A3: What is the agent currently doing? · pp. 16–17
A4: Were there any side effects or changes to the environment?
- Agents should report consequential environmental side effects, especially surprising or socially non-normative changes, as open-world tool use increases their scope.Post hoc summaries may help, but some side effects need to be communicated before action so users can intervene.Source: A4: Were there any side effects or… · p. 18
A5: Was the goal achieved?
- Users need information to verify whether complex goals were achieved, but open-world conditions and partial observability can make completion difficult to assess.Overviews, drill-down into activity, and links to primary sources are possible support for verification.Source: A5: Was the goal achieved? · p. 19
6 Conclusion
Generative, tool-using agents make common ground harder to establish and maintain as their actions grow more complex, rapid, and consequential. The authors call for sustained research to develop methods and tools that support shared understanding.
6 Conclusion
- Complex and rapidly executed agent actions, stochastic behavior, and heightened stakes make effective communication and common ground more difficult to maintain.The authors connect these difficulties to transparency and trust.Source: §6 · p. 20
- The authors call for sustained effort to develop design patterns, guidance, and principles that prioritize common ground between users and agents.They state that shared understanding directly affects system effectiveness and safety, particularly where failure is costly.Source: §6 · pp. 20–21
- The authors argue for cross-disciplinary research and practical innovation toward agents that are transparent, trustworthy, and appropriately controllable.They also call for effective communication about agents’ activities with human counterparts.Source: §6 · p. 21