Source-linked AI summary
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B. Tenenbaum, Sanja Fidler, Antonio Torralba
TL;DR
WAH addresses whether AI agents can infer a human-like agent’s household goal from one demonstration and collaborate to achieve it in a new environment. The paper introduces a multi-agent platform and benchmark with planning and learning baselines, finding that the challenge systematically evaluates social intelligence at scale and aligns objective metrics with subjective ratings.
Problem
AI agents still struggle with social perception and collaborative planning: understanding human behavior and coordinating actions to help people in complex tasks.
Method
The paper introduces WAH, VirtualHome-Social, and a benchmark in which agents infer predicate-defined goals from demonstrations and collaborate in household environments.
Results
The challenge systematically evaluates key aspects of machine social intelligence at scale, with objective metrics consistent with subjective user ratings.
Takeaways & Limitations
WAH provides a realistic framework for evaluating social perception, human-AI collaboration, and generalizable helping strategies.
Takeaways & Limitations
Bob can unintentionally block Alice because of pathfinding limitations and inability to predict other agents’ future paths.
Abstract
from arXiv · showhide
In this paper, we introduce Watch-And-Help (WAH), a challenge for testing social intelligence in agents. In WAH, an AI agent needs to help a human-like agent perform a complex household task efficiently. To succeed, the AI agent needs to i) understand the underlying goal of the task by watching a single demonstration of the human-like agent performing the same task (social perception), and ii) coordinate with the human-like agent to solve the task in an unseen environment as fast as possible (human-AI collaboration). For this challenge, we build VirtualHome-Social, a multi-agent household environment, and provide a benchmark including both planning and learning based baselines. We evaluate the performance of AI agents with the human-like agent as well as with real humans using objective metrics and subjective user ratings. Experimental results demonstrate that the proposed challenge and virtual environment enable a systematic evaluation on the important aspects of machine social intelligence at scale.
1 INTRODUCTION
Watch-And-Help targets two social-intelligence abilities: inferring another agent’s goal from one demonstration and collaborating to achieve it efficiently in a new environment. The paper contributes a challenge, a multi-agent household platform, and benchmark methods for evaluating these abilities.
- WAH requires social perception and collaborative planning: understanding human behavior and coordinating actions in the physical environment.
- In the Watch stage, Bob observes Alice once and infers her goal; in the Help stage, Bob helps Alice achieve it quickly in a different environment.
- Unlike prior setups with known or small goal spaces, WAH requires inferring a goal from one demonstration and generalizing it to a new environment.
- VirtualHome-Social supports realistic household interactions with objects, built-in human-like agents, and real-human control.
- The benchmark includes a Watch-stage goal-inference model and planning and deep-reinforcement-learning baselines for the Help stage.
- The work’s contributions are a social-intelligence challenge, a multi-agent platform, and benchmark approaches highlighting machine social intelligence.
2 RELATED WORK
WAH extends activity understanding and human-robot collaboration into realistic, diverse household environments. It combines goal inference from observed actions with multi-agent interaction capabilities that prior platforms generally lack.
- Human activity understanding: The related activity-understanding literature includes short-action recognition, trajectory prediction, group-activity recognition, and plan recognition.
- Multi-agent virtual environments: VirtualHome-Social provides a multi-agent household platform where agents observe one another, infer goals, plan, and adapt during collaboration.
- Human activity understanding: WAH focuses on inferring other people’s goals from their actions in simulated real-world environments rather than toy 2D grid worlds.
- Human-robot interaction: Prior human-robot interaction work was mainly restricted to laboratories, with shared goals or small goal spaces; WAH uses large, inferred goals in diverse households.
- Multi-agent virtual environments: Existing multi-agent platforms generally simulate simple or game-like environments and do not support human-AI collaboration on real-life activities.
3 THE WATCH-AND-HELP CHALLENGE
The WAH challenge separates goal inference from collaborative execution: Bob watches Alice’s demonstration, then helps her achieve the same predicate-defined goal in a new environment. Tasks span diverse household activities and require efficient coordination.
- Each task gives Bob a demonstration of Alice successfully performing an activity, followed by a new environment where Bob must help achieve the same goal in minimum time steps.
- VirtualHome-Social supports concurrent multi-agent actions and observations, with Alice planning from her goal and partial environmental observations.
- During training Bob receives Alice’s ground-truth goal, whereas testing requires inferring that goal from the demonstration.
- Goal Definitions: Goals are sets of 2–8 predicates with counts, and object classes allow any instance of a specified class to satisfy a predicate.
- Goal Definitions: The benchmark samples goals from five household activities and 30 predicate types, including table setting, food storage, meal preparation, dishwashing, and reading with snacks or drinks.
4 VIRTUALHOME-SOCIAL
VirtualHome-Social is a multi-agent household environment designed for concurrent interaction, mutual observation, and realistic human-like behavior. It supports symbolic and visual observations, dynamic object interaction, and a bounded-rationality planning-based human-like agent.
- Environment: The environment addresses real-life social intelligence challenges involving unconstrained goals and actions and realistic human-action display.These challenges extend beyond typical multi-agent settings.
- Environment: VirtualHome-Social lets multiple agents, including real humans, execute actions concurrently and observe one another's behavior.The platform also embeds planning-based agents as virtual humans for interaction and reasoning.
- Observations and actions: Agents receive either symbolic scene-graph observations or visual observations for learning helping behaviors under different conditions.Scene graphs represent objects as nodes and spatial relationships as edges.
- Observations and actions: Agents navigate and interact with visible objects, specifying both an action and the intended object index at each step.Because interaction requires the object to be within the agent's field of sight, the action space changes over time.
- Human-like agents: The built-in human-like agent uses a bounded-rationality planner operating on symbolic partial observations to emulate human behavior.Its hierarchical planner combines Monte Carlo Tree Search and regression planning, updates beliefs after observations, and executes plans concurrently.
5 BENCHMARK
The benchmark evaluates goal inference and helping under unseen environments, using hierarchical planning and learning baselines with success, speedup, and cumulative-reward metrics. Results show strong Watch-stage recognition on test-1, but weaker generalization to multi-activity tasks and harmful collaboration when inferred goals conflict.
- Evaluation Setup: The benchmark uses 1011 training tasks and two 100-task test sets with changed apartments, randomized initial states, unseen predicate combinations, and held-out Help-stage apartments.These conditions test generalization beyond the demonstration environment and training goals.
- Evaluation Metrics: Performance is measured by success rate, speedup relative to Alice working alone, and cumulative reward combining task completion with episode length.Speedup is defined as LAlice/LBob −1, while cumulative reward ranges from −1 for failure to 1 for achieving the goal in zero steps.
- Baselines: The benchmark also includes random agents, oracle agents with full environment or goal knowledge, hierarchical planning, hybrid RL-planning, and hierarchical RL baselines.The proposed baselines use a goal inference model and a goal-conditioned helping planner or policy, generally operating on symbolic visible-object states.
- Watch Stage: The Watch-stage goal inference model achieves 0.85 precision and 0.96 recall on test-1, compared with 0.08 and 0.09 by chance.Using only the final observation reduces precision and recall to 0.79 and 0.75, while using Alice’s actions raises both to 0.99.
- Help Stage: Planning-based approaches are most effective in the Help stage; HPTG performs best among non-oracle baselines, while inferred-goal HP and Hybrid also provide effective help.HPTG uses true goals and reasons about Alice’s future plan to avoid redundant actions and collisions.
- Generalization: On multi-activity test-2, goal inference falls to 0.68 precision and 0.64 recall, while HP reaches 88.60 ± 0.02 success rate and 0.21 ± 0.04 speedup.Alice alone achieves 95.40 ± 0.01 success rate, and the HP degradation is attributed to weaker goal recognition.
- Helping Behaviors: Helping behavior includes predicting Alice’s future actions, influencing her belief updates by opening containers, and coordinating actions to avoid conflicts.The current baselines omit low-level pathfinding, which can cause inefficient navigation and unintentionally block Alice.
6 HUMAN EXPERIMENTS
The human experiments evaluate collaboration with real people and compare subjective ratings with objective performance. Results show that baseline rankings largely transfer from the human-like agent to real humans.
- Experimental design: The study evaluates AI collaboration with real humans, replacing Alice’s built-in control with human participants.The authors conducted two experiments: humans performed tasks alone, and humans collaborated with baseline AI agents.
- Experimental design: Six subjects performed tasks alone using the human-like agent’s observation and action space across 30 tasks.Participants controlled Alice through visible-object selections, action menus, and keyboard movement controls.
- Experimental design: Twelve subjects completed 90 human-AI collaboration trials across the same 30 tasks, pairing with HP, Hybrid, or HPRG.After each trial, subjects rated the AI agent on a 1-to-7 scale using three criteria.
- Results: The three baseline agents retained the same ranking with real humans, and subjective ratings were consistent with objective scores.This consistency held when the human-like agent was replaced by real participants.
- Results: HPRG achieved a higher success rate with real humans because they completed nonconflicting subgoals instead of competing over conflicting ones.No significant objective-metric difference appeared between helping humans and helping the human-like agent except for HPRG.
7 CONCLUSION
The paper concludes that WAH and VirtualHome-Social evaluate social perception and human-AI collaboration in household activities at scale. The platform combines realistic multi-agent environments, diverse observations and actions, built-in human-like agents, and held-out helping environments.
- Conclusion: WAH evaluates whether agents can reason about other agents’ mental states and help them in unfamiliar household scenarios.The challenge targets social perception and human-AI collaboration rather than isolated single-agent task completion.
- VirtualHome-Social: VirtualHome-Social supports realistic multi-agent household interactions with objects and agents, including helping, following, and collision avoidance.Its apartments contain objects with states, coordinates, and persistent identity indices for interaction and tracking.
- Conclusion: The authors report that the challenge systematically evaluates key aspects of social intelligence at scale, while human-like-agent metrics align with subjective ratings.They also identify online goal inference and direct agent communication as future directions.
- VirtualHome-Social: The platform provides multiple apartments, humanoid avatars, visual and symbolic observations, direct messages, and high- and low-level actions.Observations include RGB, depth, segmentation, maps, skeletons, bounding boxes, and scene-graph representations.
- Evaluation scope: Two apartments are held out exclusively as helping environments during testing to assess generalization to new environments.This design separates the helping environments from the apartments used for other challenge settings.
- Human-like agent: The human-like agent updates beliefs from partial observations, samples possible states, and uses MCTS with regression planning to execute hierarchical plans.Heuristics encourage closing containers when they contain no relevant remaining goal objects.
A.4 SPECIFICATIONS
The specifications describe the platform’s execution rate and its predicate-based goal representation. VirtualHome-Social supports more predicates than the five sets currently used for defining Alice’s goals.
- Execution: The environment runs at 10 actions per second in one process, while training uses 10 processes in parallel.Parallel execution supports model training across multiple environment processes.
- Goal specification: Five predicate sets define Alice’s goals, and VirtualHome-Social supports additional predicates for future goal-definition extensions.The predicate sets correspond to five activity types.
B.2 TRAINING AND TESTING SETUP
The benchmark trains on 1,011 tasks and evaluates on two 100-task test sets designed to measure generalization across environments and unseen goals. Additional demonstrations train goal-inference models, while the baselines use planning components and learned architectures.
- Goal inference: Goal-inference training uses 5,303 demonstrations synthesized in the five training apartments without pairing them to helping environments.These demonstrations are reserved for training goal-inference models and are not used for helping tasks.
- Goal inference: The benchmark includes a goal-inference model whose architecture encodes symbolic demonstration sequences and infers predicate counts.The reported architecture uses 128-dimensional vectors, a 128-unit LSTM, and two 128-dimensional fully connected layers.
- Baselines: The HRL baseline’s low-level policy includes a Null dummy object node for actions that do not involve an object.This design is part of the object-selection policy architecture.
- Baselines: The HP baseline uses MCTS for high-level planning and regression planning for low-level actions, with its own partial observation and belief.When given Alice’s ground-truth goal, it removes Alice’s pursued subgoal from its own subgoal space.
C.3 GENERAL TRAINING PROCEDURE FOR RL-BASED APPROACHES
The RL-based approaches use goal-conditioned low- and high-level policies, trained with A2C and curriculum procedures, alongside human experiments that evaluate collaboration under fixed task constraints.
- Low-level policy: The low-level policy encodes visible objects and subgoal object classes, then uses attention to select objects relevant to the current action.Object and context representations are reshaped with attention vectors before producing interaction likelihoods.
- Low-level policy: Training samples one predicate as the final goal, rewarding proximity and successful interaction with the correct object or location.The low-level policy is trained in a single-agent environment to achieve predicates independently.
- Training procedure: The low-level policy uses a two-phase curriculum: first learning to grab target objects, then reusing that policy to place them at target locations.The second phase adds placement behavior on top of the learned grabbing policy.
- Training procedure: Both low- and high-level RL policies use off-policy A2C with RMSprop, while the high-level policy is trained first alone and then fine-tuned with the human-like agent.The high-level policy receives ground-truth goals during training.
- High-level policy: The high-level policy samples object-class and location-class subgoals, and a regression planner finds low-level plans using states from the agent’s belief.Unlike planner-generated instance-specific subgoals, the RL policy selects object classes for its subgoals.
- Human experiments: Human experiments used concurrent actions, a 250-step limit, and 30 randomly sampled test tasks distributed evenly across five categories.Participants controlled Alice and were instructed to complete the displayed goals as quickly as possible.
D.4 SUBJECTIVE EVALUATION OF SINGLE AGENT PLANS
The subjective evaluation found no significant rating difference between plans produced by the human-like agent and real humans, although human plans were sometimes judged more efficient.
- Subjective ratings: Human-like-agent plans received an average rating of 3.38, versus 3.72 for human plans, with no significant paired-test difference.The reported test was t(29) = −1.35, p = .19.
- Qualitative feedback: Free responses indicated that human plans were sometimes more efficient because they avoided unlikely searches and frequent movement between rooms.The human-like agent was generally similar but occasionally searched rooms in a counter-intuitive order because of bounded rationality and stochastic planning.
D.5 ADDITIONAL QUANTITATIVE ANALYSES OF HUMAN EXPERIMENT RESULTS
Quantitative analyses compared AI-agent performance when helping the human-like agent and real humans, supporting the use of the human-like agent for most baseline evaluations.
- Baseline comparisons: For HP and Hybrid, cumulative rewards did not differ significantly between helping the human-like agent and helping real humans.The reported tests were t(29) = −1.78, p = .1 for HP and t(29) = −0.5, p = .62 for Hybrid.
- Baseline comparisons: HPRG showed a significant cumulative-reward difference between helping the human-like agent and helping real humans.The reported test was t(29) = −2.36, p = .03.
- Implications: The authors conclude that collaboration with the human-like agent is generally comparable to collaboration with real humans, supporting a scalable training and evaluation procedure.The procedure trains with the human-like agent, then evaluates with both the human-like agent and a smaller representative real-human set.