Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challenge

• Existing embodied benchmarks largely omit accessibility differences among human partners, while outdoor scenarios and emergencies receive limited attention.Source: §1 · pp. 1–2
• The challenge includes four physically constrained agents and eight long-horizon tasks spanning indoor and outdoor scenes, emergencies, and diverse assistance situations.Source: §1 · p. 2
• Benchmark results show that current systems struggle to model partner behaviors from raw RGB images, whereas LLM-driven agents are competitive in decision-making.Source: §1 · p. 2
• CHAIC tests whether embodied agents can infer human partners’ intents and physical constraints from egocentric observations and make tailored cooperative plans.Source: §1 · pp. 1–3
• The paper benchmarks planning- and learning-based baselines and introduces a behavior-modeling agent to analyze perception and cooperation challenges.Source: §1 · pp. 2–3

• Prior multi-agent benchmarks study cooperation and inter-agent perception but do not explicitly require adapting strategies to diverse human partners with physical constraints.Source: §2 · p. 3
• CHAIC differs from contemporary assistive-agent work by using egocentric visual observations, newly created constrained agents, indoor and outdoor scenarios, and unexpected events rather than indoor oracle symbolic observations.Source: §2 · p. 3
• The benchmark builds on accessibility-oriented robotics research while introducing a large-scale embodied environment that explicitly models physical impairments.Source: §2 · p. 3

3 The Constrained Human-AI Cooperation (CHAIC) Challenge

• CHAIC assigns a helper to infer a constrained agent’s goal and capabilities, then assist in transporting target objects to a goal location.Source: §3 · p. 3, §3.3 · p. 4

3.1 Constrained Agents

• The benchmark provides four agents with constraints involving reach, strength, movement speed, obstacle mobility, and emergency-related behavior.Source: §3.1 · pp. 3–4

3.2 Tasks with Constrained Agents

• Eight tasks cover indoor and outdoor settings, including shopping, moving furniture, obstacles, object-height constraints, and emergencies.Source: §3.2 · p. 4, Figure 2 · p. 4, §3.3.1 · p. 5

3.3 Challenge Details

• Partner goals and constraints are hidden from the helper, making active perception necessary for inferring intended actions and assistance needs.Source: §3.3 · p. 4
• Agents receive egocentric RGB-D images and self-state information, then act through navigation, object interaction, and waiting actions executed asynchronously.Source: §3.3 · p. 5

3.3.1 Task Generation

• Indoor scenes use interconnected rooms with sampled objects, goals, and containers, while outdoor scenes include distributed shops or furniture transported to a nearby truck.Source: §3.3.1 · p. 5

3.3.2 Dataset Construction

• Each task has 12 training and 12 testing episodes with different environments, and episodes end at goal completion or a task-specific frame horizon.Source: §3.3.2 · p. 6

4 Language Agent Augmented with Behavior Modeling Module

• The framework combines action recognition with LLM reasoning through perception, memory, decision, and execution modules.Source: §4 · p. 6

4.1 Behavior Modeling Module

• The behavior modeling module extracts constrained-agent actions and status from egocentric image sequences using action recognition and action grounding.Source: §4.1 · p. 6
• Its action recognizer uses TSN pretrained on Kinetics-400 to classify pickup, placement, insertion, and walking behaviors, including action success or failure where applicable.Source: §4.1 · pp. 6–7
• After recognizing an action, action grounding consults the semantic map to identify the relevant predicate and object, producing the action, predicate, and status.Source: §4.1 · p. 7

5.1.1 Constrained Agent Policy

• The constrained agent follows handcrafted priorities for exploring, picking targets or containers, transporting objects, and meeting time-dependent deadlines.It prefers nearby targets, uses containers when possible, and transports immediately when little time remains.Source: §5.1.1 · p. 7

5.1.2 Evaluation Metrics

• Helpers are evaluated with TR, EI, IA, and shopping-task ER, while baselines include no helper, random, rule-based, LLM+BM, VLM, Oracle, and learning-based methods.EI is defined as ΔM/M0, and IA measures the fraction of helper-transported objects matching the constrained agent’s targets.Source: §5.1.2 · p. 7, §5.2 · p. 7

5.3 Main Results

• LLM+BM achieves the highest TR in 6 of 8 tasks, the largest EI in 7 of 8, and the best IA in 4 of 8.The evaluation covers eight constraint settings and reports TR, EI, IA, and ER for shopping.Source: §5.3 · p. 8, Table 2 · p. 8
• Behavior modeling reaches 86% validation accuracy, but incomplete or out-of-distribution action clips and weak observation-action balancing limit inference from raw RGB-D sequences.VLM helpers can underperform random or no-helper baselines by misidentifying targets and obstructing partners, while LLM+BM can still transport some objects.Source: §5.3 · pp. 8–9

6 Conclusion

• CHAIC combines four physically constrained agents with eight long-horizon indoor and outdoor tasks to evaluate social perception and cooperation.Source: §6 · p. 9
• Benchmarking planning- and learning-based baselines demonstrates the systematic evaluation that CHAIC can provide for embodied social intelligence.Source: §6 · p. 10
• The paper analyzes failure cases and provides insights for future embodied social intelligence research.Source: §6 · p. 10
• The simulated environment cannot capture all aspects of human physical constraints, and rule-based constrained-agent control limits behavioral diversity.The challenge is also not representative of every constraint users may face.Source: §6 · p. 10

A.1 Benchmark Usage

• CHAIC is available through its GitHub repository and official webpage.Source: §A.1 · p. 14

A.2 Comparison with Other Embodied Challenges

• CHAIC uniquely combines accessibility-focused evaluation, outdoor scenes with emergent events, and goal inference from observations.Table 3 compares these properties with other embodied challenges.Source: §A.2 · p. 14, Table 3 · p. 14

B.1 More Details on LLM+BM Helper

• The LLM+BM helper uses object segmentation, occupancy and semantic maps, an LLM-generated subgoal, and an executor that converts plans into navigation and exploration actions.Its navigation module uses A* with costs for free, unknown, occupied, and wall spaces, recalculating paths as the map changes.Source: §B.1 · pp. 14–15

B.2 Detailed Rules for Rule-Based Hierarchical Plan Helper (RHP)

• The rule-based hierarchical helper explores for objects, containers, and goals, prioritizes containers when time permits, and deposits carried items at the goal.It selects the nearest available object and uses different urgency thresholds as the episode ends.Source: §B.2 · p. 15

C.1 Detailed Prompt of Decision Module of LLM+BM Helper

• The LLM+BM decision prompt combines task description, self-information, other-agent information, task progress, semantic-map information, and available plans.The LLM selects a valid plan and supplies object IDs when required.Source: §C.1 · p. 15

C.1.1 Task Description

• Task descriptions state the task rules without explicitly exposing the constrained agent’s limitations.A corresponding example is shown in Figure 4.Source: §C.1.1 · p. 15, Figure 4 · p. 16

C.1.2 Self-Information

• Self-information records the helper’s prior action statuses, current position, and carried objects.A corresponding example is shown in Figure 5.Source: §C.1.2 · p. 15, Figure 5 · p. 16

C.1.3 Information about Other Agents

• Other-agent information records observed constrained-agent actions and statuses, held objects, and the last-seen position, with child positions added for shopping tasks.Source: §C.1.3 · p. 15, Figure 6 · p. 16

C.1.4 Task Progress

• Task progress records transported objects and elapsed frames, while semantic-map information records objects, containers, and goal locations with positions and heights.The prompt examples are shown in Figures 7 and 8.Source: §C.1.4 · p. 15, §C.1.5 · p. 16, Figure 7 · p. 17

C.1.6 Available Plans

• Available plans enumerate the actions the helper may take, with an example in Figure 9.Source: §C.1.6 · p. 17, Figure 9 · p. 17

C.2 Detailed Prompt of Decision Module of VLM Helper

• The VLM helper adds the last ten RGB images to a prompt otherwise similar to the LLM+BM helper.Source: §C.2 · p. 17

D.1 Detection Model for Object Detection

• The perception pipeline fine-tunes Mask R-CNN on 61K images covering 53 object types, achieving 94.4% mAP@50, while the action model uses 3,000 clips across six behaviors and reaches 86.1% top-1 validation accuracy.The action-recognition model supports behavior modeling because foundation models cannot directly discern actions in the wild.Source: §D.1 · p. 18, §D.2 · p. 18

E.1 Additional Metrics

• Completion Ratio measures the helper’s share of completed task work, while Standard Error of Transport Rate measures transport-rate uncertainty across test sets.Source: §E.1 · pp. 18, 20
• Figure 10 visualizes Transport Rate with 1-sigma standard-error bars for the baseline helpers.Source: Figure 10 · p. 19
• Table 4 reports Completion Ratio and Standard Error of Transport Rate as additional CHAIC metrics.Higher Completion Ratio generally indicates that the helper completed more task parts.Source: Table 4 · p. 19, §E.1 · p. 20
• Table 5 reports average Transport Rate and Efficiency Improvement for learning-based agents, including comparisons with agents operating without a helper.Source: Table 5 · p. 20

E.3 Comparison with Learning-Based Baselines

• Learning-based baselines perform similarly to the main agent without a helper in most tasks.The outdoor furniture task is an exception because the agents have different strength capacities and the task is relatively easy.Source: §E.3 · p. 20
• The authors attribute weak learning-based performance to the difficulty of vision-based reinforcement learning and limited online-data collection.Collecting rollouts of size 10^4 for each task requires one day on an NVIDIA A10G GPU because ThreeDWorld data collection is slow and difficult to parallelize.Source: §E.3 · p. 20
• The learning-based baselines include task-specific PPO reinforcement learning and SmartHelp with opponent modeling.The PPO agent uses RGB-D observations, semantic maps, agent state, and transport-oriented rewards; SmartHelp receives constrained-agent actions and statuses.Source: §E.3 · pp. 20–21
• SmartHelp’s opponent model achieved 93% goal-prediction accuracy and 0.02 mean squared error for constraint prediction.Source: §E.3 · p. 21

F.1 LLM+BM Helper Behaviors

• The LLM+BM helper improves transport decisions by selecting containers, inferring needed object categories, filling containers continuously, and transporting objects when efficient.Its behavior modeling and language reasoning support decisions based on observed constrained-agent behavior and task progress.Source: §F.1 · p. 21

F.2 VLM Helper Behaviors

• The VLM helper explores initially when it cannot see the constrained agent, then follows her after locating her.Following provides information about the objects she needs.Source: §F.2 · p. 22
• The VLM helper sometimes fails to infer the constrained agent’s needed objects and consequently continues following her.Source: §F.2 · p. 22
• After observing the constrained agent pick bread, the VLM helper can collect related items such as loaf bread and hamburger.Source: §F.2 · p. 22
• In the illustrated episode, the VLM helper transports one object at a time instead of using a container and both hands.The stated efficient strategy carries three objects in a container plus one additional object.Source: §F.2 · p. 22

G.1 Details of Shopping Task Generation

• Shopping episodes use six category-specific shops, randomly selected target objects, 11–13 targets, and containers placed at three randomly selected shops.The fixed goal is in front of the bicycle agent’s house, and agents start at least 0.5 meters from the nearest shop.Source: §G.1 · pp. 22–23

G.2 Details of Moving House Task Generation

• Moving-house episodes randomly place five furniture items between 50 and 900 weight units, with a truck about 10 meters from the placement area.The frail agent has strength 100, whereas the helper has strength 600.Source: §G.2 · p. 23

H.1 Broader Impacts

• The benchmark is intended to support research on accessible embodied helpers, but foundation-model baselines may produce malicious content or bad actions before deployment.The authors recommend mechanisms to detect such actions before real-world use.Source: §H.1 · pp. 23–24

H.3 LICENSE

• CHAIC is released under the MIT license, while ThreeDWorld uses BSD-2-Clause and MMAction2 and MMDetection use Apache-2.0 licenses.Source: §H.3 · p. 24

H.4 Resubmission Discussion

• The resubmission removes text access to constrained-agent action history and expands scenes and tasks, increasing task-relevant objects from about 20 to over 50.These changes respond to reviewer concerns about unnatural action-history access and insufficient object and task diversity.Source: §H.4 · p. 24

I.1 Motivation

• The dataset was created to research human-AI cooperation in embodied and realistic settings.Source: §I.1 · p. 24

I.2 Composition

• CHAIC contains 192 instances across eight tasks, with 12 training and 12 testing instances per task.Each instance is a command sequence used to initialize a ThreeDWorld scene and task.Source: §I.2 · pp. 24–25
• The dataset is a sample generated by an instance pipeline, uses a 1:1 training-testing split, and contains no labels, missing information, explicit instance relationships, or confidential data.Source: §I.2 · p. 25

I.3 Collection Process

• Data were automatically generated through an automatic scene-generation pipeline and visualized via ThreeDWorld, using random sampling; only authors were involved, and no institutional review was conducted.Collection occurred between 15 May and 10 June 2024.Source: §I.3 · p. 26

I.5 Uses

• The dataset supports embodied behavior recognition and embodied object detection, with no listed task restrictions or documented undesirable-use risks.Source: §I.5 · pp. 26–27

I.6 Distribution

• The dataset is publicly distributed through GitHub under the MIT License and currently has no DOI or third-party restrictions.The ThreeDWorld dependency uses stable version 1.12.27, and the complete dataset is available through the CHAIC GitHub repository.Source: §I.2 · p. 25, §I.6 · p. 27