Source-linked AI summary
Watch, Infer, Coordinate: Inferring Robot Partner Constraints for Zero-Shot Coordination
Suyu Ye, Zheyuan Zhang, Vaishnav Tadiparthi, Hossein Nourkhiz Mahjoub, Ehsan Moradi Pari, Tianmin Shu, Homanga Bharadhwaj, Nakul Agarwal
TL;DR
Helpers need to infer a partner’s unknown physical action constraints from prior coordination, despite ambiguity about what the partner could have done and possible teammate compensation. Watch, Infer, Coordinate scores candidate constraints using both agents’ observed behavior and uses the estimate on a new task; across three settings, it improves inference and coordination, approaching an oracle with the true constraints.
Problem
A helper’s partner may have unknown physical action constraints, and prior coordination does not directly reveal which actions were infeasible rather than unnecessary.
Source: §1 · p. 2Method
Watch, Infer, Coordinate scores candidate constraints using both agents’ observed actions, then uses the inferred capability to coordinate with the same partner on a new task.
Source: §1 · p. 2, §3.1 · p. 5Results
Takeaways & Limitations
In the evaluated carrying settings, inferred partner constraints support zero-shot coordination on a new task with performance close to an oracle given the true constraints.
Source: §6 · p. 9The study assumes a known behavioral model, goals and dynamics, full observations, and a finite constraint set; relaxing these assumptions is left as future work.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Robots operating in the physical world will increasingly need to coordinate with other robots, particularly in manipulation tasks where an object may be too large or heavy for a single robot to carry alone. Physical limitations caused by hardware degradation or actuator faults can restrict the actions a robot can reliably execute, yet these limitations may be unknown to its partner. We study whether a helper can infer a robot partner's physical constraints from observing it coordinate with another robot, then use the inferred capability to coordinate with the same partner on a new task. This is difficult because a demonstration shows what the constrained robot did, but not what it could have done. In physically coupled tasks, the other robot may also compensate for its limitations, making those limitations difficult to identify from the constrained robot's behavior alone. Our key insight is that these constraints shape the joint behavior of the team, making the actions of both robots informative about the constrained partner's capability. We introduce Watch, Infer, Coordinate, a benchmark spanning three physically coupled manipulation settings, together with an inference approach that scores candidate constraints using observed joint behavior. Across all three settings, our method substantially improves constraint inference and zero-shot coordination, approaching an oracle with access to the true constraints.
1 INTRODUCTION
The paper asks whether a helper can infer a partner’s persistent low-level physical constraints from prior coordination and use them on a new task. It introduces Watch, Infer, Coordinate and reports improved inference and zero-shot coordination across three settings, approaching an oracle with the true constraints.
1 INTRODUCTION
- A helper must infer a partner’s unknown low-level physical constraints, which can arise from hardware degradation or actuator faults and restrict reliably executable actions.The target includes limits such as position or velocity limits on individual joints.Source: §1 · p. 2
- Existing methods do not directly address inferring persistent physical action constraints from prior coordination and using them with the same partner on a new task.The helper has not observed the partner coordinating on that new task before deployment.Source: §1 · p. 2
- Inference is difficult because an unobserved action may be infeasible or unnecessary, while the demonstrator may compensate for the constrained partner.The paper’s key insight is that constraints shape joint team behavior, so actions from both agents can inform inference; these constraints may remain useful when the task changes.Source: §1 · p. 2
- Watch, Infer, Coordinate pairs three increasingly complex joint-carrying settings with an inference method that scores candidate constraints using both agents’ observed actions.Demonstrations are obstacle-free; test tasks introduce obstacles and require different coordinated motions.Source: §1 · p. 2
- The approach improves constraint inference and zero-shot coordination across all three settings, approaching an oracle given the true constraints.The paper formulates inference of persistent low-level partner constraints from prior coordination and evaluates their use on a new task.Source: §1 · p. 2
2 RELATED WORKS
Prior work addresses partner goals, behavior, or capabilities, and studies coordination and constraint inference in several settings. The paper distinguishes its focus as inferring a partner’s persistent low-level physical action limits from physically coupled team demonstrations and using them on a new task.
Comparison with Related Work
- Table 1 defines comparison dimensions including multi-agent trajectories, low-level partner constraints, physically coupled manipulation, and transfer to a new task.Its caption also defines prior-trajectory inference and continuous control.Source: Table 1 · p. 3
Physically Coupled Robot Collaboration
- Physically coupled robot collaboration spans human-robot and robot-robot teams, but the cited methods do not explicitly infer a particular partner’s persistent physical action constraints.The passage describes approaches including intent inference, language-based motion-plan revision, decentralized controllers, planning, reinforcement learning, diffusion policies, and VLM/VLA policies.Source: §2 · p. 3
Partner-Aware Assistance
- Partner-aware assistance infers goals, beliefs, strategies, or capabilities, while this work targets low-level physical action constraints from demonstrations with another agent.Examples include semantic capabilities such as reachable height or liftable weight and task-level or symbolic capability representations.Source: §2 · p. 3
Constraint Inference from Demonstrations
- Demonstration-based constraint inference includes methods for environmental constraints and agent-centric restrictions, while this work considers physical limits obscured by a teammate’s compensation.The passage distinguishes prior multi-agent work on interaction constraints from inference of a particular partner’s physical limits.Source: §2 · p. 4
3 WATCH, INFER, COORDINATE: PROBLEM SETUP AND BENCHMARK
The setup has a helper observe a constrained agent coordinating with a demonstrator, then replace the demonstrator on a new task. The benchmark and inference approach use team behavior to estimate constraints that remain fixed across tasks.
3.1 PROBLEM SETUP
- The helper observes the constrained agent and demonstrator’s states and actions, then replaces the demonstrator to coordinate with the same agent on a new task without prior observation there.The partner’s constraints remain fixed while the task and required coordinated motions change.Source: §3.1 · p. 5
3.2 WHY COORDINATED BEHAVIOR IS INFORMATIVE
- The demonstrator’s compensatory actions can reveal a constraint that is difficult to identify from the constrained agent’s actions alone.Both agents’ actions are informative because the constrained agent’s capability shapes the team’s joint behavior.Source: §3.2 · p. 5
3.3 WATCH, INFER, COORDINATE BENCHMARK
- Watch, Infer, Coordinate evaluates three joint-carrying settings with obstacle-free demonstrations and obstacle-blocked test tasks requiring different motions.The same fixed demonstration tasks are used across constraint cases, and they require a range of coordinated motions.Source: §3.3 · p. 5
- The behavioral model assumes boundedly rational agents pursuing a shared task objective, scored by a known transition model.The action-pair score evaluates how well the joint action completes the carrying task from the current state.Source: §3.3 · p. 5
- A regularized shared objective defines mutually dependent policies, so restricting the constrained agent’s feasible actions also changes the demonstrator’s expected response.Together with each setting’s physical feasibility rules, the policies define the distribution used to generate demonstrations.Source: §3.3 · p. 5
- For each candidate capability, the method scores the observed joint actions, combines that score with a prior, normalizes candidate weights, and plans using the resulting belief.At test time, the helper replaces the demonstrator while the constrained agent retains its capability and behavioral model.Source: §3.3 · pp. 5–6
- The benchmark progresses from 2D rod carrying to fixed-base and mobile dual-UR5 carrying, with 2,665, 172, and 475 candidate capabilities, respectively.The settings add joint position and velocity limits, then constraints on arm and base motion.Source: §3.3 · p. 6
4 METHOD
The method infers a partner’s capability by scoring candidate constraints against observed joint behavior, then uses the resulting belief to plan compatible actions on a new task. It uses learned models to make capability scoring tractable in more complex settings.
4.1 OVERVIEW
- Candidate capabilities are scored by how well they predict both agents’ observed actions, and the scores combine with priors to form a belief over the constrained agent’s capability.This uses observed joint behavior across states and demonstrations.Source: §4.1 · p. 6
4.2 COMPUTING CAPABILITY SCORES
- For more complex settings, learned models estimate candidate fit offline, with scoring approximations varying across the 2D rod, fixed-base dual-UR5, and mobile dual-UR5 settings.The method predicts action values for the 2D rod, step-level score contributions for fixed-base dual-UR5, and demonstration-level scores with arm and base components for mobile dual-UR5.Source: §4.2 · p. 7
4.3 PLANNING WITH THE INFERRED CAPABILITY
- At test time, MPC with CEM optimizes the helper’s actions by rolling out partner responses restricted to actions feasible under the inferred capability.The planner searches for coordination strategies compatible with the partner’s inferred physical capability, without requiring demonstrations of the new task.Source: §4.3 · p. 7
5 EXPERIMENTS AND RESULTS
Across three joint-carrying settings, the method improves capability inference and zero-shot coordination on new tasks with obstacles. The experiments also test the value of observing both agents and illustrate how inferred constraints change planning.
5.1 EXPERIMENTAL SETUP
- Inference uses 1, 2, 4, or 8 demonstrations; planning comparisons include four baselines, and report task success and SNA as percentages over three random seeds.For zero-shot coordination, inference-based methods use N = 8 demonstrations; failed episodes receive an SNA of zero.Source: §5.1 · pp. 7–8, §5 · p. 7
5.2 CONSTRAINT INFERENCE
- Our method outperforms CE-CM-Div in constraint inference across all three settings and demonstration budgets; mobile dual-UR5 error falls from 0.190 at N = 1 to 0.009 at N = 8.Table 2 reports normalized Hamming distance, for which lower is better.Source: §5.2 · p. 8, Table 2 · p. 8
- Using both agents’ behavior roughly halves fixed-base dual-UR5 inference error, reduces mobile dual-UR5 error at every budget, and helps in 2D rod carrying with one or two demonstrations.The ablation compares inference using both agents with inference using only the constrained agent’s behavior.Source: §5.2 · p. 8
5.4 QUALITATIVE EXAMPLES
- Figure 3 shows successful trajectories across all three settings and illustrates that incorporating an inferred constraint can replace an infeasible pose with a feasible alternative.The figure presents trajectories over time and a test-time planning example.Source: §5.4 · pp. 8–9, Figure 3 · p. 9
5.3 ZERO-SHOT COORDINATION
- On obstacle tasks, our method achieves 92.6%, 85.2%, and 63.0% success in the 2D rod, fixed-base dual-UR5, and mobile dual-UR5 settings, approaching Oracle CEM’s 96.3%, 88.9%, and 70.4%.Table 3 reports planning performance on obstacle-free controls and new tasks with obstacles; higher success and SNA are better.Source: §5.3 · p. 8, Table 3 · p. 9
6 CONCLUSION AND LIMITATIONS
The paper formulates inferring a partner’s low-level physical constraints from prior coordination and using them for zero-shot coordination, and introduces a benchmark spanning three physically coupled carrying settings. Experiments show improved coordination near an oracle, while the study assumes known models, goals, and dynamics, full observations, and a finite constraint set.
6 CONCLUSION AND LIMITATIONS
- The benchmark and joint-behavior inference approach improve zero-shot coordination, achieving performance close to an oracle with the true constraints.The work formulates inferring persistent low-level constraints from prior multi-agent coordination and using them on a new task.Source: §6 · p. 9
- The study assumes a known behavioral model, goals and dynamics, full observations, and a finite constraint set to isolate inference and coordination.Relaxing these assumptions is identified as an important next step toward broader generalization.Source: §6 · p. 9
- Direct baseline comparisons are limited because existing approaches use different capability representations or observations.The authors plan to release benchmark code and evaluation scripts to support broader evaluations, including future extensions to heterogeneous teams and human-robot coordination.Source: §6 · p. 10
A.1.1 BEHAVIORAL MODEL
The behavioral model predicts the demonstrator’s and constrained agent’s interdependent actions under a candidate capability. It iteratively updates boundedly rational policies, then uses the resulting policies to define a capability-conditioned joint-action distribution.
A.1.1 BEHAVIORAL MODEL
- A candidate capability determines the constrained agent’s feasible actions, and the behavioral model predicts both agents’ actions under that capability.The agents favor actions leading to good joint outcomes without assuming perfectly optimal action selection.Source: §A.1.1 · pp. 12–13
- The shared regularized objective models both agents as boundedly rational, with temperatures controlling how strongly they favor higher-value actions.Smaller temperatures concentrate probability on the highest-value actions; larger temperatures allow a wider range.Source: §A.1.1 · p. 13
- Because each agent’s useful actions depend on the other’s policy, the model alternately updates their soft responses from uniform starting policies.The updates move each policy partway toward its computed response; η = 0.5.Source: §A.1.1 · p. 13
- The final policies define a capability-conditioned joint-action distribution, and changing the candidate capability can also change the demonstrator’s predicted behavior.The model computes final soft responses using each agent’s final policy for the other.Source: §A.1.1 · p. 13
A.1.2 CAPABILITY INFERENCE DETAILS
Across the three settings, candidate capabilities are scored by how well they explain observed coordination, with physically inconsistent candidates removed before prior-weighted scores are normalized. The scoring procedure differs by setting: it uses predicted action likelihoods for the rod task and learned critics for the UR5 tasks.
A.1.2 CAPABILITY INFERENCE DETAILS
- Across settings, inference scores candidate capabilities against observed coordination, excludes physically inconsistent candidates, then combines remaining scores with a prior and normalizes them.The settings differ in how candidate scores are computed.Source: §A.1.2 · p. 14
- In the 2D rod setting, predicted distributions over 64 force directions score the observed actions of both agents, with a 0.02 uniform mixture preventing zero likelihood for an unusual action.The method sums log probabilities over steps and demonstrations, marginalizing over 49 possible temperature pairs.Source: §A.1.2 · p. 14
- In the fixed-base UR5 setting, a candidate-conditioned critic scores observed steps using states, both agents’ actions, resulting motion, and capability; consistency checks and a sparsity prior precede normalization over 172 candidates.Source: §A.1.2 · p. 14
- In the mobile setting, a recurrent critic scores complete sequences, separately scores arm and base-motion constraints, and normalizes over 475 composite capabilities after physical-consistency filtering.Source: §A.1.2 · pp. 14–15
A.1.3 PHYSICAL CONSISTENCY AND TEST-TIME PLANNING
At test time, CEM-MPC uses inferred capability information to predict feasible coordinated motions, score progress toward the goal, and replan after executing the first selected action. Motion checks enforce the setting’s physical constraints, while CEM-based comparisons hold planner and budget constant and vary the capability information provided.
A.1.3 PHYSICAL CONSISTENCY AND TEST-TIME PLANNING
- CEM-MPC predicts the constrained agent’s feasible response to candidate helper sequences, scores resulting motion for goal progress, executes the first selected action, and replans.Rod and fixed-base planning accounts for uncertainty over candidate capabilities; mobile planning uses the highest-scoring capability.Source: §A.1.3 · p. 15
- Proposed motions violating capability, robot kinematics, workspace, or collision constraints are treated as infeasible rather than modified.The rod setting checks collisions; the UR5 settings also check capability compatibility, kinematics, and workspace constraints.Source: §A.1.3 · p. 15
- CEM-based methods share the same planner and planning budget, differing only in whether they use inferred capability, no demonstration-based capability information, or the true capability.The methods are ours, CE-CM-Div, Capacity-blind CEM, and Oracle CEM, respectively.Source: §A.1.3 · p. 15
A.2 EXPERIMENT IMPLEMENTATION DETAILS
The implementation details specify setting-specific capability models and baselines, then evaluate inference and new-task coordination under fixed demonstration and trial protocols.
A.2 EXPERIMENT IMPLEMENTATION DETAILS
- Each setting uses N = 1, 2, 4, 8 demonstrations for inference and N = 8 for coordination, with three fixed evaluation constraints.Source: §A.2 · p. 15
A.2.1 2D ROD CARRYING
- The rod task tests 2,665 angular constraints, evaluates 30°, 45°, and 65° ranges, and uses a neural network to predict both agents’ action values.Source: §A.2.1 · p. 15
A.2.2 FIXED-BASE DUAL-UR5 CARRYING
- Fixed-base carrying considers 172 capabilities formed from joint restrictions; a neural network scores each candidate at observed steps and aggregates scores across demonstrations.Source: §A.2.2 · pp. 15–16
A.2.3 MOBILE DUAL-UR5 CARRYING
- Mobile carrying combines 25 base and 19 arm constraints into 475 capabilities, scored by a recurrent model that processes whole demonstrations.Source: §A.2.3 · p. 16
A.2.4 BASELINE IMPLEMENTATIONS
- Baselines compare trajectory matching, fixed-prior planning, oracle capability information, direct behavioral cloning, and inference from partner-only behavior.Source: §A.2.4 · p. 16
A.2.5 EVALUATION PROTOCOL
- Inference uses normalized Hamming distance, where lower is better; coordination reports success and SNA, which is zero for failures and rewards fewer actions on successes.Source: §A.2.5 · pp. 16–17
- Each method receives 27 obstacle-task and 9 obstacle-free trials per setting; CEM-based methods share setting configurations selected for approximately comparable runtime.Source: §A.2.5 · p. 17, Table 4 · p. 17