Observe, Share, or Predict? Disentangling Partner Information in Embodied Multi-Agent Collaboration

1 Introduction

• Decentralized embodied evaluations often conflate perception, planning, communication, and partner reasoning, leaving unclear what partner modeling improves when agents exchange percepts but not intentions or plans.Prior work also aggregates team metrics, often presuming agents with identical planners occupy symmetric roles.Source: §1 · pp. 2–3
• The study independently varies partner-state observation, object Knowledge Exchange, and next-action prediction in a controlled PARTNR framework.It uses Decoupled Partial Observability, four planners, and 1,000 validation tasks.Source: §1 · p. 3
• OAS improves task quality for every planner, while KE cuts simulation steps by up to 25% with little change in success; OANA lowers success in 15 of 16 model-context pairs.OAS and KE are largely redundant for strong planners but complementary for the weakest; KE without OAS is the worst configuration measured.Source: §1 · p. 3
• Coordination gains from shared context, especially OAS, should not by themselves be read as evidence of mental-state attribution.OANA raises partner mentions to 55–100%, but some prediction accuracy reflects recognizing an action already underway.Source: §1 · p. 3

Implicit Multi-Agent Coordination in Embodied LLMs

• Dialogue-based LLM systems demonstrate coordination through deliberation, but do not fully represent embodied settings where messaging may be limited or unavailable.This study instead allows percept exchange but not intentions, plans, or task-level messages.Source: §2 · pp. 3–4

Embodied Planning and Partial Observability

• Decentralized embodied collaboration faces partial observability and redundant effort, while holistic evaluations make it difficult to isolate perception, planning, and collaborative reasoning.The study assumes oracle action execution and privileged perception to factor out low-level errors.Source: §2 · p. 4

Theory of Mind and Partner Modeling

• Whether text-only Theory-of-Mind abilities transfer to behavior remains contested, and this study does not directly probe mental-state attribution.It tests whether anticipating a collaborator’s next action improves joint behavior instead.Source: §2 · p. 4
• Unlike fixed-observation partner-model ablations, this work varies observable partner-state access and explicit action prediction independently, scoring predictions to distinguish anticipation from recognition of ongoing actions.This provides a behavioral test of whether explicit partner modeling translates into more efficient and successful collaboration.Source: §2 · p. 4

3 Method

• The study evaluates how OAS, KE, and OANA affect team outcomes and action coordination, including whether the two agents occupy different roles.Collaboration is judged by team outcomes; coordination concerns avoiding duplicated work, aligning in time, and dividing subtasks.Source: §3 · p. 4

3.1 Setting and Agents

• A simulated human and robot use separate LLM planners on PARTNR household tasks, with no messages between them beyond the channels defined by the information conditions.Agents know the house layout initially but must find the objects; oracle skills execute high-level actions as low-level simulator steps.Source: §3.1 · p. 5

3.2 Information Conditions

• Decoupled Partial Observability updates each agent’s world graph from its own field of view, while OAS adds the co-located partner’s visible state and current percepts.OAS exposes visible behavior, not internal state; when agents are in different rooms, the observation is “Acting in another room.”Source: §3.2 · pp. 5–6
• KE synchronizes known object positions and states when agents are co-located, using the more recent record without sender attribution or sharing intentions, plans, or task-level messages.The channel is lossless and instantaneous, and reads simulator state; this is an upper bound on an equivalent real protocol.Source: §3.2 · p. 6

3.3 Other Agent’s Next Action (OANA) Prediction

• With OANA enabled, an agent reasons about both agents’ next actions, predicts the partner’s high-level action, then selects its own action; predictions are compared with current and next partner actions.The union target accepts either action, while class-level matching masks instance IDs and can score at action-type or parameter granularity.Source: Figure 2 · p. 6, §3.3 · p. 7

4.1 Experimental Setup

• The evaluation uses 1,000 PARTNR validation tasks across 13 house scenes, with one run per configuration and caution advised for differences of a few points.The experiments use Habitat 3.0 and the decentralized_zero_shot_react_summary baseline configuration.Source: §4.1 · p. 7

4.2 Task Outcomes and Coordination

• OAS reduces Extraneous Effort for every planner, while KE cuts episode length by up to 25% with little change in Success Rate.OAS also yields the best Success Rate for Gemini (81.2%) and DeepSeek (83.6%); OAS + KE is the most efficient setting for three planners, while the channels are complementary for Llama 8B.Source: §4.2 · pp. 8–9
• OANA lowers Success Rate in 15 of 16 model×context pairs while raising Other-Agent Mention Ratio from 10–53% to 55–100%.Extraneous Effort falls in 11 of 16 pairs, so added failures are not mainly due to increased duplication.Source: §4.2 · pp. 9–10

4.3 Role Asymmetry Between the Agents

• The robot reaches its first object interaction earlier in 28 of 32 runs, while the human replans longer after the robot finishes in 25 of 32 runs.Task Offloading exceeds 50% in all 32 runs, although the robot goes idle first; the division of labor stays within 52.4–60.7% as Success Rate varies from 54.3% to 83.6%.Source: §4.3 · p. 11

4.4 Partner Prediction

• Shared context improves Action Type prediction more than conditional principal-parameter accuracy: gains average +14.7/+14.6 versus +4.2/+1.0 points for robot/human.The combined OAS + KE context gives the top Action Type accuracy for every model and agent, while predicting action arguments remains difficult.Source: §4.4 · pp. 11–12, Table 2 · p. 13

4.5 Robustness: Strict Target and Full Observability

• Under the strict next-action target, Action Type accuracy falls by 36.5% on average, and full observability still does not prevent OANA from lowering success.With full observability, OANA lowers Success Rate and Percent Complete in all four configurations; OAS raises union-target Action Type accuracy for Llama 70B but leaves strict accuracy unchanged or lower.Source: §4.5 · p. 14
• Full observability shortens episodes by 43% at the object-only baseline for both Llama planners, but most Extraneous Effort remains and OAS no longer improves task quality.About 94% of Extraneous Effort remains for Llama 8B and 86% for Llama 70B; the human's completion lag halves for Llama 70B and reverses for Llama 8B.Source: §4.5 · p. 14

5 Limitations

• Experiments use Habitat 3.0, PARTNR validation tasks, privileged perception, and oracle skills, which may limit generalization to real deployments.The study covers four planners and two-agent teams; its full-observability control covers only the two Llama planners, and the simulated human is also an LLM agent.Source: §5 · p. 14
• A single run per configuration makes small differences difficult to interpret, and the 4-bit Llama 70B results may not transfer to its full-precision model.API-served model versions and execution are outside the authors’ control, so identical settings may not produce identical outputs over time.Source: §5 · p. 14
• OAS effects may partly reflect shared perception, while OANA effects include added action descriptions and longer prompts that may scale differently on longer or more constrained tasks.Predictions are scored at the class level after instance IDs are masked.Source: §5 · pp. 14–15

6 Conclusions

• The controlled study varies Decoupled Partial Observability, OAS observation, KE, and OANA independently across four planners, reporting coordination per agent and for the team.The study is set in PARTNR and uses a full factorial grid.Source: §6 · p. 15
• OAS mainly improves task quality and KE mainly improves efficiency; they are largely redundant for strong planners but complementary for the weakest.OANA prediction lowers Success Rate in nineteen of twenty model×context pairs, and some accuracy reflects recognizing the partner’s ongoing action.Source: §6 · p. 15
• Under Decoupled Partial Observability, the robot usually finishes its share first, while shared context shortens the trailing period when the simulated human works alone.Role differences appear under lenient scoring; team averages can conceal asymmetries visible in per-agent reporting.Source: §6 · p. 15
• The authors characterize partner modeling as shallow and planner-dependent, and argue that behavioral tests should separate recognition from anticipation and its use.Prediction accuracy that credits an ongoing partner action conflates recognition with anticipation.Source: §6 · p. 15

Broader Impact Statement

• For assistive household robotics, the authors suggest observable partner state and shared object knowledge, caution against relying on next-action prediction, and recommend user-visible, switchable monitoring.They also recommend confirming predictions through observation and evaluating with per-agent metrics; the simulated human is an LLM agent, not a human participant.Source: Broader Impact Statement · p. 15

A Appendix

• Appendix A covers agent actions and implementation, action comparison, extended analyses, full-observability results, strict-target prediction, and experiment prompts.Its sections A.3–A.5 supplement analyses from Sections 4.2–4.4.Source: A Appendix · p. 18

A.1 Implementation details for the ReAct agents

• The Llama planners ran locally on A100 and A40 GPUs, while Gemini 3.1 Flash Lite and DeepSeek V3.2 were accessed through OpenRouter.Some later runs used vLLM rather than Hugging Face Transformers with the same model weights and decoding settings.Source: §A.1 · pp. 18–19
• 299 seconds per episode was the fastest reported configuration, while 1008 seconds was the slowest; these runtimes are not directly comparable across configurations.With OANA prediction, episodes generally took 9% to 60% more runtime, partly due to logging, metric computation, and additional generated tokens.Source: §A.1 · p. 19
• Despite increased runtime, OANA prediction often reduces planning cycles and sometimes also reduces simulation steps.Source: §A.1 · p. 19

A.1.1 Skill API Library

• The robot cannot use the state-changing tools Clean, Fill, Pour, PowerOff, or PowerOn.The library also includes object handling, navigation, exploration, and task-completion actions.Source: §A.1.1 · p. 19
• The library also provides navigation, room exploration, opening and closing, cleaning, filling, pouring, power control, waiting, and task completion.Explore searches a specified room; Wait leaves the agent idle, and Done signals that the task is finished.Source: §A.1.1 · p. 19
• Pick, Place, and Rearrange support object handling and placement, with Place and Rearrange accepting spatial relations and optional next_to constraints.Pick limits the agent to one held object at a time; Place requires the object to be held, while Rearrange performs pick and place automatically.Source: §A.1.1 · p. 19

A.1.2 Constraint Generation

• Constrained generation restricts token sampling to sequences consistent with the specified grammar.For each Llama call, the grammar permits only valid tool calls over observed entities.Source: §A.1.2 · p. 20
• The method applies only to locally run Llama models because it requires access to output tensors.API-accessed models do not use this type of constraint generation.Source: §A.1.2 · p. 20
• Object, furniture, and room rules update dynamically from the agent’s current world graph, limiting calls to entities it knows.The human agent’s grammar includes all actions, while the robot’s omits object-state modification actions.Source: §A.1.2 · p. 20

Tool Call Grammar

• The base grammar enumerates tool-call forms for navigation, object manipulation, state changes, exploration, waiting, and task completion.Navigation targets can be furniture, rooms, or objects; placement and rearrangement include spatial relations and optional constraints.Source: Tool Call Grammar · p. 20
• For OANA prediction, object parameters use object classes rather than instance IDs, which become known only after exploration and observation.The set of object classes is available from the start of each episode.Source: Tool Call Grammar · p. 20
• API-accessed Gemini and DeepSeek models cannot use this constrained decoding and can hallucinate action-parameter entities absent from agents’ world graphs.The limitation applies because constrained decoding requires locally available output tensors.Source: Tool Call Grammar · p. 20

A.2 Action comparison

• OANA comparisons assess action type, principal parameter, all parameters, and combinations of action type with parameters.The examples ignore entity instance IDs when comparing predictions with target actions.Source: §A.2 · p. 21
• Examples distinguish matching action types from mismatches in action type, principal parameter, or full parameter sets.For example, Place predictions can match on action type and cup while differing in placement constraints; Navigate[book] does not match Pick[book].Source: §A.2 · p. 21
• OAS mainly improves task quality, while KE mainly improves execution efficiency; the best coordination uses both without OANA prediction.OANA raises partner mentioning but lowers Success Rate in all but one configuration.Source: §A.2 · p. 22, Table 3 · p. 22

• The section provides the complete metrics for Sections 4.2–4.4 and additional analyses summarized in the main text.Source: §A.3 · p. 22

A.3.1 Task Outcomes and Coordination

• OANA improves robot Action Success under prediction for DeepSeek in every setting, peaking at 93.9%, but its human agent declines in three of four settings.For the Llama 8B, Extra Planning nearly doubles from 20.3% to 39.0% for the robot without shared context.Source: §A.3.1 · p. 23
• At each model’s best setting, Extraneous Effort declines monotonically with planner strength, from 58.5% for Llama 8B to 26.7% for DeepSeek.Task Offloading remains within 52.4–60.7% across all 32 runs.Source: §A.3.1 · p. 23
• The Percent Complete–Success Rate gap widens under prediction in 15 of 16 model×context pairs, but average completion of failed episodes rises in only 8 of 16 pairs.That failed-episode measure ranges from 38% to 53% across all 32 runs.Source: §A.3.1 · p. 23
• KE without OAS raises Extraneous Effort for both Llama planners, while OAS alone lowers it substantially further for Gemini and DeepSeek.For the Llama 8B, KE raises Extraneous Effort from 62.3% to 69.6%; the authors attribute this to shared object knowledge revealing targets without showing that the partner is already acting on them.Source: §A.3.1 · p. 24
• Adding OANA partly repairs the Llama 8B’s KE-only configuration, reducing Simulation Steps from 4914 to 4121 and Extraneous Effort from 69.6% to 62.7%.Success Rate rises from 54.3% to 56.2%, a difference smaller than the stated standard error of about 2.2 points.Source: §A.3.1 · p. 24
• Gemini reaches 81.1% Success Rate with 8.0/8.5 planning cycles per episode, whereas Llama 8B reaches 62.3% with 21.5/19.8 cycles.The passage cautions that planning volume is not a proxy for competence; Gemini uses roughly 176 simulator steps per high-level action, versus 92 for Llama 8B.Source: §A.3.1 · p. 24
• For the three stronger planners, KE alone uses fewer Simulation Steps than OAS alone, while OAS alone yields higher Success Rate in three of four models.The Llama 70B is the exception: Success Rate is 76.2% with KE alone versus 76.9% with OAS alone.Source: §A.3.1 · p. 24

A.3.2 Role Asymmetry Between the Agents

• Task Offloading stays within 52.4–60.7% across all 32 runs, while Success Rate ranges from 54.3% to 83.6% and Simulation Steps from 2794 to 5486.The performance differences are consequently associated with temporal alignment, duplicated actions, and earlier object acquisition rather than a substantial change in work allocation.Source: §A.3.2 · p. 25
• The robot reaches its first object interaction earlier than the human in 28 of 32 runs; the mean Exploration Efficiency gap is 482 simulator steps for Llama 8B and 18 for Gemini.The four exceptions are Gemini’s OANA configurations, where the human is marginally faster.Source: §A.3.2 · p. 25
• The human replans more than the robot after it stops acting in 25 of 32 runs, but OAS + KE narrows the human–robot Extra Planning gap in every model without prediction.The reductions from object-only observation to OAS + KE are 14.5 to 11.4 points for Llama 8B and 9.4 to 5.6 for Gemini.Source: §A.3.2 · p. 25
• Shared context reduces human Extra Planning from the object-only baseline to OAS + KE in every model, with larger absolute reductions than for the robot.The passage links this to episode length being determined by the later-finishing agent, while noting that shared context does not change which agent can finish the task.Source: §A.3.2 · p. 25
• Under OANA, DeepSeek’s robot Extra Planning roughly doubles at baseline, from 20.3% to 39.0%, and the human-minus-robot gap becomes negative in all four contexts.For Llama 8B, the gap widens from +14.5 to +22.1 points at baseline.Source: §A.3.2 · p. 25
• Without prediction, the human has a higher Other-Agent Mention Ratio in all 16 runs; OANA flips that ordering for Gemini and Llama 70B in all four contexts each.The passages interpret mention frequency as a behavioral signal, not a quality measure.Source: §A.3.2 · p. 26
• Robot Action Success is at least as high as human Action Success in 28 of 32 runs, and OANA’s change is less favorable for the human in 13 of 16 pairs.The authors connect the human’s larger action space and restrictive preconditions with more ways for a misprediction to invalidate a human action.Source: §A.3.2 · p. 26
• Planning-cycle asymmetry depends on the planner: the robot uses more cycles in all 8 runs for each Llama model, but fewer in all 8 Gemini runs.Per-agent metrics are needed to see this reversal; agent-averaged Gemini cycles are nearly identical.Source: §A.3.2 · p. 26

A.3.3 Partner Prediction

• OAS + KE yields the top Action Type accuracy in all eight model×agent combinations, strictly in seven and tied in one.Relative to the object-only baseline, Action Type accuracy rises by +17.0/+14.0 points for Llama 70B, +14.4/+16.3 for Gemini, and +21.1/+23.5 for DeepSeek.Source: §A.3.3 · p. 26
• OAS and a synchronized object world graph can each provide information for predicting a partner’s next action, despite neither channel conveying intentions, plans, or language.The channels may overlap because observing a partner picking an object and learning that the object appeared or moved in the shared graph can support the same inference.Source: §A.3.3 · p. 26
• OAS is the stronger single channel for three of four planners, but OAS and KE are substantially redundant rather than additive in seven of eight model×agent combinations.For Gemini, separate channel gains total +22.1/+27.8 points, while the combination delivers +14.4/+16.3.Source: §A.3.3 · p. 26
• Moving from object-only observation to OAS + KE improves Action Type accuracy by +14.7 points for the robot and +14.6 for the human, but conditional principal-parameter accuracy by only +4.2 and +1.0.Even in the best-performing cell, roughly a third of correctly typed predictions carry the wrong principal parameter.Source: §A.3.3 · p. 27
• For Llama 8B, OAS raises Action Type accuracy from 31.8% to 36.1% but lowers conditional principal-parameter accuracy from 54.2% to 49.2%, leaving joint accuracy nearly unchanged.The passage attributes this pattern to improved coarse predictions alongside weaker object-level reasoning for filling in arguments.Source: §A.3.3 · p. 27
• Whether object knowledge alone suffices for prediction depends on the planner: KE alone matches or exceeds OAS alone for Gemini’s Action Type gains, while Llama gains less from KE.The passage offers self-projection from a synchronized object graph as a plausible account, and says this distinction emerges in the KE-only condition.Source: §A.3.3 · p. 27
• Among predictions with correct action type, only 52.2% have a fully correct parameter set on average, whereas action type is correct in 84.8% of parameter-correct predictions.The passage states that the asymmetry quantifies the parameter bottleneck rather than establishing it independently.Source: §A.3.3 · p. 27
• Task-solving strength and partner-prediction strength diverge: DeepSeek has the best task outcomes but its robot has the weakest baseline Action Type prediction accuracy, 27.9% versus 31.8% for Llama 8B.The authors also report that DeepSeek mentions the other agent in essentially 100% of OANA replanning steps while producing some of the least accurate predictions.Source: §A.3.3 · pp. 27–28

Within the API-accessed planners, prediction accuracy tracks the size of the success-rate

• For API-accessed planners, DeepSeek’s prediction penalty fell as baseline Action Type accuracy rose, while Gemini’s penalty tracked OAS availability more closely; this pattern did not extend to Llama.The authors present this as an account for API-accessed models, not a general law, because the design did not directly intervene on prediction quality.Source: Within the API-accessed planners,… · p. 28
• Under union scoring, the human generally led in Action Type prediction while the robot led in conditional parameter accuracy, so near-symmetric joint scores concealed distinct prediction strengths.The passage attributes the human’s action-type lead partly to the smaller target action space, while the robot’s conditional parameter lead is linked to action-repertoire structure; these patterns are specific to union scoring.Source: Within the API-accessed planners,… · pp. 28–29

A.4.1 Task Performance and Coordination under Full Observability

• 43–53%: Full observability cut Simulation Steps by 43% at baseline and 50–53% with OAS, while Llama 70B Success Rate remained essentially unchanged and Llama 8B gained only 2.4 points.For the Llama 70B, full observability yielded 76.9% Success Rate versus 77.5% under partial observability with OAS + KE; the Llama 8B rose from 62.3% to 64.7%.Source: §A.4.1 · pp. 30–31
• OAS still reduced Extraneous Effort, but OANA lowered Success Rate in all four full-observability configurations and raised Other-Agent Mention Ratio from 12–32% to 51–78%.Most redundant work survived perfect object knowledge: Extraneous Effort remained 54.8% for Llama 8B and 31.5% for Llama 70B at their best full-observability settings.Source: §A.4.1 · pp. 31–32

A.5.1 Next Action Target for Decoupled Partial Observability

• 16.0 points: The strict next-action target reduced Action Type accuracy by an average of 16.0 points, while average all-parameter accuracy conditional on type was nearly unchanged at 52.2%→52.4%.The union target also credited readable ongoing actions, such as a partner already walking, so its scores mix state recognition with genuine anticipation.Source: §A.5.1 · p. 34
• With the strict target, OAS + KE improved joint prediction for the three stronger planners but yielded −0.2/+0.8 points for Llama 8B (R/H), and the human Action Type advantage disappeared.The robot matched or led in 14 of 16 cells under the strict target, and the robot’s conditional parameter advantage survived clearly only for Gemini.Source: §A.5.1 · pp. 34–35
• Conditional prediction exposes a parameter bottleneck: correctly typed predictions had fully correct parameters 52.4% of the time under the strict target, while correct parameters implied a correct type 85.7% of the time.These conditionals were close to union-target values (52.2% and 84.8%), despite a 16-point difference in unconditional type accuracy; aggregate robot–human symmetry also masked planner-specific asymmetries.Source: §A.5.1 · p. 35
• 35.4/33.3% was the best strict-target Action Type accuracy in partial observability, while the Llama 70B’s all-parameter accuracy peaked at 23.2/22.2%.The 8B never exceeded 26.2% on Action Type or 10.4% on the strictest metric; OANA lowered Success Rate in 15 of 16 model×context pairs, consistent with predictions that were mostly incorrect.Source: §A.5.1 · p. 35

A.5.2 Action Prediction Evaluations for Full Observability

• Full observability gave Llama 70B its strongest predictions: Action Type reached 65.6/70.8% with OAS under the union target and 43.8/45.9% without OAS under the strict target.These exceed the best Decoupled Partial results of 55.5/55.8% under union and 35.4/33.3% under strict; full observability without OAS also beat the best partial configuration with OAS and KE in all eight R/H comparisons.Source: §A.5.2 · p. 36, Table 10 · p. 37, Table 11 · p. 38
• With full observability, OAS raised Llama 70B union-target Action Type by +8.6/+6.6 points but changed strict-target accuracy by −0.3/−2.8 points.OAS reveals the partner’s current state, which the union target credits but the strict next-action target does not; under partial observability, OAS also supplied otherwise missing object information.Source: Table 10 · p. 37, §A.5.2 · p. 37, Table 11 · p. 38
• Full observability largely resolved parameter prediction for Llama 70B but left action-type prediction weak: strict-target Action Type remained 43.8/45.9% without OAS and 43.5/43.1% with it.For the 8B, full observability improved conditional parameter accuracy but reduced robot-side Action Type accuracy; perfect object knowledge resolves object ambiguity, not which plausible action the partner will choose.Source: §A.5.2 · pp. 36–38, Table 11 · p. 38

A.6 Prompts for the agents

• The OANA prompt asks agents to predict the partner’s next action as well as choose their own, whereas the baseline prompt only asks for their next action while noting partner activity.The OANA instruction describes the agents as collaborating and presents both agents’ possible actions; role descriptions and available tools vary by agent role.Source: Decentralized | Zero-Shot | ReAct |… · pp. 39–41, Agent Role Description · p. 41, Tool Description · p. 42

World Description Example

• The summarized scene description appends observed object descriptions after each action, while agents can move between rooms and discover objects along the way.This describes the scene representation used in the experiments.Source: World Description Example · p. 43

Scene Summary Example Object Only Baseline

• With the object-only baseline, the agent explores when no objects are known, then begins task work after finding the required items.In the example, it finds the candle, candle holder, and plant container in the hallway.Source: Scene Summary Example Object Only… · pp. 43–44

Scene Summary Example With OAS Observation and OANA Prediction

• With OAS and OANA enabled, the agent predicts its partner will explore the living room, chooses the kitchen to avoid duplicated exploration, then navigates toward found task objects.The example also shows the agent observing the partner walking in the living room and predicting its next action as Explore[bedroom_2].Source: Scene Summary Example With OAS… · pp. 44–45