PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use

Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, Ruihe An, Peiyao Xu, Hesen Yang, Runda Liu, Chang Gong, Zhijun Cao, Zechen Bai, Wenzheng Zeng, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Kevin Qinghong Lin, Mike Zheng Shou

arXiv:2610.30108v1cs.ROcs.AIcs.CVcs.MAcs.MM

TL;DR

Research asks what changes when multimodal agents move from computer-use to robot-use, where physical interaction makes the task loop less predictable and forgiving. The paper introduces PAPAV, a five-function framework for comparing multimodal agents, robotic systems, and embodied agents. Its analysis finds that physical constraints shift more decisions into interaction, while benchmark evaluation explicitly centers on Act and leaves Anticipate and Verify largely hidden behind task success.

  • Problem

    Existing taxonomies describe computer-use and embodied systems within their respective settings but do not show how the same function changes when actions have physical consequences.

    Source: §1 · p. 2

  • Method

    PAPAV compares multimodal agents, robotic systems, and embodied agents through five task-loop functions defined by their roles rather than their implementation.

    Source: §1 · p. 3, §6 · p. 33

  • Results

    Physical embodiment preserves the task loop but leaves more choices to interaction under partial observability, limited simulatability, irreversibility, real-time coupling, and unverifiable outcomes; benchmark evaluation centers on Act, with Anticipate and Verify largely hidden behind task success.

    Source: §6 · p. 33

  • Takeaways & Limitations

    PAPAV offers a common basis for designing reliable physical agents and measuring progress across the full interaction loop, alongside end-to-end task performance.

    Source: §6 · p. 33

    In the reviewed long-horizon and household-manipulation evaluations, aggregate measures can conflate error types and do not independently establish task planning or agent verification.

    Source: §4.3.3 · p. 26

Abstract

from arXiv · show

Multimodal agents (MMAs) sustain interaction through reasoning, memory, tools, and feedback, most visibly as computer-use agents, while robotic systems (RSs) couple sensing and actuation under physical dynamics. We define multimodal embodied agents (MMEAs) as goal-directed systems that couple multimodal task reasoning with physical action and revise decisions from the resulting feedback. This raises a central question: what changes when a multimodal agent moves from computer-use to robot-use? We introduce PAPAV, a unified capability-centric framework of five recurring functions: Perceive the current state, Anticipate action effects, Plan a feasible course, Act through an interface or body, and Verify the outcome. Defined by function rather than architecture, these capabilities let PAPAV identify what all three share and then compare MMEAs with MMAs and with RSs. Five physical constraints, from partial observability to unverifiable outcomes, leave fewer choices fixed in advance and less room to reverse errors. Across 64 benchmarks, explicit evaluation centers on Act; only 5 assess Anticipate and 3 assess Verify, leaving both largely hidden behind task success. These constraints also frame the open challenges we identify. PAPAV therefore provides a common basis for designing reliable physical agents and measuring progress across all five capabilities rather than task success alone.

1 Introduction

The paper asks what changes when multimodal agents move from computer-use to robot-use, where physical interaction offers less predictable and less forgiving feedback. It introduces PAPAV to compare systems by five shared task-loop functions and analyzes how physical constraints and benchmark coverage shape that comparison.

A unified basis for cross-domain comparison

  • Existing surveys may cover both computer-use and robot-use, but none compares the settings as one capability structure; PAPAV adds that digital–physical analogy.Six of ten surveys cover both settings, and four analyze benchmarks.Source: Table 1 · p. 3, §1 · p. 3

A capability view of the task loop

  • PAPAV defines Perceive, Anticipate, Plan, Act, and Verify by their contributions to a shared task loop, not by the components or architectures that implement them.It separates predicting candidate-action effects before planning from judging post-action evidence after acting.Source: §1 · p. 3

The shared coordinate then isolates what embodiment changes

  • Five recurring physical constraints—partial observability, limited simulatability, irreversibility, real-time coupling, and unverifiable outcomes—leave more decisions to be made during interaction.Memory and orchestration connect the five functions across time as an agentic harness, not a sixth capability.Source: §1 · p. 3

Contributions

  • PAPAV provides a three-way comparison of multimodal agents, robotic systems, and MMEAs, alongside a capability-level analysis of 64 benchmarks.The benchmark analysis distinguishes explicit capability evaluation from evidence available only through task outcomes.Source: §1 · pp. 4–5
  • Explicit benchmark evaluation concentrates on Act, while few benchmarks isolate Anticipate or Verify.The paper identifies these latter capabilities as often visible only through task outcomes.Source: §1 · p. 5

2 Background and Problem Formulation

The paper compares multimodal agents and robotic systems at the task-step level, using PAPAV to describe five roles by function rather than implementation. It characterizes embodied agents as persistent feedback loops whose actions must account for physical feasibility and execution constraints.

2. Background and Problem Formulation

  • A task step—not a GUI click or control tick—is the common unit for comparing computer-use and robot-use operations.Steps pair an operation that advances the task with evidence used to select it and feedback returned afterward.Source: §2 · p. 5

2.1 Multimodal Agents

  • A multimodal agent selects actions from its goal, interaction history, and persistent memory, then incorporates new observations into later decisions.Observations can include language, images, structured states, and tool outputs; actions can include GUI operations, API calls, or tool execution.Source: §2.1 · p. 5

2.2 Robotic Systems

  • Robotic task-level actions—such as skills, waypoints, and trajectories—must be grounded in embodiment and estimated state, subject to geometric, dynamic, safety, and control constraints.Execution maps actions to control inputs over time and can also be implemented implicitly by learned or end-to-end policies.Source: §2.2 · p. 5

2.3 Multimodal Embodied Agents

  • An MMEA maintains a persistent closed loop connecting multimodal task processing, physical execution, and feedback-informed decisions.Observed effects may differ from predicted outcomes, prompting revisions to task belief, action feasibility, or the remaining plan.Source: §2.3 · p. 6
  • MMEA classification requires a persistent embodied task loop, task-level capability outputs that influence later decisions, and commitments grounded in embodiment.Missing evidence for a condition makes classification indeterminate; an explicitly absent condition excludes the configuration.Source: §2.3 · p. 6
  • The feasible action set filters task-level interventions by geometric, dynamic, and safety constraints and by the available step-time budget.Geometric feasibility includes reachability and collision-free motion; dynamic feasibility includes motion and actuator limits.Source: §2.3 · p. 6

2.4 PAPAV Roles in the Task Loop

  • PAPAV distinguishes five roles by their outputs and information flow, including anticipating decision-relevant effects and verifying outcomes in ways that inform subsequent decisions.Roles may recur at different temporal resolutions or be combined or distributed across components.Source: §2.4 · p. 7

3 Capabilities: PAPAV

PAPAV compares MMEAs, MMAs, and robotic systems through five functions defined by their role in a task loop, then examines how physical constraints change their operation. Perceive and Anticipate connect evidence and forecasts to later task decisions.

3 Capabilities: PAPAV

  • PAPAV defines five recurring capabilities by their role in the task loop, enabling comparison across MMEAs, MMAs, and robotic systems while preserving differences in embodiment and implementation.The framework places MMEAs at the intersection of physical embodiment and multimodal, goal-directed adaptive reasoning.Source: §3 · p. 7

3.1 Perceive

  • Perception grounds heterogeneous observations in a task-relevant state and revises beliefs across interactions, while distinguishing task-general belief maintenance from goal-specific outcome verification.Embodiment changes the responsibility, timing, and evidence for maintaining beliefs under partial observability.Source: §3.1 · p. 8

3.1.1 Perceive: Similarity

  • Across digital and physical systems, Perceive integrates complementary evidence and grounds it in persistent representations of entities, attributes, relations, and change.The evidence may include interface states and tool returns or physical cues such as depth, force, and touch.Source: §3.1.1 · pp. 8–9

3.1.2 Perceive: Difference

  • Physical evidence inquiry is constrained by viewpoint, contact, timing, energy, and disturbance; the task process must formulate or revise the information need and use qualified returns.Task integration depends on whether the resulting belief reaches and changes the ongoing task process, not merely on a module's sensing specialty.Source: §3.1.2 · p. 9

3.1.3 Perceive: Summary

  • Perceive closes a task-level information gap through embodied acquisition, a qualified belief return, and a consuming task decision; value of information requires comparison with what would happen without the evidence.Paired trials, branching replay, or a validated simulator are needed to estimate value of information.Source: §3.1.3 · pp. 9–10

3.2 Anticipate

  • Anticipate predicts action-conditioned future states or observations from the current state, providing prospective consequences for planning rather than ranking candidate actions itself.Forecasts may use predictive models and semantic or latent representations, but physical plausibility alone does not establish realizability or decision-worthiness.Source: §3.2 · p. 10, §3.2.1 · pp. 10–11

3.2.1 Anticipate: Similarity

  • Both multimodal agents and robots forecast how candidate actions may change heterogeneous environments, but physical forecasts depend on partly controlled dynamics, contact, embodiment, and disturbances.Digital predictions can concern webpages or generated media; robotic predictions can concern poses, contacts, object motion, and task outcomes.Source: §3.2.1 · p. 10, §3.2.2 · p. 11

3.2.2 Anticipate: Difference

  • Physical forecasts must report their applicability envelope, while task-integrated prediction requires the task decision to select a physical counterfactual and use its return before commitment.The envelope includes predicted variables and horizon, tested conditions, and signals of departure; task-aware robotic prediction is not absent, as ModularAgent illustrates.Source: §3.2.2 · pp. 11–12

3.3 Plan

Plan selects a feasible, goal-directed course and revises it as conditions change; Act grounds and executes that course through an interface or body. Their shared functions require different handling of physical commitment, embodiment, and time-sensitive feedback.

3.3 Plan

  • Planning organizes actions, prerequisites, and subgoals into a feasible course under task, spatial, temporal, and safety constraints, with clarification, assistance, or abstention available when no course is feasible.Plan may revise the remaining sequence as task conditions change; Act, rather than Plan, provides body-specific execution.Source: §3.3 · p. 12

3.3.1 Plan: Similarity

  • Multimodal agents and robots both decompose long-horizon goals into ordered, reusable subgoals expressed through environment-specific tools, programs, skills, or motion primitives.The shared abstraction preserves the conditions under which each capability is available.Source: §3.3.1 · p. 13

3.3.2 Plan: Difference

  • Physical recovery must reconstruct the actually reachable state and account for altered geometry, contact, resources, and options removed by prior actions, unlike checkpoint-assisted digital recovery.Digital actions are not universally reversible; external services, permissions, and irreversible transactions can also remove options.Source: §3.3.2 · p. 13

3.3.3 Plan: Summary

  • Plan returns an authorized course under the current goal, belief, anticipated consequences, and constraints, preserving recovery branches or selecting assistance when appropriate.Evaluation can measure precondition coverage and persistent errors from annotated traces, while option value and expected recovery cost require branching evaluation or a reachability model.Source: §3.3.3 · p. 14

3.4 Act

  • Act grounds a selected action in targets and parameters, executes it through an interface or body, and uses feedback to adjust commands as execution proceeds.Physical execution must continue while the agent computes its next action; asynchronous policies can adjust or replace remaining motion.Source: §3.4 · p. 14

3.4.1 Act: Similarity

  • Digital and robotic Act share conversion of intent into an environment-compatible action specification, but differ in targets and executable representations, from interface operations to trajectories.The representation may be a GUI operation, API call, robot action token, waypoint, or continuous trajectory.Source: §3.4.1 · pp. 14–15

3.4.2 Act: Difference

  • Body-grounded execution needs a bidirectional record linking the semantic commitment to controller progress, termination, faults, and timing while leaving implementation to embodiment-specific control.The proposed fields are observability criteria, not a standardized execution API shared by current MMEAs.Source: §3.4.2 · pp. 15–16

3.4.3 Act: Summary

  • Act maintains continuity between an authorized intervention and body-specific realization; direct measures include grounding correctness, progress observability, termination accuracy, interruption latency, and missed deadlines.ShowUI-π transfers flow-based action generation from robot-use to GUI dragging, illustrating shared Act machinery.Source: §3.4.3 · p. 16

3.5 Verify

Verify judges outcomes against task criteria using post-action evidence and returns feedback that can regulate the ongoing loop. Physical verification must account for uncertain, partial, or delayed evidence and whether a task process can still act on the verdict.

3.5 Verify

  • Verification compares post-action evidence with independently specified expectations and returns a task-relevant judgment, distinct from Perceive’s task-general belief and Act’s execution telemetry.Physical evidence may include visual, proprioceptive, contact, and force signals; it may be partial, uncertain, or delayed.Source: §3.5 · p. 16, §3.5.1 · pp. 17–18

3.5.1 Verify: Similarity

  • Verification should turn outcome mismatches into actionable feedback, distinguishing the unmet condition from its cause so the task process can select an appropriate response.A placement outside its target establishes an unmet postcondition, but does not by itself show whether grounding, planning, or execution caused it.Source: §3.5.1 · p. 17

3.5.2 Verify: Difference

  • Physical verdict correctness includes timeliness: a task process must receive the judgment while continuation, replanning, recovery, assistance, or stopping can still affect the episode.Shared sensors, representations, backbones, or training data can yield correlated actor-verifier errors, so their evidence dependencies should be reported.Source: §3.5.2 · p. 18

3.5.3 Verify: Summary

  • Verify adjudicates postconditions and uncertainty; annotated episodes support measures of false acceptance, false rejection, calibration, and latency, while intervention benefit requires causal evaluation.For self-improvement, evaluation should test retained updates on further trials under comparable conditions.Source: §3.5.3 · p. 19

3.6 Synthesis: From Capability Evidence to Evaluation

  • PAPAV guides benchmark analysis by identifying each capability’s inputs, outputs, task-decision role, and whether it is explicitly assessed, alongside uncertainty, feedback, recovery, and persistent learning.A successful retry alone does not establish lasting improvement; further comparable trials are needed to assess its effect.Source: §3.6 · p. 19

4 Benchmarks & Evaluation

The benchmark analysis distinguishes capabilities required by a task from capabilities evaluated with capability-specific evidence, applying this framework to a curated corpus of 64 benchmarks. Explicit, implicit, and not-evaluated labels use defined capability boundaries.

4 Benchmarks & Evaluation

  • The corpus contains 64 benchmarks: 10 MMEA, 23 MMA, and 31 RS, with category assignment based on primary evaluation domain.Successor benchmarks are counted separately only when tasks, environments, datasets, or protocols change substantially.Source: §4 · pp. 19–20
  • Explicit Evaluation requires capability-specific evidence; Implicit Evaluation means the capability is required but only reflected in end-to-end performance.Not Evaluated means the task or evaluation protocol does not require the capability.Source: §4 · p. 20
  • Capability boundaries exclude multimodal input alone as Perceive evidence, ordinary post-action state changes as Anticipate, and external evaluator checks as Verify.Plan requires assessed task-level organization or replanning; Act requires assessed execution, validity, or environmental effects.Source: §4 · p. 20

4.1 Multimodal Embodied Agents

Across MMEA benchmarks, explicit evaluation covers Act and usually Perceive and Plan, while Anticipate and Verify receive less direct assessment; individual benchmark protocols vary across embodied interaction, closed-loop planning, and coordination.

4.1 Multimodal Embodied Agents

  • All ten MMEA benchmarks explicitly evaluate Act, and most explicitly evaluate Perceive and Plan, while Anticipate is always implicit and only two explicitly evaluate Verify.Most rely on simulation, with limited real-world and sim-to-real evaluation.Source: §4.1 · p. 22

4.1.1 Embodied Perception-Action

  • All three embodied perception-action benchmarks explicitly evaluate Perceive and Act, but none explicitly scores predictions of future states.ESI-Bench explicitly evaluates task-level Plan; Verify is absent in IS-Bench and ESI-Bench and implicit in CaP-Bench through code refinement.Source: §4.1.1 · p. 22

4.1.2 Planning and Closed-Loop

  • TEACh explicitly evaluates Act; EMBODIEDBENCH explicitly evaluates Perceive, Plan, and Act; RoboCerebra and AgenticLab explicitly evaluate Plan, Act, and Verify.Perceive is explicit in AgenticLab and implicit in RoboCerebra; Anticipate remains implicit across these benchmarks.Source: §4.1.2 · pp. 22–23

4.1.3 Multi-Agent Coordination

  • PARTNR, RoCoBench, and Habitat-MAS explicitly evaluate Plan and Act, while Anticipate and Verify are implicit through coordination demands and feedback.Perceive is explicit in PARTNR and Habitat-MAS, but not evaluated in RoCoBench; external collision and simulator checks do not count as agent verification.Source: §4.1.3 · p. 23

4.2 Multimodal Agents

MMA benchmarks span understanding, interactive use, artifact generation, and world prediction, with different PAPAV capabilities made explicit across those task types. Verify remains limited in many interactive settings, while some world-model benchmarks directly score predicted futures.

4.2 Multimodal Agents

  • MMA benchmarks range from multimodal understanding to interactive agents, artifact generation, and world models; explicit evaluation patterns differ across these task families.Verify remains limited despite post-action feedback in several benchmarks, because agents rarely must judge success and choose whether to stop, retry, or recover.Source: §4.2 · p. 23

4.2.1 Multimodal Understanding

  • MMMU-Pro, Video-MME-v2, and VSI-Bench assess distinct multimodal understanding capabilities, but do not evaluate a complete agent loop.They target interdisciplinary question answering, video understanding, and visual-spatial reasoning, respectively.Source: §4.2.1 · p. 23

4.2.2 Web Browsing, Computer-Use, and Mobile Interaction

  • Web, computer, and mobile benchmarks generally explicitly evaluate Act, while Perceive and Verify vary by benchmark and Anticipate is usually implicit.WebArena, VisualWebArena, and WebVoyager implicitly evaluate Verify through action-result inspection; external checks in MMInA and MMSearch-Plus do not.Source: §4.2.2 · pp. 23–24

4.2.3 General and Cross-Environment Assistants

  • GAIA reflects Perceive, Plan, and Act only implicitly, while AgentVista and OmniGAIA explicitly evaluate Perceive and Act but not Anticipate or Verify.CRAB explicitly evaluates Act through graph-based progress checks, with its other capabilities implicit.Source: §4.2.3 · p. 24

4.2.4 Interactive Games and Spatially Intelligent Agents

  • GameWorld and iVISPAR explicitly evaluate Act and implicitly evaluate Anticipate, but neither evaluates Verify or task-level Plan.iVISPAR explicitly evaluates Perceive through controlled representation comparisons; GameWorld evaluates it implicitly.Source: §4.2.4 · p. 24

4.2.5 Artifact Generation

  • Artifact-generation benchmarks treat executable outputs as Act, while Perceive and Plan remain implicit and Anticipate and Verify are not evaluated.Artifact testing is external and does not require the agent to predict, detect failure, or revise.Source: §4.2.5 · p. 24

4.2.6 World Model and Physical Prediction

  • WorldModelBench, PBench, and HarnessEval-W explicitly evaluate Anticipate through scored future predictions, but do not assess task-level Plan or agent Verify.Act is not evaluated for generated predicted observations; HarnessEval-W's planning and evidence validation belong to its external evaluator.Source: §4.2.6 · pp. 24–25

4.3 Robotic Systems

Robotic-system benchmarks concentrate explicit evaluation on Act, while task-level Plan and agent Verify are not explicitly assessed in the corpus-wide analysis. Across benchmark families, aggregate outcomes often combine multiple sources of error or leave assessment to external evaluators.

4.3 Robotic Systems

  • Of 31 RS benchmarks, 29 provide explicit Act evidence, while none explicitly evaluates task-level Plan or agent Verify.Aggregate scores combine execution errors with perception or planning errors; path and motion planning count as execution unless task-level organization is assessed.Source: §4.3 · p. 25

4.3.1 Navigation and Exploration

  • R2R and GOAT-Bench explicitly evaluate Perceive and Act, but their aggregate scores combine recognition, localization, memory, and movement errors.Anticipate and Verify are implicit, and path search is not task-level Plan.Source: §4.3.1 · p. 25

4.3.2 Skill Learning and Policy Generalization

  • Manipulation suites assess retention, perturbation robustness, skill composition, and physical variation, but none evaluates Verify because completion checks are external.Explicit Perceive is found in CALVIN, LIBERO-PRO, LIBERO-Plus, and RoboLab; Plan is implicit in some suites and not evaluated in others.Source: §4.3.2 · p. 25

4.3.3 Long-Horizon and Household Manipulation

  • Phase-level progress can distinguish early failures from near completion, but external progress measures do not establish task planning or agent verification.Such measures may still conflate semantic, perceptual, navigation, and manipulation errors.Source: §4.3.3 · pp. 25–26

4.3.4 Safety-Constrained Control and Human Assistance

  • Safety benchmarks extend Act evaluation to action safety, while Anticipate is implicit and external monitoring means none evaluates agent Verify.SafeManip alone provides implicit Plan evidence through ordered constraints.Source: §4.3.4 · p. 26

4.3.5 Real-World and Sim-to-Real Evaluation

  • Real-world and sim-to-real benchmarks explicitly evaluate Act, but Plan and Verify are not evaluated because task generation, state checking, and ranking remain external.Ranking agreement measures sim-to-real validity, while site-level results reveal variation from cameras, control frequency, reset procedures, or operators.Source: §4.3.5 · p. 26

4.3.6 Robot World Models and Physical Prediction

  • Robot world-model benchmarks explicitly evaluate Anticipate, but predicted futures stop short of a full decision loop with action selection, outcome observation, and updating.A separate control policy remains responsible for action execution; recording selected predictions and later updates could connect anticipation with closed-loop behavior.Source: §4.3.6 · pp. 26–27

4.4 Cross-Domain Comparison

Across the benchmark corpus, explicit evaluation concentrates on Act, while Anticipate and Verify receive little direct assessment. Process-diagnostic measures can help distinguish capability failures, and benchmark settings increasingly converge across domains.

4.4.1 Overall PAPAV Coverage

  • Act is Explicit in 10/10 (100%) MMEA, 16/23 (70%) MMA, and 29/31 (94%) RS benchmarks; Anticipate and Verify are Explicit in only 5/64 (8%) and 3/64 (5%).Perceive is Explicit in 7/10 (70%) MMEA, 10/23 (43%) MMA, and 10/31 (32%) RS benchmarks; Explicit Perceive requires a dedicated understanding, grounding, or evidence-acquisition task.Source: §4.4.1 · p. 27
  • Plan is Explicit in 7/10 (70%) MMEA benchmarks but Implicit in 16/23 (70%) MMA and 13/31 (42%) RS benchmarks; remaining MMA and RS benchmarks do not evaluate it.The pattern reflects protocol design, not planning ability, and comparisons across the 10 MMEA, 23 MMA, and 31 RS benchmarks are descriptive rather than estimates of the complete literature.Source: §4.4.1 · p. 27

4.4.2 Diagnosticity and Capability Bottlenecks

  • Focused process measures, including checking and replanning signals, provide more direct evidence for distinguishing poor planning, missed failures, and faulty execution than aggregate task outcomes.The cited protocols include physical future prediction, task success detection, hierarchical decisions, and replanning after execution deviations; prediction benchmarks alone may stop before action and feedback update.Source: §4.4.2 · pp. 27–28

4.4.3 Convergence across Domains

  • Benchmarks increasingly share persistent states, multi-step interaction, multimodal feedback, and coordinated action, while differences in capability coverage mainly reflect benchmark design rather than system capability.Across domains, Act has the strongest explicit coverage, while Anticipate and Verify remain less observable.Source: §4.4.3 · p. 28

4.5 Evaluation Metrics and Reproducibility

The section distinguishes benchmarks by whether their metrics report outcomes or diagnose capability-specific processes, then considers what makes evaluations reproducible. Outcome-based metrics are easier to interpret but weaker at locating failures, while process diagnosis requires finer instrumentation.

4.5.1 Outcome- and Process-Based Evaluation

  • Outcome-only benchmarks report terminal results, outcome-augmented benchmarks add progress or efficiency measures, and process-diagnostic benchmarks link at least one measure to a specific capability.Outcome-augmented measures do not link performance to a specific capability.Source: §4.5.1 · pp. 28–29
  • The corpus contains 24 outcome-only, 25 outcome-augmented, and 15 process-diagnostic benchmarks; process diagnosis improves failure attribution but requires finer annotations, trajectory logging, and evaluator design.Process diagnosis appears in 4/10 (40%) MMEA, 4/23 (17.39%) MMA, and 7/31 (22.58%) RS benchmarks.Source: §4.5.1 · p. 29

4.5.2 Reproducibility

  • Reproducibility depends on stable task data, execution environments, and evaluators, with real-robot studies adding variation from hardware, calibration, control frequency, scenes, and human intervention.Reporting task instances, versions, trajectories, configurations, budgets, seeds, trial counts, hardware settings, and evaluation criteria supports reproducibility; the corpus was frozen at the stated cutoff date.Source: §4.5.2 · p. 29

4.6 Discussion and Future Benchmark Design

The discussion argues that aggregate outcomes often fail to identify which capability caused an error. It recommends evaluations that retain end-to-end performance while adding attributable signals and testing verification and recovery under realistic variation.

4.6 Discussion and Future Benchmark Design

  • Aggregate outcomes rarely identify failure sources in tasks that couple PAPAV capabilities, so future benchmarks should retain end-to-end scores while adding attributable signals such as grounding correctness and failure recognition.RoboCerebra and AgenticLab illustrate checking and replanning signals for failure attribution.Source: §4.6 · p. 29
  • Verification evaluations should connect judgments to later behavior, use evidence that can reveal errors missed by the actor, and compare actor-verifier performance against independently established reference judgments.Measures include false success and failure judgments, detection delay, recovery success, and extra actions or time needed for recovery.Source: §4.6 · pp. 29–30
  • Benchmarks should pair controlled comparisons with physical trials or environmental perturbations to test whether verification and recovery remain effective as observations, hardware, or execution conditions change.Results may vary across sites and reconstructed environments.Source: §4.6 · p. 30

5 Open Challenges

The open challenges concern keeping PAPAV capabilities coordinated in partially observed, changing, and difficult-to-verify environments. They address what to remember and predict, when to replan or seek information, and how verification can support lasting improvement.

5 Open Challenges

  • The five challenges concern coordination across PAPAV capabilities under partial observation, continual change, difficult simulation, and outcomes that can be hard to verify.The agent must decide what to retain, predict, revise, and verify as it interacts.Source: §5 · p. 30

5.1 OC1. Persistent Physical Belief: What Should the Agent Remember?

  • Physical memory must preserve temporarily unobservable state while revising stale or contradictory beliefs, avoiding both repeated rediscovery and unbounded accumulation of information.The challenge is the quality of the belief memory supports, not memory capacity alone.Source: §5.1 · pp. 30–31

5.2 OC2. Decision-Relevant Prediction: What Is Worth Predicting?

  • Prediction should preserve future distinctions that matter to the decision, since detailed forecasts can include irrelevant information while compact latent forecasts can hide safety- or feasibility-critical factors.What counts as sufficient prediction depends on the task and decision; a practical test is whether removing detail changes action ranking, safety judgments, or outcomes.Source: §5.2 · p. 31

5.3 OC3. Adaptive Commitment: When Should the Agent Replan?

  • Action horizons should adapt: long commitments risk slow reactions to change, while constant replanning is costly and can make behavior hesitant.Predictable free-space motion may tolerate longer commitment; contact, unexpected motion, uncertainty, or irreversible actions may warrant earlier replanning, assessed alongside reaction delay and inference cost.Source: §5.3 · pp. 31–32

5.4 OC4. Uncertainty-Conditioned Control: Act, Observe, or Ask?

  • Under uncertainty, an agent must choose among acting, gathering information, or asking for help, balancing the likelihood that more information changes the decision against the consequences of error.The challenge is to avoid both reckless autonomy and continual re-observation or escalation.Source: §5.4 · p. 32

5.5 OC5. Learning from Verification: How Can the Agent Improve?

  • Verification-guided improvement must connect observed outcomes to the PAPAV decision that caused failure, test revisions through further interaction, and retain them only when they remain useful.A successful retry does not establish lasting improvement: scene changes can invalidate a correction, and repeated feedback can reinforce an incorrect explanation.Source: §5.5 · p. 33

6 Conclusion

PAPAV compares multimodal agents, robotic systems, and multimodal embodied agents through five functions in a shared task loop. Embodiment preserves that loop but makes more decisions open during interaction, while capability-level evaluation remains uneven.

6 Conclusion

  • PAPAV makes the digital–physical transition comparable by comparing multimodal agents, robotic systems, and multimodal embodied agents through five functions defined by their contribution to a shared loop.The functions are defined by what they contribute rather than by architecture.Source: §6 · p. 33
  • Embodiment preserves the task loop but adds five physical constraints, moving more decisions into interaction and making reliability depend on managing consequences as they unfold.The constraints are partial observability, limited simulatability, irreversibility, real-time coupling, and unverifiable outcomes.Source: §6 · p. 33
  • Explicit evaluation concentrates on Act, while Anticipate and Verify remain largely hidden behind task success; combining capability-level tests with end-to-end tasks could make failures easier to locate.The paper identifies these evaluation gaps and proposes combining capability-level tests with end-to-end tasks to make reported progress easier to interpret.Source: §6 · p. 33

Related papers

Loading 2610.30108v1…