PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Agentic Robots: Model, Data, Environment, and Harness

Chonghan Qin, Dongming Wu, Kaituo Feng, Jiaxu Wang, Xinlei Yu, Peiwen Sun, Yijia Fan, Xilin He, Bingao Chen, Weitong Lian, Ryan Ji, Wenqi Ren, Xiaobin Hu, Dahua Lin, Yu-Gang Jiang, Xiangyu Yue

arXiv:2610.20442v1cs.RO

TL;DR

Foundation-model policies can improve robotic understanding and action prediction, but long-horizon execution requires progress tracking, plan adjustment, and failure recovery, while research on agentic robots remains fragmented. This paper traces the field’s development and synthesizes it through a four-domain taxonomy and analysis of agentic methods. It outlines future research directions spanning fair evaluation, physical deployment, efficiency, and cross-embodiment generalization.

  • Problem

    Research on agentic robots remains fragmented, while reliable long-horizon tasks require capabilities beyond action prediction, including progress tracking, plan adjustment, and failure recovery.

    Source: §1 · pp. 3–4

  • Method

    The paper traces the evolution from robot policies to agentic systems and synthesizes approaches through a taxonomy and analysis of agent decisions across the system.

    Source: §1 · p. 4

  • Results

    The paper organizes existing work into seventeen overlapping categories across Model, Data, Environment, and Harness, and analyzes how agent decisions depend on tools, robot executors, and interaction feedback.

    Source: §1 · p. 4, §9 · p. 48

  • Takeaways & Limitations

    The research agenda calls for fair benchmarking, real-robot deployment, efficient robot use, and evaluation of cross-embodiment generalization.

    Source: §9 · p. 48

    Adaptive evaluation conclusions depend on test selection, access to prior results, and score validation.

    Source: §6.3 · p. 30

Abstract

from arXiv · show

Foundation models are driving a paradigm shift in robotic intelligence, moving beyond low-level action prediction to goal-directed orchestration of data collection, model training, task planning, and real-world execution. We characterize this emerging paradigm as an agentic robots, in which agents reason over human instructions, interact with robotic environments, rectify execution errors autonomously, and dynamically adjust operational strategies. Although this field develops fast, current studies are not well connected. Researchers have no consistent view of how to combine different agentic functions into a complete robot system. This paper provides a unified overview of existing research on agentic robots. We first trace the development from robot policies to agentic systems, covering vision-language-action models, world models, and world-action models. We then clarify the definition and scope of agentic robots and organize existing research into seventeen categories across four functional domains: Model, Data, Environment, and Harness. Across these domains, we examine how agents shape model learning and robot decisions, collect and curate robot data, construct worlds, tasks, and tests for learning and evaluation, and organize capabilities for robot use. Finally, we discuss open challenges and outline directions for future research. Looking forward, we hope this work contributes to a clearer understanding of agentic robots and supports new approaches to making them more capable, reliable, and adaptive.

1 Introduction

Foundation-model policies improve scene and task understanding, but action prediction alone does not ensure reliable long-horizon execution. This paper unifies fragmented research on agentic robots with a four-domain taxonomy and an analysis of how agents coordinate and improve robot capabilities.

Why Agentic Robots Matter?

  • VLA action prediction alone cannot consistently track long-horizon task progress, adjust plans to environmental changes, and recover from failures.The introduction notes that these capabilities are needed for reliable execution beyond predicting actions.Source: §1 · p. 3

A Unified Illustration for Agentic Robotics

  • The paper organizes agentic robotics into Model, Data, Environment, and Harness, tracing agent decisions within each domain and their effects across the system.The taxonomy distinguishes learning and robot-facing decisions, data acquisition and curation, worlds, tasks and evaluation, and capability coordination, reuse and improvement.Source: §1 · p. 4
  • The paper traces the evolution from robot policies to agentic systems and clarifies agents’ roles in task execution and capability development.It brings these research directions into a common framework.Source: §1 · p. 4

A Systematic Analysis of Agentic Methods

  • The paper compares representative approaches and analyzes how agent decisions depend on available tools, robot executors, and interaction feedback.It also examines the strengths and limitations of these approaches.Source: §1 · p. 4

A Research Agenda for Practical Agentic Robots

  • The research agenda connects fair evaluation with physical deployment and generalization across robotic platforms.It outlines directions for establishing agents’ contributions and translating advances into reliable, efficient robot operation.Source: §1 · p. 4

2 Evolution of Robot Learning

Robot learning progresses from policies that map observations to actions toward systems that organize execution and capability improvement. The section reviews policies, vision-language-action models, world models, and world-action models as overlapping traditions whose capabilities can coexist in agentic systems.

2 Evolution of Robot Learning

  • Robot learning spans executable behavior, multimodal knowledge integration, physical-outcome prediction, and coordination of execution and improvement.These stages describe overlapping research traditions and complementary capabilities that can coexist in one system.Source: §2 · pp. 4–5

2.1 Conventional Robot Policies

  • Conventional policies map observations to actions, with capabilities depending largely on robot experience available during training.Image-based and 3D-aware policies differ in scene representation; scene representation, learning objective, and action decoder are distinct design choices.Source: §2.1 · p. 5

2.2 Vision-Language-Action Models

  • Vision-language-action models connect pretrained visual and linguistic knowledge to action generation, using varied action decoders, memory, reasoning, and future prediction.Approaches include discrete action tokens, continuous diffusion- or flow-based decoding, historical context, intermediate guidance, and predictive objectives.Source: §2.2 · pp. 5–6

2.3 World Models and World-Action Models

  • World models predict action-conditioned environmental changes for planning and learning, while world-action models jointly connect future visual prediction with robot action learning.World models also serve as simulated environments or visual planners; some approaches translate predicted futures into robot actions through inverse dynamics.Source: §2.3 · pp. 6–7

2.4 Agentic Robots

  • Agentic systems coordinate capabilities, interpret execution feedback, and adjust subsequent operations, extending model-mediated decisions to experience acquisition and capability improvement.The Figure 3 taxonomy places these functions across Model, Data, Environment, and Harness.Source: §2.4 · p. 7, Figure 3 · p. 8

3 Foundations of Agentic Robots

The section defines agentic robots by how agents organize goal-directed execution and maps their roles across capability development and robot use. It distinguishes this organizational focus from adjacent fields.

3.1 Definition and Scope

  • Agents autonomously coordinate goal-directed execution through observations and feedback, and may also develop capabilities by collecting data, constructing environments, or revising training and skills.They use new observations and outcomes to continue, gather information, revise plans, or recover from failure.Source: §3.1 · p. 8

3.2 Taxonomy

  • The taxonomy organizes agent roles into four functional domains and 17 non-exclusive categories, allowing contributions to span domains and categories.Model, Data, Environment, and Harness cover capability development and task execution; a single work may contribute to multiple categories.Source: §3.2 · p. 9, Table 1 · p. 10

3.3 Comparison with Related Concepts

  • Unlike embodied AI’s broad study of learning and acting through environmental interaction, the agentic perspective emphasizes capability selection, task coordination, and feedback-based revision.The two fields both connect perception, reasoning, and action in physical or simulated environments.Source: §3.3 · p. 9
  • Autonomous robotics already includes planning, acting, monitoring, observation, and learning; “agentic” specifies how foundation-model agents organize decisions, not greater autonomy or freedom from human guidance.Agents may interpret goals, coordinate skills or tools, and use execution feedback in later decisions.Source: §3.3 · pp. 9–10
  • Robot foundation models acquire reusable task capabilities, while agentic robotics focuses on how goals, available actions, and execution feedback govern decisions during a task.The agentic organization may be implemented within one model or across several components.Source: §3.3 · p. 10

4 Model: What Should the Agent Learn, Understand, Decide, and Control?

This section examines model-side interventions in what agents learn, perceive, plan, and control, including how model outputs and feedback are matched to executable capabilities. It also describes adaptation through changes to decision context, planner configurations, or learned parameters over different timescales.

4.1 Training

  • Training agents can revise rewards, policies, pipelines, or experiments, but reliable iteration requires keeping the measurements used to judge revisions stable.The scope of editable artifacts ranges from reward signals to policy code and the experimental workflow; typed artifacts and configuration-resolved contracts support traceability across data, training, evaluation, and deployment.Source: §4.1 · pp. 11–12

4.2 Perception

  • Perception agents seek or interpret task-relevant evidence, including by moving cameras or objects, and must renew grounding when remembered locations no longer support an action.The representation—such as geometry, relations, or beliefs—must support the downstream decision; gathering evidence trades its value against motion cost and risk.Source: §4.2 · pp. 13–14

4.3 Planning

  • Planning spans generating action sequences or subgoals, checking goals and plans before execution, and revising actions or the planning model using execution feedback.Longer-horizon plans suit predictable outcomes; uncertain outcomes or scene changes favor shorter plans guided by new observations.Source: §4.3 · pp. 14, 16

4.4 Robot Control

  • Robot-control models can issue commands, specify target poses, write control programs, or define motion constraints, with required detail determined by the execution layer.Execution layers may handle inverse kinematics, collision checks, and low-level control; checks and feedback can support retries or recovery when motions fail.Source: §4.4 · pp. 17–18
  • Control feedback may update spatial maps or action-model and language-model reasoning (, [411], [329]); unobserved object movement still requires fresh observations.VoxPoser updates spatial value maps from observations during model-predictive control, while COMPASS sends controller feedback tokens to the language model.Source: §4.4 · p. 18
  • Robot-control outputs should match executor capabilities, ranging from skill names to poses, programs, constraints, or costs, with feasibility checks and feedback guiding execution.For contact-rich manipulation, CoRAL uses interaction feedback to revise physical models, cost functions, or contact strategies when requirements prove unsuitable.Source: §4.4 · pp. 18–19

4.5 Adaptation

  • Stored experience can condition decisions or tune planners without changing policy parameters, but context-dependent gains may not persist and planner tuning does not address policy limitations.Adaptation choices depend on the failure source: perception errors may need new observations, planning errors replanning, and persistent execution failures policy updates.Source: §4.5 · pp. 19–20
  • Persistent adaptation updates action labels, high-level or residual policies, and action selection; autonomous interaction supplies learning signals through rewards, simulated teaching, or retrospective evaluation.These methods target different error sources and use different supervision mechanisms; some update a separate policy while keeping an evaluator or low-level controller fixed.Source: §4.5 · pp. 19–20
  • Across tasks and deployments, systems reuse or filter experience for training, with outcomes depending on supervision quality, experience diversity, and interaction costs.Evaluation should distinguish within-episode recovery, cross-task generalization, and improvement after retraining, while checking whether acquired capabilities are preserved.Source: §4.5 · p. 20

5 Data: How Should the Agent Collect, Annotate, and Refine Robot Data?

The Data domain treats agentic robot learning as a workflow of collecting experience, adding structured supervision, and refining records. It surveys how agents support each stage.

5.1 Collection

  • Collection methods gather real-robot or human experience, generate trajectories in simulation, or synthesize artificial observations and motions.Some simulation and synthesis approaches also use execution failures or generated scene and task information to guide what data is produced next.Source: §5.1 · pp. 21–22

5.2 Annotation

  • Robot records often lack textual, spatial, interaction, and outcome labels, so agents produce and validate annotations in four corresponding groups.Text annotations include task-stage descriptions and captions; the other groups localize objects or contacts and label progress, success, failure, or safety.Source: §5.2 · pp. 23–24

5.3 Refinement

  • Refinement selects useful trajectories or segments and revises supervision when execution evidence changes how earlier behavior should be judged.Selection can retain productive portions of unsuccessful attempts, while later feedback or human-corrected criteria can update earlier labels and scores.Source: §5.3 · pp. 25–26

6 Environment: What Worlds, Tasks, and Tests Should the Agent Build?

The Environment domain distinguishes scene reconstruction, task generation, and benchmarking as separate parts of shaping robot interaction. The paper emphasizes matching checks and revisions to the object being built or tested.

6 Environment: What Worlds, Tasks, and Tests Should the Agent Build?

  • A scene may be unstable, a task infeasible, or an evaluator uninformative, so construction should connect each failed check to the object needing revision.For a drawer task, the scene must permit joint motion, the task must specify the object to retrieve, and the evaluator must check the final relation.Source: §6 · p. 26

6.1 Reconstruction

  • Reconstruction must produce an executable scene with task-relevant structure, interaction changes, and an interface for inspection and revision; visual similarity alone is insufficient.The resulting simulated behavior and task feasibility do not establish real-world validity.Source: §6.1 · pp. 26–27

6.2 Task Generation

  • Task generation specifies goals, initial states, constraints, and success conditions, but feasibility repairs can alter task difficulty or meaning.For adaptive distributions, evaluation on separately specified tasks is needed because a learner’s own success signal can make tasks easier without improving the target ability.Source: §6.2 · pp. 28–29

6.3 Benchmarking

  • Benchmarking spans planning, tool and programming decisions, active perception, memory, and safety, with protocols determining what each evaluation measures.For example, open-loop skill composition does not measure feedback-driven replanning, and hazard-response decisions are distinct from real-time reaction when simulation pauses during inference.Source: §6.3 · pp. 29–30
  • A benchmark score depends on the full execution contract, including the executor, feedback path, observation access, controller assistance, and interaction budget.Changing these conditions can change the ability measured by the same task.Source: §6.3 · pp. 29–30
  • Adaptive evaluation can reveal weaknesses, but conclusions depend on test selection, access to prior results, and score validation.Fixing test conditions independently of the evaluated system supports broader claims; improving the evaluator and improving the robot are separate outcomes.Source: §6.3 · p. 30

7 Harness: How Should Robot Capabilities Be Orchestrated, Recovered, and Evolved?

This section examines how the harness makes agentic robot capabilities usable and coordinates their execution, recovery, and evolution. It covers component invocation and information flow, behavior across attempts, skill libraries and infrastructure, and layered coordination among planners, learned policies, and analytic controllers.

7.1 Orchestration

  • Orchestration coordinates executor routing, function-call protocols, state-compatible handoffs, and control-return timing so components can act reliably in sequence.Protocols can specify valid inputs and outputs and independent success conditions; handoffs may repair or prevent state mismatches, while timing may depend on progress, execution budgets, or clocks.Source: §7.1 · pp. 32–33

7.2 Memory

  • Memory records, updates, and retrieves prior information without changing model weights, supplying relevant evidence, task state, or reusable results to later decisions.Records may be refreshed or checked against changing scenes; retrieval can pass a skill’s setup conditions or a condensed task summary, while earlier solutions may be validated before reuse.Source: §7.2 · pp. 34–35

7.3 Monitoring and Recovery

  • Monitoring and recovery span execution checks, response selection and validation, recovery actions, resume-state choices, and rules for when repeated attempts stop.Checks before, during, or after execution respectively select among candidates, signal stopping or adjustment, or explain outcomes; recovery may continue from a disturbed scene or restore it first.Source: §7.3 · pp. 36–38

7.4 Skills

  • Skills package reusable procedural knowledge as context-loaded guidance or callable operations, evolving from fixed primitives and task-specific generated programs toward libraries reused across tasks.Library entries differ in what they store and how they are admitted: examples include evaluator-approved action code, human-correction knowledge, and skills tested on held-out states; flawed entries may persist if not revised.Source: §7.4 · pp. 38–39
  • Skill revisions need validation and retirement gates: systems accept changes after tests such as replay, held-out scores, or repeated success, while ungated updates can be non-monotonic and fail to accumulate into task success.Examples include ASPIRE [228], RATs [435], RoboRSI [373], and SkillOpt [402]; a 123-round study found accepted changes did not add up to success on its target task [339].Source: §7.4 · p. 39
  • Most systems that transfer harness experience into weights train the executor; training the coder is rarer, and continual learning from rollouts or evolved skills remains an open direction.RoboClaw [178] and ENPIRE [383] fine-tune policies, while RoboCodeX [242] and Uni-Skill [385] fine-tune code-generating models; CaP-X [74] trains a coding model on three simulated tasks before physical transfer.Source: §7.4 · p. 40

7.5 Infrastructure

  • 74.0% to 59.1% question-answering accuracy follows removal of NavMCP’s complete return protocol, while halving Show-Harness step size raises zero-shot success from 60% to 82%.Capability interfaces determine what agents can express and what feedback they receive; EmbodiedTool [466] also shows that valid tool calls do not ensure choosing the right tool.Source: §7.5 · p. 41
  • 14% to 62% macro-average success on evaluated LIBERO-Pro perturbations follows adding ASPIRE’s trace, debugging, and repair-validation engine.Perception is a major source of physical failures, and privileged observations can mask that cost: Kuroki and Tang [156] report 65.0% success with full simulator observations versus 22.5% with RGB alone.Source: §7.5 · p. 42
  • About 30% lower task time comes from SmolVLA’s asynchronous execution, at a small cost in success; policy serving also raises latency, capacity, placement, and fleet scheduling concerns.Systems include remote policy servers, edge/cloud placement, shared GPU scheduling, and runtime optimizations; the section notes that serving latency should become part of the planning budget.Source: §7.5 · p. 42
  • Safety gates outside the model can block unsafe commands, while harness revisions can be required to pass contract checks, isolated review, replay, and held-out evaluation.RoboGuard [277] reduces jailbreak success from 92.3% to 2.3%; ETA [50] reports that no revised harness candidate passed its checks.Source: §7.5 · pp. 42–43

7.6 Layered Systems

  • Layered systems connect System 2 planners, System 1 learned visuomotor policies, and System 0 analytic controllers, with the harness determining what passes between layers and at what rate.Ground Slow, Move Fast [367] uses a 2 Hz planner, 30 Hz policy, and model predictive control; HarnessWAM [89] finds progress-event deliberation outperforms fixed-interval planner calls.Source: §7.6 · p. 44
  • Separate orchestrators can call VLAs alongside analytic primitives, preserving replaceable, inspectable layers, while code can provide an upper layer with monitored task graphs and precompiled recovery branches.A future direction is a stack with callable VLA lower layers, programs as the upper layer, and offline agent revision; comparisons should isolate the changed layer and test behavior beyond optimization feedback.Source: §7.6 · pp. 44–45

8 Outlook: What Comes Next for Agentic Robots?

The outlook identifies unresolved questions about what agents contribute, physical executability, and performance under limited resources or unfamiliar platforms. It organizes future work around fair benchmarking, real-robot deployment, efficiency, and cross-embodiment generalization.

8.1 Fair Benchmarking

  • Fair benchmarks should compare systems under shared executors, observations, and interaction budgets, report resource costs, and test recovery, environmental changes, and improvement on unseen tasks.Task success alone can conceal whether an agent or its executor produced the result; evaluations should also account for additional observations, computation, time, and human assistance.Source: §8.1 · pp. 46–47

8.2 Real-robot Agentic Robotics

  • Physical deployment requires capability-aware interfaces and feedback confirming action outcomes, because commands can fail, scenes change, and real sensing is noisier than simulated state estimates.Evaluations should clarify what information agents receive, what lower-level controllers handle, and whether agents recover when their assumptions fail.Source: §8.2 · p. 47

8.3 Efficient Robot Use

  • Efficiency depends on timing model calls against progress, uncertainty, and delay consequences, and on total execution costs rather than inference speed or call count alone.Local models may respond faster with limited resources, while cloud models may offer stronger reasoning but introduce communication delays.Source: §8.3 · p. 47

8.4 Cross-Embodiment Generalization

  • Cross-embodiment generalization requires agents to inspect unfamiliar robots’ capabilities, adapt plans, and reuse high-level strategies without assuming motion commands or control parameters transfer.Evaluations could use previously unseen robots and measure success, required configuration or training, and whether experience from other robots improves performance.Source: §8.4 · p. 48

9 Conclusion

The paper traces agentic robot research and organizes it into a taxonomy spanning four functional domains. It connects execution and learning from experience, then identifies four directions for future research.

9 Conclusion

  • The paper organizes existing work into seventeen overlapping categories across Model, Data, Environment, and Harness, covering learning and decisions, data preparation, constructed worlds and tests, and software coordination.Source: §9 · p. 48
  • Agent decisions depend on available observations, robot capabilities, and feedback; stored experience can guide later task decisions and changes to data, training, skills, models, and software.Changes should be tested on new tasks and conditions while checking that previously learned capabilities are preserved.Source: §9 · p. 48
  • The paper identifies fair benchmarking, real-robot deployment, efficient robot use, and cross-embodiment generalization as four directions for future research.It aims to connect these questions toward more capable agentic robots.Source: §9 · p. 48

Related papers

Loading 2610.20442v1…