PaperlayerView exact revision on arXiv

Source-linked AI summary

ABot-ER: Unified Embodied Reasoning through Agentic Data-Model Co-Evolution

Qiuyu Ding, Heng-Da Xu, Changda Xia, Jiajun Shen, Zihao Guan, Zhengchen Yuan, Jianfang Zhang, Di Yang, Jiayi Tian, Yuting Xu, Fei Wang, Shiao Liu, Wei Zhang, Xiaolong Wu, Wenbin Tang, Zedong Chu, Mu Xu

arXiv:2610.18355v2cs.AI

TL;DR

Unified embodied reasoning must coordinate perception, planning, and execution despite uneven capabilities and competing training demands. ABot-ER combines evaluation-driven data evolution with task-conditioned residuals over a shared backbone, and reports the highest aggregate performance in all three competencies. Its converged corpus also supports a single-pass post-training run with substantially less accumulated training.

  • Problem

    Unified embodied reasoning must reconcile uneven capabilities across perception, planning, and execution with data mixtures that may drift and heterogeneous outputs that compete for shared parameters.

    Source: §1 · p. 3

  • Method

    ABot-ER pairs an evaluation-feedback engine that revises supervision and data mixtures with task-conditioned low-rank residuals over a shared backbone.

    Source: §1 · pp. 3–4

  • Results

    ABot-ER achieves the highest aggregate score in perception, planning, and execution across twelve benchmarks covering five capability dimensions.

    Source: §1 · p. 4

  • Takeaways & Limitations

    A single post-training run on the converged corpus matches the six-round model’s twelve-benchmark mean with one-fifth of the accumulated post-training.

    Source: §1 · p. 4

    The reported interpretation of task conditioning is not a controlled result because the compared corpora differ in more than one respect.

    Source: §6.4 · p. 26

Abstract

from arXiv · show

We introduce ABot Embodied Reasoning (ABot-ER), a unified embodied reasoning foundation model that breaks the tension between breadth and depth in embodied reasoning. Embodied intelligence demands the joint mastery of perception, planning, and execution, yet reconciling generality with specialization across this spectrum remains an open challenge. Within a single unified model, ABot-ER commands specialist-level depth across all three competencies, spanning spatial, scene, and temporal reasoning in perception; long-horizon action organization in planning; and point grounding and affordance in execution. The Agentic Data-Model Self-Evolution Engine continuously adapts data strategies to the model's evolving capability state. The Task-Conditioned Adaptive Training Strategy uses task tokens to activate lightweight low-rank residuals over a shared backbone, separating updates from heterogeneous output formats while retaining shared embodied knowledge. Across a systematic evaluation on twelve benchmarks covering three core competencies and five capability dimensions, ABot-ER achieves state-of-the-art overall performance across perception, planning, and execution within a unified parametric framework, demonstrating that breadth and depth can coexist in unified embodied reasoning.

1 Introduction

Embodied reasoning must coordinate perception, planning, and execution, but unified models face data-mixture drift and interference between heterogeneous outputs. ABot-ER combines adaptive data revision and task-conditioned residuals, and reports state-of-the-art aggregate performance across all three competencies.

1 Introduction

  • Unified embodied reasoning faces two challenges: fixed data mixtures can drift from evolving learning needs, and heterogeneous outputs can compete for shared parameters.The paper terms these challenges Data-Model Drift and Modality Entanglement, respectively.Source: §1 · p. 3
  • The Agentic Data-Model Self-Evolution Engine revises supervision and capability-wise data mixtures in response to evaluation diagnoses of the model’s evolving needs.Its bounded, iterative process is intended to mitigate Data-Model Drift.Source: §1 · p. 3
  • Task tokens activate lightweight low-rank residuals for output-specific adaptation while a shared backbone retains cross-task learning.The strategy is intended to mitigate interference between heterogeneous outputs.Source: §1 · p. 4
  • ABot-ER achieves state-of-the-art overall performance, with the highest aggregate score in perception, planning, and execution within one unified framework.The evaluation covers twelve benchmarks and five capability dimensions.Source: §1 · pp. 3–4
  • A single post-training run on the engine-produced corpus matches the six-round model’s mean across twelve benchmarks with one-fifth of the accumulated post-training.The same corpus and recipe are also applied to backbones from 2B to 27B parameters without repeating the data-revision loop for each backbone.Source: §1 · p. 4

2 The Role of an Embodied Reasoning Model

An embodied reasoning model mediates between observations and downstream control by returning assessments, plans, or grounded targets rather than motor commands. ABot-ER serves five interdependent capability dimensions from one model, supporting both action selection and assessment of outcomes.

2 The Role of an Embodied Reasoning Model

  • The model returns task-relevant assessments, ordered plans, or grounded image coordinates; the runtime manages task progression and downstream policies execute motion.The model itself does not issue motor commands.Source: §2 · p. 4

2.1 What the Model Answers for the Robot

  • The five dimensions comprise three perception capabilities—spatial and relational reasoning, scene and state understanding, and temporal and event reasoning—and planning and execution-oriented grounding.Planning and execution-grounding outputs are treated as action-space outputs, distinct from semantic outputs by task function and interface.Source: §2.1 · p. 4
  • The dimensions are interdependent in a single action: placing an object draws on spatial, state, and grounding judgments, while recovering from a failed grasp also requires temporal recognition and replanning.A grounded release point must agree with the scene assessment that the surface is clear.Source: §2.1 · p. 4

2.2 Why One Dedicated Model

  • One model serves all five dimensions, providing both proposed actions and assessments of whether actions worked so the runtime can proceed, retry, or re-plan.ABot-ER serves these dimensions from a single 9B-parameter checkpoint.Source: §2.2 · p. 5

3 Related Work

Prior work spans unified embodied models, specialized perception and planning methods, and adaptive training, but uses varied task interfaces and supervision. ABot-ER brings these capabilities together through capability-level data revision and task-conditioned adaptation over a shared backbone.

3 Related Work

  • The related-work review organizes prior research around unified embodied models, perception and interaction grounding, planning and feedback, and adaptive data construction and multi-task training.Table 1 summarizes five capability dimensions and their answer forms, including participant-view directions and normalized robot-camera coordinates.Source: §3 · pp. 5–6, Table 1 · p. 6

3.1 Embodied Vision-Language Foundation Models

  • Prior unified models consolidate embodied capabilities in different ways, while ABot-ER exposes semantic assessments, action plans, and interaction targets without directly generating motor commands.The reviewed approaches include sensor-token integration, action generation, planning with separate control, and progressive multi-stage training.Source: §3.1 · p. 6

3.2 Embodied Perception and Interaction Grounding

  • Perception and grounding work covers spatial relations, interaction targets, scene states, and temporal reasoning, but uses task-specific supervision and output conventions.The review notes that integrating these studies requires explicit alignment of task definitions and interfaces.Source: §3.2 · p. 7

3.3 Task Planning and Feedback-Guided Execution

  • Planning approaches vary in plan representation, perception interface, and feedback mechanism, from semantic feasibility and code generation to online plan revision.Interaction benchmarks also require action selection conditioned on successive observations.Source: §3.3 · p. 7

3.4 Adaptive Data Construction and Multi-Task Training

  • ABot-ER’s engine revises supervision content and capability mixtures from evaluation-based diagnosis, while its low-rank branches are jointly trained with a shared backbone.The task token selects a semantic or action-space branch, rather than a separate branch for each benchmark or capability.Source: §3.4 · pp. 7–8

4 ABot-ER

ABot-ER formulates five embodied capabilities within a shared conditional-generation framework and uses a unified vision-language architecture. Task-conditioned low-rank residuals add task-family-specific capacity while the backbone remains shared.

4 ABot-ER

  • The five capability dimensions are covered by one model and formulated as conditional generation over shared embodied knowledge, with sequence-normalized causal cross-entropy training.The dimensions organize coverage rather than define separate training losses; relative exposure is controlled through data construction and sampling.Source: §4.1 · pp. 8–9

4.1 Capability Coverage and Unified Modeling Objective

  • The framework covers spatial and relational reasoning, scene and state understanding, temporal and event reasoning, long-horizon action organization, and point grounding and affordance.The objective is constrained by Modality Entanglement and Data-Model Drift, addressed respectively by task-conditioned training and the data-evolution engine.Source: §4.1 · pp. 8–9

4.2 Model Architecture

  • A decoder-only vision-language architecture shares one autoregressive backbone and output head across all dimensions, with visual inputs routed through an encoder and projector when present.The two task families are EQA and TASK_PLAN; coordinates use a shared normalized numeric convention while serialization follows benchmark-specific formats.Source: §4.2 · pp. 9–10

4.3 Task-Conditioned Adaptive Training Strategy

  • Task-conditioned low-rank residuals augment query and value projections in selected full-attention blocks, while all other parameters remain shared and are trained jointly.A task marker selects the branch once per sequence, and the selected branch remains active throughout decoding.Source: §4.3 · p. 10

5 Agentic Data-Model Self-Evolution Engine

The Agentic Data-Model Self-Evolution Engine uses evaluation-driven diagnosis to adapt supervision, output protocols, data exposure, and checkpoints as capabilities change. Its targeted training rounds address dimension-specific trade-offs and select checkpoints under regression guardrails, with evaluation across five dimensions and twelve benchmarks.

5 Agentic Data-Model Self-Evolution Engine

  • The engine diagnoses data faults and applies distinct or coordinated revisions: missing targets call for content construction, conflicting representations for protocol alignment, and under-exposure for mixture reallocation.It distinguishes training-data faults from evaluation-interface or integrity faults, which instead call for label checks or source audits.Source: §5 · p. 10, §5.1.1 · p. 12

5.1 Evaluation-Driven Capability Diagnosis

  • Diagnosis aggregates errors at the five capability-dimension level, admitting a gap when benchmark trends agree or item-level analysis supports it for a single-benchmark dimension.Fixed rules classify observable output and error records; they separate benchmark symptoms from capability conclusions to reduce Goodhart risk.Source: §5.1 · p. 11, §5.1.1 · p. 12

5.1.2 Agent Proposals and Deterministic Verification

  • A coordinating language-model agent proposes root causes, data requirements, and candidate sources, while deterministic checks gate builds before formal training.Evaluation items are excluded as construction sources, and archived cases trace missing-supervision, protocol-conflict, under-exposure, and contamination faults to later revisions.Source: §5.1.2 · pp. 12–13

5.2 Content Evolution: Targeted Supervision

  • Content evolution constructs or repairs task-relevant point targets and complete planning trajectories with checked preconditions and verified terminal states.Targets are evidence-grounded and independently verified; elastic construction bands permit documented deviations when the remaining items are costly to produce and do not materially change coverage or risk.Source: §5.2 · pp. 13–14

5.3 Protocol Evolution

  • VABench-P’s within-run range narrowed from 45.17 points to 3.46 points after the combined target-semantics rebuild and protocol alignment.The engine unifies coordinates as normalized [0, 1] values while preserving task-specific serialization; all 104,960 released post-training labels pass contract checks.Source: Table 4 · p. 15, §5.3 · p. 15

5.4 Mixture Evolution

  • The released point grounding and affordance dimension scored 67.06, while quota adjustment alone raised its score from 50.58 to 57.39 but did not resolve checkpoint variation.Exposure is adjusted only after content and output requirements are checked; R6 allocations differed by accounting basis, with question-answering-to-planning at 75 : 25 by unique records and 29.71 : 70.29 by assistant supervision tokens.Source: §5.4 · pp. 16–17

5.5 Data Governance and Evaluation Integrity

  • A contamination audit found that an R3 EB-ALFRED score of 100 came from only 12 of 300 episodes, while the recomputed score of 27 used contaminated weights.Three implicated trajectory sources were quarantined in full, removing 23,565 candidate-pool records; R3 scores are excluded from public performance claims.Source: §5.5 · p. 17

5.6 Training, Selection, and Evolution Across Rounds

  • The released model uses Stage A to establish joint competence and Stage B to emphasize remaining bottlenecks while retaining other-task data to mitigate forgetting.Stage A showed asymmetric convergence: perception saturated before planning and execution, which continued to improve with longer exposure.Source: §5.6 · p. 18
  • Stage B uses supervised fine-tuning from the selected Stage A checkpoint on data rebalanced toward bottleneck dimensions, with a reduced learning rate and saturated-dimension anchors to limit forgetting.It uses a substantially shorter schedule than Stage A.Source: §5.6 · p. 18
  • Checkpoint selection filters candidates through per-round retention guardrails, then ranks survivors by leading-result count, targeted mean, and nine-benchmark question-answering mean.Leading results are strict wins over frozen public models, counted within one checkpoint; targets and guardrails together cover all twelve benchmarks.Source: §5.6 · p. 18
  • Rounds close retrospectively when two successive checkpoints fail to add targeted leading results and either the targeted mean declines or a guardrail is breached.Otherwise, a round closes at its configured budget; the data version is frozen when remaining bottlenecks cannot be addressed by revising the mixture or source pool.Source: §5.6 · pp. 18–19
  • The released model is 0.93 points higher in point grounding and affordance and 3.96 points lower in long-horizon action organization than R6.Source: §5.6 · p. 19
  • Across rounds, gains shifted from structural fixes to formalized data and evaluation controls, then targeted single-digit dimension gains, with non-monotonic trade-offs.Source: §5.6 · pp. 19–20
  • A single post-training pass achieves a similar twelve-benchmark mean with one-fifth the accumulated post-training, but favors question answering and single-shot planning over interactive planning.Both conditions share an ancestor checkpoint and use the same corpus for their final pass; Table 6 compares them at the same point in that pass.Source: Table 6 · p. 20, §5.6 · p. 20

6 Experiments

The experiments assess ABot-ER across twelve benchmarks and five capability dimensions, using shared training releases and an evaluation protocol intended to make comparisons consistent. The section also specifies how benchmark results are aggregated and how public baselines are compared.

6 Experiments

  • The experiments test unified-model competitiveness, transfer of one training recipe across backbone sizes, and the effects of task conditioning and data-model co-evolution.Real-world demonstrations illustrate model outputs within robotic systems.Source: §6 · p. 20

6.1.1 Capability Dimensions and Benchmarks

  • The evaluation covers twelve benchmarks across perception, long-horizon planning, and execution-oriented point grounding and affordance.The benchmark groups overlap: the EQA suite includes perceptual judgments and pointing tasks, and point-hit rates do not measure physical task completion.Source: §6.1.1 · pp. 20–21

6.1.2 Training Data

  • Both training stages use the converged releases, assembled from public training splits and simulator rollouts and organized by five capability dimensions.Stage A contains about 462k physical records; Stage B about 105k, with planning receiving a quarter of its records and close to two thirds of assistant supervision tokens.Source: §6.1.2 · p. 21

6.1.3 Metrics and Aggregation

  • Metrics vary by task, while competency means use fixed benchmark groups and the twelve-benchmark mean is not an official leaderboard aggregate.Perception averages seven benchmarks, planning three, and execution two; the overall mean mixes score-like metrics and task-success rates.Source: §6.1.3 · p. 22, Table 7 · p. 22

6.1.4 Inference Protocol

  • Inference uses one shared model with task-family tokens; point-grounding benchmarks use the EQA condition, and audited EQA runs use greedy decoding without multi-sample ensembling.The audited point predictions are normalized to the original image, while provenance of the EB archives’ original online calls remains unverified.Source: §6.1.4 · p. 23

6.1.5 Baseline Models

  • Public-baseline comparisons use available benchmark results, but differences in model size, training data, and original evaluation settings limit mechanistic attribution.Public scores come from dated Embodied Arena and local snapshots; the paper’s inference settings apply only to explicitly reevaluated results.Source: §6.1.5 · p. 23

6.2 Results Across Embodied Capabilities

ABot-ER combines strong results across the three competencies, and the experiments examine whether a shared training recipe transfers across model sizes and how post-training and task conditioning affect results.

6.2 Results Across Embodied Capabilities

  • ABot-ER improves over the strongest listed baselines by 5.70 points on RoboVQA and 5.78 on Cosmos, and records the best listed ERQA result.The results describe strength in scene understanding and temporal reasoning.Source: §6.2 · p. 23
  • ABot-ER has the highest planning-suite mean among listed models with complete results, leads on EB-ALFRED and EB-Habitat, and has a 9.34-point EB-ALFRED margin.The reported task outcomes do not establish stronger feedback correction or native closed-loop execution.Source: §6.2 · p. 23
  • ABot-ER leads the perception, planning, and execution means by 3.94, 4.20, and 11.81 points, respectively, and achieves the best listed result on six of twelve benchmarks.The three strongest baseline values come from different models; aggregate comparisons use identical benchmark sets and are not official leaderboard scores.Source: §6.2 · p. 23, Table 8 · p. 24
  • ABot-ER exceeds Cosmos3-Nano by 15.79 percentage points on VABench-P, but its relative standing on Where2Place is less favorable.Point-hit rates are distinct from control precision and physical manipulation success.Source: §6.2 · p. 24

6.3 One Recipe Across Model Sizes

  • One shared pair of releases transfers across 2B, 4B, and 27B dense backbones and a 35B mixture-of-experts model, without size-specific data diagnosis or revision.All five dimensions are populated after training; the 2B model reaches usable levels across them, and larger backbones retain that profile while lifting it.Source: §6.3 · pp. 24–25
  • From 2B to 27B, perception rises 9.85 points, execution 15.16, and planning 17.45; with 3B active parameters, the mixture-of-experts model falls between the 9B and 27B dense models on suite means.The three competencies do not move together with scale.Source: §6.3 · pp. 24–25

6.4 Effect of the Task-Conditioned Adaptive Training Strategy

  • The post-training stage raises the planning mean by 2.18 points and the nine-benchmark question-answering mean by 0.30, while Where2Place regresses by 2.11 points.The planning gains accompany a quarter of the release’s physical records and close to two thirds of its assistant supervision tokens devoted to planning; perception benchmarks move by at most 1.92 in either direction.Source: §6.4 · pp. 25–26, Table 11 · p. 26
  • On an unaligned corpus, conditioning raises the twelve-benchmark mean by 1.36 and improves eleven benchmarks; after repair, suite means slightly favor the unconditioned run, while ET-Plan-Bench favors conditioning at all 17 matched checkpoints by 3.3 points on average.The authors treat the corpora comparison as an interpretation rather than a controlled result because the corpora differ in more than one respect.Source: §6.4 · p. 26, Table 12 · p. 27

6.5 Analysis of Data-Model Self-Evolution

The self-evolution analysis tracks capability changes across versions and benchmarks, emphasizing tradeoffs and the need to retain existing strengths. Real-world demonstrations show ABot-ER supplying visual judgments and grounded targets to a robot runtime that coordinates decisions and actions.

6.5 Analysis of Data-Model Self-Evolution

  • The compared versions use different training budgets and often restart from the pretrained model, so their differences do not isolate data-strategy effects.The analysis describes the development of the released model line rather than a controlled sweep over data recipes.Source: §6.5 · p. 27, Table 13 · p. 27
  • Spatial and relational scores stabilize after an early increase, while planning and execution change more visibly in later stages.Figure 5 summarizes reported trends, not directly comparable absolute abilities or validated radar-area totals; historical scoring and protocol effects remain.Source: §6.5 · pp. 27–28, Figure 5 · p. 28
  • Every transition mixes gains and losses; from R5 to R6, EB-ALFRED rises 8.7 points while ET-Plan-Bench, Where2Place, and EB-Habitat fall 8.0, 4.3, and 4.7.The relevant objective is broad competitiveness within one model, not combining benchmark peaks from different versions.Source: §6.5 · p. 28, Figure 6 · p. 29
  • Within R2, ET-Plan-Bench recovers from 32.6 to 68.6 but OpenEQA ends at 57.1 below its R1 level of 60.7; seven benchmarks follow the OpenEQA pattern.VABench-P swings between 8.6 and 53.8 across consecutive checkpoints, and its instability is traced to conflicting coordinate supervision.Source: §6.5 · p. 28, Figure 6 · p. 29
  • Checkpoint selection requires leading results on targeted benchmarks without breaching a retention guardrail, balancing improvement breadth against retention.The resulting ledger includes local regressions alongside gains.Source: §6.5 · p. 28

6.6 Real-World Deployment

  • In deployment, ABot-ER provides scene, spatial, and temporal judgments plus grasp and placement points; ABot-AgentOS uses those outputs to decide whether to continue or revise actions.The demonstrations include block arithmetic and gesture validity assessment, while the runtime—not ABot-ER—takes the decision.Source: §6.6 · pp. 29–30, Figure 7 · p. 30

7 Conclusion and Future Work

ABot-ER unifies perception, planning, and execution in one model, pairing a shared embodied-reasoning backbone with task-conditioned adaptation and evaluation-driven data revision. Across twelve benchmarks, it leads the public models compared on all three competency means; the authors identify environment-feedback learning and tighter reasoning-control coupling as future directions.

7 Conclusion and Future Work

  • ABot-ER combines a shared embodied-reasoning backbone with task-family-routed low-rank residuals and an evaluation-feedback loop for diagnosing and revising training data.The model uses one set of weights for perception, planning, and execution, without switching weights or adding an inference decoding stage.Source: §7 · p. 31
  • Across twelve benchmarks, ABot-ER leads public models on the perception, planning, and execution competency means.The evaluation covers three core competencies and five capability dimensions.Source: §7 · p. 31
  • The authors plan to extend self-evolution from supervised fine-tuning to environment feedback and to couple reasoning and control more tightly while keeping them separate.They also identify the corpus, evaluation contract, and audit trail as artifacts to maintain alongside the checkpoint.Source: §7 · p. 31

A Self-Evolution Engine: Full Records

The appendix gathers construction quantities, release audits, data compositions, training configurations, and the engine’s per-round ledger.

A Self-Evolution Engine: Full Records

  • The appendix collects the engine’s construction quantities, release audits, data compositions, training configurations, and per-round ledger.Accounting bases and round attributions are stated in the relevant captions and table notes.Source: A Self-Evolution Engine: Full Records · p. 38

A.1 Supervision Construction and Release Audit

The release audit reports screening coverage, hard-gate outcomes, and the scope of the supervision-construction table. Both released stages pass the hard gates, with documented residual risk.

A.1 Supervision Construction and Release Audit

  • Table 14 describes supervision construction by capability dimension; its R6 development-release count is 160,000 unique identities materialized twice as 320,000 physical records.The caption distinguishes this development release from the released model’s Stage A composition.Source: Table 14 · p. 38
  • The external perceptual-hash index covers 408,502 of 707,758 distinct media paths in the joint audit (57.72%); uncovered entries remain residual risk.No evaluation-media overlap was found within the covered index.Source: §A.1 · p. 38
  • Both released stages pass the reported hard gates, with documented residual risk; unrecorded quantities are not reported as zero.The audit distinguishes quantities that were not recorded separately from measured zero values.Source: Table 15 · p. 39

A.2 Stage Compositions and Training Configuration

The released training stages have distinct composition and selection descriptions, with targeted post-training governed by a question-answering floor and retention bounds. Task-planning supervision occupies a larger share of assistant tokens than of physical records.

A.2 Stage Compositions and Training Configuration

  • 64.32 is the nine-benchmark question-answering mean floor for the targeted post-training run; separate retention, parse-rate, and test-overlap bounds also applied.The run targeted VSI, VABench-P, Where2Place, Cosmos, and OpenEQA.Source: §A.2 · p. 39
  • Stage A’s physical-record composition is summarized in Table 16; its long-horizon planning partition contains 42,236 general, 1,598 Habitat-grounded, and 349 ALFRED-grounded plans.The release has no regenerated token-proxy report, so no token-proxy share is stated.Source: Table 16 · p. 40
  • Stage B’s Final Joint closure set is rebalanced toward planning recovery under a question-answering guardrail, with each unique identity materialized twice.Source: Table 17 · p. 40
  • 25.00% of physical records but 63.76% of assistant supervision tokens are task planning; point supervision accounts for 25.53% and 29.96%, respectively.The passage attributes the task-planning share difference to long structured-plan labels.Source: §A.2 · p. 40

A.3 Per-Round Ledger

The per-round ledger distinguishes diagnostic and continuation-chain results, while clarifying how its training history relates to the released-stage settings.

A.3 Per-Round Ledger

  • Per-round hyperparameters are omitted; Table 18 gives the released-stage differences and shared settings.The shared settings include a frozen vision encoder and trainable connector, decoder, and task-conditioned residuals.Source: Table 18 · p. 41, §A.3 · p. 41, Table 19 · p. 42
  • Weight origin makes clear that the per-round history is not a single continuation chain.R3 is diagnostic, and R4 scores come from a same-recipe continuation chain rather than R4’s own checkpoint.Source: Table 19 · p. 42
  • R3 is diagnostic, and R4 scores were measured on a same-recipe continuation chain rather than its own checkpoint.Source: Table 19 · p. 42

Related papers

Loading 2610.18355v2…