PaperlayerView exact revision on arXiv

Source-linked AI summary

ASENA: Self-evolving Agents for Embodied Navigation

An-Chieh Cheng, Isabella Liu, Edmund Bu, Johan Bjorck, Hongxu Yin, Zhengyi Luo, Jan Kautz, Linxi "Jim" Fan, Yuke Zhu, Sifei Liu

arXiv:2609.39207v1cs.RO

TL;DR

Embodied tasks can require observation-dependent search, inspection, and communication beyond reaching a destination. ASENA provides coding agents with supervised robot programming and persistent skills, alongside an optional monocular navigation policy. The paper reports leading navigation and EQA performance, with workspace evolution and real-robot demonstrations, while noting that real-world tasks take 10–20 minutes.

  • Problem

    Embodied tasks can require deciding what to inspect, compute, and communicate based on new observations, and may extend beyond predefined skills.

    Source: §1 · p. 2

  • Method

    ASENA connects coding agents to robot sensing, supervised execution, and a persistent workspace, and offers an optional 4B monocular policy trained on route and geometry-derived atomic navigation data.

    Source: §1 · p. 2, §3.3 · p. 6

  • Results

    ASENA-VLN achieves leading R2R/RxR success rates with fewer agent interactions, and ASENA achieves leading EQA accuracy; coding agents also navigate effectively without a learned policy.

    Source: §4.3 · p. 8, §5 · pp. 10–11

  • Takeaways & Limitations

    On a Unitree G1, online programming extends navigation to search, inspection, spatial reasoning, and synthesized gestures without a pre-built map.

    Source: §5 · p. 11

    Real-world tasks take 10–20 minutes, motivating faster reasoning and execution; future work will strengthen safeguards for dynamic environments and extend workspace evolution to learned policy updates.

    Source: §5 · p. 11

Abstract

from arXiv · show

We present ASENA, an embodied agent system that connects general-purpose coding agents to robot sensing, computation, supervised execution, and persistent experience. Agents can write and execute programs, inspect recorded outcomes, repair failures, and reuse notes and executable skills while keeping their model weights fixed. We further introduce ASENA-VLN, a 4B monocular navigation policy that serves as an optional tool within this programmable system. ASENA-VLN predicts body-frame trajectories for both extended routes and short-horizon behaviors using a shared vision-language decoder trained on route instructions, visual question answering, and a newly curated dataset of geometry-derived atomic navigation tasks. As a standalone policy, ASENA-VLN achieves state-of-the-art success rates of 68.7% on R2R and 70.2% on RxR. When integrated with a coding agent, learned navigation improves ASENA's success rate by 11 percentage points on both agentic benchmarks while reducing execution time. Through persistent workspace evolution and simulator feedback, ten passes over recurring 100-task subsets further improve success from 72% to 98% on R2R and from 65% to 89% on RxR. On embodied question answering, ASENA achieves state-of-the-art accuracy with fewer interaction steps. Finally, real-world demonstrations on a Unitree G1 combine search, visual inspection, spatial reasoning, and synthesized gestures without a pre-built map, illustrating how online programming extends robot behavior beyond route following and predefined skills.

1. Introduction

ASENA gives coding agents a supervised, persistent programming interface for embodied tasks whose next steps depend on observations. It pairs this system with an optional learned navigation policy and evaluates fixed-weight workspace evolution.

1. Introduction

  • A vending-machine task illustrates why embodied work can require search, recognition, remembering the user’s location, and reporting—not just reaching a destination.The next step depends on what the robot observes.Source: §1 · p. 2
  • Coding agents can construct and revise procedures during a mission using sensing, computation, simulation, and action, including tasks beyond predefined skills.Physical use also calls for feedback tied to outcomes, safeguards, and a workspace that retains useful procedures.Source: §1 · p. 2
  • ASENA connects robot sensing, computation, code execution, outcome inspection, and persistent notes and skills through shared interfaces and supervised execution.Programs can be revised while model weights remain fixed.Source: §1 · p. 2
  • ASENA-VLN is a 4B monocular policy trained on route instructions and geometry-derived navigation atoms, predicting body-frame trajectories for routes and short-horizon goals.A new atomic-navigation dataset supervises reusable fine-grained behaviors; the policy can be called, combined with geometric navigation, or omitted.Source: §1 · p. 2
  • The paper evaluates navigation and embodied question answering in simulation and demonstrates complex instruction following and user interaction on a Unitree G1 without a pre-built map.Its stated contributions include inspectable supervised programming, a route-and-atomic-skill navigation policy, and fixed-weight workspace evolution.Source: §1 · p. 2

2. Related Work

Related work spans programmatic robot control, agent self-improvement, language-guided navigation, and embodied question answering. ASENA studies persistent workspace evolution around a pretrained navigation policy while keeping model weights fixed.

2. Related Work

  • Code as Policies, CaP-X [7], and ASPIRE [8] use programs to compose perception and control, benchmark API abstraction and feedback, or diagnose and repair robot programs.ASPIRE also retains reusable skills.Source: §2 · p. 2
  • Voyager, Reflexion, and ENPIRE [11] retain skills or feedback, with ENPIRE also coupling real-robot verification to program synthesis and policy training.ASENA instead studies workspace evolution around a pretrained navigation policy with all model weights fixed.Source: §2 · pp. 2–3
  • ASENA connects coding agents to robots through sensing, a persistent workspace, supervised execution, and an optional learned navigation policy.Runs record robot states, actions, feedback, code, and visualizations for inspection and replay.Source: Figure 2 · p. 3
  • Prior language-guided navigation includes topological methods, video-to-action policies, model composition, motion primitives, and unified language–vision–action translation [18].ASENA-VLN is described as a lightweight 4B video-to-action policy with composable atomic capabilities.Source: §2 · p. 3
  • Embodied question-answering work explores active evidence gathering, spatial memories, long-horizon multi-target questions, and efficient exploration [22].ASENA uses these settings to assess composition of perception, navigation, and memory tools, including persistent notes and executable skills.Source: §2 · p. 3

3. Method

ASENA grounds coding-agent programs in robot feedback, execution safeguards, and retained workspace experience. An optional learned policy adds language-conditioned motion through a shared decoder trained on route and atomic-navigation data.

3. Method

  • ASENA provides a unified write–run–inspect workflow for sensing, computation, visualization, simulation, and action, with physical safeguards and recorded runs for later repair and reuse.A learned navigation policy is available as an optional tool within the same interface.Source: §3 · p. 3

3.1. Grounding Programs in Robot Feedback

  • ASENA exposes robot sensing and visualization for geometry estimation, object inspection, and execution monitoring, so agents can use recorded evidence to revise programs when actions diverge from intent.Generated physical actions pass through an execution interface and controller; callable atomic skills include approaching objects, passing doorways, speaking, and gesturing.Source: §3.1 · p. 4
  • Synthesized gestures are checked in MuJoCo for pose continuity, motion limits, and sampled self-collision, while navigation uses occupancy-map clearance checks and independent motion safeguards.Validated motion requires operator approval; heartbeat stops and manual emergency halts add further protection.Source: §3.1 · p. 4

3.2. Turning Execution into Retained Experience

  • Execution traces and feedback update a workspace of notes and executable skills while model and controller weights remain fixed.On physical hardware, revised programs still require independent safety checks and operator approval; the supplied Table 1 and Table 2 captions identify standalone VLN and object-goal evaluation scopes without giving result values.Source: §3.2 · p. 5, Table 1 · p. 5, Table 2 · p. 5

3.3. Learning a Reusable Visual Navigation Policy

  • ASENA-VLN is an optional policy that predicts bounded body-frame trajectories from language and recent monocular observations, while the coding agent decides when to invoke it and monitors progress.Its shared decoder predicts waypoint trajectories; curated atomic tasks use scene geometry, geodesic distance, and visibility to define feasible, measurable motions, complementing route training.Source: §3.3 · p. 6
  • Training also includes multi-view visual question answering, and trajectory predictions—not pixel goals—are used in the reported evaluations.Pixel goals require an additional motion planner or simulator shortest-path follower; training uses a 3:1 mixture of trajectory prediction and spatial VQA.Source: §3.3 · pp. 6–7

4. Evaluation and Applications

The evaluation examines ASENA-VLN as a standalone route-following policy and as a tool in object search and coding-agent navigation, then tests workspace evolution, embodied question answering, and real-world missions. Results show complementary benefits from learned navigation and retained experience, alongside broader robot capabilities on a Unitree G1.

4. Evaluation and Applications

  • The evaluation covers standalone route following on shorter R2R and longer RxR instructions, object-goal search, agentic navigation, workspace evolution, EQA, and physical-robot missions.Standalone policy metrics include navigation error, oracle success, success rate, path efficiency, and route similarity.Source: §4 · p. 7, §4.1 · p. 7

4.2. Object-Goal Navigation

  • On HM3D-v2 and HM3D-OVON, combining global-search reasoning with ASENA-VLN achieves state-of-the-art object-goal results, consistently across seen, synonymous, and unseen object categories.A separate VLM plans where to search and updates the plan from new observations; ASENA-VLN supplies local navigation.Source: §4.2 · p. 7

4.5. Embodied Question Answering

  • 81.2% HM-EQA accuracy at pass 17 and 66.5% MT-HM3D accuracy at pass 4 are the highest frozen-workspace results reported for those benchmarks.At low reasoning effort, evolved workspaces use steps comparable to the hand-designed efficient configuration while achieving higher accuracy; outcomes vary across revisions.Source: Table 5 · p. 8, Table 4 · p. 8, §4.5 · p. 10

4.3. Complementary Agent and Policy Capabilities

  • Coding agents can navigate effectively without a learned policy: GPT-6 Astra achieves state-of-the-art performance and approaches reported human performance of 90% SR on R2R.Its geometric and programmatic tools support instruction interpretation, route construction, progress monitoring, and error recovery.Source: §4.3 · p. 8
  • 11 percentage points of SR improvement on both R2R and RxR come from adding ASENA-VLN to Claude Sonnet 5; GPT-6 Astra benefits less and slightly loses RxR success.The policy improves success and path efficiency for Claude Sonnet 5, while its value depends on the coding backend and integration into plans.Source: §4.3 · p. 8

4.4. Retained Experience Improves Recurring Tasks

  • Across all four recurring-task campaigns, success improves; tool-equipped workspaces retain higher SPL on both benchmarks while worker and improver costs decrease.Variants with and without ASENA-VLN reach similar final R2R success, while the policy retains a clearer advantage on longer RxR instructions; model, policy, and controller weights remain fixed.Source: §4.4 · p. 9

4.5. Embodied Question Answering

  • Approximately 19% lower median normalized exploration steps on 132 MT-HM3D questions accompany little change in accuracy when ASENA-VLN is enabled.The paired ablation aligns with earlier R2R and RxR results linking learned navigation with fewer repeated agent interactions.Source: §4.5 · p. 10

4.6. Real-World Applications

  • Three supervised G1 missions in unseen environments without a pre-built map demonstrate search, online evidence gathering, spatial reasoning, memory, and multimodal interaction.Tasks include returning directions, reporting inspected coffee machines, and comparing a snack before generating a whole-body gesture; the figure shows search and return traces with odometry-estimated distances.Source: Figure 5 · p. 9, §4.6 · p. 10

5. Discussion and Conclusion

ASENA combines supervised coding-agent programming with persistent skills, while ASENA-VLN supplies optional learned navigation. The paper reports fixed-weight workspace improvement and real-world demonstrations, while noting that mission duration motivates faster reasoning and execution.

5. Discussion and Conclusion

  • ASENA unifies coding-agent programming, supervised execution, and persistent skills, improving recurring-task performance while model weights remain fixed.Its optional 4B monocular ASENA-VLN policy uses curated atomic-navigation data and achieves leading R2R/RxR success rates with fewer agent interactions; ASENA also achieves leading EQA accuracy.Source: §5 · pp. 10–11
  • On a G1, online programming extends navigation to search, inspection, and interaction, including whole-body responses synthesized and validated online.Source: §5 · p. 11
  • Real-world tasks take 10–20 minutes, motivating faster reasoning and execution; future work includes stronger dynamic-environment safeguards and workspace evolution that updates learned policies.Source: §5 · p. 11

A.1. Atomic Instruction Format and Training Data

The appendix describes ASENA-VLN’s stored navigation annotations, training pools, and sampling schedule. Training has sampled only part of the larger trajectory pool by the reported checkpoint, with extra trajectory-ending windows used to emphasize stopping behavior.

A.2. Atomic Instruction Format and Training Data

  • Stored navigation annotations pair goal text with quality and optional end-pose anchors, while coordinate-based point goals use a separate input format.Source: Table 6 · p. 14
  • Training pools account for quality and backward-action filtering, alternative views, chain-of-thought variants, repeated trajectory-ending windows, and sampling weights; pool counts describe availability, not samples consumed.Source: Table 7 · p. 14
  • At step 8,400, training had drawn 4,300,800 samples including repeats and reached epoch 0.1178, so only part of the larger trajectory pool was sampled in an epoch.The VQA pool sets the epoch length at 71,309 updates; five extra copies of each eligible trajectory-ending window emphasize stopping behavior.Source: §A.1 · p. 14

A.2. Jetson Thor Quantization and Runtime

The appendix evaluates quantized ASENA-VLN inference on the G1’s Jetson AGX Thor using an earlier checkpoint, and compares runtime components and prediction agreement with BF16. Reduced-vocabulary FP8 lowers summed component latency, while agreement checks are explicitly distinct from navigation-success evaluation.

A.2. Jetson Thor Quantization and Runtime

  • Runtime measurements use an earlier step-8,400 checkpoint, batch size 1, an eight-frame history, and 32 greedy decoding steps; every FP8 variant quantizes the language model.Variants labeled BF16 vision keep the 415M-parameter vision encoder in BF16; the embedding and output projection remain unquantized.Source: §A.2 · p. 14, Table 8 · p. 15
  • 2,806 to 877 ms is the reduction in summed component latency for FP8 with reduced vocabulary, a 3.20× speedup over PyTorch BF16 under the same input settings.Decoding remains the largest FP8 cost at 683.4 ms, or 78% of the total; among BF16-vision variants, reduced vocabulary cuts decoding from 847.4 to 694.1 ms (18.1%).Source: §A.2 · p. 15, Table 8 · p. 15
  • Keeping the vision encoder in BF16 improves first-waypoint and full-trajectory agreement with BF16, but these held-out prediction-consistency checks do not measure navigation success.The checks use 256 held-out inputs; on-device checks also confirmed valid navigation-token IDs and correctly formatted eight-waypoint trajectories.Source: §A.2 · p. 15, Table 9 · p. 15

A.3. Atomic Navigation Dataset Samples

The atomic navigation examples pair observations with instructions and overlaid trajectories, spanning both metric motion and varied spatial goals. Sample instructions include movement, turning, object-relative navigation, and room finding.

A.3. Atomic Navigation Dataset Samples

  • “Stop after you take 4 steps forward” and “turn all the way around” illustrate distance-based motion and rotation instructions.Source: §A.3 · p. 16
  • Other examples specify object-relative movement, room finding, doorway entry, and corridor traversal.They include walking to a dishwasher, sidestepping toward a bathroom fan, finding a bedroom, entering a kitchen through a door, and stopping at a corridor’s end.Source: §A.3 · p. 16
  • The examples span metric motion, rotation, relative goals, alignment, room goals, doors, corridors, and floor changes.Each observation is paired with an instruction and an overlaid trajectory; the overlays do not use metric camera projection.Source: Figure 6 · p. 16

A.4. Navigation Instructions and Policy Inputs

Navigation policy inputs combine goal text and a quality setting, with an optional end-pose anchor specifying where the robot should finish. Directions and constraints remain in the instruction text, and the policy also receives visual history.

A.4. Navigation Instructions and Policy Inputs

  • Each policy input contains goal text and a quality setting, with an optional end-pose anchor describing the desired finish relative to an object, room, or starting position.Table 10 presents examples of these JSON inputs.Source: Table 10 · p. 17, §A.4 · p. 17
  • Directions, route cues, and constraints stay in the instruction text, while the policy receives the input together with visual history.Source: §A.4 · p. 17
  • End-pose distances are expressed in meters and rounded to two decimal places; room goals use the relation “inside” with distance 0.0.All five examples use quality 5.Source: §A.4 · p. 17

A.5. G1 Hardware and Navigation Settings

The G1 hardware setup combines panoramic vision, LiDAR, and local sensor and motion services. Navigation settings control walking behavior, while the robot’s head and body partially occlude the inverted LiDAR’s view.

A.5. G1 Hardware and Navigation Settings

  • The G1 carries a mast-mounted panoramic camera and inverted Livox Mid-360 LiDAR, with a Jetson AGX Thor running local sensor and motion services.The camera interface supports the Insta360 X5 and Ricoh Theta X; Tables 11 and 12 summarize sensor/state outputs and navigation settings.Source: §A.5 · p. 18, Table 11 · p. 18, Table 12 · p. 18

A Front view

  • The sensor-mounting views show the G1’s sensor placement, while its head and body partially occlude the inverted LiDAR’s view.Camera yaw correction sets the panorama’s horizontal orientation.Source: Figure 7 · p. 18, A Front view · p. 18
  • Slow-walk settings control speed, stopping tolerances, command size, and timeouts, and can be adjusted at runtime.A separate walking gait is also available.Source: A Front view · p. 18, Table 12 · p. 18
Loading 2609.39207v1…