PaperlayerView exact revision on arXiv

Source-linked AI summary

AnywhereVLA: Language-Conditioned Exploration and Mobile Manipulation

Konstantin Gubernatorov, Artem Voronov, Roman Voronov, Sergei Pasynkov, Stepan Perminov, Ziang Guo, Dzmitry Tsetserukou

arXiv:2509.21006v1cs.ROcs.AI

TL;DR

Natural-language mobile manipulation in unseen indoor environments requires robots to integrate instruction understanding, exploration, and physical interaction. AnywhereVLA combines classical SLAM, exploration, and navigation with a fine-tuned VLA manipulation module. It achieves 46% overall success with fine-tuned SmolVLA, compared with 10% without that fine-tuning.

  • Problem

    Integrating language-based control, exploration, and manipulation in expansive environments remains challenging, while many VLA models have limited spatial awareness in unseen or occluded regions.

    Source: §I · p. 1

  • Method

    AnywhereVLA combines classical SLAM, exploration, and navigation with VLA manipulation in a modular pipeline that translates language instructions into control actions and navigation trajectories.

    Source: §I · pp. 1–2

  • Results

    46% overall SR with fine-tuned SmolVLA versus 10% overall SR without SmolVLA fine-tuning on the purpose-built pick-and-place dataset.

    Source: §VI · p. 6

  • Takeaways & Limitations

    The framework reports real-time performance exceeding 10 Hz across modules deployed on consumer-available edge computing units.

    Source: §I · p. 2

    The discussion identifies unresolved object-level versus spatial-prepositional reasoning and proposes future semantic parsing and relational reasoning to address it.

    Source: §VII · pp. 6–7

Abstract

from arXiv · show

We address natural language pick-and-place in unseen, unpredictable indoor environments with AnywhereVLA, a modular framework for mobile manipulation. A user text prompt serves as an entry point and is parsed into a structured task graph that conditions classical SLAM with LiDAR and cameras, metric semantic mapping, and a task-aware frontier exploration policy. An approach planner then selects visibility and reachability aware pre grasp base poses. For interaction, a compact SmolVLA manipulation head is fine tuned on platform pick and place trajectories for the SO-101 by TheRobotStudio, grounding local visual context and sub-goals into grasp and place proposals. The full system runs fully onboard on consumer-level hardware, with Jetson Orin NX for perception and VLA and an Intel NUC for SLAM, exploration, and control, sustaining real-time operation. We evaluated AnywhereVLA in a multi-room lab under static scenes and normal human motion. In this setting, the system achieves a $46\%$ overall task success rate while maintaining throughput on embedded compute. By combining a classical stack with a fine-tuned VLA manipulation, the system inherits the reliability of geometry-based navigation with the agility and task generalization of language-conditioned manipulation.

I. INTRODUCTION

The introduction frames large-scale language-guided mobile manipulation as an integration challenge: robots must explore unfamiliar spaces while understanding instructions and manipulating objects. AnywhereVLA combines classical navigation and SLAM with VLA-based task grounding, using one language instruction to condition exploration and manipulation.

I. INTRODUCTION

  • Integrating language-based control, environment exploration, and manipulation in expansive environments remains a significant challenge.The introduction situates this challenge amid growing interest in natural-language task specification for service robotics.Source: §I · p. 1
  • Existing VLA models often have limited spatial awareness and remain confined to localized settings, impeding operation in unseen or occluded regions.The introduction contrasts this with classical navigation stacks, which map and explore unknown spaces but lack language comprehension and semantic reasoning.Source: §I · p. 1
  • AnywhereVLA combines classical SLAM and navigation with VLA scene understanding and task grounding for mobile manipulation in unseen environments.Its pipeline converts high-level language instructions into low-level control commands and computes navigation trajectories.Source: §I · pp. 1–2
  • A single language-based instruction conditions environment exploration and navigation while also driving VLA manipulation.The architecture parses the task into simpler actions that further condition active environment exploration.Source: §I · p. 2, Figure 1 · p. 2
  • The framework reports real-time performance exceeding 10 Hz across modules deployed on consumer-available edge computing units.The stated goal is efficient, responsive operation in dynamic settings.Source: §I · p. 2

II. RELATED WORK

Related work spans VLA manipulation, efficient embedded policies, and integrated mobile-manipulation systems. The section distinguishes these approaches by their spatial awareness, need for prior maps, and exploration efficiency.

II. RELATED WORK

  • VLA methods ground instructions and generalize across tasks, but have limited spatial awareness [11], [12],.MoManipVLA jointly plans base and arm motions, while π0 and π0.5 use pretrained vision-language knowledge and robotic data [11], [12],.Source: §II · p. 2
  • SmolVLA, EdgeVLA, and TinyVLA target efficient deployment, but reported generalization is primarily within manipulation domains [17],, [19].The section reports a 7× inference speedup for EdgeVLA while maintaining task accuracy.Source: §II · p. 2
  • RDT-1B and AC-DiT focus on manipulation, so they require complementary modules for spatial memory, target discovery, and long-horizon navigation, [21].Source: §II · p. 2
  • BUMBLE navigates building-scale environments but relies on a known whole-environment map and preprovided landmarks [14].ASC demonstrates apartment-scale pick-and-place with spatial memory, while complete exploration and manipulation take 10–15 minutes [22].Source: §II · p. 3

III. ANYWHEREVLA ARCHITECTURE

AnywhereVLA uses instruction-conditioned exploration and manipulation over a semantic 3D map. Its map construction associates LiDAR points with image detections, then aggregates object points into persistent class-level geometry and confidence for downstream planning.

III. ANYWHEREVLA ARCHITECTURE

  • The four-module workflow parses a language instruction, builds a semantic 3D point-cloud map, explores for the target, approaches it, and executes manipulation with fine-tuned SmolVLA.Active Environment Exploration uses target-class-conditioned frontier exploration and halts when the target is detected and localized; Approach uses a projected 2D grid map.Source: §III · p. 3

A. 3D Semantic Mapping with Confidence (SM)

  • Semantic mapping synchronizes RGB images, undistorted LiDAR clouds, and 2D detections, associating projected LiDAR points with detections to build a metric-semantic map.The process densifies scans, voxelizes points, backprojects near-surface points, transforms them to the world frame, and accumulates them per class.Source: A. 3D Semantic Mapping with Confidence… · p. 3

B. Object Aggregation

  • Object aggregation clusters per-class points with DBSCAN, filters outliers, and summarizes each cluster by centroid, covariance, and confidence.Confidence fuses point density, angular coverage, inlier count, and detector scores; the resulting map provides persistent object labels and geometry for planning and manipulation.Source: B. Object Aggregation · pp. 3–4

C. LiDAR densification

LiDAR densification addresses sparse returns within image detections by interpolating between adjacent samples in the same azimuth bin.

C. LiDAR densification

  • Sparse, discontinuous LiDAR scans can leave few or no valid returns inside a 2D detection box.Source: C. LiDAR densification · p. 4
  • The method inserts M interior points between valid ring-adjacent samples within the same azimuth bin.The interpolation is subject to range-jump and spatial-gap gates.Source: C. LiDAR densification · p. 4
  • For endpoints S and E, the stated interpolation produces interior points indexed by t = 1,...,M.Source: C. LiDAR densification · p. 4

D. Confidence estimation

Confidence estimation uses point density, multi-view angular coverage, inlier count, and mean detector score, with a logistic function and tunable parameters.

D. Confidence estimation

  • Confidence estimation considers point density, multi-view angular coverage, inlier count, and mean detector score.Angular coverage is the union of yaw arcs over 2π, and density and inlier count are normalized by ρ0 and N0, respectively.Source: D. Confidence estimation · p. 4
  • Fig. 3 depicts a sparse point cloud from the Velodyne VLP-16 LiDAR.Source: Figure 3 · p. 4
  • The confidence mapping uses a logistic function with tunable weights and bias.Source: D. Confidence estimation · p. 4
  • Fig. 4 depicts a densified LiDAR point cloud after interpolation.Source: Figure 4 · p. 4

E. Active Environment Exploration (AEE)

The AEE module searches unknown regions with frontier-based goals derived from SLAM occupancy grids, then validates selected goals with Nav2.

E. Active Environment Exploration (AEE)

F. Approach module

The approach module converts a labeled object pose on a flat support surface into a safe, reachable base pose for VLA manipulation.

F. Approach module

  • The module estimates a support-surface boundary and normal, then places the robot beyond the edge at a user-defined offset with its base perpendicular to the surface.The surface normal is estimated with principal component analysis.Source: F. Approach module · p. 4
  • Nav2 validates reachability and collision-free access; infeasible poses prompt iteration over nearby edge candidates until valid options are exhausted.Source: F. Approach module · pp. 4–5
  • Frontier-goal selection filters and chunks map-derived candidates, optimizes yaw, and checks path feasibility before navigation.Candidates are ranked by distance after non-maximum suppression.Source: F. Approach module · p. 5

IV. VLA MODEL FINE-TUNING

The SmolVLA 450M model was fine-tuned on SO-101 pick-and-place episodes, using a specified optimizer and learning-rate schedule.

IV. VLA MODEL FINE-TUNING

  • SmolVLA 450M was fine-tuned on 50 SO-101 pick-and-place episodes collected by leader-manipulator teleportation.Fine-tuning used an NVIDIA RTX 4090 GPU with 16 GB VRAM.Source: §IV · p. 5
  • Training used AdamW with weight decay 0.01, a cosine-decay schedule, learning rate 0.0001, warmup 100, and gradient clipping at norm 10.0.Batch size was 16.Source: §IV · p. 5

V. HERMESBOT MOBILE MANIPULATOR

HermesBot combines a mobile base and SO-101 manipulator with separate navigation and VLA camera systems. The sensors support navigation and visual coverage of the manipulator and its workspace.

V. HERMESBOT MOBILE MANIPULATOR

  • HermesBot pairs a two-wheeled differential-drive platform with a mounted SO-101 manipulator.The platform was developed to support integrated multimodal sensing and computation.Source: §V · p. 5

A. Sensor Configuration

  • Navigation uses a Velodyne VLP-16 LiDAR and Intel RealSense D435i RGB-D camera integrated with [29] for Visual-LiDAR-Inertial SLAM.Source: A. Sensor Configuration · p. 5
  • Three Intel RealSense D435 cameras provide third-person overview, wrist-mounted gripper, and forward-facing workspace views for the VLA subsystem.The views are shown in Fig. 6(a), (b), and (c), respectively.Source: A. Sensor Configuration · p. 5, Figure 6 · p. 5

B. Computational Hardware

AnywhereVLA ran on a Jetson Orin NX and Intel NUC, and was evaluated for success and completion time in unseen indoor environments. The reported outcomes include component-level failure modes and episode durations across exploration radii.

B. Computational Hardware

  • A Jetson Orin NX 16Gb handles perception-heavy semantic mapping and VLA manipulation, while an Intel NUC Core i7 32Gb handles SLAM and navigation.Table I reports module inference frequencies and latencies measured during real-world deployment.Source: B. Computational Hardware · p. 6, Table I · p. 6

VI. EXPERIMENTAL RESULTS

  • 46% overall success rate and 85% VLA Manipulation module success rate were achieved with fine-tuned SmolVLA, versus 10% overall success without fine-tuning.The evaluation comprised 50 pick-and-place episodes in unseen university open space with dynamic clutter and human presence.Source: §VI · p. 6
  • VLA manipulation errors mainly involved bottles slipping from the gripper, while AEE failed in 25% of cases when cluttered tight spaces and corridors hid the desired object within radius Re.Source: §VI · p. 6
  • Within 5 m, episodes averaged under 133 seconds; within 10 m, tasks were completed in under 10 minutes.Completion time was measured across exploration radii of 2.5, 5, 7.5, and 10 m; see Fig. 7.Source: §VI · p. 6, Figure 7 · p. 6

VII. CONCLUSION, LIMITATIONS AND FUTURE WORK

AnywhereVLA combines classical SLAM, exploration, and navigation with a lightweight VLA for pick-and-place in novel indoor environments. The conclusion identifies difficulty enforcing precise spatial-semantic constraints and proposes semantic reasoning extensions.

VII. CONCLUSION, LIMITATIONS AND FUTURE WORK

  • The system targets open, large-scale indoor exploration but does not reliably enforce precise spatial-semantic constraints in instructions.For example, it may find the first detected bottle regardless of whether it is on the instructed table.Source: §VII · p. 6
  • A proposed dynamic scene graph would represent detected objects and affordances as nodes and probabilistic spatial relations as edges, with exploration rewards for path efficiency and relational fidelity.Source: §VII · p. 6
  • A lightweight VLM could resolve relational queries, such as whether a bottle is supported by a table, before success attribution.The authors suggest this could improve compositional generalization without substantial computational overhead.Source: §VII · p. 7
Loading 2509.21006v1…