PaperlayerView exact revision on arXiv

Source-linked AI summary

FoveaVLA: Foveation Actions Replace the Wrist Camera in VLAs

Bingao Chen, Lihan Zha, C. Karen Liu, Guy Tevet

arXiv:2610.20064v2cs.ROcs.AI

TL;DR

Wrist cameras offer close-up detail but restrict scene coverage, while low-resolution external views lose fine manipulation detail. FoveaVLA predicts the next interaction region and supplies its crop alongside full external views. Across four real-world suites, both configurations outperform wrist-equipped π0.5 overall.

  • Problem

    Wrist views restrict scene coverage, while resizing whole external images to low policy resolution reduces the detail available for manipulation.

    Source: §I · p. 1

  • Method

    FoveaVLA predicts the next interaction region alongside each action chunk and gives the policy original-resolution crops together with full views.

    Source: §I · p. 2

  • Results

    70.8% overall success for FoveaVLA-Single with one external RGB-D view versus 52.1% for wrist-equipped π0.5, while Dual reaches 77.1%.

    Source: B. Replacing the Wrist Camera (Q1) · p. 5

  • Takeaways & Limitations

    FoveaVLA operates without a wrist view, and the single-view results indicate that both foveal detail and depth are needed.

    Source: C. Local Detail and Geometric Cues · p. 5, §VI · p. 6

Abstract

from arXiv · show

We present FoveaVLA, a Vision-Language-Action (VLA) model that replaces the wrist camera with learned foveation actions. Wrist cameras provide close-up detail but restrict the view to the end-effector's surroundings. External cameras see the whole scene, but the policy receives them at low resolution, so the fine detail needed for manipulation is lost. FoveaVLA recovers this detail by predicting, alongside each action chunk, the interaction region to observe at the next policy query. The policy then receives crops of these regions, taken from the original-resolution images, together with the full views. We build FoveaVLA on $\pi_{0.5}$ in two configurations, Single and Dual, using one or two external stereo cameras with full and foveal RGB and depth inputs, and post-train both on DROID with RoboInter interaction annotations. Across four real-world task suites, FoveaVLA-Single achieves 70.8% success with one external RGB-D view, and FoveaVLA-Dual achieves 77.1%, versus 52.1% for the wrist-equipped $\pi_{0.5}$. Both match the baseline on plain pick-and-place and improve success with distractors, cloth manipulation, and far-away objects.

I. INTRODUCTION

FoveaVLA replaces the wrist camera with learned crops from external images, retaining scene context while adding close-up detail. On four real-world suites, both configurations outperform wrist-equipped π0.5 overall; the related work contrasts this fixed-camera approach with synthesized, virtual, and actively controlled viewpoints.

I. INTRODUCTION

  • Wrist cameras provide close-up detail but can miss distant targets and scene context; resizing external images to policy resolution loses manipulation detail.External views also vary in geometric relation to the robot, and the wrist view adds hardware and mismatches demonstrations recorded without one.Source: §I · p. 1
  • FoveaVLA predicts the next interaction region alongside each action chunk, then supplies an original-resolution crop alongside the full external view.The method uses RoboInter interaction annotations on DROID demonstrations and comes in one- and two-external-camera configurations.Source: §I · p. 2
  • 70.8% overall success for FoveaVLA-Single with one external RGB-D view, versus 52.1% for wrist-equipped π0.5; Dual reaches 77.1%.Single improves success on Distractors from 25.0% to 58.3% and on Move from 41.7% to 75.0%.Source: §I · p. 2

II. RELATED WORK

  • Prior approaches synthesize or render viewpoints, actuate cameras, or use recorded human gaze; FoveaVLA selects regions from images captured by fixed cameras.Unlike prior auxiliary localization targets, its predicted box determines the crop used in the next robot observation.Source: §I · p. 1, §II · p. 2

III. PRELIMINARIES

  • A VLA maps camera images, robot state, and a language instruction to an action chunk; FoveaVLA builds on π0.5’s PaliGemma backbone and flow-matching action expert.PaliGemma’s location tokens are reused to represent foveation actions.Source: §III · pp. 2–3

IV. METHOD

FoveaVLA extends π0.5 to predict action chunks and future interaction boxes, using those boxes to select the next external-image crops. Full views preserve scene layout, while foveal RGB-D crops provide detail and distance cues.

IV. METHOD

  • FoveaVLA predicts an action chunk and one interaction region per external view for the next policy query; the predicted regions determine the crops in the next input.Single uses one external stereo camera and Dual uses two; both are post-trained from π0.5 on DROID with RoboInter annotations.Source: §IV · p. 3

A. Full and Foveal Observations

  • Full views resize and pad source images to encoder resolution, while foveal crops retain the interaction region at source resolution before resizing to that same input size.The full view preserves scene layout; the crop allocates the encoder’s pixels to the interaction region. Aligned depth is estimated from stereo images and processed with the same resizing and cropping.Source: A. Full and Foveal Observations · p. 3

B. Foveation Actions

  • The policy predicts boxes for the next query time t+w alongside its action chunk, using separate action and box-prediction heads that share the observation representation.The predicted boxes determine the next foveal crops; the box branch decodes location tokens.Source: B. Foveation Actions · p. 3

C. Learning from Interaction Annotations

FoveaVLA learns interaction regions from RoboInter annotations on DROID, using them to supervise crops for future observations. Training pairs current observations and action chunks with future boxes, while jointly optimizing action generation and foveation prediction.

C. Learning from Interaction Annotations

  • Each interaction box is the smallest axis-aligned rectangle enclosing the components required by an annotated skill; the corresponding square crop supplies the foveal input.Examples include the gripper, manipulated object, and annotated target region.Source: C. Learning from Interaction Annotations · p. 3, Figure 3 · p. 4
  • Training pairs an observation and demonstrated action chunk at t with interaction boxes annotated at the future query t + w.Single-view examples require valid annotations in one view; Dual examples require them in both views.Source: C. Learning from Interaction Annotations · p. 3
  • Ground-truth crops are perturbed in center and scale, and sometimes replaced with full views, to improve robustness to imperfect predictions and match the initial query.RGB and depth use the same crop geometry.Source: C. Learning from Interaction Annotations · p. 3
  • Training jointly optimizes action generation and future-box prediction, with both losses updating the shared backbone.The action objective interpolates demonstrated chunks with Gaussian noise; foveation tokens are predicted with teacher forcing, and depth-stream dropout supports partial depth availability.Source: C. Learning from Interaction Annotations · pp. 3–4

D. Closed-Loop Inference

At each query, FoveaVLA acts from the current observation and predicts boxes for the next one. The experiments evaluate camera replacement, crop contributions, depth and extra views, transfer, and prediction of where to look next.

D. Closed-Loop Inference

  • The action chunk is generated first; while it executes, box tokens are decoded from the cached prefix so crops can be ready for the next observation.Initial foveal inputs are full views; subsequent inputs crop current RGB and aligned depth using preceding box predictions.Source: D. Closed-Loop Inference · p. 4

V. EXPERIMENTS

  • The evaluation asks whether learned foveation replaces the wrist camera, what crops add beyond box supervision, and what depth and a second view contribute.It also tests transfer to unseen environments and whether the policy can predict where to look next.Source: §V · p. 4

A. Experimental Setup

The study evaluates external-camera FoveaVLA on four real-world task suites and compares it with wrist-camera and external-view baselines. The results show higher overall success than wrist-equipped π0.5, while ablations examine crops, depth, and additional views.

B. Replacing the Wrist Camera (Q1)

  • 70.8% overall success for FoveaVLA-Single and 77.1% for FoveaVLA-Dual compare with 52.1% for wrist-equipped π0.5.Both FoveaVLA variants match π0.5 at 100.0% on plain Pick-and-Place and achieve higher success on the other three suites.Source: Figure 4 · p. 5, B. Replacing the Wrist Camera (Q1) · p. 5

A. Experimental Setup

  • The real-world protocol uses four task suites, 12 shared randomized trials per suite, and reports mean success ± SEM; Overall pools all 48 trials.The models are post-trained from π0.5 on DROID with RoboInter box supervision; FoveaVLA uses one or two external RGB-D views.Source: A. Experimental Setup · pp. 4–5, Table I · p. 6

B. Replacing the Wrist Camera (Q1)

  • 58.3% versus 25.0% on Pick-and-Place with Distractors and 75.0% versus 41.7% on Move with far objects are the Single-versus-wrist-baseline success rates.Dual records the strongest Fold/Unfold and Move results, at 75.0% and 83.3%.Source: B. Replacing the Wrist Camera (Q1) · p. 5
  • FoveaVLA-Single takes approximately 125 ms per policy query and 50 ms for cached box decoding on an RTX 5080; Dual takes roughly twice as long.At the target control rate, box decoding fits within action-chunk execution.Source: B. Replacing the Wrist Camera (Q1) · p. 5

C. Local Detail and Geometric Cues

  • Foveal crops improve success over full views alone: with RGB-D, Single and Dual rise from 6.3% and 41.7% to 70.8% and 77.1%.With two RGB views, box supervision without crops reaches 31.3%, while adding crops reaches 66.7%.Source: C. Local Detail and Geometric Cues · p. 5
  • With one view, crops without depth reach 20.8% overall success and depth without crops reaches 6.3%; combining both reaches 70.8%.Depth also raises dual-view success from 25.0% to 41.7% with full views and from 66.7% to 77.1% with foveation.Source: C. Local Detail and Geometric Cues · p. 5
  • With foveal crops but no depth, a second external view raises overall success from 20.8% to 66.7%, near Single with RGB-D at 70.8%.With RGB-D, the second view raises overall success from 70.8% to 77.1%; Single remains best on Distractors, 58.3% versus 50.0%.Source: C. Local Detail and Geometric Cues · p. 6

D. Zero-Shot Generalization (Q4)

FoveaVLA-Dual transfers zero-shot to MolmoSpaces, while future-box prediction localizes held-out interaction regions competitively and avoids waiting for the target frame. The conclusion reports strong real-world success for both configurations without wrist views.

D. Zero-Shot Generalization (Q4)

  • 22.4% average zero-shot success on MolmoSpaces exceeds the 17.8% published π0.5-DROID reference, with higher scores on seven of nine tasks.The largest differences are Pick-v1.5 (33.7% versus 18.1%) and Pick&Place-Color-v2 (20.8% versus 10.4%).Source: D. Zero-Shot Generalization (Q4) · p. 6, Table II · p. 6

E. Foveation Prediction (Q5)

  • 0.465 mIoU and 0.483 Acc@.5 for FoveaVLA future-box predictions are comparable to YOLO11s at 0.492 and 0.488 despite predicting before the target frame.The model exceeds π0.5-DROID, PaliGemma + PG, and SAM3 + PG; unlike YOLO11s, its box decoding adds no wait after a new observation arrives.Source: E. Foveation Prediction (Q5) · p. 6

VI. CONCLUSION

  • 70.8% success for FoveaVLA-Single and 77.1% for FoveaVLA-Dual compare with 52.1% for the wrist-equipped π0.5 across four real-world suites.The conclusion also reports that Dual transfers zero-shot to simulated MolmoSpaces scenes.Source: §VI · p. 6

APPENDIX A DATASET CONSTRUCTION

The dataset pipeline aligns DROID demonstrations with RoboInter interaction annotations, constructs boxes for primitive-required components, and pairs current observations with future targets. High-resolution source images provide square crops, with crop perturbations and full views used during training.

A. Matching demonstrations and annotations

  • DROID robot state, actions, and instructions are joined to raw external-camera images by episode, camera view, and frame index, while wrist images are excluded.The external RGB reference is the rectified left imager of each stereo camera.Source: A. Matching demonstrations and… · p. 9

B. Interaction-box construction

  • Interaction boxes enclose the gripper, manipulated object or tool, and target region required by each annotated primitive skill.Invalid required components and unsupported primitives are excluded; primitive labels construct supervision but are not provided at inference.Source: B. Interaction-box construction · p. 9, Table IV · p. 9

C. Temporal pairing and sample validity

  • Training pairs the current interaction box with the box eight steps later, requiring valid annotations at both times and an available future frame.Dual additionally requires valid targets for both external views, while the future box may change with the annotated skill.Source: C. Temporal pairing and sample validity · p. 9

D. Foveal crops and augmentation

  • The tight target box is enclosed by a larger square crop extracted at 1280×720 and resized to 224 × 224; full views are padded to the same encoder resolution.Training uses 45% nominal crops, 45% perturbed crops that retain the interaction region, and 10% full views.Source: D. Foveal crops and augmentation · p. 9

E. Aligned metric depth

The depth pipeline estimates aligned metric depth from each external stereo pair and applies the same crop geometry to depth and RGB. The models encode depth as a separate input stream and use shared-scale colorization and validity handling.

E. Aligned metric depth

  • Stereo disparity from each rectified external-camera pair is converted to metric optical-axis depth using z = fxB/d, with focal length, stereo baseline, and disparity at the corresponding resolution.Depth maps are aligned to reference RGB, and deployment uses the same disparity conversion and crop geometry.Source: E. Aligned metric depth · p. 9

A. Model and optimization

  • Single receives four image streams and Dual eight, separating full and foveal RGB and depth while retaining the π0.5 visual encoder, language backbone, and action expert.All model parameters are updated, and both flow and foveation losses propagate into the shared backbone.Source: A. Model and optimization · p. 9, Table V · p. 10

B. Depth representation and visual augmentation

  • Depth uses a fixed metric scale and Turbo coloring, with invalid pixels black and depth patches masked when fewer than 25% of pixels are valid.Depth streams are independently dropped during training; color jitter applies to RGB only to preserve crop, depth, and box alignment.Source: B. Depth representation and visual… · p. 10

C. Box coordinates and decoding

Foveation boxes are decoded in full-view coordinates and converted back to source-image crops, while the controller executes part of each predicted action chunk before querying again. The MolmoSpaces evaluation uses adapted external views, depth, and a zero-shot protocol.

C. Box coordinates and decoding

  • Decoded boxes use coordinate order (ymin, xmin, ymax, xmax), are mapped back through full-view resizing and padding, and produce square crops; invalid boxes reuse the last valid box or full view.Dual decodes the physical-left box before the physical-right box.Source: C. Box coordinates and decoding · p. 10

D. Prediction and execution horizons

  • The action expert predicts 16 actions per query, while the controller executes the first eight before requesting new observations.The flow objective supervises the full predicted chunk.Source: D. Prediction and execution horizons · p. 10

A. Tasks and zero-shot transfer

  • The Dual checkpoint is evaluated on nine MolmoSpaces tasks without training on MolmoSpaces data, covering picking, constrained pick-and-place, and articulated-object opening or closing.Task instances, scene initializations, and success definitions follow the benchmark.Source: A. Tasks and zero-shot transfer · p. 10

B. External-camera observations

  • MolmoSpaces inputs use two adapted external views rendered at 1280×720, with full views and predicted crops resized to 224 × 224 and no wrist image.Camera layouts vary by task, including fixed shoulder views and randomized views on opposite sides of the robot base.Source: B. External-camera observations · p. 10, Table VI · p. 10

C. Depth and action interfaces

  • Simulation supplies metric depth aligned with each external RGB image, and the policy receives DROID-format joint positions, gripper state, and task instruction.Eight actions run per query before new images are captured; the control period is 66 ms.Source: C. Depth and action interfaces · pp. 10–11

D. Scoring and published reference

The evaluation uses four real-world task suites with shared instances across methods, plus MolmoSpaces scoring reported against a published π0.5-DROID reference. Task-specific criteria define success across pick-and-place, moving, folding or unfolding, and distractor conditions.

D. Scoring and published reference

  • MolmoSpaces results use the official evaluator and the unweighted mean of nine task success rates; π0.5-DROID is a published reference, not a controlled comparison.Its camera layout, input resolution, and depth availability differ from this evaluation.Source: D. Scoring and published reference · p. 11

A. Task instances and execution

  • The real-world benchmark has four task suites with 12 shared instances each, totaling 48; trials use the DROID setup at a target 15 Hz for at most 900 control steps.Object lists and positions set up scenes but are not given to the policy.Source: A. Task instances and execution · p. 11

B. Task definitions and success criteria

  • The four suites assess placing a specified object in a container, moving it near a reference, folding or unfolding a specified towel, and placing a named marker in a named bowl amid distractors.Success requires task-specific manipulation and release criteria; distractors may be disturbed if the instructed manipulation is completed.Source: B. Task definitions and success criteria · p. 11

C. Outcome annotation

Operators assign binary success using task-specific criteria, and suite rates average 12 trials while Overall pools all 48. Trial tables specify the instances, and figures show representative outcomes and foveal observations.

C. Outcome annotation

D. Complete trial specifications

  • Tables VII–X enumerate trial specifications for the four task suites, while Figure 6 shows representative initial and completed states.The figures include Single examples in the first three rows and a Dual example in the last row; Fold/Unfold trials begin flat or half-folded according to task.Source: D. Complete trial specifications · p. 11, Figure 6 · p. 12, Table VII · p. 13
  • Figure 7 follows a successful distractor trial across three moments, pairing each full external RGB view with its foveal crop.The crops show the target marker during approach and transport, then in the bowl after release; depth is omitted.Source: Figure 7 · p. 14

Related papers

Loading 2610.20064v2…