PaperlayerView exact revision on arXiv

Source-linked AI summary

Cue the Flow: Steering Flow-Matching Policies for Open-World Delivery Manipulation

Haoxuan Wang, Griffin Galimi, Junhua Huang, Selina Song, Wayne Wu, Yan Yan, Bolei Zhou

arXiv:2609.38989v1cs.RO

TL;DR

Open-world delivery manipulation must handle novel objects, instructions, and changing scenes, while existing control approaches can struggle with noisy perception and dynamic targets. This paper steers a frozen flow-matching VLA using grounding-derived spatial cues, reporting relative success-rate gains over baselines across tabletop and mobile-base settings.

  • Problem

    Open-world delivery creates a mismatch between policies trained in fixed environments with limited vocabularies and deployment with novel objects, unseen referring expressions, and variable scenes.

    Source: §1 · p. 1

  • Method

    A lightweight cue-conditioned adapter uses grounding-derived spatial cues to steer a frozen flow-matching VLA’s action chunks through diagonal affine transformations.

    Source: §1 · p. 2

  • Results

    86% and 44% relative gains over the best-performing baselines in average task success rate under tabletop and mobile-base settings, respectively.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Spatial-cue steering provides a practical interface between open-vocabulary grounding and reactive VLA control, with negligible additional inference cost.

    Source: §5 · p. 9

    The method depends on the underlying flow-matching policy and remains sensitive to out-of-distribution affordance regions and object positions, constraining the mobile base’s stopping location and orientation relative to the target.

    Source: §6 · p. 9

Abstract

from arXiv · show

Open-world goods delivery requires mobile manipulators to follow free-form user instructions and manipulate potentially novel objects. Existing dual-system approaches use high-level grounding models to convert language into grounded visual prompts, but their low-level controllers can remain brittle under noisy perception, dynamic scenes, and contact-rich interactions. We instead use a pretrained flow-matching vision-language-action model as the low-level control interface, leveraging its reactivity and robustness to environmental changes while treating the grounding output as a spatial cue for policy steering. Our key insight is that the pretrained VLA already provides a strong manipulation prior, while the spatial cue supplies the missing target information needed to guide actions under novel language--object mappings. Concretely, we introduce a lightweight cue-conditioned adapter. The adapter is first trained with contrastive objectives to produce salient and spatially discriminative cue representations, and is then supervised to predict a diagonal affine transformation over the generated action chunk, aligning policy steering with the cued target. Across tabletop and mobile-base settings, our method improves instruction following and manipulation success on both in-domain and out-of-domain objects, achieving up to near $2\times$ improvement in average task success rate with negligible inference overhead.

1 Introduction

Open-world delivery creates a train–deployment mismatch and challenges controllers that rely on geometric plans or implicit visual prompting. The paper steers a frozen flow-matching VLA with grounding-derived spatial cues, reporting gains over baselines across tabletop and mobile-base settings.

1 Introduction

  • Policies trained in fixed environments with limited vocabularies face open-world deployment with novel objects, unseen referring expressions, and variable scenes.The paper frames this train–deployment mismatch as a core delivery-manipulation challenge.Source: §1 · p. 1
  • Geometric low-level controllers can be fragile when perception is noisy, targets move, or fixed plans cannot recover from grounding or localization errors.Pretrained flow-matching VLAs offer learned priors for object interaction, contact-rich manipulation, and reactive correction.Source: §1 · p. 2
  • Input-level visual prompting leaves cue influence implicit, motivating a direct mechanism to align generated trajectories with grounding guidance.The passage identifies the challenge as converting grounded signals into actionable guidance during flow-based generation.Source: §1 · p. 2
  • A lightweight cue-conditioned adapter uses contrastive cue representations to predict diagonal affine transformations over action chunks from a frozen flow-matching VLA.This steers actions toward the cued target while preserving the policy’s manipulation prior and avoiding full policy fine-tuning.Source: §1 · p. 2
  • 86% and 44% relative gains over the best-performing baselines in average task success rate under tabletop and mobile-base settings, respectively.The method was validated across in-domain and out-of-domain object settings.Source: §1 · p. 3

2 Related Work

Related work covers flow-matching VLAs, policy steering, and visual prompting as routes to general-purpose robot manipulation. The paper distinguishes its approach by using spatial cues to target-specify and directly modulate a frozen VLA’s action chunks.

Flow-matching VLAs

  • Flow-matching VLAs model action generation as a learned continuous flow conditioned on fused visual-language features, but steering them toward novel language–object mappings remains underexplored.Prior work has begun placing these models in dual-system architectures.Source: §2 · p. 3

Policy steering

  • Policy-steering methods modify sampling or refinement while keeping a pretrained generative policy fixed, commonly using external objectives or auxiliary models.The paper contrasts its use of spatial cues as target-specification signals for generalizing to out-of-domain language–object mappings.Source: §2 · p. 3

Visual prompting for robotic policies

  • Visual-prompting approaches connect planners to controllers or inject prompts into VLA inputs, where prompt influence on action generation can remain implicit.This method instead separates a spatial cue from the input image and directly modulates a frozen VLA’s action chunk.Source: §2 · p. 3

3 Method

The method separates grounding a target from generating its manipulation actions: a frozen flow-matching VLA is steered by an explicit spatial cue. A lightweight adapter learns this steering while preserving the pretrained policy’s manipulation capability.

3.1 Problem Definition

  • The problem is to redirect an existing manipulation prior toward a target specified by a novel language–object mapping, rather than learn a new motor skill.The formulation modulates a frozen policy’s action chunk to follow a grounding module’s spatial cue while preserving its learned manipulation capability.Source: §3.1 · pp. 3–4

3.2 Dual-System Design

  • A grounding module turns free-form instructions into an explicit spatial cue, allowing a frozen flow-matching VLA to generate actions without relying solely on implicit language grounding.The cue is a Gaussian heatmap derived from the target mask; the VLA is kept frozen and attached to a lightweight adapter that modulates its action chunk during flow integration.Source: §3.2 · p. 4

3.3 Contrastive Adapter

  • Spatial contrastive training aligns action features with the correct cue and contrasts it against in-batch and spatially shifted cues, emphasizing target location over scene appearance.Spatially shifted negatives keep the scene fixed while changing the target location, helping distinguish cue position from appearance context.Source: §3.3 · p. 5
  • A residual cue representation, computed from real and zero heatmaps, targets cue-relevant action transformations and discourages shortcuts based on action features alone.The prediction head outputs diagonal affine modulation terms under flow-matching supervision, while regularization discourages large corrections that could corrupt the pretrained action distribution.Source: §3.3 · p. 5

3.4 Implementation Details

  • The adapter is trained on the policy’s demonstrations with ground-truth spatial cues; Table 2 compares success rates across tabletop and mobile-base settings and three object–instruction conditions.LoRA tended to overfit cue–action pairs in the low-data setting; deployment adds about 0.02% parameters and negligible latency, with the spatial cue recomputed after substantial scene changes.Source: §3.4 · pp. 5–6, Table 2 · p. 6

4 Experiments

Experiments evaluate three distribution-shift settings on tabletop and mobile-base tasks; the proposed method is best or tied-best across settings, with analyses of adapter components, cue formats, reactivity, and simulation.

4.1 Experimental Setup

  • Evaluation spans three shifts, from seen objects with paraphrased language to unseen objects with novel referring expressions; mobile-base deployment adds viewpoint and scene-geometry shifts.During the manipulation phase, the mobile base remains stationary at a predefined pose while the arm operates autonomously.Source: §4.1 · p. 6

4.2 Results

  • In Setting III rollouts, our method follows the cue while Base acts untargetedly and VP-VLA oscillates or reaches the wrong object; the delivery example combines manual navigation with autonomous manipulation.The cue-conditioned skill also follows the target across changes in scene layout and visual appearance.Source: Figure 3 · p. 7, §4.2 · p. 8, Figure 4 · p. 8
  • MOKA averages 18.0% success on Tabletop and 8.3% on Mobile Base, while VP-VLA reaches 44.4% on Mobile Base but only 8.3% in Setting III.VP-VLA also reaches only 4.2% in Tabletop Setting III; MOKA failures often originate from inaccurate visual prompts.Source: §4.2 · p. 7
  • In Setting III, our method reaches 66.7% on Tabletop versus 29.2% for the strongest baseline, and 41.7% on Mobile Base versus 8.3% for the strongest baselines; it is best or tied-best in every setting.The reported gains are attributed to sparse spatial cues and an adapter that steers the frozen policy while preserving its pretrained prior and fine-tuned manipulation skill.Source: §4.2 · p. 8

4.3 Analysis

  • In Setting III, the adapter alone reaches 16.7% success, adding Lsp raises it to 33.3%, and adding Lres yields 41.7%.Lsp provides the largest individual gain, while Lres further refines the steering signal.Source: §4.3 · p. 8
  • Gaussian heatmaps and binary masks perform similarly, while axis-aligned bounding boxes perform worse, suggesting localized cues steer more effectively than coarse boxes.Coarse box boundaries can point toward invalid or non-manipulable object regions.Source: §4.3 · p. 9
  • With the target position perturbed during execution, the policy reacts using a cue grounded once at rollout start, without re-querying the high-level planner.The authors attribute this reactivity to the underlying VLA and contrast it with fixed-trajectory or repeated-replanning geometric controllers.Source: §4.3 · p. 9
  • Across 200 LIBERO-PRO Object-suite episodes, our method achieves 15.5% success, versus 10.5% for Base, 9.5% for Base-L, and 10.5% for VP-VLA.The best tested regularization setting is λreg = 0.01 at 15.5%; no regularization can overcorrect, while stronger regularization can suppress useful steering.Source: §4.3 · p. 9

5 Conclusion

The paper presents spatial-cue steering for open-world delivery manipulation, using a pretrained flow-matching VLA and a lightweight adapter to guide a frozen policy. Evaluations across tabletop and mobile manipulation show improved in- and out-of-domain performance with negligible additional inference cost.

5 Conclusion

  • Across tabletop and mobile manipulation evaluations, spatial-cue steering improves in-domain and out-of-domain performance with negligible additional inference cost.The framework uses System 2 outputs as action-steering signals while keeping the VLA frozen.Source: §5 · p. 9
  • Replacing the geometric low-level controller with a pretrained flow-matching VLA improves robustness to visual perturbations and reactivity to dynamic environments.Source: §5 · p. 9
  • The framework uses a lightweight adapter to turn System 2 outputs into action-steering signals while retaining the VLA’s manipulation prior.The paper presents this as a practical interface between open-vocabulary grounding and reactive VLA control.Source: §5 · p. 9

6 Limitations

The method depends on the underlying flow-matching policy and remains sensitive to out-of-distribution affordance regions and object positions. These sensitivities constrain where and how the mobile base can stop relative to the target.

6 Limitations

  • The method depends on the underlying flow-matching policy and remains sensitive to out-of-distribution affordance regions and object positions.Source: §6 · p. 9
  • Sensitivity to affordance regions and object positions constrains the mobile base’s stopping location and orientation relative to the target.Source: §6 · p. 9
  • Future work will address human interaction and handover, item selection from densely clustered bags, and tighter navigation-policy integration.The stated aim is to improve manipulation reliability.Source: §6 · p. 9

A.1 MOKA

The appendix evaluates MOKA and adapts VP-VLA as comparison methods, then describes the authors’ mobile-demonstration procedure and its out-of-domain behavior.

A.1 MOKA

  • Despite perception and prompting adaptations and an oracle execution assumption, MOKA remains limited by target misclassification, incorrect grasp points, and infeasible motions.MOKA’s 2D affordance representation transfers only limitedly to shelf-based delivery manipulation.Source: §A.1 · p. 14

A.2 VP-VLA

  • VP-VLA is adapted for flow-matching policies by adding its visual prompt as an input and training with auxiliary grounding supervision.Unlike this paper’s approach, VP-VLA uses the System 2 signal as an input-level prompt rather than an action-steering signal.Source: §A.2 · p. 14

A.3 Ours

  • Mobile demonstrations use stationary-arm grasping, a hard-coded transport pose, manual or map-based navigation, then resumed policy inference for placement.In indoor and outdoor conditions, performance and steering effectiveness decrease, though target identification and task completion remain possible.Source: §A.3 · p. 14

B Experiment Details

Experiments use a wheeled quadruped with a robotic arm and camera streams sent to a remote workstation for policy inference. The hardware layout is shown in Figure 7.

B.1 Hardware Setup

  • The experimental platform combines a DEEP Robotics Lynx M20 Pro wheeled quadruped with an AgileX PiPER arm.Figure 7 shows the robot hardware setup.Source: §B.1 · p. 14, Figure 7 · p. 15
  • A ZED 2 provides the third-person view and a RealSense D435i is wrist-mounted, but depth and stereo measurements are not used.Source: §B.1 · p. 14
  • The Jetson AGX Orin relays video and action commands while a remote RTX 5080 workstation runs the policy, with approximately 80 ms round-trip latency.The latency is modestly higher than wired operation.Source: §B.1 · p. 15

B.2 Evaluation Setup

Evaluation varies both instruction phrasing and spatial arrangements while preserving the underlying pick-and-place task. Three regimes distinguish seen objects with paraphrased language, unseen objects with known language, and unseen objects with novel language.

B.2 Evaluation Setup

  • Evaluation prompts vary grasp verbs, object descriptors, and placement phrasing while preserving the same pick-and-place semantics.Seen-object descriptors include “black” or “plastic”; unseen-object descriptors include novel terms such as “brown” or “red.”Source: §B.2 · p. 15, Table 3 · p. 16
  • Spatial configurations vary object placement, spacing, distractors, and camera viewpoints, with the same configurations used for all compared methods.The seen-object setting uses training objects in novel arrangements; unseen-object settings introduce novel targets and increasingly challenging referring expressions.Source: §B.2 · p. 15, Figure 8 · p. 16
  • The three regimes test seen objects with paraphrased language, unseen objects with known language, and unseen objects with novel language.Novel-language prompts use ordinal identifiers 1, 2, or 3 to test compositional reference grounding under previously unseen instruction patterns.Source: Figure 8 · p. 16, Table 3 · p. 16

C Additional Results

Visual comparisons show the policy steering toward the grounded target across all three evaluation settings, with trajectories redirected toward it as the end-effector approaches. The method can fail when its manipulation prior is unreliable or consistent steering over-modulates precise actions.

C Additional Results

  • Across all three settings, the policy is progressively redirected toward the cued target as the end-effector approaches the objects.Figure 9 uses a red heatmap to indicate the correctly grounded target object.Source: C Additional Results · p. 16, Figure 9 · p. 17

D Failure Cases

  • The method can fail when the frozen policy deviates from a feasible trajectory or consistent steering over-modulates precise actions.The latter can lead to overshooting or unstable grasping.Source: D Failure Cases · p. 17
Loading 2609.38989v1…