PaperlayerView exact revision on arXiv

Source-linked AI summary

Localized Visual Feature Aggregation via Focus Pooling for Visuomotor Policies

Ruiyu Wang, Zheyu Zhuang, Danica Kragic, Florian T. Pokorny

arXiv:2609.08408v1cs.RO

TL;DR

Small object- or contact-level cues can determine actions, but end-to-end use of localized intermediate CNN features remains underexplored. FocusPool applies robot-state-conditioned attention pooling to those features. It improves policy success by 36.2% in simulation and 41.2% in the real world over pooling and explicit local-focus baselines under the same data budget.

  • Problem

    Small object- or contact-level cues can determine actions, yet pooling may wash them out; end-to-end use of localized intermediate CNN features remains underexplored.

    Source: §1 · p. 1

  • Method

    FocusPool uses a robot-state-conditioned query and iteratively refined cross-attention to pool localized intermediate CNN features into compact representations.

    Source: §1 · pp. 1–2

  • Results

    FocusPool improves policy success by 36.2% in simulation and 41.2% in the real world over pooling and explicit local-focus baselines under the same data budget.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Selective readout of intermediate CNN features can yield more efficient control representations through end-to-end training without explicit input cropping.

    Source: §7 · p. 8

    The analysis is anchored on ResNet-18; conclusions for other CNN backbones are tentative because their depth–locality trade-offs may differ.

    Source: §5 · p. 8

Abstract

from arXiv · show

Focusing on spatially localized, control-relevant visual cues has been shown to improve data efficiency in visuomotor policies by reducing the need to model task-irrelevant visual variation. Existing methods often impose this focus through input preprocessing, such as cropping control- or object-centric regions in RGB images or point-clouds. However, it remains underexplored whether such localized features can be exposed directly from commonly used convolutional neural network (CNN) encoded features. In this paper, we show that intermediate CNN features preserve localized visual context for control, but existing pooling methods fail to aggregate it effectively. We introduce FocusPool, an attention pooling module that selectively aggregates intermediate visual features according to their relevance to the robot's current proprioceptive context. The resulting pooled representation captures task-progressive, control-relevant local information and is used directly for policy learning. Across simulation and real-world experiments, FocusPool improves policy success rates over pooling and explicit local focus methods by 36.2% and 41.2%, with training only 5.8% of encoder parameters.

1 Introduction

The introduction motivates end-to-end pooling because spatial compression can dilute localized control cues, while existing pooling does not selectively exploit intermediate CNN features. FocusPool applies state-conditioned attention to aggregate those features, with reported gains in simulation and real-world policy success.

1 Introduction

  • Small object- or contact-level cues can determine the next action, yet pooling with background or irrelevant objects can wash them out.Prior cropping approaches improve data efficiency but require additional training stages and can suffer from error accumulation.Source: §1 · p. 1
  • FocusPool uses a robot-state-conditioned query and iterative cross-attention to pool intermediate features into progressive local masks, using 5.8% of ResNet-18 encoder parameters.Intermediate features retain spatial locality, but average pooling is nonselective and spatial softmax can introduce noisy keypoints.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • 36.2% in simulation and 41.2% in the real world: FocusPool improves policy success over pooling and explicit local-focus baselines under the same data budget.It also matches baseline performance with half the data and trains only 5.8% of the encoder parameters.Source: §1 · p. 2

2 Related Work

Prior work establishes local visual focus through input cropping and uses attention to aggregate visual features across vision and robotics. FocusPool differs by applying trainable pooling directly to localized intermediate CNN features.

2 Related Work

  • Prior local-focus methods predict control-relevant regions and crop RGB or point-cloud inputs, including Q-attention [3] [5], RVT-2, Hu et al. [7], and PALM [4].These coarse-to-fine designs restrict the visual input around predicted locations or the end-effector.Source: §2 · p. 2
  • Attention-based aggregation spans ViT patch or query methods [18] [19] [20] and robotics approaches for local feature pooling or multimodal fusion [22] [23].Robot-centric Pooling [14] and AFA [15] also use selective feature aggregation for robotics tasks.Source: §2 · pp. 2–3
  • Unlike prior selective aggregation methods, FocusPool targets localized aggregation of intermediate CNN features without preprocessing the input.This contrasts with local-focus pipelines that first predict a region and then crop the visual input.Source: §2 · pp. 2–3

3 Methodology

FocusPool builds a compact policy representation by pooling localized features from intermediate CNN layers, using robot-state-conditioned attention and a temporary early-training prior.

3.1 Where to Aggregate: Intermediate Visual Features for Locality

  • Intermediate maps retain spatial structure with sufficient visual abstraction, while deeper maps become less localized; FocusPool's gains diminish at deeper feature levels.ResNet-18 statistics compare feature-map resolution and theoretical receptive field across residual stages.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 How to Aggregate: FocusPool

  • A robot-state-conditioned learnable query probes spatial tokens and is iteratively refined with visual context, producing the policy representation from selective control-relevant patches.The final context serves directly as the visual input to the policy.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Early Attention Regularization

  • A weak prior stabilizes early attention by targeting projected future-keyframe end-effector heatmaps, then decays to zero after warm-up.The heuristic target therefore has no further impact after the warm-up period.Source: §3.3 · p. 4

4 Experiment

The experiments compare localized feature pooling with pooling baselines and explicit local focus in simulation and on a real robot. FocusPool improves success over the per-task best baselines in both settings.

4 Experiment

  • The simulation evaluation uses six MimicGen tasks, a 100-demonstration budget except for the data-efficiency study, and a shared CNN-based Diffusion Policy setup.The real-world evaluation uses multi-view RGB with a UFactory xArm 7 and 100 Table Cleanup demonstrations or 50 demonstrations for the other tasks.Source: §4.1 · p. 5, §4 · p. 5, §4.2 · p. 7

4.1 Simulation Evaluation

  • 36.2% average relative gain over the per-task best baseline in simulation; real-world FocusPool gains 41.2% over the per-task best baseline.The real-world comparison is evaluated over 20 rollouts using the latest checkpoint.Source: §4.1 · p. 5, §4.2 · p. 7, Table 4 · p. 7
  • FocusPool reaches comparable baseline performance with half the data, while Threading success is 40.7% for FocusPool versus 12.7% for RVT-2.In Threading, the interaction region is offset from the end-effector, where the RVT-2-style keypoint prior can be imperfect.Source: §4.1 · p. 5, Figure 3 · p. 5
  • FocusPool peaks at L1–L2 while pooling baselines perform best at later stages; at L1, FocusPool reaches 52.0% average success using 1.5% of encoder parameters.At L2, FocusPool attends to task-progressive control-relevant regions, whereas baselines produce diffuse attention or noisy keypoints.Source: §4.1 · p. 6, Figure 4 · p. 6
  • Removing state conditioning or iterative refinement makes attention less selective and reduces success by 6 or 8 percentage points; early attention regularization mainly improves training stability.Regularization increases mean success by 6.5 points but best success by only 3.6 points; its noisy-prior variant performs similarly at 50.8% versus 52.7%.Source: §4.1 · pp. 6–7, Figure 5 · p. 6, Table 3 · p. 6

4.2 Real Evaluation

  • On Giraffe Build, FocusPool improves performance by 100%, and L1 outperforms L2 as control-relevant cues occupy small image regions.FocusPool-L1 and L2 outperform all baselines overall in the real-world evaluation.Source: §4.2 · p. 7, Table 4 · p. 7, Figure 6 · p. 7

5 Discussion

The discussion reports results beyond the ResNet-18 setting while noting that architecture-specific choices remain open. FocusPool also pairs with a flow-matching policy head.

5 Discussion

  • A simple ViT adaptation on final-block patch tokens outperforms both the standard [CLS] representation and AFA.The adaptation uses DINOv3-pretrained ViT-S tokens and no attention regularization.Source: §5 · p. 8
  • FocusPool paired with a multimodal flow-matching policy head retains similar gains.Source: §5 · p. 8
  • The optimal readout stage, pretraining choice, and use of pooled features versus full tokens may vary across architectures and remain for future study.The main analysis is anchored on ResNet-18; other CNN backbones may have different depth-locality trade-offs.Source: §5 · p. 8

6 Limitation

ImageNet pretraining benefits FocusPool, though the method remains less dependent on pretrained local visual structure than one compared baseline.

6 Limitation

  • Without pretraining, FocusPool achieves 41.7% success and outperforms the best baseline.Source: §6 · p. 8
  • ImageNet pretraining improves existing pooling methods as well as FocusPool; diffusion policy with spatial softmax gains 40.5% relative versus FocusPool's 27.0%.Source: §6 · p. 8
  • How different pretrained weights affect FocusPool remains for future investigation.Source: §6 · p. 8

7 Conclusion

FocusPool selectively extracts task-progressive, control-relevant local information from intermediate CNN feature maps. The appendix algorithm replaces a pooling layer and returns a policy representation and attention map.

7 Conclusion

  • FocusPool uses iteratively refined, robot-state-conditioned cross-attention to turn intermediate feature maps into control-relevant representations without explicit input cropping.Its selective readout is intended to avoid diluting localized spatial cues.Source: §7 · p. 8

A Pseudocode

  • Algorithm 1 replaces the pooling layer after residual stage l and returns the policy representation cS and averaged attention map A for early attention regularization.Source: A Pseudocode · p. 12
  • The algorithm conditions an initial query on proprioception, probes spatial visual patches through repeated cross-attention, and averages attention weights into A.Algorithm 2 uses attention map A and target points u to produce a prior loss.Source: A Pseudocode · p. 12

B.1 Simulated Tasks

The evaluation covers six spatially varied MimicGen tasks and three real-world manipulation tasks, using a CNN-based diffusion policy and an xArm 7 setup.

B.1 Simulated Tasks

  • The simulation benchmark contains six MimicGen tasks with high spatial variation.The real-world tasks are Coffee Transport, Table Cleanup, and Giraffe Build.Source: §B.1 · p. 12, §B.2 · p. 13

B.2 Real-world Setup

  • The three real-world tasks run on a UFactory xArm 7 with fixed third-person and eye-in-hand RGB cameras.Demonstration counts are 50 for Coffee Transport and Giraffe Build and 100 for Table Cleanup.Source: §B.2 · p. 13, Figure A.2 · p. 13

B.3 Training Detail

  • The main evaluation uses an ImageNet-pretrained ResNet-18 visual encoder with Diffusion Policy as the policy head.The proprioceptive state is end-effector position, rotation, and gripper opening; actions use the corresponding workspace-frame end-effector commands.Source: §B.3 · p. 13

B.4 Model Implementation

  • FocusPool pools intermediate ResNet features with a robot-state-conditioned query, while RcP and AFA provide alternative attention-pooling baselines.The RVT-2-style RGB baseline predicts a local-focus heatmap and crops around its maximum-probability patch.Source: §B.4 · p. 14

C.1 Constructing Attention Pooling Masks

The ablations examine how robot-state conditioning and iterative refinement shape FocusPool’s attention masks and policy performance.

C.1 Constructing Attention Pooling Masks

  • Without iterative refinement, attention may shift to a later block before the current insertion is complete, whereas visual context can disambiguate task progress.Performance is comparable with two and three refinement steps; the authors use three.Source: §C.1 · p. 15, Figure A.3 · p. 15
  • On Stack Three, removing state conditioning drops success by 16%, compared with an 8% drop without iterative refinement.The blocks look similar but differ in height, making robot state an explicit cue to the current stage.Source: §C.1 · p. 16
  • Removing both iterative refinement and robot-state conditioning makes attention less localized and less task-progressive, with the largest policy-success drop.The comparison includes the baseline AFA, which lacks both components.Source: §C.1 · pp. 15–16

C.2 Effect of Visual Pre-training

The pretraining analysis compares spatial softmax and FocusPool with and without ImageNet encoder weights across six MimicGen tasks.

C.2 Effect of Visual Pre-training

  • Pretraining improves performance for both spatial softmax and FocusPool.The analysis compares each method with and without encoder pretraining.Source: §C.2 · p. 16
  • Spatial softmax gains from pretraining across tasks, including 19% on Square and 29% on Stack Three.Source: §C.2 · p. 16
  • FocusPool performs similarly with and without pretraining on Square and Stack Three, and its gains from pretraining are most pronounced on harder-action tasks.The authors describe FocusPool as less dependent on pretrained features for aggregating task-relevant information from demonstrations.Source: §C.2 · p. 16

D.1 Data Efficiency

The appendix reports per-task MimicGen success rates, comparisons across ResNet pooling stages, and visualizations of pooling focus during rollouts.

D.1 Data Efficiency

  • 40.7±3.2%, 33.3±1.8%, 59.3±4.8%, 81.3±2.2%, 47.3±6.7%, and 54.0±2.2% are FocusPool’s success rates across the six MimicGen tasks.In table order, the tasks are Threading, Square, Mug Cleanup, Stack Three, 3-Piece Assembly, and Pick Place.Source: §D.1 · p. 17

D.2 Pooling Stage

  • The four pooling methods’ reported average success profiles across ResNet stages L1–L4 are 53/54/43/36, 8/23/33/31, 12/31/37/37, and 22/34/38/37%.The table compares pooling methods read out after different residual stages across six MimicGen tasks.Source: §D.2 · p. 17

E Real-world Visualization

F Simulation Visualization

Related papers

Loading 2609.08408v1…