PaperlayerView exact revision on arXiv

Source-linked AI summary

VISTA: Visually Inferred Spatial ConTact Attention for Contact-Rich Manipulation

Jiayi Chen, Wenlong Dong, Yan Huang, Xianglin Chen, Zijian Lin, Jiaqi Yin, Yushan Liu, Wenbo Ding

arXiv:2608.25872v1cs.RO

TL;DR

Contact-rich manipulation lacks reliable, low-cost interaction feedback: vision can be ambiguous, while tactile and force sensing add hardware burdens. VISTA-Policy uses compliant-gripper deformation as VDF feedback within an integrated imitation-learning policy, and evaluations report superior performance to pure-vision and tactile baselines across three tasks, with OOD generalization and robustness.

  • Problem

    Vision-based contact cues can be ambiguous under occlusion or subtle interaction, whereas dedicated tactile and force sensors introduce hardware complexity and deployment costs.

  • Method

    VISTA-Policy extracts a 3D Visual Deformation Field from a passive compliant gripper and integrates encoding, energy-based denoising, and deformation-augmented policy control.

  • Results

    VISTA-Policy substantially outperforms pure-vision and tactile baselines on Cross-Scale Object Grasping, Cap Unscrewing, and Calligraphy Writing, with OOD generalization and robustness.

  • Takeaways & Limitations

    The results support VDF as a durable, cost-effective route to visuo-physical feedback for fine-grained manipulation without dedicated tactile or force sensors.

  • Takeaways & Limitations

    VISTA relies on observable compliant-gripper deformation, which may degrade under complete occlusion or extremely low-light conditions.

Abstract

from arXiv · show

Contact-rich manipulation requires precise interaction feedback. While vision-centric imitation learning is prevalent, external visual observations provide indirect and ambiguous cues about contact states, particularly under occlusion or subtle object--gripper interactions; dedicated tactile or force sensors can provide rich contact information but introduce additional hardware complexity, calibration requirements, and deployment costs. To bridge this gap, we propose VISTA-Policy, an imitation learning paradigm that utilizes the Visual Deformation Field (VDF), a 3D displacement representation of a compliant gripper, as high-dimensional visuo-physical feedback. The framework integrates: 1) a Physics-Aware Encoding Engine for real-time VDF decoding; 2) an Energy Aggregation Denoising Mechanism to isolate true interaction signals; and 3) a Deformation-Augmented Policy Network with incremental gripper actions for precise closed-loop correction. Extensive evaluations on Cross-Scale Object Grasping, Cap Unscrewing, and Calligraphy Writing demonstrate that VISTA-Policy outperforms the strong pure-vision baseline 3D Diffusion Policy and the tactile baseline. VISTA-Policy further demonstrates substantial out-of-distribution generalization to unseen object scales and robustness against dynamic disturbances, offering a durable and cost-effective route toward general-purpose fine-grained manipulation in unstructured environments. Project videos and supplementary materials are available at: https://sites.google.com/view/vista-policy.

I. INTRODUCTION

VISTA-Policy addresses ambiguous visual contact feedback and costly tactile hardware by extracting compliant-gripper deformation as visuo-physical feedback for contact-rich imitation learning. Its integrated perception, denoising, and policy components outperform vision and tactile baselines across evaluated tasks.

  • Motivation: Vision-centric policies provide ambiguous contact states under occlusion or subtle interaction, while tactile and force sensors add durability, cost, and deployment challenges.These limitations motivate an alternative feedback source that remains visually observable and physically tied to contact.
  • Approach: VISTA-Policy uses the Visual Deformation Field, a 3D displacement representation of a passive compliant gripper, as structured visuo-physical feedback.The deformation is extracted from external visual observations and represents a physical response induced by contact.
  • Results: Evaluations on Cross-Scale Object Grasping, Cap Unscrewing, and Calligraphy Writing show substantial gains over 3D Diffusion Policy and a tactile baseline.The reported improvements cover success rate, data efficiency, and generalization.
  • Approach: The framework integrates a Physics-Aware Encoding Engine, Energy Aggregation Denoising Mechanism, and Deformation-Augmented Policy Network.These components extract 3D deformation, isolate contact signals from noise, and fuse deformation feedback for action generation.
  • Approach: The passive compliant gripper provides observable reactive deformation along the x-axis and z-axis under external contact.These deformations form the physical basis for the VDF representation.

II. RELATED WORK

Related work highlights limits of vision-only and tactile approaches for high-frequency contact dynamics. VISTA-Policy combines deformation-based feedback with relative incremental gripper actions to address these perception and control challenges.

  • VISTA-Policy: The policy reformulates gripper control as relative increments between consecutive timesteps while retaining the original arm action representation.This design addresses coupled perception and control issues by avoiding dependence on absolute geometric anchors.
  • Vision and tactile feedback: Vision-centric policies struggle with high-frequency contact dynamics because visual observations are distal to physical interaction.This motivates multimodal feedback for fine-grained manipulation.
  • Vision and tactile feedback: Tactile sensing can be fragile, computationally costly, and unreliable during planar contacts, while low-dimensional measurements complicate spatial noise separation.Temporal filtering may introduce phase delays, perception lag, and over-execution.
  • Implicit sensing: Implicit vision-based force sensing removes internal sensors but often depends on specialized mechanisms and complex kinematics, limiting generalizability.Integrating such representations into robust data-driven manipulation remains challenging.
  • VISTA-Policy: VISTA-Policy extracts dense VDF features aligned with visual states, using coherent regional deformation to distinguish contact from low-amplitude vibration and tracking noise.The representation provides high spatial resolution and supports spatial noise decoupling.

C. OOD Generalization and Physical Invariants

OOD generalization remains difficult for imitation-learning policies facing unseen categories and scale variations. VISTA-Policy targets this challenge with a low-cost physical-invariant VDF and a three-stage architecture for extracting, denoising, and using deformation feedback.

  • Challenge: OOD generalization across unseen categories and drastic scale variations remains a critical challenge for multimodal imitation-learning policies.Existing approaches include large-scale VLA models and geometric inductive biases, each with practical limitations.
  • Physical invariants: VDF is proposed as an explicit, low-cost physical invariant because compliant-gripper deformations follow mechanics across diverse geometries and materials.The representation is intended to support cross-category and cross-scale grasping without external sensory payloads.
  • Architecture: The Physics-Aware Encoding Engine extracts a structured 3D VDF from real-time camera inputs.This is the first stage of the VISTA-Policy pipeline.
  • Architecture: The Energy Aggregation Denoising Mechanism derives continuous contact confidence from the VDF and uses it to gate a fused global representation.This stage filters deformation information before policy prediction.
  • Architecture: The Deformation-Augmented Policy Network combines spatio-temporal encoding with relative incremental gripper actions to produce manipulation commands.This is the final stage of the architecture shown in Fig. 3.

B. Physics-Aware Encoding Engine

The Physics-Aware Encoding Engine converts camera observations of a compliant gripper into a structured, real-time 3D Visual Deformation Field (VDF) for closed-loop manipulation.

  • Real-time tracking: A streaming tracker processes each frame with implicit memory, reducing temporal memory complexity from O(T) to O(1).Asynchronous computation supports operation at 20 Hz with minimal impact on precision.
  • Gripper geometry: A lightweight segmentation and post-processing pipeline extracts resilient gripper edge profiles under occlusion, illumination changes, and mask fractures.Morphological filtering and B-spline fitting reconstruct smooth geometric profiles for 3D estimation.
  • 3D reconstruction: Aligned RGB-D maps backproject sampled 2D edge pixels into a local 3D coordinate system anchored at the gripper’s rigid root.The system samples N points along the longitudinal y-axis and represents deformation in the xz-plane.
  • Invariant representation: Relative displacement measurements remove correlations between absolute gripper position and contact state, producing physically invariant deformation features.Depth completion uses independent 1D linear interpolation across feature columns to fill missing z measurements.
  • VDF output: The final VDF is a spatially continuous tensor D ∈ R^T×N×C with C = 4 displacement channels for downstream policy input.The representation is designed to provide consistent deformation data across the observation horizon and spatial sampling dimension.

C. Energy Aggregation Denoising Mechanism

The Energy Aggregation Denoising Mechanism aggregates spatial deformation after noise-floor filtering, converts it into a temporally smoothed contact confidence, and gates deformation features before multimodal policy fusion.

  • Energy aggregation: Spatial energy aggregation distinguishes coherent contact deformation from isolated tracking noise and converts the VDF into a compact contact confidence.The mechanism avoids relying on a single deformation point and integrates effective deformation across the global field.
  • Noise-floor filtering: The ReLU deadzone removes deformation magnitudes below the calibrated noise threshold before computing effective spatial energy.N is the number of spatial sampling points, and ||d_i,t||_2 denotes the deformation magnitude at point i.
  • Confidence mapping: A sigmoid maps the physical energy scalar to a normalized raw contact confidence, with gain k controlling trigger sensitivity around the nominal center E_mid.The confidence transitions sharply when average deformation exceeds the critical threshold.
  • Temporal smoothing: An exponential moving average produces temporally continuous contact confidence and suppresses transient jumps from occlusion or motion blur.The smoothing factor α controls the temporal update.
  • Gated fusion: Soft gating uses contact confidence to blend deformation features with a learnable neutral embedding before concatenation with visual and proprioceptive state features.The resulting multimodal representation F is passed to the downstream policy network.

D. Deformation-Augmented Policy Network

The Deformation-Augmented Policy Network encodes spatio-temporal VDF patterns and uses relative incremental gripper actions to generate contact-aware manipulation commands.

  • Action Representation: Relative incremental gripper actions superpose predicted positional changes onto the current opening, removing dependence on absolute action-space anchors.This representation is designed to address limited OOD action generation and cross-scale generalization.
  • Spatio-Temporal Deformation Encoding: A 1D CNN treats VDF sampling points as a physically ordered sequence to learn localized deformation patterns.Dual pooling captures both peak deformation and overall contact distribution.
  • Spatio-Temporal Deformation Encoding: Multi-head self-attention aggregates spatio-temporal contact dynamics across the interaction sequence.The resulting readout token provides a global contact-state representation for the downstream policy.
  • Task and Evaluation Design: The experiments cover cross-scale grasping, cap unscrewing, and calligraphy writing across varied object dimensions and platform heights.These tasks evaluate cross-scale generalization, action-space ablations, sample efficiency, robustness, and physical consistency.

V. EXPERIMENTS

The experiments test VISTA-Policy across contact-rich tasks and evaluate both perception robustness and policy performance using task, generalization, and deformation analyses.

  • Experimental Questions: The evaluation addresses whether VDF improves contact representation, execution performance, OOD generalization, robustness, sample efficiency, and physical safety.These objectives are organized as three research questions.
  • Perception Evaluation: The Physics-Aware Encoding Engine is evaluated for robust edge extraction before policy evaluation.Further experimental details are provided in the supplementary material.
  • Physical Consistency: VDF statistics during contact phases compare deformation consistency across the three tasks and competing methods.For methods without VDF policy input, VDF is computed only for post-hoc evaluation.
  • Policy Evaluation: The experiments report evaluation results for the three tasks and separately examine OOD scale generalization with gripper action-space ablations.These comparisons are summarized in Tables II and III.

A. General Analysis

VISTA-Policy comprehensively outperforms the evaluated baselines across the three contact-rich tasks and produces more consistent contact-phase deformation.

  • Overall Performance: VISTA-Policy comprehensively outperforms all baselines across the evaluated aspects of the three contact-rich tasks.The comparison supports improved performance across the task suite.
  • Physical Consistency: VISTA-Policy exhibits the narrowest VDF fluctuation range across all tasks compared with the baselines.The authors associate this physical consistency with capturing contact regularities through VDF.
  • OOD Scale Generalization: Training on demonstrations from a single 4 cm-wide object tests extrapolation to unseen object scales.The baseline prematurely enters lifting near the training width, while VISTA-Policy adaptively modulates closure across scales.
  • OOD Scale Generalization: 100% SR is achieved by VISTA-Policy for objects smaller than the training samples.This result demonstrates robust extrapolation to narrower object scales.

2) Gripper Action Representation Ablation:

The ablations show that relative incremental actions are necessary for VDF-guided cross-scale grasping, while VISTA-Policy also remains effective under multi-scale, disturbance, and fragile-object tests.

  • Gripper Action Representation Ablation: Absolute-action variants grasp the 4 cm training object but generalize poorly to unseen scales.VISTA-Abs over-grips larger objects and stalls on narrow targets, whereas DP3-Wrist-Abs ascends without contact.
  • Gripper Action Representation Ablation: Relative increments free the policy from the 4 cm numerical boundary and enable VDF-guided adaptive gripper regulation.The comparison supports combining VDF feedback with relative incremental actions.
  • Multi-Scale Training: VISTA-Policy achieves 100% grasping SR across all scales with zero drops in the multi-object grasping evaluation.The evaluation uses demonstrations spanning object diameters from 1.5 to 6 cm.
  • Multi-Scale Training: Visual baselines still exhibit excessive gripping on larger objects despite multi-scale exposure and doubled training samples.This indicates limited mastery of manipulation regularities without precise contact-state guidance.
  • Robustness and Delicate Manipulation: Dynamic disturbances and fragile tofu and playing-card tasks evaluate recovery, structural safety, and precise interaction regulation.The authors report completion without structural damage for the delicate manipulation scenarios.

C. Cap Unscrewing

Cap Unscrewing tests show that VISTA-Policy handles asymmetric contact and multi-stage recovery more reliably than the cited baselines. The calligraphy evidence further indicates higher-fidelity, more stable strokes, while the supplied figures document evaluation of quality, sample efficiency, and robustness.

  • Cap Unscrewing: DP3-Wrist successfully unscrewed the training-scale cap in only 1 out of 5 trials.This result came from training only on 2 cm-diameter caps.
  • Cap Unscrewing: VISTA-Policy performs smoother secondary attempts after failed cap-opening attempts, unlike baseline axis-misalignment failures that make subsequent twisting ineffective.The cited failure is especially noted for medium-sized 4 cm caps.
  • Evaluation Scope: The supplied evaluation figures cover calligraphy sample efficiency and robustness, while Table VII reports quantitative calligraphy results.The passages identify Fig. 9 and Table VII but do not provide their numerical values.
  • Calligraphy Writing: VISTA-Policy shifts toward the upper-right SQI–stroke-length region, approaching expert demonstrations and indicating stronger writing fidelity.The baseline groups include short dotlike strokes that can inflate stroke fullness despite poor completion.
  • Calligraphy Writing: VISTA-Policy closely tracks training-set averages with minimal variance, showing more stable execution quality than baselines and ablations.Figure 8(b) compares stroke length, width, and SQI with error bars.

2) OOD Generalization and Dynamic Disturbance Robustness:

VISTA-Policy generalizes beyond trained conditions and remains robust under disturbances, while feature analysis links this behavior to deformation-based contact representations. Its VDF features separate contact phases and remain consistent across object scales and geometries.

  • OOD Generalization: VISTA-Policy enables reliable paper contact and short-stroke writing at an extreme lower-bound height of 15 cm.DP3-Wrist instead exhibits excessive pressing beyond its trained range and insufficient downward extension at lower heights.
  • Ablation and Robustness: Ablation variants degrade across tasks: VISTA-MLP prematurely lifts during grasping, while VISTA-Cat produces instability and stuttering.The reported failures include drops during transport, descent misalignment, and gripper-closure stuttering.
  • Feature Representation: Across objects with different scales and geometries, VDF features show high pairwise similarity for similar localized contact states.The analysis interprets this as a unified, physically invariant contact representation.
  • Feature Representation: VDF features form distinct phase-separated similarity structure across non-contact and progressive-tightening contact frames, unlike cluttered DP3 correlations.Frames 1–3 are non-contact and frames 4–6 are contact states.

G. Discussion and Practical Implications

The discussion attributes VISTA-Policy’s advantages to explicit deformation-based physical feedback and reports practical benefits over pure-vision and tactile alternatives. The authors also identify visibility and representation limits that motivate future work.

  • Mechanistic Interpretation: Explicit VDF feedback supplies a structured physical prior that supports data efficiency and generalization beyond geometric observations alone.The discussion connects pure-vision failures such as premature lifting and over-gripping to limited physical closed-loop feedback.
  • Comparison with Tactile Hardware: TDF-DM exhibits persistent stuttering, scraping, and protective stops, with an 80% DR in Cap Unscrewing and 20% SR in Calligraphy Writing.VISTA-Policy is reported to outperform this tactile baseline across metrics, including safety.
  • Practical Implications: The compliant gripper costs $77 and withstands over 1,000 high-intensity contact-rich trials without structural degradation.These figures support the paper’s deployment-feasibility and cost-benefit discussion.
  • Overall Implications: VISTA-Policy combines real-time VDF extraction, spatial energy denoising, incremental gripper actions, and policy augmentation across three contact-rich tasks.The paper reports strong OOD generalization and robustness relative to pure-vision and tactile baselines.
  • Limitations: VISTA’s deformation features may degrade under complete occlusion or extremely low-light conditions, and the current representation primarily provides contact-state feedback.The authors propose temporal memory, multiview fusion, force estimation, and broader VLA integration as future directions.
Loading 2608.25872v1…