PaperlayerView exact revision on arXiv

Source-linked AI summary

CRAFT: Adapting VLA Models to Contact-rich Manipulation via Force-aware Curriculum Fine-tuning

Yike Zhang, Yaonan Wang, Xinxin Sun, Kaizhen Huang, Zhiyuan Xu, Junjie Ji, Zhengping Che, Jian Tang, Jingtao Sun

arXiv:2602.12532v1cs.RO

TL;DR

Contact-rich manipulation challenges VLA models because force signals are task-critical yet can be overshadowed by higher-entropy visual and language inputs. CRAFT regulates those inputs with a force-aware curriculum and uses synchronized teleoperation data; experiments report improved success and generalization across tasks and VLA architectures.

  • Problem

    VLA models face an unresolved challenge in prioritizing low-entropy force signals alongside vision and language for robust, generalizable contact-rich manipulation.

    Source: §I · p. 2, B. Information Bottleneck in… · p. 3

  • Method

    CRAFT uses a variational information bottleneck to initially constrain visual and language embeddings, then progressively restores their contribution, alongside force-aware teleoperation data collection.

    Source: §I · p. 2, B. Force-aware Curriculum Fine-tuning · p. 5

  • Results

    Across five real-world tasks, π0-base with CRAFT achieved an average increase of 35.36%, while RDT + CRAFT improved by 25.66% points on average; experiments also report generalization to unseen objects and novel tasks.

    Source: §I · p. 2, B. Main Results · p. 6

  • Takeaways & Limitations

    Experiments support force-aware curriculum fine-tuning as a lightweight approach for robust, generalizable contact-rich manipulation across VLA architectures.

    Source: §V · p. 7

Abstract

from arXiv · show

Vision-Language-Action (VLA) models have shown a strong capability in enabling robots to execute general instructions, yet they struggle with contact-rich manipulation tasks, where success requires precise alignment, stable contact maintenance, and effective handling of deformable objects. A fundamental challenge arises from the imbalance between high-entropy vision and language inputs and low-entropy but critical force signals, which often leads to over-reliance on perception and unstable control. To address this, we introduce CRAFT, a force-aware curriculum fine-tuning framework that integrates a variational information bottleneck module to regulate vision and language embeddings during early training. This curriculum strategy encourages the model to prioritize force signals initially, before progressively restoring access to the full multimodal information. To enable force-aware learning, we further design a homologous leader-follower teleoperation system that collects synchronized vision, language, and force data across diverse contact-rich tasks. Real-world experiments demonstrate that CRAFT consistently improves task success, generalizes to unseen objects and novel task variations, and adapts effectively across diverse VLA architectures, enabling robust and generalizable contact-rich manipulation.

I. Introduction

VLA models can follow broad instructions, but contact-rich manipulation remains difficult because perception alone misses subtle physical interactions and models often lack task-critical force feedback. CRAFT addresses this with force-aware curriculum fine-tuning and synchronized teleoperation data; real-world experiments report gains in task success and generalization.

I. Introduction

  • Contact-rich tasks demand precise alignment and force regulation, but VLA models often lack force feedback and therefore exhibit unstable, imprecise, or unsafe behavior.Visual and language inputs alone do not capture subtle physical interactions at the point of contact.Source: §I · p. 1
  • Existing force-aware approaches still lack a unified mechanism to regulate modality contributions, leaving robust and generalizable contact-rich manipulation an open challenge.The cited challenge is prioritizing low-entropy force signals amid higher-entropy perceptual inputs.Source: §I · p. 2
  • CRAFT uses a variational information bottleneck to suppress visual and language signals early, then gradually restores them so policies first prioritize force inputs.The curriculum anneals the bottleneck’s regularization during training.Source: §I · p. 2
  • CRAFT improves success and generalization across five real-world contact-rich tasks and applies to both RDT and π0 architectures.The tasks include fine-alignment insertion, continuous contact wiping, and deformable-object manipulation.Source: §I · p. 2
  • A homologous leader–follower teleoperation system records synchronized vision, language, and force data across contact-rich tasks for policy learning and evaluation.The contribution describes a system based on an off-the-shelf robot arm without additional sensor installation.Source: §I · p. 2

II. RELATED WORK

VLA research has moved from demonstration-heavy vision-to-action models toward semantic, generative, and diffusion-based approaches for broader generalization. Related force-aware methods add tactile or force inputs in different ways, while CRAFT uses an information bottleneck to regulate visual and language information during force-aware learning.

A. Vision-Language-Action Models

  • Early VLA models learned vision-to-action mappings from large real-world datasets, while later methods used semantic priors and generative or diffusion-based action prediction to improve generalization.Representative diffusion- or flow-based approaches predict complex action sequences from high-level instructions.Source: A. Vision-Language-Action Models · p. 2
  • Most VLA models largely ignore force or tactile feedback, while RDP, VLA-Touch, and ForceVLA incorporate these signals through visual-tactile learning, dual-level touch, or a force-aware mixture of experts.CRAFT contrasts its information-bottleneck approach with FACTR’s heuristic schedule of decreasing visual blur.Source: A. Vision-Language-Action Models · p. 2

B. Information Bottleneck in Multimodal Learning

Prior information-bottleneck work compresses or regulates multimodal representations, but few methods balance vision, language, and force in VLA policies for contact-rich manipulation. CRAFT applies a variational bottleneck and force-aware curriculum, supported by a synchronized leader–follower teleoperation setup.

B. Information Bottleneck in Multimodal Learning

  • Information bottleneck methods compress input redundancy while preserving task-relevant information, and the variational extension approximates the objective with tractable distributions for deep networks.Prior multimodal methods also use bottlenecks to learn compact embeddings or control modality contributions.Source: B. Information Bottleneck in… · pp. 2–3
  • Existing information-bottleneck and force-input approaches provide limited regulation of vision, language, and force contributions in VLA action prediction, motivating CRAFT.The passage identifies scalable and generalizable force-aware VLA learning as an unresolved gap.Source: B. Information Bottleneck in… · p. 3

III. Method

  • CRAFT places a variational information bottleneck after vision and language embeddings, initially prioritizing lower-entropy force signals before restoring visual and language contributions.The passage describes force as carrying direct physical-interaction information needed for stable and precise contact-rich manipulation.Source: §III · p. 3

A. Homologous Leader-Follower Teleoperation System

  • The leader–follower setup uses an off-the-shelf Franka Panda follower and a low-cost force-controlled leader, with follower contact forces fed back to the operator in real time.The setup does not require additional tactile or force sensors or specialized high-precision haptic equipment.Source: A. Homologous Leader-Follower… · p. 3
  • Two synchronized cameras capture wrist-mounted first-person and static third-person views, while images, proprioception, joint torques, and actions are synchronized for policy learning.The passage identifies joint torque as the force modality and describes it as more informative than joint position alone.Source: A. Homologous Leader-Follower… · p. 3

B. Force-aware Curriculum Fine-tuning

CRAFT regulates high-entropy vision and language inputs with a variational information bottleneck (VIB), initially emphasizing force signals before progressively restoring multimodal information. Joint torque provides force-aware proprioception that captures motion discrepancies and external interactions.

B. Force-aware Curriculum Fine-tuning

  • CRAFT initially constrains visual and language information with a VIB, then relaxes that constraint so the policy can reintegrate multimodal signals while retaining force-aware representations.The schedule applies strong compression early in training and progressively reduces it.Source: B. Force-aware Curriculum Fine-tuning · pp. 3, 5, Figure 2 · p. 4
  • At each timestep, the policy receives visual observations and torque with a language instruction, then produces an action sequence over a horizon.The described entropy imbalance can lead a naive policy to over-rely on perceptual modalities and ignore force.Source: B. Force-aware Curriculum Fine-tuning · p. 3
  • The VIB compresses fused visual and language embeddings into compact features, with its loss using a variational approximation because the true marginal is intractable.The compact features are sampled during fine-tuning using the reparameterization trick.Source: B. Force-aware Curriculum Fine-tuning · pp. 3–4
  • Joint torque encodes kinematic state, deviations from desired motion, inertial effects, and external contact forces, providing richer force-aware proprioception than joint positions alone.Feeding torque alongside vision and language improves observability of internal dynamics and external contacts.Source: B. Force-aware Curriculum Fine-tuning · p. 4

C. Implementation

CRAFT adapts flow-based π0 and diffusion-based RDT using force-aware demonstrations and torque inputs, adding a lightweight VIB after the pretrained vision-language encoder without architectural modifications.

C. Implementation

  • CRAFT is implemented on flow-based π0 and diffusion-based RDT, using force-aware demonstrations collected with homologous leader-follower teleoperation.Joint torques become proprioceptive inputs, and a lightweight VIB regulates visual and language information after the pretrained encoder.Source: C. Implementation · p. 5
  • The VIB can be integrated into other VLA models without architectural modifications and can extend to tactile feedback and similar task-critical proprioceptive inputs.The implementation focuses on joint torques.Source: C. Implementation · p. 5
  • Fine-tuning adapts models pretrained only on vision and language using force-aware demonstrations across contact-rich tasks.The demonstrations are collected with the homologous leader-follower teleoperation system.Source: C. Implementation · p. 5

IV. Experiments

The paper evaluates CRAFT in real-world contact-rich manipulation experiments across five representative tasks spanning insertion, surface interaction, and deformable-object manipulation.

IV. Experiments

  • The real-world experiments evaluate CRAFT's effectiveness in augmenting VLA models through three key research questions.The supplied experiment passage introduces the questions but does not state their specific contents.Source: §IV · p. 5
  • The five representative tasks are USB Insertion, Flip Carton, Wipe Whiteboard, Rolling Plasticine, and Shaft-to-Hole Insertion.Together, they involve precise alignment, continuous surface interaction, and manipulation of deformable objects.Source: Figure 3 · p. 5

1) Main Results:

The main-results section frames evaluation around performance against original VLA models, generalization to unseen objects and task variations, and the contribution of the curriculum's components.

1) Main Results:

  • The first research question asks whether CRAFT-enhanced VLA models outperform their original counterparts on contact-rich manipulation tasks.The passage poses this as a question and supplies no result.Source: 1) Main Results · p. 5
  • The generalization study asks whether CRAFT-enhanced models generalize to unseen objects and novel task variations.The passage states the evaluation question, not its outcome.Source: 1) Main Results · p. 5
  • The ablation study asks why force-aware curriculum fine-tuning is critical and what each component contributes to CRAFT.The passage poses this as a question and gives no ablation findings.Source: 1) Main Results · p. 5

A. Experimental Setup

The experiments evaluate five contact-rich tasks involving precise alignment, continuous surface interaction, and deformable-object manipulation, comparing CRAFT-enhanced VLA models with representative baselines.

A. Experimental Setup

  • The five tasks are USB Insertion, Flip Carton, Wipe Whiteboard, Rolling Plasticine, and Shaft-to-Hole Insertion.They cover precise alignment, continuous surface interaction, and deformable-object manipulation.Source: A. Experimental Setup · p. 5
  • Insertion succeeds when the plug or shaft is fully seated; carton flipping requires a complete flip without dropping or damaging the carton.Whiteboard wiping requires at least 90% uniform coverage, while plasticine must become cylindrical without breaking or excessive deformation.Source: A. Experimental Setup · p. 5
  • CRAFT-enhanced models are compared with FACTR, π0-base, and RDT across the task suite.The evaluation uses 20 trials for USB Insertion, Shaft-to-Hole Insertion, and Flip Carton, and 12 trials for Wipe Whiteboard and Rolling Plasticine.Source: A. Experimental Setup · p. 6

B. Main Results

CRAFT improves success rates across contact-rich tasks for π0-base and RDT, while using force feedback to adjust manipulation and supporting different VLA architectures.

B. Main Results

  • 35.36% is the average increase for π0-base + CRAFT across five tasks; RDT + CRAFT rises from 22.66% to 48.32%, an average improvement of 25.66% points.π0-base + CRAFT achieves the highest success rate across most tasks; TABLE I summarizes average success rates across the five tasks.Source: B. Main Results · p. 6, Table I · p. 6
  • With USB insertion and carton flipping, CRAFT adjusts motion using contact-force signals, while the baseline misaligns insertion and fails to coordinate force and motion during flipping.The CRAFT model guides the direction and magnitude of applied torques during flipping.Source: B. Main Results · p. 6, Figure 5 · p. 6
  • CRAFT is a lightweight, plug-and-play module that leaves vision-language encoders unchanged and applies to both π0 and RDT.The enhanced variants achieve higher success rates on contact-rich tasks across both architectures.Source: B. Main Results · pp. 6–7
  • Figure 6 presents original setups and object-level and task-level variants used in the generalization experiments.The variants are introduced beneath the original experimental setups.Source: Figure 6 · p. 7

C. Generalization Studies

CRAFT is evaluated on object-level and task-level variations of four contact-rich tasks, with higher reported success rates across the OOD settings.

C. Generalization Studies

  • CRAFT raises average success rates across object- and task-level variants from 22.50% to 58.75%, a 36.25% improvement.TABLE II reports generalization performance on these variants.Source: C. Generalization Studies · p. 7, Table II · p. 7
  • OOD evaluations vary carton and plasticine appearance, USB socket height and visuals, and whiteboard placement angle across four tasks.The four tasks are USB Insertion, Flip Carton, Wipe Whiteboard, and Rolling Plasticine.Source: C. Generalization Studies · p. 7
  • The reported improvements are attributed to the force-aware curriculum and torque-centered representations, which focus attention on external torques and task-relevant visual and language representations.The paper concludes that CRAFT strengthens robustness and generalization under OOD settings.Source: C. Generalization Studies · p. 7

D. Ablation Studies

CRAFT’s ablations indicate that both the variational information bottleneck and torque-based proprioception contribute to performance, while the conclusion reports gains across five contact-rich tasks and broader generalization.

D. Ablation Studies

  • The ablation evaluates three variants on USB Insertion, Wipe Whiteboard, and Rolling Plasticine, using 10 trials per task and average success rate across tasks.The variants are the base model, VIB with joint positions, and VIB with joint torque.Source: D. Ablation Studies · p. 7
  • Adding VIB substantially improves performance over the base model, and combining VIB with joint torque further boosts success rates.The ablation authors attribute the additional benefit to torque’s richer, task-relevant feedback.Source: D. Ablation Studies · p. 7, Table III · p. 7

V. Conclusion

  • Across five contact-rich tasks, CRAFT improves success rates, generalizes to unseen objects and task variations, and applies across different VLA architectures.The authors characterize it as a lightweight approach to robust, generalizable contact-rich manipulation.Source: §V · p. 7

Related papers

Loading 2602.12532v1…