PaperlayerView exact revision on arXiv

Source-linked AI summary

CompliantVLA-adaptor: VLM-Guided Variable Impedance Action for Safe Contact-Rich Manipulation

Heng Zhang, Wei-Hsing Huang, Qiyi Tong, Gokhan Solak, Puze Liu, Kaidi Zhang, Sheng Liu, Jan Peters, Yu She, Arash Ajoudani

arXiv:2601.15541v2cs.RO

TL;DR

Existing VLA models lack force-aware adaptation for contact-rich manipulation. CompliantVLA-adaptor adds VLM-guided variable impedance control regulated by force feedback; the paper reports higher success and fewer force violations across most evaluated tasks.

  • Problem

    VLA models typically produce position commands without force-aware adaptation, limiting safe interaction in contact-rich tasks.

    Source: §I · pp. 1–2

  • Method

    CompliantVLA-adaptor uses a VLM to generate context-aware stiffness and damping for VIC, with force-based scaling to reduce stiffness above safe thresholds.

    Source: §I · p. 2, 2) Multimodal-informed impedance… · p. 4

  • Results

    9.86% to 17.29% average success rate across all tasks; task success improved and force violations fell across most contact-rich tasks (7/8).

    Source: A. Simulation results · p. 6

  • Takeaways & Limitations

    The adaptor is a lightweight, plug-and-play safety layer for existing VLA models and a practical step toward safe, generalizable contact-rich manipulation.

    Source: §VII · p. 7

    VLMs may generalize poorly to unseen objects or environments, leading to suboptimal performance in novel situations.

    Source: §VI · p. 7

Abstract

from arXiv · show

We propose a CompliantVLA-adaptor that augments the state-of-the-art Vision-Language-Action (VLA) models with vision-language model (VLM)-informed context-aware variable impedance control (VIC) to improve the safety and effectiveness of contact-rich robotic manipulation tasks. Existing VLA systems (e.g., RDT, Pi0.5, OpenVLA-oft) typically output position, but lack force-aware adaptation, leading to unsafe or failed interactions in physical tasks involving contact, compliance, or uncertainty. In the proposed CompliantVLA-adaptor, a VLM interprets task context from images and natural language to adapt the stiffness and damping parameters of a VIC controller. These parameters are further regulated using real-time force/torque feedback to ensure interaction forces remain within safe thresholds. We demonstrate that our method outperforms the VLA baselines on a suite of complex contact-rich tasks, both in simulation and the real world, with improved success rates and reduced force violations. This work presents a promising path towards a safe foundation model for physical contact-rich manipulation. We release our code, prompts, and force-torque-impedance-scenario context datasets at https://sites.google.com/view/compliantvla.

I. INTRODUCTION

VLA models can understand task semantics but lack force-aware adaptation for safe contact-rich interaction. CompliantVLA-adaptor adds VLM-guided variable impedance control, and its failures are described as more graceful than baseline failures.

I. INTRODUCTION

  • Position- or trajectory-based VLA control does not account for contact dynamics, so unexpected resistance can prompt unsafe forces or damage.Contact-rich tasks require precise force regulation and adaptive compliance, including when handling delicate objects.Source: §I · pp. 1–2
  • Existing VIC can respond compliantly to external forces, but often depends on manual tuning, task-specific schedules, training, or demonstrations.The paper positions VLM-guided impedance as a way to use visual, force, and language context for dynamic parameter regulation.Source: §I · p. 2
  • CompliantVLA-adaptor failures are mainly minor misalignments or slippage, unlike baseline failures involving force-threshold violations and toppled objects.The paper presents the adaptor as a plug-and-play solution that augments existing VLA models.Source: §I · p. 2
  • The adaptor uses a VLM to translate high-level semantic understanding into context-aware stiffness and damping parameters for variable impedance control.The proposed system integrates VLM parameter generation with VIC execution while retaining VLM reasoning and adding compliant control.Source: §I · p. 2

II. RELATED WORK

VLA models provide semantic understanding and high-level planning but generally do not adapt compliance to interaction forces. Existing VIC approaches offer force-responsive control, yet their training and feature requirements constrain semantic generalization.

A. Vision-language-action models for robotic manipulation

  • VLA models map visual and linguistic inputs to actions, but can fail in contact-rich tasks because they do not adapt compliance to interaction forces.The related-work discussion notes examples including OpenVLA's operational space controller and InstructVLA's simple PD controller.Source: A. Vision-language-action models for… · p. 2

B. Variable impedance control for safe physical interaction

  • Existing VIC methods use force feedback, demonstrations, or context-aware schemes, but rely on task-specific training, expert data, or hand-crafted features.These approaches cannot exploit rich semantic cues such as visually identifying fragile electronics.Source: B. Variable impedance control for safe… · p. 3
  • CompliantVLA bridges semantic reasoning and compliant control by using VLM guidance to adapt impedance, addressing a semantic gap in prior VIC approaches.The paper says the gap can force conservative tuning or risk unsafe contact with novel objects.Source: B. Variable impedance control for safe… · p. 3

C. VLMs enhance impedance parameter generation for safe physical interaction

CompliantVLA-adaptor uses a VLM’s semantic understanding and real-time force feedback to generate context-aware impedance parameters for VLA-driven contact-rich manipulation. It combines this approach with a force-regulated safety layer, addressing limits of task-specific schedules, simplified evaluations, and specialized-sensor approaches.

C. VLMs enhance impedance parameter generation for safe physical interaction

  • Prior methods use expert-defined task schedules, language-model selection tested only on simplified pick-and-place, or multimodal sensing that requires specialized hardware.These approaches respectively limit adaptability, leave complex contact dynamics unevaluated, or constrain deployment scalability.Source: C. VLMs enhance impedance parameter… · p. 3
  • The adaptor combines VLM semantic understanding with real-time force feedback to generate context-aware impedance parameters, supporting zero-shot generalization across diverse contact-rich tasks while maintaining safety through force-regulated adaptation.It maps visual-linguistic context to impedance values using pre-trained VLMs.Source: C. VLMs enhance impedance parameter… · p. 3

III. METHODOLOGY

  • The architecture combines a VLA-VIC controller, VLM-based impedance generation from visual-linguistic context, and a real-time force-regulated safety layer.The VLM processes visual observations, language instructions, and force feedback to generate stiffness and damping parameters that modulate the VIC executing VLA actions.Source: §III · p. 3, Figure 2 · p. 3

A. Problem formulation

  • The task formulation represents manipulation as M = (S, A, T, C), with visual observations in the state space, natural-language task specifications, and physical constraints including force limits..Source: A. Problem formulation · p. 3
  • For a 7 DoF arm, a traditional VLA maps observations and instructions to end-effector displacement and gripper actions but ignores interaction forces; the adaptor supplies actions, stiffness, and damping to the controller.The formulation identifies external contact force and torque F, translational stiffness K, and translational damping D as relevant quantities.Source: A. Problem formulation · pp. 3–4
  • Orientation stiffness and damping use empirically tuned proportional coefficients, with ε = 0.15 and ζ = 0.707, to scale them relative to translational parameters for compliance and control stability.The paper describes this formulation as enabling simultaneous generation of desired actions and compliance parameters.Source: A. Problem formulation · p. 3

B. CompliantVLA-adaptor

A frozen VLM processes visual observations, language, and external force to produce impedance parameters. Contact-phase recognition and force regulation form a dual-layer safety system.

B. CompliantVLA-adaptor

  • A frozen VLM extracts multimodal features from external force, visual observations, and physics-injected language instructions, then maps them to impedance parameters.The system pairs VLM-based parameter modulation with contact-phase recognition as a dual-layer safety system.Source: B. CompliantVLA-adaptor · p. 4
  • The method combines semantic impedance modulation with contact-phase recognition to support safe interaction forces.These are identified as the two parts of the dual-layer safety system.Source: B. CompliantVLA-adaptor · p. 4
  • The VLM receives wrist and overview RGB images, a language instruction, and external force, reasons about the current task phase, and generates impedance parameters.The process uses a pretrained VLM such as ChatGPT-4o-mini.Source: B. CompliantVLA-adaptor · p. 4

1) Contact Phase Recognition through multimodal information:

Contact-phase recognition combines visual-language understanding with force sensing, while impedance generation maps task context into variable parameters that are adjusted using real-time force feedback.

1) Contact Phase Recognition through multimodal information:

  • Hybrid phase recognition combines VLM visual understanding with force feedback to distinguish Free-motion, Approaching, Contact, and Retreat without contact-dynamics models.The passage notes that vision alone can miss contact under occlusion or subtle transitions, while force-only detection omits semantic context.Source: 1) Contact Phase Recognition through… · p. 4

2) Multimodal-informed impedance parameter generation:

  • A force-based scaling factor αforce in [0.2, 1] reduces stiffness when measured forces exceed safe thresholds, complementing VLM-generated impedance values.The resulting parameters are fed to the VIC controller, which remains compliant to unexpected contact forces.Source: 2) Multimodal-informed impedance… · p. 4
  • Force/torque readings are monitored continuously at 1000 Hz, and damping is computed for critical damping using effective mass and ζ = 0.7.The reactive strategy adapts using both measured forces and VLM context reasoning.Source: 2) Multimodal-informed impedance… · p. 4
  • A physical-context prompt lets the VLM infer anisotropic impedance parameters from task requirements, current situation, and environmental physical constraints, reducing impedance along intended motion and elevating it perpendicular to motion for alignment.Damping coefficients are proportional to stiffness values (10–20%).Source: 2) Multimodal-informed impedance… · p. 4
  • The phase hierarchy assigns highest impedance in free motion, medium impedance while approaching or retreating, and lowest impedance during contact.Context-aware settings are intended to avoid unnecessary conservatism and dangerous over-forcing in the primary motion direction, with ranges scaled to hardware.Source: 2) Multimodal-informed impedance… · p. 4

C. Hybrid VLA-VIC control architecture

The hybrid architecture retains VLA action generation and adds VLM-informed VIC execution, with impedance generation, action chunks, and low-level control operating at different rates.

C. Hybrid VLA-VIC control architecture

  • The architecture integrates VLA action generation with VLM-enhanced VIC execution to retain VLA generalization while adding compliant safety.The system operates across three temporal scales.Source: C. Hybrid VLA-VIC control architecture · p. 4
  • At approximately 1 Hz, the VLM processes visual-linguistic context and force feedback to generate impedance parameters.This is the VLM-informed impedance-generation scale.Source: C. Hybrid VLA-VIC control architecture · p. 5
  • At approximately 3 Hz, the VLA generates desired end-effector displacements from current observations and instructions.The action chunk is tracked by the low-level controller at 1000 Hz, using adaptive compliance to maintain safe contact.Source: C. Hybrid VLA-VIC control architecture · p. 5

IV. EXPERIMENTS

The experiments assess whether CompliantVLA-adaptor improves VLA task success under force thresholds and whether its VLM generates context-aware impedance parameters for safe interaction.

IV. EXPERIMENTS

  • The evaluation compares CompliantVLA-adaptor with state-of-the-art VLA models that use position control without force adaptation.The tasks include object insertion and contact-sensitive pick-and-place.Source: §IV · p. 5
  • The experiments ask whether the adaptor improves task success over baseline VLA models under force thresholds.Source: §IV · p. 5
  • The evaluation also examines how effectively the VLM generates context-aware impedance parameters for safe interaction.Source: §IV · p. 5

A. Simulation tasks

The simulation evaluation selects eight contact-rich tasks from LIBERO and ManiSkill and compares augmented VLA models with baseline control under a shared force-safety protocol.

A. Simulation tasks

  • Eight representative tasks from LIBERO and ManiSkill cover contact-rich scenarios such as precision insertion, shape sorting, drawer operation, and constrained object placement.The subset is used instead of the full set of 100+ tasks to avoid redundancy and resource constraints.Source: A. Simulation tasks · p. 5
  • The comparison uses Pi0, RDT-1B, and OpenVLA-oft, representing different architectures and training paradigms.The experiments are conducted in the same environment as the baselines.Source: A. Simulation tasks · p. 5
  • A two-stage protocol evaluates each baseline's default control and then its augmented operation with the VLM-enhanced VIC controller under the same force check criteria.Baseline control has no explicit force feedback; executions use a 30N threshold and terminate after three consecutive violations or a timeout.Source: A. Simulation tasks · p. 5

B. Contact force measurement and safety criteria

The evaluation defines force-safety criteria and describes how force feedback, impedance updates, and real-world hardware support contact-rich experiments.

B. Contact force measurement and safety criteria

  • For contact-sensitive tasks, the force threshold is 30N; a trial ends as failed if force exceeds it three consecutive times.Forces are monitored across Fx, Fy, and Fz to filter transient sensor spikes.Source: B. Contact force measurement and… · pp. 5–6

C. Contact force regulation by CompliantVLA-adaptor

  • Force is monitored continuously, while the VLM is queried every two action steps (about 0.5s) to update impedance parameters.The query interval balances responsiveness with computational load.Source: C. Contact force regulation by… · p. 6

D. Real-world experiments

  • Real-world experiments use a 7-DoF Franka Emika Panda arm with a parallel gripper for contact-rich manipulation, including pushing a heavy box to a target position and wiping a whiteboard clean.A client handles data acquisition and low-level control, while an NVIDIA RTX A6000 server hosts the VLA model.Source: D. Real-world experiments · p. 6

1) Visual perception:

  • A global Logitech camera and wrist-mounted RealSense D435i provide complementary views, with both images resized to 224 × 224 pixels for the VLA.The wrist camera provides an egocentric view of the scene.Source: 1) Visual perception · p. 6

2) Data collection for fine tuning:

The section describes real-world fine-tuning data and reports simulation comparisons under a contact-force constraint. The adaptor improves success and reduces force violations across most evaluated tasks, with results varying by task category.

2) Data collection for fine tuning:

  • The real-world fine-tuning dataset contains RGB images from both cameras, proprioceptive states, and language instructions, with 50 successful trials collected per task.The data are intended for fine-tuning the Pi0.5 VLA backbone.Source: 2) Data collection for fine tuning · p. 6

V. RESULTS

  • The simulation evaluation compares CompliantVLA-adaptor with state-of-the-art VLA baselines across eight contact-rich tasks from LIBERO and ManiSkill.The reported comparison covers Pi0, RDT-1B, and OpenVLA-oft.Source: A. Simulation results · p. 6

A. Simulation results

  • Success rates improve and force violations fall across 7/8 contact-rich simulation tasks with the adaptor.Fig. 4 compares adaptor and baseline runs across eight tasks and the RDT, Pi0, and OpenVLA-oft models.Source: A. Simulation results · p. 6, Figure 4 · p. 7
  • The average success rate across all tasks rises from 9.86% to 17.29%, while the adaptor's maximum success rate is 76%.The baseline maximum is 54%; mechanical-constraint tasks show the largest relative improvements, while higher-baseline pick-and-place tasks improve more modestly.Source: A. Simulation results · p. 6
  • Baseline force violations lead to task termination in plug-charger, drawer, and push tasks; the adaptor reduces these violations through impedance adjustment.The reported examples include RDT on plug charging, Pi0 and OpenVLA-oft on drawer manipulation, and OpenVLA-oft on pushing.Source: A. Simulation results · p. 6

B. Real-world experiment results

In real-world tests, the adaptor regulates stiffness and keeps interaction forces within safe limits. The paper also identifies latency, operating cost, and generalization to novel settings as limitations.

B. Real-world experiment results

  • During real-world wiping, stiffness ranges from 400 to 600; during box pushing, it ranges from 800 to 1000, with forces reported within safe limits.The stiffness and force profiles show smooth transitions without abrupt spikes.Source: B. Real-world experiment results · p. 7

VI. DISCUSSION AND LIMITATIONS

  • VLMs may generalize poorly to unseen objects or environments, and API calls are costly while onboard VLMs are computationally heavy.VLM inference is slower than the servo loop, but abnormal-contact handling remains in the 1000 Hz force-regulated VIC; latency limits semantic-impedance-prior refresh.Source: §VI · p. 7

VII. CONCLUSION

  • The conclusion presents the adaptor as a lightweight, plug-and-play safety layer for existing VLA models, combining visual-language context with VIC.The paper reports higher task success rates and fewer force violations in simulation and real-world experiments.Source: §VII · p. 7
Loading 2601.15541v2…