Source-linked AI summary
Imp-ACT: Adaptive Impedance Control and Action Chunking with Transformers to Learn Contact-Rich Manipulation from Demonstrations
Luca Zanetti, Doganay Sirintuna, Idil Ozdamar, Pietro Balatti, Heng Zhang, Arash Ajoudani
TL;DR
Contact-rich manipulation depends on interaction response as well as motion, and stiffness targets can be difficult to align with demonstrations. Imp-ACT records adaptive stiffness during teleoperation and learns to predict it with motion; successful wiping episodes had higher reported peak stiffness than plug-insertion episodes.
Problem
Contact-rich task behavior depends on interaction-force response as well as motion, and imitation learning needs stiffness targets associated with demonstrated motion.
Source: §I · p. 1Method
Imp-ACT applies and records motion-direction stiffness during teleoperation, then trains an ACT policy to predict pose, gripper action, and stiffness from observations.
Source: §I · p. 2Results
547 N/m on average (range 485–705 N/m) was the peak stiffness across successful wiping episodes, compared with 299 N/m (range 272–329 N/m) for plug insertion.
Source: A. Stiffness adaptation · p. 6Takeaways & Limitations
The data-collection principle is policy-agnostic in spirit, with future work proposed for Diffusion Policy or VLAs.
Source: §VI · p. 8
Abstract
from arXiv · showhide
Contact-rich manipulation requires robots to balance accurate motion tracking with compliant interaction, yet most visual-action policies leave compliance fixed at the controller level. We present Imp-ACT, a methodologically grounded and practical approach to incorporating direction-dependent Cartesian stiffness modulation directly into demonstration collection, without manual stiffness selection or offline target reconstruction. During teleoperation, a self-tuning impedance controller adapts stiffness along the instantaneous direction of motion while maintaining compliance in orthogonal directions. The adapted stiffness is applied and recorded alongside visual observations and motion commands, capturing motion and compliance under the same dynamics. We implement this pipeline using Action Chunking with Transformer (ACT) to predict end-effector pose, gripper action, and motion-direction stiffness from visual, proprioceptive, and wrench observations. The performance of Imp-ACT is evaluated on wiping and plug insertion using both success rate and quantitative measures of contact behavior. Compared with fixed low- and high-stiffness baselines, Imp-ACT achieves comparable or higher success while maintaining low interaction forces. In wiping, it reduces contact-force vibration by approximately $29\times$ relative to the compliant baseline and $180\times$ relative to the stiff baseline. In plug insertion, it reduces forces orthogonal to the insertion direction by $43\%$ relative to the better fixed-stiffness baseline. These results highlight the benefit of maintaining sufficient stiffness along the direction needed for task execution while preserving compliance in other directions to limit contact forces and accommodate environmental constraints.
I. INTRODUCTION
Contact-rich manipulation depends on how robots respond to interaction forces, not only on motion. Imp-ACT generates and records direction-dependent stiffness during demonstrations, then learns motion and compliance together.
I. INTRODUCTION
- Contact-rich task execution depends on the robot’s response to interaction forces as well as its motion commands.Stiffness trades off accurate tracking and compliant interaction, so suitable stiffness targets must accompany demonstrated motion.Source: §I · p. 1
- Manual stiffness selection burdens operators, while post-demonstration compliance reconstruction can mismatch collection and deployment dynamics and shift policy states.The paper motivates applying stiffness during demonstration so motion and compliance are recorded under the same interaction dynamics.Source: §I · pp. 1–2
- Imp-ACT combines human-demonstrated motion with online direction-dependent stiffness, recorded alongside motion and generated without explicit operator stiffness commands.The policy jointly predicts pose, gripper action, and motion-direction stiffness from visual, robot-state, and wrench observations.Source: §I · p. 2
II. RELATED WORK
Prior impedance work adapts stiffness to interaction demands, while visuomotor imitation policies largely leave contact compliance to low-level control. Imp-ACT records adaptive stiffness during demonstrations and trains a policy to predict it with motion.
A. Variable Impedance Control
- Variable-impedance approaches adapt stiffness and damping to task or interaction demands, including self-tuning Cartesian control along the direction of motion.The cited literature also includes task-specific rehabilitation and prosthetic-knee controllers and operator-EMG-based teleimpedance.Source: A. Variable Impedance Control · p. 2
B. Imitation Learning
- Visuomotor policies mainly predict motion, while contact response remains with the low-level controller; compliance-action methods have required manual modes, offline reconstruction, or a separate deployment module.Introducing compliance after collection can change interaction dynamics and shift the states encountered by the policy.Source: B. Imitation Learning · p. 2
III. METHODOLOGY
- Imp-ACT adapts stiffness along the motion direction while maintaining orthogonal compliance, records the applied stiffness with motion, and learns both from visual, proprioceptive, and wrench observations.The operator commands pose and gripper action without selecting stiffness or reconstructing targets offline.Source: B. Imitation Learning · p. 2
- The methodology covers impedance control, online self-tuning, demonstration representation, and Imp-ACT policy learning and deployment.Demonstration observations include camera images, measured pose, wrench, and gripper state; targets include desired pose, gripper command, and applied stiffness, which the policy predicts at deployment.Source: §III · pp. 2–3, Figure 2 · p. 3
A. Cartesian Impedance Controller
Cartesian impedance control regulates interaction through stiffness and damping, while the self-tuning approach raises stiffness along the motion direction and preserves compliance orthogonally. It adjusts that directional stiffness using tracking error and changes in external force.
A. Cartesian Impedance Controller
- The controller uses Cartesian stiffness and damping to regulate the robot’s response to external forces and tracking errors.Its Cartesian wrench depends on desired and actual pose and velocity profiles.Source: A. Cartesian Impedance Controller · p. 3
B. Self-Tuning Impedance Approach
- Higher stiffness along the desired motion direction supports trajectory tracking, while lower stiffness in orthogonal directions preserves responses to contact forces and environmental constraints.The orthogonal stiffness and damping coefficients are set to kmin and dmin, respectively.Source: B. Self-Tuning Impedance Approach · p. 3
- The controller increases motion-direction stiffness when translational tracking error exceeds its threshold, and the error tolerance prevents unnecessary increases.The coefficient is bounded between kmin and kmax, with corresponding damping calculated from stiffness.Source: B. Self-Tuning Impedance Approach · p. 4
- When tracking error is below threshold and external force decreases quickly enough, stiffness is reduced to avoid remaining unnecessarily high after contact diminishes.The force-drop rate and its threshold determine when the decrease condition is met.Source: B. Self-Tuning Impedance Approach · p. 4
C. Demonstration Collection and Representation
Demonstrations pair robot and sensor observations with action targets, including stiffness adapted online by the controller. The representation also encodes orientation continuously for learning and decodes predicted rotations for the low-level controller.
C. Demonstration Collection and Representation
- Each recorded frame pairs robot and sensor observations with its action target, including measured TCP pose, gripper state, camera images, external wrench, equilibrium pose, and gripper command.The external wrench is mapped from the F/T sensor mounted between the robot flange and gripper.Source: C. Demonstration Collection and… · p. 4
- The self-tuning algorithm initializes stiffness at kmin and increases it when tracking error exceeds threshold, or decreases it when force-drop conditions are met.The algorithm summary specifies the update conditions for both changes.Source: C. Demonstration Collection and… · p. 4
- Online-adapted stiffness kst is appended to the action vector during demonstrations without requiring explicit operator input.Measured and target orientations use a continuous 6D representation; predicted rotations are decoded with Gram–Schmidt orthonormalization.Source: C. Demonstration Collection and… · p. 4
D. Imp-ACT Policy Learning and Deployment
Imp-ACT extends ACT to predict action chunks containing pose, gripper command, and motion-direction stiffness from camera and robot-state observations. At deployment, those predictions are smoothed, converted to controller units, and used to set translational stiffness and damping.
D. Imp-ACT Policy Learning and Deployment
- Imp-ACT predicts 50-step action chunks from two 480 × 640 RGB images and an 18-dimensional state containing pose, gripper, wrench, motor-current, and object-detection information.The chunk horizon is one second.Source: D. Imp-ACT Policy Learning and… · p. 4
- Each 11-dimensional action includes equilibrium position, orientation, gripper command, and motion-direction stiffness, while orthogonal translational stiffness remains fixed.The policy runs at 50 Hz and uses temporal ensembling to smooth overlapping predictions.Source: D. Imp-ACT Policy Learning and… · p. 4
- At deployment, predicted actions are denormalized before reaching the low-level controller, and Eqs. (4)–(8) map predicted stiffness to translational stiffness and damping.Each action dimension, including stiffness, is normalized using training-demonstration means and standard deviations.Source: D. Imp-ACT Policy Learning and… · p. 4
A. Experimental Setup
Experiments use a torque-controlled Panda arm with force sensing and wrist and back cameras. A VR teleoperation interface provides relative pose control, gripper control, and contact-force cues for operators.
A. Experimental Setup
- Experiments use a torque-controlled 7-DoF Franka Panda arm with a two-finger gripper, an added F/T sensor, and wrist and back cameras.The back camera supplies broader task context while the wrist camera captures close-range interactions.Source: A. Experimental Setup · pp. 4–5
B. Teleoperation Interface
- A VR controller provides six-degree-of-freedom teleoperation, with a clutch for relative Cartesian target updates and a trigger for proportional gripper control.Releasing the clutch pauses target updates and recording while retaining the last target.Source: B. Teleoperation Interface · p. 5
- A wrist-camera view and force arrow and gauge help operators unload contact and retract the target before releasing an inserted object, reducing risk of unintended motion into surrounding structure.Source: B. Teleoperation Interface · p. 5
C. Experimental Scenarios
The experiments compare adaptive stiffness with fixed low- and high-stiffness control on wiping and plug insertion, using separate demonstrations and matched training and evaluation conditions. The system adapts stiffness along the motion direction while retaining compliance in orthogonal directions.
C. Experimental Scenarios
- The comparison uses fixed 200 N/m and 1000 N/m controllers alongside adaptive stiffness, with separate task datasets and identical demonstration counts, variation protocols, training settings, and evaluation conditions across policies.Each task is evaluated under all three stiffness conditions.Source: C. Experimental Scenarios · p. 5
1) Wiping:
- Wiping removes marks of varying positions and shapes; 40 demonstration episodes were collected for each stiffness condition.The task requires maintaining surface contact while limiting excessive force and stick-slip vibration.Source: 1) Wiping · p. 5
2) Plug insertion:
- Plug insertion uses randomized initial poses and 80 demonstrations per stiffness condition, with approximately 1 mm tolerance for aligning both pins with the socket.The initial position varied by ±2 cm in translation and ±10° in rotation about the robot base-frame X-axis.Source: 2) Plug insertion · p. 5
D. System Parameters
- Adaptive stiffness ranges from 200 to 1000 N/m along the motion direction, while both orthogonal directions remain at 200 N/m.Rotational stiffness is fixed at 30 Nm/rad on all three axes.Source: D. System Parameters · pp. 5–6
- The controller uses damping set by D_c = 2ζ√K_c with ζ = 0.7 and an adaptation gain α = 1000; each policy trains for 300,000 steps.The reported adaptation gain balances rapid response to external interactions against abrupt stiffness changes.Source: D. System Parameters · p. 6
V. RESULTS
The learned stiffness varies by task and tracks motion direction, increasing along that direction while remaining compliant in the two orthogonal directions. During wiping, adaptation continues across free-space motion and contact rather than switching between discrete regimes.
A. Stiffness adaptation
- The stiffness ellipsoid’s main axis aligns with the instantaneous desired motion direction, and larger interaction forces are associated with greater stiffness along that direction.The two orthogonal directions remain at the compliant baseline.Source: A. Stiffness adaptation · p. 6
- During a representative wiping episode, motion-direction stiffness averages 317 N/m in free space and grasping, rises to 377 N/m on average in contact, peaks at 554 N/m, then decreases as contact is released.Adaptation remains active throughout the episode, including free-space motion and grasping.Source: A. Stiffness adaptation · p. 6
- Across successful episodes, wiping stiffness peaks at 547 N/m on average (range 485–705 N/m), compared with 299 N/m for plug insertion (range 272–329 N/m).Wiping involves sustained surface interaction, while plug insertion involves shorter, more localized contacts.Source: A. Stiffness adaptation · p. 6
B. Wiping
Imp-ACT matches the high-stiffness baseline’s wiping success while producing much lower contact-force vibration; in plug insertion, it has the highest success rate and lowest transverse force. Completion times remain comparable to the faster fixed-stiffness baseline in both tasks.
B. Wiping
- Wiping success was 100% for Imp-ACT and the high-stiffness baseline, versus 96% for the low-stiffness baseline.Each condition was evaluated over 25 trials.Source: B. Wiping · p. 6
- Wiping mean peak force was 43.2 N for Imp-ACT, versus 45 N for low stiffness and 94.4 N for high stiffness; completion time was 20.3 s.Mean interaction force was 24.3 N for Imp-ACT, 24.0 N for low stiffness, and 41.4 N for high stiffness; completion times were 16.9 s for high stiffness and 25.6 s for low stiffness.Source: B. Wiping · pp. 6–7
- Wiping band power was 0.6 N2 for Imp-ACT, versus 17.7 N2 for the low-stiffness baseline and 109.6 N2 for the high-stiffness baseline.These correspond to reductions of approximately 29× and 180×, respectively; band power measures vertical-force vibration over 8–40 Hz.Source: B. Wiping · p. 7
C. Plug insertion
- Plug insertion success was 76% for Imp-ACT, versus 72% for the high-stiffness baseline and 60% for the low-stiffness baseline.Source: C. Plug insertion · p. 7
- Plug-insertion transverse force was 4.4 N for Imp-ACT, versus 7.8 N for the low-stiffness baseline and 11.9 N for the high-stiffness baseline, a 43% reduction relative to the better baseline.Transverse force quantifies lateral loading associated with misalignment and contact with the socket walls.Source: C. Plug insertion · p. 7
- Plug-insertion completion time was 15.8 s for Imp-ACT, 15.1 s for the high-stiffness baseline, and 18.2 s for the low-stiffness baseline.Source: C. Plug insertion · p. 7
VI. CONCLUSION
Imp-ACT incorporates online stiffness modulation into demonstrations without requiring operator stiffness commands or post-collection targets. The experiments support adapting stiffness to task direction, while future work could extend the approach to rotational compliance and other policy pipelines.
VI. CONCLUSION
- Imp-ACT integrates stiffness modulation into imitation-learning demonstrations, with a self-tuning module adjusting stiffness online during intuitive kinematics-based teleoperation.The operator commands motion without explicit stiffness commands or targets assigned after collection.Source: §VI · p. 7
- Wiping and plug-insertion experiments support adapting stiffness to task direction: maintain stiffness along intended motion while preserving orthogonal compliance to limit transverse forces and accommodate constraints.Source: §VI · p. 7
- Future work could extend adaptation to rotational degrees of freedom and add a passivity filter to regulate energy injection and establish stable interaction conditions.Source: §VI · pp. 7–8
- The data-collection principle could extend beyond ACT to other visual-action pipelines, including Diffusion Policy and vision-language-action models.Source: §VI · p. 8