PaperlayerView exact revision on arXiv

Source-linked AI summary

Contact-Aware Imitation Learning Through Contact Factorization

Jiho Hong, Daeun Song, Sanghyun Kim, Mingyo Seo

arXiv:2610.09533v1cs.RO

TL;DR

Raw force measurements can make contact-rich imitation dependent on the conditions represented in demonstrations. FACE factorizes interaction in normalized contact-relative coordinates and adapts physical commands using online contact estimates, with the policy held fixed. In real-robot trials across unseen surfaces, it succeeds more often than No Force-Factorization.

  • Problem

    Raw-force imitation can tie a learned policy to demonstrated conditions, although the same intended behavior may require different force magnitudes and directions as contact geometry and surface properties change.

    Source: §I · p. 1

  • Method

    FACE represents normal and tangential forces in normalized contact-relative coordinates and uses estimated contact normal and friction scale to reparameterize observations and fixed-policy outputs.

    Source: §I · p. 1

  • Results

    29/30 trials (96.7%) across three unseen surface conditions succeeded with FACE, compared with 15/30 (50%) for No Force-Factorization.

    Source: C. Quantitative Evaluation · p. 7

  • Takeaways & Limitations

    Real-robot experiments support separating task behavior from contact factors, with controlled comparisons highlighting the complementary roles of force factorization and contact-normal adaptation.

    Source: §V · p. 8

Abstract

from arXiv · show

Generalizable contact-rich manipulation requires robots to preserve intended task behavior while adapting its physical realization to changing contact conditions. However, interaction forces can vary substantially with small changes in surface geometry, orientation, and friction, making policies trained directly on raw force measurements difficult to transfer beyond demonstrated conditions. We introduce FACE, a contact-factorized imitation learning framework that separates intended task behavior from environment-dependent contact factors. Our representation expresses interaction forces in normalized, contact-relative coordinates, while a learned contact-normal estimator and an online friction estimator infer the local contact normal and effective friction scale. Together, these estimators enable force observations to be encoded and policy outputs to be decoded into physical motion and force commands during execution. In this way, FACE adapts execution to current contact conditions while preserving the intended task behavior, without updating the policy parameters. We evaluate FACE on real-robot contact-rich manipulation under unseen variations in surface properties and geometry, demonstrating robust generalization across contact conditions through controlled comparisons with variants that adapt prior approaches to our setting. Videos and additional materials can be found on the project page: https://rcilab.khu.ac.kr/face.

I. INTRODUCTION

Contact-rich policies can overfit to the forces observed during demonstration, even when the intended behavior is unchanged. FACE factorizes interaction forces and adapts their physical realization using estimated contact conditions, while keeping the task policy fixed.

I. INTRODUCTION

  • Raw-force imitation can tie a policy to demonstrated conditions because required force magnitudes and directions change with surface properties and contact geometry.Prior force-aware methods also use physical force quantities whose scale and orientation depend on the environment.Source: §I · p. 1
  • FACE represents normal loading with bounded coordinates and tangential loading relative to friction-scaled normal loading in the local contact frame.The same factorization is used for observations and actions, with contact orientation and friction scale estimated separately.Source: §I · p. 1
  • At deployment, estimated friction and local contact normal reparameterize a fixed policy’s outputs into physical motion and force commands.The policy remains fixed while execution adapts across surfaces with different frictional properties and curvatures.Source: §I · p. 1
  • The evaluation uses real-robot contact-rich manipulation under unseen variations in friction, surface geometry, and their combination.The demonstrations are collected with a handheld force-sensing interface.Source: §I · pp. 1–2

II. RELATED WORK

Prior work has incorporated force and tactile feedback into demonstrations, policies, and contact-aware action representations. FACE focuses on representing interaction across changing contact conditions through friction-relative tangential loading and online contact-factor estimates.

A. Force-Aware Demonstration and Imitation Learning

  • Force-aware demonstration and imitation methods capture coupled motion and force information, but their learned force or compliance targets can remain dependent on demonstrated contact conditions.Teleoperation-based collection can require an instrumented interface and closed-loop robot execution during demonstration.Source: A. Force-Aware Demonstration and… · p. 2

B. Contact-Aware Action Representation

  • Contact-aware action methods use task-related or inferred constraint frames, learned compliance, and interaction-frame estimation, while FACE focuses on friction-normalized tangential loading.At inference, FACE uses estimated contact normal and friction scale to encode force observations and decode fixed-policy outputs into physical commands.Source: B. Contact-Aware Action Representation · p. 2

III. METHOD

FACE formulates contact-rich manipulation with unknown physical properties and local geometry as a partially observed decision process. It estimates a latent contact factor from observation history, then uses that estimate to factorize interaction observations and produce physical references from policy actions.

A. Problem Formulation

  • The latent state is decomposed into task state and a contact factor comprising local contact normal and effective friction scale, which may vary across objects, surfaces, and locations.The contact factor is generally unknown before interaction.Source: A. Problem Formulation · p. 3
  • FACE estimates the contact factor from observation history and uses it to separate demonstrated behavior from its contact-dependent physical realization.RGB, end-effector pose, and measured interaction wrench form the observation; the estimate factorizes the wrench and transforms policy actions into physical motion and force references.Source: A. Problem Formulation · p. 3

B. Contact Factorization for Interaction Representation

FACE represents interaction forces in local contact coordinates, separating task-level normal loading from tangential loading scaled by friction and normal force. These normalized coordinates reduce dependence on global pose and changing surface orientation.

B. Contact Factorization for Interaction Representation

  • Normal loading is normalized by a prescribed operating limit into a bounded, dimensionless coordinate, with zero denoting no loading and one denoting loading at or above the limit.The coordinate clips values to [0,1].Source: B. Contact Factorization for… · p. 3
  • Tangential force is factored into the effective friction scale, normal force, and a residual representing demonstrated loading relative to friction-scaled normal loading.The residual is evaluated only above a small positive normal-force threshold; during steady Coulomb sliding it approaches one when the friction scale matches the effective value.Source: B. Contact Factorization for… · p. 3
  • The tangential action residual is represented in bounded log coordinates, mapping its nominal value of one to zero for policy learning.Tangential action supervision excludes samples at or below the normal-loading threshold.Source: B. Contact Factorization for… · p. 3
  • The same physical factorization is used for interaction observations and actions, while relative poses remove dependence on the absolute robot configuration and world frame.The contact normal and friction scale account for local surface orientation and the dominant scaling of tangential force.Source: B. Contact Factorization for… · p. 3

C. Policy Learning

The policy learns demonstrated motion and interaction in contact-factorized coordinates. A fixed nominal friction reference defines the interaction behavior to learn, while time-varying friction estimates are reserved for observations and execution adaptation.

C. Policy Learning

  • A fixed median friction reference from valid sliding samples defines interaction targets, removing nominal friction scale while preserving temporal variation in demonstrated interaction.The reference is computed for each training contact condition and held fixed throughout its demonstration.Source: C. Policy Learning · p. 3
  • Normal-loading targets are supervised over the full trajectory, including zero loading in free space.Tangential-interaction supervision is limited to valid contact, and padded episode-boundary steps are excluded.Source: C. Policy Learning · p. 4
  • Interaction targets use the fixed reference rather than the causal, time-varying deployment estimate, which could absorb demonstrated interaction variation into the contact factor.The online estimate is reserved for causal observation and execution adaptation.Source: C. Policy Learning · p. 4
  • A conditional flow-matching policy generates a demonstrated action sequence from the current policy observation using interpolated noise and target velocity.Each action combines demonstrated relative motion with its factorized interaction target.Source: C. Policy Learning · p. 4

D. Online Contact Estimation & Execution Adaptation

At deployment, FACE estimates the contact normal and friction scale online, then uses them to convert fixed-policy outputs into physical motion and force references. Each committed action is reparameterized with the latest estimates for adaptation to changing contact conditions.

D. Online Contact Estimation & Execution Adaptation

  • The contact activation is derived from measured force magnitude, and a separately trained estimator uses recent pose, rotation, and wrench history to estimate the local contact normal.The estimator is trained on the task demonstrations with an activation-weighted cosine loss against normals from registered training geometry.Source: D. Online Contact Estimation &… · p. 4
  • During active contact with sufficient tangential motion, a scalar Kalman filter tracks friction from tangential and normal forces and returns an online estimate initialized from a condition-independent prior.The estimate is used for contact adaptation at deployment.Source: D. Online Contact Estimation &… · p. 4
  • The estimated friction scale restores the friction-dependent scale of predicted tangential interaction when converting it to physical force references.Source: D. Online Contact Estimation &… · p. 4
  • The estimated normal adapts predicted motion to local contact geometry; hybrid position-force control applies tangential force feedforward and regulates normal force along that normal.Each committed action uses the latest normal and friction estimates without additional policy inference within the action chunk.Source: D. Online Contact Estimation &… · p. 4

IV. EXPERIMENTS

The experiments evaluate FACE on a real-robot drawing task under unseen changes in object geometry and frictional properties. They examine contact-factor estimation, decoupling of learned behavior from physical realization, and generalization across contact conditions.

IV. EXPERIMENTS

  • The evaluation uses an FR3 manipulator on a real-robot drawing task with unseen objects varying in contact geometry and frictional properties.Source: §IV · p. 4
  • The study asks whether contact factors can be estimated reliably on unseen objects and surfaces.Source: §IV · p. 4
  • The experiments test whether factorization decouples learned behavior from contact-dependent physical realization and improves generalization across geometry and friction changes.They also examine how each component contributes.Source: §IV · p. 4

A. Experimental Setup

The study tests whether contact-rich marking transfers from nominal demonstrations to altered object geometries and surfaces. A handheld sensing interface records demonstrations, and matched sensing hardware is used for robot deployment.

A. Experimental Setup

  • The real-robot task marks an “S” on fragile objects, using compliant paper cups and rigid, brittle eggs with varied geometries.Cup surfaces also vary from higher-friction PVC tape to lower-friction plastic and wet plastic; variations appear only at deployment.Source: A. Experimental Setup · pp. 4–5
  • The handle is removed for deployment, and the same gripper and sensing assembly are mounted on the robot to reduce end-effector mismatch.Source: Figure 3 · p. 5, A. Experimental Setup · p. 5
  • Successful marking produces a continuous trajectory, while contact failures include insufficient force, excessive force, and contact loss.The protocol also categorizes incorrect motion as a failure mode.Source: Figure 4 · p. 5, A. Experimental Setup · p. 5
  • Demonstrations use a handheld gripper with force/torque sensing and visual-inertial tracking, recording synchronized motion, visual observations, and contact forces.RGB, pose, and gripper state are recorded at 30 Hz, while wrench is sampled at 500 Hz.Source: A. Experimental Setup · p. 5, Figure 3 · p. 5
  • Methods train on the same 200 demonstrations per object category and are tested on unseen conditions, with success rate as the primary metric.Surface variations use 10 trials per condition; cup geometry variations and egg instances use 5 trials each.Source: A. Experimental Setup · p. 5

B. Surface Estimation and Interaction Representation

The adaptation module estimates contact factors during sliding, while factorized force coordinates reduce variation across materials and geometries without removing temporal structure. Its estimates remain effective on held-out contact conditions.

B. Surface Estimation and Interaction Representation

  • A GRU estimates contact normals from motion and wrench histories, then a scalar Kalman filter updates friction estimates during steady sliding.The contact-normal estimator is trained on paper and PTFE over flat and curved configurations; cotton and an unseen convex geometry are held out for evaluation.Source: B. Surface Estimation and Interaction… · p. 6
  • Raw tangential loading varies across materials and geometries, while factorized residuals stay centered near zero during steady sliding and retain phase-dependent variation.Departures from ρ = 1 outside steady sliding support learning ρ rather than fixing it to unity.Source: B. Surface Estimation and Interaction… · p. 6
  • 0.030 rad and 0.045 rad are the average contact-normal errors on seen and unseen geometries, respectively; with cotton and convex B both unseen, error is 0.047 rad and median friction ratio is 1.20.The friction ratio is measured during steady sliding.Source: B. Surface Estimation and Interaction… · p. 6
  • Initialized from a condition-independent prior, friction estimates approach their reference scales during steady sliding, including for held-out material and geometry conditions.The reported evaluation includes both unseen geometries and unseen materials.Source: B. Surface Estimation and Interaction… · p. 6

C. Quantitative Evaluation

FACE is compared with prior approaches adapted to the setting and controlled variants that remove or alter individual components. It performs strongly across unseen surface conditions, while the results also identify variation across geometry conditions and remaining contact-regulation failures.

C. Quantitative Evaluation

  • The controlled comparisons assess visual-only imitation, raw force as an observation, force in physical units, and analytic instead of learned contact-normal estimation against FACE.The comparisons are intended to isolate contributions under varying contact conditions.Source: C. Quantitative Evaluation · pp. 6–7
  • 65% (13/20) versus 0% is the success-rate comparison between FACE and the analytic contact-normal variant across unseen paper-cup deformations.On unseen eggs, FACE achieves 68% versus 4%; excessive-force failures decrease from 56% with the analytic estimator to 8% with FACE, while insufficient-force failures remain at 24%.Source: Table I · p. 7, C. Quantitative Evaluation · p. 7
  • 29/30 trials (96.7%) is FACE’s success rate across unseen surface conditions, compared with 15/30 (50%) for No Force-Factorization.The table reports success and failure-mode breakdowns over 10 trials per condition; FACE’s only reported failure is insufficient force on the wet plastic shell.Source: Table II · p. 7, C. Quantitative Evaluation · p. 7
  • Visual Observation Only exhibits contact loss in 96% of egg trials, whereas FACE has no such failures; No Force-Factorization achieves 45% across cup deformations and exceeds FACE by one trial in Vertical (mild).Source: C. Quantitative Evaluation · p. 7
  • The results support separating learned interaction from contact-dependent physical execution, with ablations highlighting force factorization and learned contact-normal estimation and remaining insufficient-force and contact-loss failures indicating limits in contact regulation.Source: C. Quantitative Evaluation · p. 8

V. CONCLUSION

FACE separates intended task behavior from environment-dependent contact factors and uses contact-relative normalized force coordinates with online estimates to adapt execution without updating the policy. Real-robot experiments show improved transfer under unseen surface and geometry variations.

V. CONCLUSION

  • FACE represents force observations and actions in normalized coordinates defined by the local contact frame and friction scale.Source: §V · p. 8
  • A learned contact-normal estimator and online friction filter adapt execution to current contact conditions while leaving the task policy unchanged.Source: §V · p. 8
  • The authors identify friction estimation during limited sliding as a potential limitation and propose visual cues as a path toward addressing partial observability and extending the framework to deformable and multi-contact interactions.Source: §V · p. 8
Loading 2610.09533v1…