PaperlayerView exact revision on arXiv

Source-linked AI summary

M3-Tele: A Unified Multimodal Teleoperational Framework for Compliant Whole-Body Mobile Manipulation

Hengxiang Chen, Shenwen Deng, Yujian Ma, Gan Ma, Qiang Li, Nutan Chen

arXiv:2609.07859v2cs.RO

TL;DR

Imitation learning for contact-rich mobile manipulation needs demonstrations that are high quality, while existing work often separates multimodal perception from coordinated whole-body operation. M3-Tele integrates compliant whole-body control and tactile feedback in a multimodal teleoperation and data-collection system. Experiments report improved contact regulation, task completion rates of 80% to 94%, and demonstrations that support learning-based manipulation policies.

  • Problem

    Existing teleoperation and data-collection work often separates multimodal perception from coordinated whole-body operation, while imitation learning depends strongly on demonstration quality, coverage, and consistency.

    Source: §1 · pp. 1–2

  • Method

    M3-Tele combines force-adaptive task-space admittance, constrained whole-body inverse kinematics, tactile-adaptive gripper control, and multimodal teleoperational data collection.

    Source: §1 · p. 2

  • Results

    M3 reduced force tracking error to 0.35 N versus 4.12 N without admittance control, and M3-Tele task completion rates ranged from 80% to 94%.

    Source: §4.2 · p. 9, §5 · p. 11

  • Takeaways & Limitations

    The collected multimodal demonstrations effectively support learning-based manipulation policies.

    Source: §5 · p. 11

    The proposed compliant control framework is currently scoped to parallel grippers, with extension to dexterous hands left for future work.

    Source: §5 · p. 11

Abstract

from arXiv · show

Executing contact-rich tasks efficiently requires the seamless integration of whole-body coordination and physical compliance regulation. However, existing teleoperation and data-collection frameworks often overlook the joint consideration of multimodal perception and coordinated whole-body operation. This limitation can reduce the efficiency and quality of demonstration collection, thereby affecting the effectiveness of downstream policy learning. In this work, we present \textbf{M3-Tele}: A Unified \underline{M}ultimodal \underline{Tele}operational Framework for Compliant Whole-Body \underline{M}obile \underline{M}anipulation, enabling stable physical interaction and capturing aligned visual, tactile, force, and proprioceptive observations during task execution. Extensive experiments demonstrate that the proposed framework significantly improves contact-rich teleoperation performance. The proposed controller reduces the force tracking error from 4.132~N to 0.346~N, the contact loss from 2.46 to 0.02 events per trial and the tactile deformation error by 65\%. User studies across four mobile manipulation tasks also verify the reliability and usability of the proposed system. Furthermore, Diffusion Policy experiments highlight the value of joint tactile and force sensing.

1. Introduction

Mobile manipulators are increasingly used both to perform tasks and to collect demonstrations, whose quality, coverage, and consistency matter for imitation learning. M3-Tele addresses the separation between multimodal learning and compliant whole-body mobile manipulation with a teleoperation and data-collection system built around coordinated control and tactile feedback.

  • Mobile manipulators are increasingly treated as data engines for reusable visuomotor policies, alongside their roles in household assistance, logistics, inspection, and service robotics.Source: §1 · p. 1
  • Demonstration quality, coverage, and consistency strongly affect imitation learning, while inconsistent actions and insufficient state coverage can induce distribution shift and compounding deployment errors,.Source: §1 · p. 1
  • Whole-body coordination can reduce base–arm mode switching, but kinematic coordination alone does not ensure compliant contact or integrate dense optical tactile feedback into teleoperation-to-learning workflows.Source: §1 · p. 2
  • M3-Tele combines retargeted commands, force-adaptive task-space admittance, constrained whole-body inverse kinematics, and tactile-adaptive gripper control for contact-rich teleoperation.It provides aligned simulation and real-world interfaces for multimodal data acquisition, policy training, and reproducible evaluation.Source: §1 · p. 2, Figure 1 · p. 2

2. Related Work

The related work covers whole-body inverse kinematics, compliant control, and imitation learning for mobile manipulation. M3 links coordinated base–arm control and directional force compliance with tactile observations in an imitation-learning pipeline.

2.1. Whole-Body Inverse Kinematics

  • Whole-body inverse kinematics distributes end-effector motion across base and arm while respecting constraints; M3 adds posture regulation, base damping, and coordinated velocity commands.Source: §2.1 · p. 3

2.2. Compliant Control

  • M3 projects measured contact force onto the contact-normal z-axis and adjusts only that position reference, preserving tangential and orientation tracking stiffness.The corrected task-space target then passes to whole-body inverse kinematics.Source: §2.2 · p. 3

2.3. Mobile Manipulation Imitation Learning

  • Prior mobile-manipulation policies primarily use vision or robot states, while visuo-tactile imitation work mainly targets fixed-base platforms; M3 bridges these directions with whole-body and multimodal learning.Source: §2.3 · p. 3

3. Methodology

M3-Tele formalizes multimodal demonstrations and a controller pipeline that retargets leader motion, regulates contact, and coordinates base-arm motion.

3. Methodology

  • The methodology introduces M3-Tele as a multimodal teleoperation framework and identifies M3 as its complete controller formulation.Source: §3 · p. 3

3.1. Problem Formulation

  • M3 records multimodal observations, task-space actions, and teleoperated demonstrations for downstream mobile-manipulation policy learning.Source: §3.1 · pp. 3–4

3.2. Teleoperation Controller Formulation

  • The controller maps leader motion to end-effector targets, handles compliant contact and robust grasp, then coordinates base-arm motion through constrained whole-body inverse kinematics.Source: §3.2 · p. 4

3.2.1. Leader-to-Follower Retargeting

  • Leader motion is transformed and scaled into the follower convention; targets generated independently from fixed initial poses avoid accumulated retargeting drift.Source: §3.2.1 · p. 4

3.2.2. Unified High-level Controller

  • The high-level controller transforms measured force into the world frame and uses force and tactile-deformation errors for reference corrections; tactile deformation is a proxy, not calibrated force.Source: §3.2.2 · pp. 4–5

3.2.3. QP-based Whole-Body Inverse Kinematics

  • Velocity-level constrained inverse kinematics allocates task motion across base and arm while minimizing pose error, regularizing posture and motion, and enforcing the differential-drive nonholonomic constraint.Source: §3.2.3 · p. 5

4. Experiments

The experiments evaluate contact control, coordinated mobile manipulation, teleoperation usability, and the downstream utility of collected demonstrations. Across these evaluations, M3-Tele shows improved contact regulation and strong performance across tasks, while joint tactile and force observations benefit policy learning, particularly with fewer demonstrations.

4.1.1. Hardware

  • The platform combines a mobile base and 7-DoF arm with RGB cameras, a wrist force–torque sensor, and GelSight tactile sensors; demonstrations include visual, tactile, wrench, and proprioceptive observations.The same tactile stream yields force-distribution, contact-motion, and marker-deformation representations, and sensing and control modules run asynchronously.Source: §4.1.1 · p. 6

4.1.2. Baselines

  • Six configurations vary arm-side contact control, base–arm coordination, and operator interface to compare admittance control, tactile feedback, and feedback-enabled teleoperation.AC, TC, and FE denote admittance control, tactile-feedback control, and feedback-enabled configurations.Source: §4.1.2 · p. 6, Table 1 · p. 7

4.1.3. Tasks

  • Four tasks span pick-and-place and bar rotation through drawer opening and long-horizon wiping; wiping is the representative controller benchmark and policy-learning task.The tasks cover object transport, rotational manipulation, constrained pulling, and sustained surface contact.Source: §4.1.3 · p. 6, Figure 6 · p. 7, §4.1.5 · p. 8

4.1.4. Metrics; 4.1.5. Experimental Details

  • Metrics cover contact, force, tactile, task-performance, and usability outcomes; controller and system metrics use successful trials, while data validation includes task failures and completion quality.The study used 300 trials for configuration comparisons and 200 cross-task executions; board-wiping policies were evaluated over 100 rollouts using four observation configurations and varying demonstration amounts.Source: §4.1.4 · p. 8, §4.1.5 · p. 8

4.2. Experimental Results

  • Admittance control reduced force-tracking error from 4.12 N to 0.35 N; tactile feedback reduced deformation RMSE from 6.369 × 10−3 to 2.230 × 10−3.Without admittance control, contact was less stable; removing tactile control also increased force-tracking error from 0.346 N to 0.906 N.Source: §4.2 · p. 9, Figure 9 · p. 10
  • Whole-body coordination reduced completion time from 72.37 s to 58.73 s; FE-iPhone ease of use was 8.4±0.9 versus 5.8 ± 0.8 for standard iPhone.The Xbox interface had a shorter bar-rotation time of 35.4±3.9 s, but lacked the proposed interface’s physical interaction feedback.Source: §4.2 · pp. 9–10, Figure 9 · p. 10
  • Completion rates were 94.0%, 92.0%, 86.0%, and 80.0% for pick-and-place, bar rotation, drawer opening, and wiping, respectively; usability scores exceeded 7.5/10 for most tasks.Wiping took 59.63 ± 6.13 s, compared with 34.87 ± 3.60 s for pick-and-place.Source: Figure 10 · p. 10, §4.2 · p. 10
  • The complete vision–tactile–force policy achieved the highest coverage ratio across training-set sizes and generally best or near-best other metrics, especially with 25% or 50% of demonstrations.Its isolated metric exceptions did not yield better overall task execution than the complete multimodal input.Source: §4.2 · p. 10, Figure 11 · p. 11

5. Conclusion

The conclusion presents M3 as a force–tactile, whole-body control framework for more stable contact-rich manipulation, and M3-Tele as a demonstration-collection system evaluated in user studies and policy learning.

5. Conclusion

  • M3 integrates force–tactile feedback with coordinated whole-body control to improve contact regulation under physical constraints.Board-wiping experiments report lower force fluctuation and tactile-deformation variation than degraded control configurations.Source: §5 · p. 11
  • M3-Tele builds on the control framework and achieves 80%–94% completion rates across mobile-manipulation user studies.The reported studies cover simple and contact-rich scenarios.Source: §5 · p. 11
  • Diffusion Policy experiments show that the collected multimodal demonstrations can support learning-based manipulation policies.Source: §5 · p. 11
  • The proposed compliant control framework is intended for future extension from parallel grippers to dexterous robotic hands.Source: §5 · p. 11

A. Tactile Sensor Representation

The tactile representation describes contact changes through feature displacements and distinguishes overall tactile deformation from coherent tangential deformation.

A. Tactile Sensor Representation

B. Details of Whole-Body IK Formulation

The whole-body IK formulation optimizes generalized motion to track a compliant end-effector target while regularizing arm posture and motion and enforcing the mobile base’s non-holonomic constraint.

B. Details of Whole-Body IK Formulation

C. Metrics Definition

The metrics quantify contact continuity and force regulation, wiping outcomes, task timing, and operator-rated ease of use.

C. Metrics Definition

  • Contact loss is counted when filtered contact is absent, and force is averaged over valid contact samples.The contact state is obtained from measured normal force.Source: C. Metrics Definition · p. 14
  • Wiping outcomes use detected stained areas before and after execution, alongside successful and total trial counts.Source: C. Metrics Definition · p. 14
  • Ease of use is rated by operators on a 0–9 scale, while completion time is measured for successful trials.The rating considers command intuitiveness, motion coordination, contact-state awareness, and completion effort.Source: C. Metrics Definition · p. 15

D. Controller Parameters

The experimental controller parameters are summarized in Table 3.

D. Controller Parameters

E. Mobile Teleoperation Interface

The mobile interface supports episode management, compliance selection, and real-time multimodal monitoring during teleoperation. It provides connection and contact-state feedback to the operator.

E. Mobile Teleoperation Interface

  • The interface initializes demonstrations with synchronized robot control and workstation data recording, while reporting communication round-trip time for wireless-connection monitoring.Source: E. Mobile Teleoperation Interface · p. 15
  • Soft, medium, and hard profiles map to admittance-control parameters, and real-time warnings flag abnormal tactile or force readings.The profiles offer an operator-facing way to adapt interaction behavior to task phases and contact conditions.Source: E. Mobile Teleoperation Interface · p. 15
  • Configurable wrist-camera and tactile streams provide synchronized scene-level motion-alignment and local contact/deformation feedback.Operators can enable or disable individual channels and view selected observations in real time.Source: E. Mobile Teleoperation Interface · p. 15
Loading 2609.07859v2…