PaperlayerView exact revision on arXiv

Source-linked AI summary

SwapBot: Tool-Centric Robot Manipulation via Interchangeable End-Effectors and Tactile Sensing

Yunchu Zhang, Ray Song, Junhan Jiao, Liyiming Ke, Yuchen Yang, Ruihan Gao, Emma Romig, Siddhartha Srinivasa, Yiyue Luo

arXiv:2610.40646v1cs.RO

TL;DR

Household tool use requires varied contact-rich interactions, while existing end effectors face capability-controllability tradeoffs and prior tool-use methods may lack tool-object tactile sensing. SwapBot combines low-cost adapters for diverse household tools, integrated tactile sensing, teleoperation demonstrations, and hierarchical policies for tool selection and tool-specific control. Tactile feedback improved imitation-learning policy performance for pan-fried tofu cooking, increasing task success rates by 25 absolute percentage points, from 35% to 60%.

  • Problem

    Household tool use requires varied contact-rich interactions, while existing end effectors face capability-controllability tradeoffs and prior tool-use methods may lack tool-object tactile sensing.

    Source: §I · p. 2

  • Method

    SwapBot combines low-cost adapters for diverse household tools, integrated tactile sensing, teleoperation demonstrations, and hierarchical policies for tool selection and tool-specific control.

    Source: §I · p. 2, D. Policy Learning: From… · p. 5

  • Results

    Tactile feedback improved imitation-learning policy performance for pan-fried tofu cooking, increasing task success rates by 25 absolute percentage points, from 35% to 60%.

    Source: §I · p. 2

  • Takeaways & Limitations

    The results demonstrate tool-centric teleoperation and imitation learning on long-horizon cooking, with tactile feedback improving reported policy performance.

    Source: §I · p. 2, B. Evaluating Intuitive Tool-Centric… · p. 6, C. Policy Learning with Tactile… · p. 7

    Current sensors are low resolution and prone to wear, placement relies on human intuition, and policy learning depends on predefined task segmentation.

    Source: §V · p. 8

Abstract

from arXiv · show

Robot manipulation in everyday environments inherently involves substantial tool usage. Seemingly simple household tasks, such as cooking pan-fried tofu, require forceful cutting, delicate brushing, and precise placement using different tools under contact-rich interaction. Thus, the ability to autonomously swap and use task-appropriate tools is important for tool-centric manipulation. We introduce SwapBot, a modular, tool-centric end effector system featuring low-cost adapters that quickly convert common household tools into robot end effectors. Rather than pursuing a single universal gripper, SwapBot adopts a unified design principle that combines a fixed robot-side coupling interface with low-cost, tool-side adapters, enabling compatibility with diverse tools, including articulated ones, and multiple robot platforms. Each adapter integrates compact tactile sensors, supports fast autonomous swapping, and interfaces with an intuitive teleoperation system for efficient demonstration collection. User studies show that, during teleoperation, SwapBot reduces task completion time and operator workload. Using collected demonstrations on the robot, we train imitation learning policies to successfully perform tool-centric tasks like cooking tofu, stirring salad, flipping meat. Our ablation study shows that using tactile-augmented demonstrations improves policy performance across contact-rich skills, increasing success rates by 15 to 25 percentage points over RGB-only policies. These results highlight modular, tactile-equipped tools as a practical and extensible pathway toward robust and capable robot manipulation. Project website https://personalrobotics.github.io/SwapBot.

I. INTRODUCTION

Tool-centric household tasks demand varied contact-rich interactions that current end effectors handle poorly. SwapBot addresses this with low-cost interchangeable tools, tactile sensing, and teleoperation, and its evaluations report improved teleoperation and tactile-policy performance.

I. INTRODUCTION

  • Existing end effectors trade capability against controllability: anthropomorphic hands are costly to maintain and control, while parallel grippers struggle with nuanced contact tasks.Examples include cutting and spreading.Source: §I · p. 2
  • Prior tool-use methods either inherit dexterous-hand complexity and stability challenges or lack tactile sensing at the tool-object interface.The paper identifies this sensing gap as a hindrance to effective control and learning for contact-rich tasks.Source: §I · p. 2
  • SwapBot combines a fixed robot-side interface with low-cost adapters to equip household tools, including articulated ones, with compact tactile sensors.The approach targets the limitations of costly robot-specific changers that do not readily support articulated tools or tactile sensing.Source: §I · p. 2
  • The system supports Franka, UR5, and Tycho platforms, and autonomously swaps tools in under 10 seconds.Tool-matched leader devices and visualized tactile feedback support teleoperation and demonstration collection.Source: §I · p. 2
  • Tactile feedback increased tofu-cooking task success from 35% to 60%, a 25-percentage-point gain over RGB-only demonstrations.The system also qualitatively demonstrated stirring salad and flipping meat.Source: §I · p. 2
  • In studies with 12 participants, SwapBot reduced teleoperation completion time and workload while improving success compared with a generic tool.The study focused on brushing and cutting; robot experiments also demonstrated long-horizon tofu cooking.Source: §I · p. 2

II. THE SWAPBOT SYSTEM

The SwapBot system combines interchangeable end effectors, tactile sensing, teleoperation, and policy learning in a plug-and-play design. Its coupling aims for low-cost, rapid tool exchange across robots and both passive and articulated tools.

II. THE SWAPBOT SYSTEM

  • SwapBot comprises interchangeable end effectors, compact tactile sensing, a teleoperation interface, and policy learning from demonstrations.The plug-and-play design supports low-cost prototyping with daily-use tools and downstream learning.Source: §II · p. 2
  • SwapBot is shown adapted to Tycho, UR5, and Franka robots.The system overview also depicts the robot-mounted adapter, coupling, tool dock, and tactile I/O board.Source: Figure 3 · p. 3, Figure 2 · p. 3
  • Tool-side modules target a cost below $10, rapid swapping, compatibility across robot platforms, and support for passive and articulated tools.The design is intended for accessible fabrication and straightforward assembly with basic hand tools.Source: §II · p. 3
  • A self-aligning electromagnetic coupling joins the robot-side hub to tool adapters and supports both passive tools and articulated tools through a pass-through mechanism.A guided gear supports engagement under small alignment errors, while the electromagnet holds the tool during forceful manipulation.Source: §II · p. 3
  • The system completes a tool swap in about 10 seconds by docking, releasing the current tool, and attaching another.Its robot-side interface remains unchanged when a different adapter is needed for substantially thinner tools.Source: §II · p. 3

B. Compact Tactile Sensing: Instrumented Tools

SwapBot uses compact, tool-specific piezoresistive sensors to capture contact forces without adding bulk or compromising tool usability. Sensor placement varies with tool geometry, including indirect interaction measurements for tools too small or thin to instrument at the tip.

B. Compact Tactile Sensing: Instrumented Tools

  • Tool-specific sensor layouts capture force information without adding bulk or compromising usability.The sensors are compact and tailored to each tool.Source: B. Compact Tactile Sensing:… · pp. 3–4
  • Low-level tool-specific policies use visual observations, tactile feedback, and proprioceptive state to produce end-effector actions.The framework composes these skills with ongoing tool selection during long-horizon execution.Source: B. Compact Tactile Sensing:… · p. 4
  • The piezoresistive sensor sandwiches Velostat between conductive-fabric electrodes, converting normal force into an electrical signal through resistance changes.A matrix layout forms sensing nodes at electrode intersections and uses 2n electrodes for an n × n array.Source: B. Compact Tactile Sensing:… · p. 4
  • Larger tools can carry sensors on both handle and blade or tip, while small or thin tools are instrumented at the handle for indirect tool-object measurements.This placement keeps modules compact and lightweight when tip instrumentation is impractical.Source: B. Compact Tactile Sensing:… · p. 4

C. Teleoperation Interface: High-Quality Demonstration Collection

SwapBot's teleoperation interface pairs familiar leader tools with corresponding robot-side tools and provides live tactile visualization. The design aims to reduce retargeting effort and give operators contact information for collecting demonstrations.

C. Teleoperation Interface: High-Quality Demonstration Collection

  • The interface maps a leader device's 6-DoF pose and actuation commands to the corresponding robot-side tool, using similar tools on both sides.For articulated tools, relative motion from two IMUs supplies the actuation command.Source: C. Teleoperation Interface:… · p. 4
  • Synchronized data streams at 20 Hz include multi-view RGB, tactile signals, robot joint states, actions, and task labels.Live force and tactile-map overlays give operators contact information during demonstration collection.Source: C. Teleoperation Interface:… · p. 4
  • Tactile feedback highlights first contact and load-distribution changes that can be ambiguous in vision alone.The interface is designed to reduce cognitive load and improve precision in tasks such as cutting, grasping, and transporting.Source: C. Teleoperation Interface:… · pp. 4–5
  • SwapBot generally improves teleoperation performance over the baseline across both tasks, with tactile visualization informing operators about contact forces and load changes.The interface also supports demonstration collection for complex cooking tasks.Source: Figure 5 · p. 5

D. Policy Learning: From Demonstrations to Long-Horizon Manipulation

SwapBot uses a hierarchy that selects a tool for each stage and activates a tool-specific controller for fine-grained manipulation. With manually specified subgoals, the system sequences these policies for long-horizon tasks such as cooking pan-fried tofu.

D. Policy Learning: From Demonstrations to Long-Horizon Manipulation

  • A high-level selector predicts a tool from a top-view image and goal image, then activates the corresponding tool-specific controller.The selector is a ResNet-plus-MLP classifier with a SoftMax output.Source: D. Policy Learning: From… · p. 5
  • The low-level controller generates actions from multi-view images, tactile signals, and robot proprioception using a U-net-based diffusion policy.Source: D. Policy Learning: From… · p. 5
  • Long-horizon execution assumes a subgoal for each stage; in the current implementation, the subgoal-image sequence is specified manually before execution.A skill is considered complete after end-effector movement stays below 0.005 m for 20 consecutive control steps, after which the robot resets its pose and selects the next tool.Source: D. Policy Learning: From… · p. 5

III. EXPERIMENTS

The experiments assess tactile sensing, teleoperation, and tactile-enabled policy learning. Sensor characterization finds a trade-off between sensitivity and signal stability, leading to the selection of 2 mm electrodes.

III. EXPERIMENTS

  • The experiments ask whether tool-specific tactile sensors balance sensitivity and detection range, whether teleoperation improves demonstration efficiency, and whether tactile feedback improves imitation learning.Source: §III · p. 5

A. Characterizing Tool-Specific Tactile Sensing

B. Evaluating Intuitive Tool-Centric Teleoperation

A controlled study compared a general VR-style tool with SwapBot teleoperation, with and without visualized tactile feedback, for brushing and cutting. SwapBot generally improved performance over the baseline, with higher success and lower reported cognitive load.

B. Evaluating Intuitive Tool-Centric Teleoperation

  • The study involved 12 participants across beginner, intermediate, and expert skill levels, each performing brushing and cutting under three teleoperation conditions.Each participant completed 10 trials per task-condition pair; completion time was calculated on successful trials, while failures counted toward success rate.Source: B. Evaluating Intuitive Tool-Centric… · pp. 5–6
  • SwapBot generally improved teleoperation performance over the general-tool baseline across both brushing and cutting tasks.The figure compares three conditions: a general teleoperation tool, SwapBot without visualized tactile feedback, and SwapBot with it; it reports faster completion, higher success, and lower workload for SwapBot than the general-tool baseline.Source: B. Evaluating Intuitive Tool-Centric… · p. 6, Figure 6 · p. 6
  • Participants reported higher task success and lower cognitive load with SwapBot than with the baseline general teleoperation tool.Source: B. Evaluating Intuitive Tool-Centric… · p. 6

C. Policy Learning with Tactile Feedback for Long-Horizon Tasks

Tactile-enabled, multi-view policies performed strongly on tofu manipulation skills, and the complete system chained tool-specific policies to carry out the long-horizon task. Tactile signals provided consistent gains over RGB-only baselines, especially with less demonstration data.

C. Policy Learning with Tactile Feedback for Long-Horizon Tasks

  • The tactile-policy evaluation used 100 demonstrations for each of brushing, transport to the pot, and cutting, and compared multiple input modalities.Reverse transport was not evaluated quantitatively; RGB-only policies often failed when tool-object contact was occluded, while tactile policies were robust to small tofu-shape and initial-position variations.Source: C. Policy Learning with Tactile… · p. 6
  • Full RGBs + tactile policies achieved up to 85% success in brushing, 70% in tong picking, and 90% in knife cutting.The task required brushing sauce onto tofu, transporting it with tongs, and returning and cutting it without damage.Source: C. Policy Learning with Tactile… · p. 6
  • Across data regimes, tactile policies matched or outperformed RGB-only baselines, with the largest gains in low-data settings.Figure 7 reports success rates across demonstration-data proportions; paired t-tests are reported as confirming statistically significant improvements.Source: C. Policy Learning with Tactile… · p. 7, Figure 7 · p. 7
  • The chained system completed the full tofu task with a 60% average success rate across 20 rollouts.The tool selector achieved 96.8% test-set classification accuracy; failures were mostly due to tofu slippage or out-of-distribution states.Source: C. Policy Learning with Tactile… · p. 7

IV. RELATED WORK

Prior work studies tool grasping and tool use, while SwapBot focuses on an integrated hardware-and-learning system for interchangeable, tactile-instrumented tools and contact-rich manipulation.

A. Tool Manipulation

  • Prior work learns task-oriented grasps and functional affordances, [17], whereas SwapBot converts household tools into interchangeable, tactile-instrumented end effectors and learns visuotactile policies.The system combines tool-matched teleoperation with tactile feedback to collect demonstrations.Source: A. Tool Manipulation · p. 7
  • Earlier work demonstrates long-horizon manipulation with diverse tools through planning and learning [8], [9]; SwapBot combines hardware, tactile-feedback demonstrations, and policy learning.Source: A. Tool Manipulation · p. 7
  • Unlike approaches learning synergies between tool grasp points and tasks,, SwapBot does not rely on tool-affordance representations, instead using a mechanically simple tool changer.Source: A. Tool Manipulation · p. 7

B. Tool Design for Robot Tool Usage

SwapBot pairs a unified tool-changing system with tactile sensing at the tool–object interface, addressing limits of robot-specific tool changers and gripper-specific tactile approaches.

B. Tool Design for Robot Tool Usage

  • SwapBot supports passive and articulated household tools with secure coupling, torque transmission, tactile sensing, and tool-matched teleoperation for demonstration collection and policy learning.The system is presented as a low-cost alternative to expensive tool changers tailored to specific robots and tools.Source: B. Tool Design for Robot Tool Usage · p. 7

C. Tactile Sensing for Manipulation

  • Table II reports single-seed success rates for modality ablations, with additional seeds and variance and significance comparisons for RGB-only and SwapBot results in Fig. 7.Source: Table II · p. 8
  • Table III reports whole-task success rates from chaining the best-performing policy checkpoint from each stage.Source: Table III · p. 8
  • SwapBot adapts scalable tactile sensor designs to diverse tools, enabling flexible measurement at the tool–object interface beyond gripper-specific approaches.Prior tactile-policy approaches on parallel grippers are described as typically limited to specific gripper designs.Source: C. Tactile Sensing for Manipulation · p. 8

V. CONCLUSION

SwapBot combines modular tools and tactile sensing to improve demonstration collection and policy learning for long-horizon tasks. The conclusion highlights tactile feedback while identifying sensor and policy-learning constraints.

V. CONCLUSION

  • SwapBot is a modular, tool-centric manipulation system with integrated tactile sensing that improves demonstration collection and policy learning for long-horizon tasks.Source: §V · p. 8
  • Tactile feedback proved crucial for teleoperation quality and imitation-learning robustness, especially with limited data.Source: §V · p. 8
  • Current limitations include low-resolution, wear-prone tactile sensors, intuition-based sensor placement, and reliance on predefined task segmentation.Future work targets more durable, higher-resolution sensing, data-driven sensor placement, and autonomous sequencing without manual subgoals.Source: §V · p. 8
Loading 2610.40646v1…