PolyUMI: Accessible Visual-Tactile-Audio Data Collection for Object Inference and Manipulation

• The paper addresses the limited access to contact cues in vision- and proprioception-focused demonstrations and the challenge of synchronizing vision, touch, and audio across collection and deployment.Prior work had incorporated tactile and auditory sensing, but the introduction identifies unified sensing and consistent collection-to-deployment geometry as remaining challenges.Source: §I · p. 1
• PolyUMI is an open-source wireless interface for synchronized visual–tactile–audio demonstrations and robot deployment, using the same sensing finger in handheld and robot-mounted embodiments.The platform and learning software are released with hardware designs, electronics, firmware, and fabrication instructions.Source: §I · pp. 1–2
• VisTA uses modality-specific input stems and a joint Transformer encoder to integrate sensor observations across modalities and time, conditioning a flow-matching model to predict relative end-effector commands.Source: §I · p. 2
• VisTA outperforms evaluated baselines on board wiping and matches the strongest multimodal baseline on lightbulb turning, while tactile and auditory observations support object inference and reactive slip control.The results also indicate that contact feedback is most useful when combined with visual guidance.Source: §I · p. 2

A. Multimodal Demonstration Collection

• UMI supports portable handheld demonstrations and policy transfer; later interfaces add tactile sensing, or force-related measurements,, while PolyUMI combines synchronized vision, touch, and contact audio.Source: A. Multimodal Demonstration Collection · p. 2

B. Tactile and Auditory Contact Sensing

• ManiWAV learns from in-the-wild audio-visual demonstrations, while PolyTouch and DIGIT 360 integrate tactile sensing with additional contact-sensing channels.PolyUMI follows PolyTouch’s sensing principle through an independently developed implementation designed for handheld collection, modular maintenance, and transfer between handheld and robot grippers.Source: B. Tactile and Auditory Contact Sensing · p. 2

C. Multimodal Policy Learning

• Prior policies use joint representations, masked multimodal learning, self-attention, stage-guided fusion, or multisensory touch representations; VisTA jointly processes spatial and temporal observations to condition a flow-matching action head.Source: C. Multimodal Policy Learning · p. 2

III. VISUAL-TACTILE-AUDIO DATA COLLECTION FOR OBJECT INFERENCE AND MANIPULATION

• PolyUMI reuses one sensing finger across handheld demonstration collection and robot execution, addressing contact information that vision may miss and enabling multimodal data acquisition and policy deployment.VisTA jointly reasons over visual, tactile, auditory, and proprioceptive observations to generate contact-aware actions.Source: §III · p. 2

1) System Overview:

• The handheld gripper and robot-mounted end-effector share a transferable multimodal finger; the handheld device has an onboard battery, computer, and audio interface, so collection needs no tethered workstation.The wrist camera is triggered over Bluetooth to coordinate recording, and the mount can be adapted to other grippers by replacing the platform-specific mount.Source: 1) System Overview · pp. 2–3
• PolyUMI is the only compared handheld interface that collects synchronous vision, tactile, and audio data, and it provides live audio monitoring for operator feedback.Source: Table I · p. 3

2) Modular Multimodal Sensing Finger:

• The optical tactile sensor views a large contact region through a curved mirror; its seven-layer reflective surface makes local deformation visible, and the tactile stream records at 20 fps and 1152 × 648 resolution.Camera parameters adjust online to maintain usable observations as ambient illumination changes.Source: Table II · p. 3, 2) Modular Multimodal Sensing Finger · p. 3
• A finger-coupled microphone records vibrations transmitted through the finger and grasped object, providing temporally precise cues for contact onset, sliding, impact, and slip.The audio is sampled as 16 kHz mono PCM, and headphone monitoring supported more consistent demonstrations during contact phases.Source: 2) Modular Multimodal Sensing Finger · p. 3
• The wrist camera records 1920 × 1080 video at 60 fps, while handheld and robot proprioception are expressed in a common end-effector representation despite being estimated differently.Handheld pose comes from ORB-SLAM3 using video and IMU; robot pose comes from joint state, and gripper width is read from the hand or finger markers.Source: 2) Modular Multimodal Sensing Finger · pp. 3–4
• The multimodal sensing finger adds approximately $240 in hardware and four hours of manual assembly relative to the original UMI platform.Source: Table III · p. 4

• Adding the multimodal sensing finger to UMI requires approximately $240 in additional hardware and four hours of assembly; transferring it between gripper embodiments takes approximately 10 minutes.The design uses modular components and minimal software dependencies to reduce setup effort.Source: 3) Manufacturing and Assembly · p. 4

4) Synchronization and Data Processing:

• PolyUMI aligns observations to a common trajectory timeline at 10 Hz, compensates for sensing-to-action latency, and combines recent visual, tactile, audio, and proprioceptive inputs.It retains two image observations and approximately 0.5 seconds of contact audio.Source: 4) Synchronization and Data Processing · p. 4

5) Control and Execution:

• The robot executes 10 Hz gripper-frame pose commands through a 1 kHz Cartesian impedance controller, with the Franka Hand modified for 200 Hz continuous position control.The first actions in each chunk are skipped to compensate for combined latency; the skip count uses online measurement and offline calibration.Source: 5) Control and Execution · p. 4, 4) Synchronization and Data Processing · p. 4

B. VisTA Multimodal Policy

• The fused tokens condition a flow-matching action model that predicts 16 relative end-effector actions, each containing translation, rotation, and gripper-width commands.The policy head uses a DiT-style transformer and conditional flow matching.Source: B. VisTA Multimodal Policy · pp. 4–5
• VisTA encodes camera and tactile images, audio spectrograms, and robot state into modality-specific tokens with spatial and temporal embeddings.Joint self-attention exchanges information across sensors, locations, and observation times; unlike pooled sensor features, this preserves local tactile and short audio cues.Source: B. VisTA Multimodal Policy · p. 4

IV. EXPERIMENTS

• The evaluation asks whether tactile and contact-audio streams expose object properties that are difficult to infer from wrist vision alone.The study uses V for wrist vision, T for optical tactile images, and A for contact audio.Source: §IV · p. 5
• A second evaluation tests whether tactile and contact-audio observations provide precise feedback for closed-loop slip control.Source: §IV · p. 5
• A third evaluation tests whether multimodal observations improve imitation policies on contact-rich manipulation tasks.Source: §IV · p. 5

A. Experimental Procedure

• Across experiments, baselines and modality ablations share demonstrations, train–test splits, horizons, preprocessing, augmentation, and optimization budgets.Models train for 120 epochs; manipulation and control report successful rollouts, while classification reports held-out accuracy.Source: A. Experimental Procedure · p. 5

B. Object Properties and Slip Control

• The object-property and slip-control evaluation first tests whether contact sensing captures task-relevant information before assessing longer-horizon manipulation.Source: B. Object Properties and Slip Control · p. 5

1) Tactile Shape Recognition:

• The shape-recognition task addresses fine surface features that are occluded by the gripper or smaller than an external camera’s effective resolution.It classifies Flat, Square, Circle, Triangle, and the letter N from one tactile reading.Source: 1) Tactile Shape Recognition · p. 5
• A CNN tactile encoder and MLP head predicted class labels from single tactile images, reaching 92.3% accuracy for tactile shape recognition across five classes on 714 samples from seven held-out scenes per class.Training used 1986 samples from 20 scenes per class; no scene appeared in both splits.Source: 1) Tactile Shape Recognition · p. 5, Figure 5 · p. 5

2) Object-in-Box Classification:

• The task places gears, thin screws, or thick screws in the same closed box and shakes it for 20 oscillations along a sinusoidal trajectory.The initial box position varies over a 6 cm range along each of the x-, y-, and z-axes.Source: 2) Object-in-Box Classification · p. 5
• The classifier uses separate CNN encoders for each available modality and an MLP head, with three seeds and 90 total predictions per sensor configuration.Thirty demonstrations were collected per class, with 20 for training and 10 for validation.Source: 2) Object-in-Box Classification · pp. 5–6
• ≈80.0% accuracy across sensor configurations with contact audio contrasts with 42–44% without audio, only modestly above 33% random guessing.The tactile-audio models classify thick screws with 100% accuracy, and the visual-audio model identifies thin screws in 83% of trials.Source: 2) Object-in-Box Classification · p. 6, Figure 6 · p. 6

3) Closed-Loop Slip Control:

• Slippage can be subtle, visually occluded, or too rapid for wrist-camera detection, while tactile deformation and structure-borne vibrations measure relative object–gripper motion.The task starts with a screwdriver held approximately horizontal and targets a final orientation between 0° and 10° from vertical.Source: 3) Closed-Loop Slip Control · p. 6
• On the Slip Control Task, the full V+T+A policy succeeds in 8/10 trials, versus 2/10 for vision-only; contact-based observations provide important feedback for fine-grained gripper–object interaction control.Source: 3) Closed-Loop Slip Control · p. 6

C. Contact-Rich Manipulation

• Board wiping tests sustained contact and surface following, while lightbulb turning combines alignment, insertion, and rotational engagement.All policies receive proprioception alongside the sensor observations.Source: C. Contact-Rich Manipulation · pp. 6–7, Figure 9 · p. 7

1) Baselines and Implementation:

• The comparison includes a vision-only Diffusion Policy and three multimodal baselines that receive the same visual, tactile, auditory, and proprioceptive inputs as VisTA.MulSA uses modality encoders and attention, Sparsh-X uses token stems and attention bottlenecks, and PolyTouch combines pretrained modality representations through cross-attention.Source: 1) Baselines and Implementation · p. 7

2) Board Wiping:

• Board wiping requires contact regulation as well as visual tracking; insufficient contact leaves the line unerased, while deviations produce partial completion.The task uses a slightly randomized line of approximately 25 cm; losing board contact counts as failure.Source: 2) Board Wiping · p. 7
• VisTA establishes and maintains board contact while following the line closely enough to erase it completely, outperforming the evaluated baselines in board wiping.Vision-only Diffusion Policy and MulSA frequently lose contact; Sparsh-X generally maintains contact but often deviates from the line.Source: 2) Board Wiping · p. 7, Figure 10 · p. 7

3) Lightbulb Turning:

• The task varies required bulb rotation and initial gripper height by ±1 cm; success requires the bulb to switch on within 90 s and remain seated after release.The robot starts with a loosely inserted bulb and must maintain a stable grasp and regulate axial position while rotating it.Source: 3) Lightbulb Turning · p. 7
• All methods achieve at least 80% success on lightbulb turning; VisTA closely follows the best-performing vision-only Diffusion Policy and Sparsh-X.Failures occur primarily during transitions between rotation stages; MulSA sometimes loses the bulb while re-grasping, and PolyTouch sometimes retreats before full seating.Source: 3) Lightbulb Turning · pp. 7–8

V. CONCLUSION

• The conclusion reports that VisTA outperforms evaluated baselines on board wiping and matches the strongest multimodal baseline on lightbulb turning.The authors describe the lightbulb task as one in which tactile and auditory sensing offer limited additional benefit because bulb state and completion have clear visual cues.Source: §V · p. 8, 3) Lightbulb Turning · p. 8