Source-linked AI summary
$N_0$-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation
NeoteAI Team, Fudan TEAI Team
TL;DR
Contact-rich manipulation requires touch and anticipation beyond what vision-only policies provide. N0-TWAM jointly predicts future vision, contact, and action, achieving the strongest overall results across simulated and real contact-rich benchmarks.
Problem
Contact-rich manipulation requires tactile information and future-interaction anticipation that vision-only policies do not naturally provide.
Method
N0-TWAM jointly predicts future vision and tactile contact with action using a modality-specific Mixture-of-Transformers world-action model trained on large-scale visuo-tactile demonstrations.
Results
N0-TWAM is strongest overall across simulated and real contact-rich benchmarks, reaching 84.5% on UniVTAC, 49.4% on NeoSim, and 46.3% across eight real-robot tasks.
Takeaways & Limitations
The results support tactile-native world-action modeling as a basis for contact-rich manipulation across simulated and real-robot benchmarks.
Takeaways & Limitations
The authors identify longer prediction horizons and broader tactile-sensor coverage as future directions.
Abstract
from arXiv · showhide
We present $N_0$-TWAM, a tactile-native world-action model for contact-rich manipulation that predicts both future vision and future contact. To our knowledge, it is the first tactile world-action model trained at large scale, and it shows strong capability on contact-rich tasks. We pre-train $N_0$-TWAM at large scale with visuo-tactile joint training over tactile-rich demonstrations spanning six embodiments and 450 tasks. We use NeoForce, a unified force-based tactile representation, to form a physically grounded contact signal that conditions action generation. To improve long-horizon and multi-stage manipulation, we introduce tactile contact events for task staging and advance through them during execution. For real-time efficiency, we adopt an asymmetric Mixture-of-Transformers architecture that pairs a full-width expert for video prediction with slim experts for downstream action and tactile prediction. Evaluations on both real and simulated benchmarks justify the capabilities of $N_0$-TWAM across a range of contact-rich tasks, and demonstrate the benefit of data scaling for precise tactile and action prediction. In summary, $N_0$-TWAM endows a world-action model with predictive capabilities to foresee vision, touch and action, building a solid foundation for fine-grained manipulation on open contact-rich tasks. The codebase and model checkpoints will be made publicly available to foster further research and development in tactile-enabled robotic manipulation.
1 Introduction
N0-TWAM addresses contact-rich manipulation by jointly predicting future vision and touch while using observed tactile signals for action generation. It combines large-scale visuo-tactile pretraining, an asymmetric Mixture-of-Transformers design, tactile event-based staging, and strong simulation and real-robot results.
- Motivation: Contact-rich manipulation depends on fingertip events that cameras often cannot resolve, motivating policies with direct tactile access and predictive modeling.Examples include seating a screw, peeling a cup from a stack, and grasping soft objects.
- Gap: Prior tactile approaches either consume touch without predicting it, bolt on separate tactile predictors, or isolate tactile tokens from the visual stream.These approaches do not predict touch jointly with vision at the same causal step.
- Contributions: N0-TWAM is a native tactile world-action model whose dedicated tactile expert jointly predicts future touch and vision under shared self-attention.It is presented as the first tactile world-action model trained at large scale.
- Results: 84.5% on UniVTAC, 49.4% on NeoSim, and 46.3% average across eight real-robot tasks establish strong performance against 30.0% for the strongest vision-language-action baseline.Ablations attribute gains to both predicted and observed tactile pathways, while analyses report benefits from more pretraining data and robustness under distribution shift.
- Method: N0-TWAM pretrains on tens of thousands of real-robot hours spanning six embodiments and 450 tasks, while its dual tactile pathways generate foresight and condition action through NeoForce.Tactile contact events also segment demonstrations and schedule execution stages, with predicted contact triggering transitions and observed contact confirming them.
2 Model
N0-TWAM is a generative world-action model that predicts the robot’s future visual and tactile experience, then derives actions from that predicted future. Its design jointly models vision and touch, separates modality-specific capacity while sharing attention, and uses touch both as foresight and current observation.
- World-action formulation: N0-TWAM rolls the robot’s future visual and tactile experience forward and reads actions from the predicted future.It replaces direct action regression from observations with a single generative model.
- Multimodal architecture: Vision and touch are modeled jointly, with separate modality-specific experts that share a single self-attention mechanism.Private weights preserve modality-specific capacity while allowing vision and touch to attend fully to one another.
- Tactile roles: Touch is both predicted ahead of time as a foresight target and observed at the current time as an input for action generation.These two tactile roles support the model’s tactile-aware execution design.
2.1 Model architecture
N0-TWAM uses a three-stream visuo-tactile-action architecture in which vision and touch jointly predict the future scene before action denoising. Its asymmetric Mixture-of-Transformers backbone shares attention across modality-specific experts while keeping video full-width and action/tactile experts slim.
- Problem setup: N0-TWAM predicts future video, tactile observations, and actions jointly from language instructions and observation history.Vision and touch are treated as equally important, while actions are conditioned on the just-predicted future video and tactile signals.
- Mixture-of-Transformers backbone: Three modality experts—video, tactile, and action—interact through one shared self-attention at each layer.Video and tactile experts co-generate future scene and contact, and the action expert denoises its chunk conditioned on that predicted pair.
- Parameter-efficient experts: The asymmetric backbone uses a full-width video expert with dv=3072 and slim action and tactile experts with da=dt=1024.The private residual/FFN widths differ by expert, while shared self-attention operates at d=3072; the video expert is warm-started and the slim experts are trained from scratch.
- Diffusion-forcing cascade: A causal attention mask implements the predict-then-act cascade within one forward pass, with noisy video and tactile generation preceding noisy action prediction.Clean history is teacher-forced during training, and each new chunk attends to the full clean history without attending to future chunks.
- Action representation: Each arm contributes a 10-dimensional per-frame action, yielding a 20-dimensional bimanual interface with masked unused dimensions for single-arm embodiments.Position and orientation use chunk-start-anchored deltas, while the gripper command remains absolute.
2.2 Modeling touch
N0-TWAM models touch both as a predicted future in shared video-tactile latent space and as a present force-space signal for action. These pathways support contact foresight, physically grounded observation, and staged training across large-scale data and task-specific behaviors.
- Predicted tactile pathway: Predicted touch is VAE-encoded and patchified like video, entering a shared latent space where vision and tactile tokens are directly comparable.A learned sensor-id embedding supports up to four sensors, while tactile streams retain private weights for tactile-specific statistics.
- Dual tactile roles: Touch serves two roles: N0-TWAM predicts future contact and reads present tactile observations during action.Contact states such as secure grasp, incipient slip, and excessive force can determine the next action while remaining invisible to scene cameras.
- Predicted tactile pathway: The tactile expert predicts future contact as residual changes from the initial frame, emphasizing informative variations at contact onset and release.Predicted tactile and scene latents are denoised together, allowing each to condition the other while preserving causal history.
- Observed tactile pathway: Observed touch is converted from raw tactile images into a dense three-axis force map, encoded by NeoForce, and cross-attended into the action stream.The force map contains [fx, fy, fz] and a contact mask; the observed encoder is instantiated per domain.
- Staged training: Pre-training uses only predicted touch to learn contact dynamics jointly with future scenes, while the observed pathway is introduced separately for action conditioning.The force estimator remains frozen, whereas the NeoForce observed encoder is warm-started and fine-tuned with the action loss.
2.3 Tactile-aware execution
N0-TWAM performs tactile-aware autoregressive execution by predicting video and tactile signals before conditioning action generation, while grounding its rolling context in incoming visual and tactile observations. For long-horizon tasks, tactile contact events segment demonstrations and drive inference-time sub-task advancement.
- Tactile-aware execution: N0-TWAM autoregressively denoises future video and tactile latents, conditions actions on those predictions, then decodes and executes each action chunk.The rollout uses a rolling key/value cache and asynchronous prediction pipeline.
- Tactile-aware execution: Real frames and tactile readings replace corresponding predictions in frame-id causal order, keeping the rolling context anchored to what the robot actually saw and felt.The observed tactile pathway operates outside the denoising loop.
- Tactile-aware execution: The model supports real-time execution at 7B parameters because predicted contact is generated during scene prediction while observed tactile features use lightweight sensor-rate cross-attention outside denoising.The cascade’s one-way dependencies also leave video and tactile keys and values unchanged after their prediction phase.
- Long-horizon execution: Tactile punctuation segments long-horizon demonstrations into sub-task clips by detecting contact-onset and release events, with gripper aperture as a fallback.A tactile rise confirms that a grasp made contact, distinguishing successful grasps from aperture-only failed closures.
- Long-horizon execution: At inference, a lightweight scheduler maintains a sub-task queue and advances to the next prompt when the current sub-task completes using predicted and observed tactile events.The predicted future tactile signal triggers advancement one step early, followed by confirmation from the observed release or contact-onset event.
3 Data
N0-TWAM uses NeoData’s large-scale, tactile-rich manipulation corpus and converts multimodal episodes into cached latent inputs, overlapping temporal windows, unified actions, and contact-event stages. These preprocessing choices support training across heterogeneous robots and long-horizon contact-rich tasks.
- Dataset: NeoData provides over 30,000 hours across six embodiments and 450 contact-rich tasks, with many episodes containing synchronized per-finger tactile and multi-view RGB.Its tactile coverage enables learning to predict contact rather than only appearance.
- Latent encoding: Camera and tactile streams are encoded as latents by one frozen causal video VAE, while language uses a frozen umT5 encoder; all inputs are cached before training.Tactile streams are treated as small videos so touch shares vision’s latent space, and encoders do not run during training forward passes.
- Chunking: Episodes are segmented into 33-latent-frame windows spanning 387 raw frames, with 24-frame strides and approximately 27% overlap between consecutive windows.Raw frames are sampled at 10 fps from 30 fps and compressed through 4× temporal downsampling; tail windows are right-aligned.
- Unified action space: Actions from every embodiment are canonicalized into a 20-dimensional end-effector space, with 10 dimensions per arm, chunk-anchored pose deltas, absolute grippers, and per-robot normalization.Quantile clipping is applied to the heavy-tailed gripper channel.
- Contact-event stage segmentation: Stage boundaries are identified primarily from sharp tactile changes, using gripper aperture for ambiguous cues and a final human check to correct remaining boundaries.The resulting labels and inference-time scheduling are keyed to meaningful contact events such as contact, release, slip, and insertion seating.
4 Training and Experiments
Section 4 describes large-scale visuo-tactile pre-training, evaluation across simulated and real contact-rich manipulation suites, and ablations showing that tactile foresight, observed tactile conditioning, and NeoForce improve performance. N0-TWAM achieves its strongest advantages on contact-dependent and visually perturbed tasks, while remaining competitive on placement-dominated tasks.
- Training: N0-TWAM is pre-trained on tens of thousands of manipulation hours spanning six robot embodiments, with video, touch, and action trained using equal 1:1:1 modality weights.Training uses future tactile as a foresight target while the observed tactile pathway is disabled during pre-training.
- Benchmarks and metric: The evaluation covers eight UniVTAC tasks, twelve NeoSim tasks, and eight real-robot contact-rich tasks, reporting task success rate and macro-average success.The real-robot suite contains four dual-arm PiPER and four single-arm Flexiv tasks.
- Simulation results: 84.5% average success makes N0-TWAM the top method, roughly 17 points above InternVLA-A1 at 67.1%, while vision-only world-action baselines perform substantially worse.The reported vision-only baselines are LingBot-VA at 31.4, FastWAM at 48.0, and GigaWorld-Policy at 16.5.
- Simulation results: 49.4 average success makes N0-TWAM best on NeoSim, ahead of π0.5 at 45.8, with especially clear leads on contact-heavy stacking and insertion tasks.Simulated tactile differs from the sensor distribution used for NeoForce pre-training, so the observed pathway uses a lightweight simulation encoder.
- Robustness: 45 success under visual perturbation makes N0-TWAM more robust than π0.5 at 25 and LingBot-VA at 30, while its overall average reaches 51.7.On unseen objects, π0.5 leads at 80, whereas on unseen positions N0-TWAM matches π0.5 at 45.
5 Related Work
The section situates N_0-TWAM among vision-language-action models, tactile perception systems, and robotic world models. It distinguishes N_0-TWAM by cascading predicted touch into action generation and using asymmetric expert widths for efficient inference.
- Vision-language-action models: Vision-language-action models generate manipulation actions conditionally from observations and language, with large-scale pretraining transferring semantic knowledge into robot control.Recent systems add real-time asynchronous execution, larger cross-embodiment corpora, embodied backbones, and unified policies.
- Tactile perception for manipulation: Tactile-aware systems align touch with language and action, inject force feedback, specialize in contact-heavy tasks, or use touch-derived feasibility signals.Force-aware policies also combine RGB with high-rate force/torque streams, though the supplied passage ends before describing further details.
- World models for robotics: Robotic world models predict environment states conditioned on text, driving commands, or robot actions, while world foundation models generate future observations from large video corpora.N_0-TWAM extends this space with tactile prediction alongside video and action prediction.
- World models for robotics: N_0-TWAM cascades video, tactile, and action experts so predicted touch becomes an intermediate consumed by the action expert, rather than a side signal or attention bias.It retains full pretrained capacity in the visual expert while training slimmer tactile and action experts; this asymmetry keeps streaming inference cheap.
6 Conclusion
N0-TWAM integrates tactile foresight and observations into a video world-action model that predicts near-future scenes and contact before generating actions. The conclusion identifies faster inference, longer prediction horizons, and broader tactile-sensor coverage as future directions.
- Conclusion: N0-TWAM brings touch into the predicted future of a video world-action model, predicting near-future scenes and contact before reading actions from both.Tactile serves as both a denoised foresight target and an observed reading.
- Conclusion: A shared self-attention mechanism ties three per-modality experts together in a frame-id causal cascade.The asymmetric design keeps the model at 7.2B and supports cheap streaming inference.
- Future work: Future work targets faster inference so the policy can operate at higher control rates.Streaming decoding of the one-directional cascade is identified as the acceleration opportunity.
- Future work: Extending the vision and tactile prediction window could let the model anticipate contact events further ahead.This would lengthen the predicted horizon.
- Future work: Training on more types and varieties of tactile sensors could broaden transfer across embodiments.The stated goal is broader sensor coverage for the learned tactile representation.
A Per-task ablation results
The section reports per-task ablation success rates for N0-TWAM on UniVTAC and NeoSim. UniVTAC covers eight tasks, while NeoSim covers twelve tasks across single-arm and dual-arm settings.
- UniVTAC ablations: Table 7 reports UniVTAC success rates for the full N0-TWAM model and three ablations across eight tasks.These per-task results provide the task-level values underlying Figure 9’s averages.
- NeoSim ablations: Table 8 reports NeoSim success rates for N0-TWAM and its ablations across twelve tasks.The NeoSim suite contains four single-arm and eight dual-arm tasks.
- Evaluation scope: The 20% pre-training variant was evaluated on UniVTAC only.This condition is specified as a scope limitation for the per-task ablation results.
B Per-task tactile-realism results
Per-task evaluations show that gel-rendered tactile appearance causes little in-domain accuracy loss, while NeoForce processing further improves performance, especially when tasks provide weight-bearing or shear cues.
- Overall comparison: Gel-rendered results remain close to clean-field performance on most tasks, indicating little in-domain accuracy cost from realistic tactile appearance.The comparison covers UniVTAC and NeoSim per-task success rates.
- NeoForce gains: 82→96: Put Bottle in Shelf success improves when the gel-rendered image is read through NeoForce.The passage identifies this task as having a clear weight-bearing or shear cue.
- Evaluation scope: Per-task success is reported across eight UniVTAC tasks and twelve NeoSim tasks, including four single-arm and eight dual-arm tasks.Tables 9 and 10 compare clean-field, gel-rendered, and NeoForce-encoded gel-rendered conditions.
C Per-modality training dynamics
Per-modality pre-training losses initially decrease and stabilize, but mixing hand-collected UMI data affects the streams differently: video loss increases and converges higher, while tactile loss remains essentially unchanged.
- Data recipes: The study compares real-robot teleoperation-only pre-training with a recipe that mixes UMI data before returning to pure teleoperation data.The mixed recipe uses hand-collected UMI data, with the full run lasting 30,000 steps.
- Pre-marker dynamics: Before the +UMI marker, video, action, and tactile flow-matching losses all decrease and stabilize.The MoT backbone logs separate losses for its video, action, and tactile experts.
- Effect of UMI mixing: After UMI data enters about 60% of the batch, the video loss rises and converges higher, whereas the tactile loss is essentially unchanged.Figure 12 tracks the first 18k of the 30,000-step run.