PaperlayerView exact revision on arXiv

Source-linked AI summary

$N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens

NeoteAI Team, Fudan TEAI Team

arXiv:2607.23782v1cs.RO

TL;DR

Contact-rich manipulation lacks large-scale tactile pretraining and effective tactile feedback. N0-VTLA addresses this with latent tactile prediction and ALTER, outperforming baselines across benchmarks and achieving the highest success on three tasks.

  • Problem

    Existing VLA policies largely lack tactile integration, while prior tactile extensions use only task-scale collections, limiting evidence for large-scale tactile pretraining.

  • Method

    N0-VTLA combines large-scale visuo-tactile pretraining with a predictive latent tactile pathway and ALTER, an advantage-conditioned offline reinforcement learning method.

  • Results

    N0-VTLA leads strong baselines across NeoReal and simulation benchmarks, while N0-VTLA+ALTER achieves the highest success on all three evaluated tasks.

  • Takeaways & Limitations

    The results support predictive latent touch as a promising approach for tactile-driven contact-rich manipulation and offline policy improvement.

  • Takeaways & Limitations

    ALTER’s effectiveness remains to be studied across a broader range of manipulation tasks and settings.

Abstract

from arXiv · show

We present $N_0$-VTLA, a vision-tactile-language-action (VTLA) foundation model capable of (1) fine-grained contact-rich manipulation with tactile perception and tactile-feedback control, and (2) offline policy improvement from stored deployment data. Building on current vision-based backbones, we propose a training recipe for tactile integration consisting of visuo-tactile pre-training, staged tactile-pathway integration, and advantage-conditioned offline policy improvement. During pre-training, the policy learns broad contact priors from NeoData, our large-scale visuo-tactile robot dataset; to our knowledge, $N_0$-VTLA is the first VTLA model pretrained on tactile data at scale. During post-training, we augment the policy with a predictive tactile pathway that distills the contact patterns learned at scale into the fine motion adjustments required by downstream tactile-centric manipulation. For offline policy improvement, we introduce ALTER, an advantage-conditioned offline reinforcement learning method that converts relative progress and trajectory-event comparisons into binary advantage labels for policy training on a fixed deployment corpus, further improving task-specific learning on contact-rich skills such as deformable object manipulation. Across contact-rich benchmarks, $N_0$-VTLA outperforms strong baselines by wide margins: it wins all nine real-robot NeoReal tasks and reaches 63.8% mean success on a twenty-task simulation suite, against 44.0% for the strongest baseline. $N_0$-VTLA policies trained with ALTER reach 75-95% success on three long-horizon real-robot tasks. These results lay a foundation for versatile tactile-driven manipulation policies.

1 Introduction

N0-VTLA addresses the lack of scalable tactile learning in contact-rich manipulation with latent tactile prediction, large-scale visuo-tactile pretraining, and ALTER for offline policy improvement. Its latent pathway retrieves future tactile targets at 92.3% top-1 accuracy versus 3.2% chance, while N0-VTLA+ALTER achieves the highest success on all three tasks.

  • Motivation: Current tactile extensions train on at most tens of hours from handfuls of skills, limiting progress in contact-rich manipulation.This contrasts with vision–language–action models’ broader instruction following and transfer across tasks, scenes, and embodiments.
  • Latent tactile tokens: N0-VTLA predicts latent tactile tokens for the net tactile change over the coming action chunk and conditions the action expert on them.Tactile difference images are encoded through a frozen pretrained visual encoder and lightweight trainable projection, with a three-stage integration recipe.
  • Offline policy improvement with ALTER: ALTER uses pairwise progress modeling and tactile-grounded annotations, object-drop events, and logged human corrections to produce stage-relative advantage labels for offline policy learning.N0-VTLA+ALTER achieves the highest success on all three tasks, and the method also applies to the base VLA policy.
  • Large-scale tactile pretraining: N0-VTLA is pretrained on NeoData, a large-scale multi-platform visuo-tactile robot corpus unified by a canonical cross-embodiment action space.The data span single- and dual-arm configurations and use a quality-verified pipeline.
  • Latent pathway verification: 92.3% top-1 accuracy retrieves matching future-tactile targets after Stage 1, compared with 3.2% chance.This experiment verifies the latent pathway before evaluating the full system.

2 Model

N0-VTLA combines a PaliGemma-based vision–language–action backbone and flow-matching action expert with a latent tactile pathway for contact-rich manipulation. The pathway is trained through future-tactile prediction, masked tactile-conditioned action learning, and end-to-end policy training.

  • Architecture: N0-VTLA reads camera views, language, robot state, and touch, then generates a future action chunk through a latent tactile pathway.Its backbone uses PaliGemma, while a flow-matching action expert produces the actions.
  • Training recipe: The three-stage recipe trains a tactile predictor, aligns the action expert using latent tactile tokens under a masked vision–language pathway, then trains the full policy end to end.The deployed controller is the model produced after Step 3.
  • Latent tactile pathway: The predictor converts current tactile-difference tokens and contextualized vision–language features into 10 latent tactile tokens representing future contact.The future target averages tactile-encoder outputs over tactile changes across the next H steps and active views; supervision combines symmetric InfoNCE and auxiliary L1 reconstruction.
  • Latent tactile pathway: Current-contact tokens never enter the action expert directly; instead, predicted latent tokens are projected to the expert width and prepended before noisy action tokens.The expert attends to the latent tokens and the vision–language prefix during action generation.
  • Design rationale: 92.3% predictor retrieval accuracy exceeds 57% for current-tactile encoding alone when ranking future-tactile targets.The passage states that the margin widens as the candidate pool grows.

3 Data

N0-VTLA is pretrained on NeoData, a curated multi-platform visuo-tactile corpus spanning multiple robot embodiments and handheld collection. The data pipeline standardizes actions, states, tactile streams, and simulated episodes while verifying repository quality before training.

  • Corpus: NeoData spans single- and dual-arm manipulators plus a UMI-style handheld collection gripper, with tactile images captured by sensors on participating gripper fingers.Contact is represented as tactile-image streams rather than low-dimensional force signals.
  • Canonical representation: All embodiments use a fixed 32-dimensional state-and-action container inherited from π0.5, with two 10-dimensional arm slots and 12 unused zero dimensions.Each arm slot contains end-effector position, rot6d rotation, and a gripper channel.
  • Tactile collection: Each participating gripper finger contributes a tactile stream synchronized with RGB and proprioception, referenced to a zero-contact baseline lasting at least 0.5 s.The gripper is open and static during the baseline, and subsequent tactile frames are interpreted relative to it.
  • Data quality: Converted repositories undergo completeness, schema-conformance, and statistics-and-media checks, with failed repositories repaired or excluded before training.The verification process applies the invariants required by the training pipeline.
  • Simulated data: UniVTAC simulator episodes enter through the same canonical schema and verification process as real data, enabling one policy interface for both.These simulated episodes supply the NeoSim suite evaluated in Section 5.3.

4 Training

N0-VTLA reaches deployment through staged visuo-tactile pretraining, tactile-pathway integration, task-specific supervised fine-tuning, and optional offline policy improvement. The staged curriculum grounds latent contact representations and aligns them with action control without destabilizing the pretrained policy.

  • Training recipe: The training recipe grows the trainable surface only after each new interface is grounded, progressing from pretraining to three-stage tactile integration, supervised specialization, and optional deployment-data post-training.Each stage starts from the checkpoint produced by the previous stage, enabling the tactile pathway to come online without destabilizing the pretrained policy.
  • Tactile-pathway integration: Stage 1 trains a predictor, tactile projection, and reconstruction head against future tactile latents using symmetric InfoNCE and an L1 reconstruction objective.The contrastive term retrieves the correct future contact, while reconstruction anchors the latent to the spatial layout of contact changes.
  • Tactile-pathway integration: Stage 2 aligns latent tactile tokens with the action expert under the base action objective, then Stage 3 jointly trains the policy while keeping the tactile encoder backbone frozen.Stage 2 masks the vision-language prefix from action queries; Stage 3 removes that mask and restores the direct prefix-to-expert path.
  • Task adaptation: Downstream policies are adapted by supervised fine-tuning from the pretrained checkpoint using a few hundred demonstrations, with normalization statistics recomputed on each task’s data.The same reduced-scale recipe is used for real-robot tasks and the simulated task suite, with one policy per task.
  • Offline policy improvement: ALTER performs advantage-conditioned offline reinforcement learning on a fixed deployment corpus by converting dense progress and sparse trajectory-event comparisons into binary labels for policy training.It uses tactile contact changes alongside kinematic, gripper, and visual cues to annotate progress and preferences, then ranks samples by local execution change within predicted stages.

5 Experiments

Experiments evaluate N0-VTLA on real-robot and simulation benchmarks, offline policy improvement, and learned tactile representations. N0-VTLA outperforms baselines across contact-rich manipulation, while ALTER further improves task-specific policies.

  • Evaluation protocol: Evaluation uses rollout success rate, supplemented by a 100-point progress score for intermediate subtask checkpoints.NeoReal reports both metrics, while NeoSim reports success percentage under fixed task-completion criteria.
  • NeoReal benchmark: 47.2% mean NeoReal success and 56.8 progress points let N0-VTLA surpass π0.5 at 29.4% and 42.3 points.N0-VTLA beats the strongest baseline on every task in success rate and leads the progress-score mean.
  • Simulation benchmarks: 63.8% mean success across twenty simulation tasks exceeds π0.5 at 44.0%, with N0-VTLA averaging 83.1% on UniVTAC and 50.8% on NeoSim.On UniVTAC, N0-VTLA exceeds InternVLA-A1 at 67.1%; on NeoSim, it exceeds π0.5 at 45.8%.
  • Offline policy improvement: ALTER yields N0-VTLA success rates of 95%, 80%, and 75% across Towel Folding, Bag Packing, and Cardboard Box Folding.The corresponding π0.5+ALTER rates are 90%, 75%, and 60%, respectively.
  • Representation analysis: 92.3% of held-out queries retrieve matching future-tactile targets top-1, versus 3.2% chance and a 57% current-touch baseline.The representation remains tactile-leaning after joint training, and perturbation tests argue against a vision–language shortcut.

6 Findings

The findings show that tactile input improves contact-rich manipulation by enabling contact-aware retries, fine gripper-force control, and action changes concentrated at contact-critical moments. Stronger tactile pretraining also retains its advantage under ALTER, yielding higher success across three folding and packing tasks.

  • Contact-aware insertion: 85% on Socket Plugging versus 60% for π0.5, while Board Insertion reaches 25% against 0% for both ACT and π0.5.Touch turns insertion into a contact-aware retry loop rather than a one-shot visual commitment.
  • Fine gripper-force control: N0-VTLA stabilizes the compliant bottle upright by continuously adjusting gripper aperture instead of gripping hard enough to lift it.The tactile sensor’s deformation signal supports fine control even though it does not measure force directly.
  • Tactile pathway mechanism: With touch, the predicted action path diverges from the touch-removed counterpart at firm contact and grasp but coincides with it in free space.Holding the observation and sampling noise fixed isolates the tactile pathway’s contribution to the sampled action.
  • Offline policy improvement: 95%, 80%, and 75% are N0-VTLA’s ALTER results on Towel Folding, Bag Packing, and Cardboard Box Folding, versus 90%, 75%, and 60% for π0.5.Under clean-demonstration SFT, N0-VTLA leads π0.5 by 10, 15, and 15 points on the same tasks.

7 Related Work

Related work spans scalable vision–language–action policies, tactile-conditioned and predictive VTLA models, and offline policy improvement using deployment data. These efforts advance action generation, transfer across tasks and embodiments, tactile integration, and progress- or value-guided learning.

  • Vision–language–action policies: ACT and Diffusion Policy established action-chunk and generative visuomotor architectures, while RT-1 and RT-2 advanced scalable transformer control and vision–language transfer.ACT predicts temporally coherent action chunks; Diffusion Policy models multimodal action sequences through iterative denoising.
  • Vision–language–action policies: UniVLA and Qwen-VLA extend transfer across tasks, environments, and embodiments, while StarVLA, MiMo-Embodied, and Pragmatic VLA emphasize modularity, cross-embodiment scaling, and deployment.Xiaomi-Robotics-1 further pushes foundation-scale pretraining with more than 100,000 hours of real-world trajectories.
  • Tactile-conditioned and predictive VTLA policies: Tactile-conditioned VTLA research combines transferable tactile representations and scalable data collection with policies that react to measured touch or predict contact evolution.Touch and Go and Visuo-Tactile Transformers learn joint visual–tactile features, while VITaL, Sparsh, AnyTouch, UniTouch, and FTP-1 emphasize transfer or scaling.
  • Offline RL and VLA policy improvement from deployment data: Offline policy-improvement methods use demonstrations, autonomous experience, interventions, value guidance, or trajectory-level reinforcement learning to address deployment shifts and long-tail failures.Progress and value estimates also support demonstration filtering, flow-matching reweighting, action guidance, and offline-to-online adaptation.

8 Conclusion

N0-VTLA treats touch as a prediction target: latent tactile tokens predict contact and condition action generation through a compact latent representation. The conclusion identifies broader predictive-latent tactile representations and extensions of ALTER as future directions.

  • Conclusion: N0-VTLA predicts contact outcomes with latent tactile tokens rather than reacting directly to touch.A visual backbone and trainable projection convert contact-difference images into tokens, which a lightweight predictor compresses into a latent z.
  • Conclusion: The latent z conditions a flow-matching action expert directly without entering the vision–language prefix.This design treats touch as a prediction target rather than raw sensory input to the vision–language context.
  • Future work: Future work proposes exploring predictive-latent tactile representations beyond the specific predictor architecture used here.The framing conditions action generation on a compact estimate of net contact change over the chunk horizon rather than raw sensory tokens.
  • Future work: A second future direction is extending ALTER beyond its current formulation.The supplied passage identifies ALTER extension as a direct direction for future work but does not specify the extension.

A Data Card · B Compute Infrastructure

NeoData is presented as a released corpus with a dedicated data card, while N0-VTLA training uses distributed cluster-scale infrastructure and smaller-scale runs for downstream jobs. Infrastructure reporting remains general and omits detailed hardware and throughput specifications.

  • A Data Card: The NeoData data card covers collection protocols.These protocols are documented in the companion data technical report.
  • A Data Card: The NeoData data card covers sensor specifications.The companion data technical report documents these specifications in full.
  • A Data Card: The NeoData data card covers per-repository provenance.Per-repository provenance is documented in the companion data technical report.
  • A Data Card: NeoData is released as a corpus with a dedicated data card.Table 3 summarizes the corpus at data-card level.
  • B Compute Infrastructure: N0-VTLA training uses multi-node distributed training at cluster scale on modern accelerators.This describes the infrastructure used for the model’s training.
  • B Compute Infrastructure: Simulation post-training and smaller fine-tuning jobs run at correspondingly smaller scale on the same class of hardware.The hardware class remains modern accelerators, while job scale is reduced.
  • B Compute Infrastructure: The infrastructure description omits hardware models, device counts, memory footprints, and throughput figures.The paper explicitly reports the infrastructure only at a general level.

C Data-Engineering Pitfalls · D Deployment Protocol

Before training, converted repositories are checked against data-quality invariants whose violations map to specific symptoms. At deployment, the policy uses full 50-step action chunks, resets tactile baselines per episode, and serves predictions through a message-based interface.

  • C Data-Engineering Pitfalls: Every converted repository is checked against data-quality invariants before entering training.The invariants pair each red line with the symptom produced when it is violated.
  • C Data-Engineering Pitfalls: Table 4 summarizes the enforced data-engineering invariants and their violation symptoms.
  • D Deployment Protocol: 50-step action chunks are emitted from one prefix encoding and one ten-Euler-step denoising solve per policy request.The horizon H = 50 is the full training horizon.
  • D Deployment Protocol: The controller must execute every step of each returned chunk before requesting the next prediction.Re-predicting after fewer than 50 steps violates the deployment contract.
  • D Deployment Protocol: At episode start, reset clears the tactile baseline and the first post-reset observation becomes the zero-contact reference for each active tactile view.The resulting tactile difference at that instant is exactly zero.
  • D Deployment Protocol: A message-based interface accepts reset and predict requests and returns action chunks with timing information.Predict requests include camera images, active tactile views, robot state, and an optional language instruction.

E Per-Task Results and Scoring Rubrics

This section defines reproducible point-based progress scoring for rubric-scored NeoReal tasks and documents representative execution rollouts. Each task uses checkpoint-based scoring, with stage rubrics distributing 100 points across fixed subtask sequences.

  • Scoring Rubrics: Progress scores sum credit across fixed subtask checkpoints, replacing subjective judgment with reproducible point-based rubrics.The rubrics define the progress score for Section 5.1 on rubric-scored NeoReal tasks.
  • Scoring Rubrics: 100 points are distributed across checkpoints for every NeoReal task, with two representative long-horizon sequences listed in Table 5.Per-checkpoint weights are documented in the companion data report.
  • Representative Rollouts: Figure 11 shows keyframe strips from N0-VTLA executions on three NeoReal tasks and one NeoSim task.The figure complements aggregate scores with representative rollouts and is referenced from Section 5.2.

F ALTER Offline Policy-Learning Details

ALTER’s offline-policy details specify the deployment-corpus composition, task-specific progress stages, event-supervision and temporal-smoothing procedures, and progress-model behavior under degradation and recovery.

  • Deployment-corpus composition: Towel Folding, Bag Packing, and Cardboard Box Folding use 351/286/180, 213/243/238, and 550/587/397 clean, autonomous, and HIL episodes, respectively.HIL totals include separately collected staged-recovery episodes beginning from selected error states.
  • Representative stage annotations: Towel Folding and Bag Packing use L1 stages, whereas Cardboard Box Folding uses L2 stages because its L1 decomposition is overly fine-grained.These stages construct clean-demonstration progress targets.
  • Implementation details: Event supervision uses margin m = 0.02, with event pairs comprising approximately 5% of sampled pairs and receiving a 0.1 per-sample loss multiplier.A five-frame median filter and non-decreasing stage-index enforcement reduce short-term prediction noise.
  • Progress trajectories on additional tasks: On both additional tasks, predicted progress decreases when execution degrades and rises again after the robot returns to a productive execution state.Figure 14 evaluates one held-out episode per task using the pairwise progress model.
Loading 2607.23782v1…