PaperlayerView exact revision on arXiv

Source-linked AI summary

PINT: A Unified Model with Shared Predictive Intent for Robot Manipulation

Xinyu Guo, Chao Jing, Li Yi, Xingyu Chen, Zhengxing Wu

arXiv:2610.16355v1cs.RO

TL;DR

Pretrained VLMs support semantic reasoning and VGMs provide physical-dynamics priors, but combining their specialized capabilities remains a challenge. PINT coordinates them through a shared chunk-level Predictive Intent while preserving expert-specific pathways. Without embodied pretraining, it reports improved OOD performance across simulation and real-world manipulation.

  • Problem

    VLMs provide semantic reasoning and VGMs physical-dynamics priors, but effectively combining their specialized capabilities for robot manipulation remains an open challenge.

    Source: §1 · p. 2

  • Method

    PINT uses learnable Intent Tokens in the VLM sequence to form a shared chunk-level Predictive Intent that conditions the VGM and Action Expert while preserving expert-specific pathways.

    Source: §1 · p. 2

  • Results

    15.5% success on RoboTwin 2.0-Hard vs. 1.8% for Fast-WAM and 3.2% for StarVLA; PINT also reports improved OOD generalization across simulation and real-world manipulation without embodied pretraining.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Across simulation and real-world manipulation, the results suggest that a compact shared predictive interface can coordinate specialized pretrained models for robot manipulation.

    Source: §5 · p. 10

    Evaluation focuses on robot manipulation and a single VLM–VGM backbone pairing; embodied pretraining, broader embodied settings, and larger-scale training remain future work.

    Source: §5 · p. 10

Abstract

from arXiv · show

Generalist robot manipulation can benefit from pretrained vision-language models (VLMs) for semantic reasoning and video generation models (VGMs) for world modeling. However, existing unified models that directly fuse them through shared attention at multiple layers can make it harder to draw on their specialized capabilities. We introduce PINT, a unified model that coordinates semantic reasoning, world modeling, and action generation through a shared chunk-level Predictive Intent while preserving expert-specific pathways. PINT inserts learnable Intent Tokens into the VLM sequence to form this representation, conditioning both the VGM and an Action Expert (AE). It replaces only the VGM's prompt conditioning and retains the VGM at inference, where the AE reads its representations through one-way attention. Rather than prescribing what Predictive Intent should encode, PINT learns it jointly from the native prediction objectives of the VLM, VGM, and AE, which provide complementary views of the same upcoming task execution. Training uses two stages: Stage 1 freezes the VGM so that future video prediction shapes Predictive Intent with task-relevant future dynamics, while Stage 2 fixes the learned Intent and adapts the VGM and AE. Without embodied pretraining, PINT achieves strong in-distribution (ID) performance and substantially improves out-of-distribution (OOD) generalization, reaching 15.5% on RoboTwin 2.0-Hard compared with 1.8% for Fast-WAM and 3.2% for StarVLA, and 79.5% on LIBERO-Plus compared with 51.5% and 75.0%, respectively. Across five real-world bimanual tasks, PINT reaches 68% ID and 24% OOD success, compared with 35%/9% for pretrained Motus and 67%/35% for pretrained $\pi_{0.5}$.

1 Introduction

PINT coordinates semantic reasoning, world modeling, and action generation through a shared chunk-level Predictive Intent while retaining expert-specific pathways. Without embodied pretraining, it reports strong in-distribution performance and improved out-of-distribution generalization across simulation and real-world manipulation.

1 Introduction

  • The motivating integration challenge is combining VLM semantic reasoning with VGM physical-dynamics priors without making it harder to use either expert’s specialized capabilities.Standard VLAs do not explicitly model future physical dynamics, while WAMs typically do not exploit VLM semantic reasoning.Source: §1 · p. 2
  • 15.5% success on RoboTwin 2.0-Hard vs. 1.8% for Fast-WAM and 3.2% for StarVLA; LIBERO-Plus: 79.5% vs. 51.5% and 75.0%, respectively.Across five real-world bimanual tasks, PINT reaches 68% ID and 24% OOD success, compared with 35%/9% for pretrained Motus and 67%/35% for pretrained π0.5.Source: §1 · p. 2
  • PINT introduces Predictive Intent, a compact chunk-level representation learned jointly from the VLM, VGM, and AE native prediction objectives without prescribing its content.It serves as a shared interface across specialized experts.Source: §1 · p. 2
  • Learnable Intent Tokens form Predictive Intent in the VLM sequence and condition both the VGM and Action Expert, while replacing only the VGM’s prompt conditioning.The VGM remains active at inference, and the AE reads its representations through one-way attention.Source: §1 · p. 3

2 Related Work

Prior work develops VLA and WAM approaches that connect semantic understanding, future visual prediction, and action generation. Unified models and compact future-oriented representations offer different ways to coordinate these capabilities.

2.1 Vision-Language-Action and World-Action Models

  • VLA research adapts pretrained VLMs for action generation, while WAMs more directly couple future visual prediction with action generation using video generative pretraining.Examples include RT-2 and OpenVLA for VLA, and GR-1, GR-2, WorldVLA, DreamZero, and Fast-WAM for future visual prediction and action.Source: §2.1 · p. 3

2.2 Unified Semantic, World, and Action Modeling

  • Unified policies coordinate semantic reasoning, world modeling, and action generation through shared or interacting components, including multi-expert interaction and unified generative backbones.Examples include Motus, MotuBrain, LingBot-VA, LingBot-VA 2.0, and Cosmos 3.Source: §2.2 · p. 3
  • Compact future-oriented representations route future information to video prediction and action, with prior approaches varying in where they form intent and whether video remains at inference.Unlike World Tokens and InternVLA-A1.5, PINT forms Predictive Intent within the VLM sequence; it keeps the VGM active at inference and supplies its representations to the AE through one-way attention.Source: §2.2 · p. 3

3 Method

PINT unifies future-video prediction and action generation around a shared Predictive Intent derived from multimodal context, while retaining expert-specific pathways. Its method defines this representation, conditions the VGM and Action Expert with it, and learns it through a two-stage training schedule.

3.1 Problem Formulation

  • A policy maps multi-view images, a task instruction, and proprioceptive state to future video and a short-horizon action chunk, with discrete action-token prediction used only during training.The video and action horizons specify the respective prediction lengths.Source: §3.1 · pp. 3–4

3.2 Overview

  • Learnable Intent Tokens at the end of the VLM prefix form Predictive Intent, conditioning future-video prediction and action generation while the AE also uses state and current-frame VGM features.The VLM receives an autoregressive action-token suffix during training; PINT is trained first with a frozen VGM, then with Intent fixed while adapting the VGM and AE.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Predictive Intent

  • Predictive Intent consists of the final-layer hidden states at the Intent Token positions and depends only on the multimodal prefix.Native next-token prediction over a discrete action-token suffix joins video and action objectives in shaping the shared representation.Source: §3.3 · pp. 4–5

3.4 Intent-Conditioned Future Video Prediction

  • The VGM replaces its original prompt conditioning with Predictive Intent, using the current frame and that representation to predict future video while preserving its pretrained generative mechanism.During Stage 1, the frozen VGM’s video objective backpropagates through the Intent pathway; at inference, the default configuration caches current-frame VGM features for AE denoising without future-video denoising.Source: §3.4 · p. 5

3.5 Intent-Conditioned Action Generation

  • The AE combines Predictive Intent and proprioceptive state with layer-wise current-frame VGM key/value features, attending one-way to the VGM representations.During Stage 1, the action objective supplies continuous-control supervision to Predictive Intent alongside autoregressive and future-video objectives.Source: §3.5 · pp. 5–6

3.6 Two-Stage Training

  • Stage 1 freezes the VGM while optimizing the VLM, Intent Tokens, and AE so future-video, autoregressive, and action supervision shape Predictive Intent; Stage 2 fixes Intent and adapts the VGM and AE.Stage 2 optimizes video and action objectives to refine generation conditioned on the fixed representation.Source: §3.6 · p. 6

4 Experiments

Experiments assess PINT on simulation benchmarks and real-world manipulation, then examine design ablations and whether Predictive Intent reflects behavioral structure. PINT performs strongly across ID and OOD evaluations, while the ablations and visualization probe its components and representation.

4.1 Experimental Setup

  • Evaluation covers four simulation benchmarks and five real-world tasks; LIBERO policies transfer to LIBERO-Plus, RoboTwin training uses 50 clean tasks, and real-world OOD jointly varies lighting, background, and object location.Methods are grouped by embodied pretraining, with non-pretrained methods as the primary comparison and pretrained methods as references.Source: §4.1 · p. 7

4.2 Main Results

  • 15.5% Hard success lets PINT exceed Fast-WAM’s 1.8%, despite lower Easy success (68.6% vs. 72.4%), and lifts average success to 42.1% vs. 37.1%.PINT-VA further improves Easy, Hard, and average success to 71.9%, 18.1%, and 45.0%, respectively.Source: Table 2 · p. 8, §4.2 · p. 8
  • 79.5% success on LIBERO-Plus exceeds Fast-WAM (51.5%), StarVLA-OFT (75.0%), and ResVLA (75.3%); PINT also reaches 98.0% on LIBERO without embodied pretraining.PINT-VA further raises the averages to 98.2% on LIBERO and 81.0% on LIBERO-Plus.Source: §4.2 · p. 8
  • 68% ID success across five real-world bimanual tasks matches pretrained π0.5 (67%) and exceeds pretrained Motus (35%); PINT’s 24% OOD success exceeds Motus (9%) but trails π0.5 (35%).The real-world results cover five tasks.Source: §4.2 · p. 8, Table 3 · p. 9

4.3 Ablation Studies

  • 62.4%/11.1% Easy/Hard success with prompt-conditioned VGM is below PINT’s 68.6%/15.5%; Split Intent reaches 65.1%/11.8%.On LIBERO/LIBERO-Plus, prompt conditioning also lowers success from 98.0%/79.5% to 95.9%/63.4%.Source: §4.3 · pp. 8–9, Table 4 · p. 9
  • Removing Stage 1 future-video supervision yields 63.2%/11.5% RoboTwin and 97.1%/75.2% LIBERO/LIBERO-Plus success; removing VGM-to-AE features yields 65.9%/14.6% on RoboTwin.Removing autoregressive supervision yields 67.3%/12.2%; Stage 1 Only and Single-Stage Training reach 58.5%/13.1% and 65.5%/14.4%, respectively.Source: §4.3 · p. 9

4.4 Predictive Intent Visualization

  • The t-SNE action-chunk projection is colored by Predictive Intent’s three-dimensional PCA projection; structured color variation across highlighted behaviors suggests behavioral structure across trajectories.The highlighted regions are qualitative inspection areas, not supervised behavior labels or automatically discovered semantic clusters.Source: §4.4 · p. 9, Figure 4 · p. 10

5 Conclusion

PINT coordinates semantic reasoning, world modeling, and action generation through shared Predictive Intent while preserving expert-specific pathways. It uses a two-stage training strategy and retains the VGM during inference.

5 Conclusion

  • PINT learns chunk-level Predictive Intent from the VLM, VGM, and AE objectives, coordinating their capabilities while preserving expert-specific pathways.Stage 1 shapes Intent with a frozen pretrained VGM; Stage 2 adapts the VGM and AE.Source: §5 · p. 10

A.1 Model Configuration and Inference

  • At inference, the VLM produces Predictive Intent, the VGM processes the current frame, and the AE generates actions using layer-wise VGM features through one-way attention.The VGM’s layer-wise current-frame features are cached during action generation; future-video denoising is omitted.Source: §A.1 · pp. 15–16

A.2 Optimization and Hyperparameters

  • Stage 1 trains the VLM, Intent Tokens, AE, and projectors with the VGM DiT and video VAE frozen; Stage 2 freezes the VLM, Intent Tokens, and video VAE.Stage 2 trains the VGM DiT, AE, and projectors; evaluation uses the final Stage 2 checkpoint.Source: §A.2 · pp. 16–17
  • Both stages use AdamW, linear warm-up followed by cosine learning-rate decay, and gradient clipping; Stage 1 uses autoregressive, video, and action objectives, while Stage 2 uses video and action objectives.The reported final Stage 2 checkpoint is evaluated without selecting checkpoints by validation or test performance.Source: §A.2 · pp. 16–17

A.3 Two-Stage Training and Variant Details

The appendix specifies PINT’s two-stage training and variant configurations, then describes its RoboTwin and LIBERO evaluation setups. PINT records 79.5% on LIBERO-Plus, while PINT-VA records 81.0%.

A.3 Two-Stage Training and Variant Details

  • Stage 1 jointly trains the VLM, Intent Tokens, AE, and projectors with the VGM and video VAE frozen; Stage 2 freezes the VLM, Intent Tokens, and video VAE while training the VGM DiT, AE, and projectors.Stage 2 starts from the final Stage 1 checkpoint without reinitializing the AE or projectors.Source: §A.3 · p. 17
  • The prompt-conditioned VGM variant gives the VGM the task instruction through Wan’s text pathway, while Split Intent assigns separate token groups to the VGM and AE.In Split Intent, AE-specific tokens can attend to the preceding VGM-specific group, but not vice versa.Source: §A.3 · p. 17
  • Loss ablations remove Stage 1 video supervision or the autoregressive objective, while interaction and schedule ablations remove VGM-to-AE features or alter the two-stage training schedule.The Stage 1 Only variant uses default inference; Single-Stage Training jointly optimizes the modules for 10 RoboTwin epochs, matching the default two-stage total budget.Source: §A.3 · pp. 17–18

B. Benchmarks, Data, and Evaluation

  • RoboTwin training uses 2,500 clean demonstrations across 50 bimanual tasks; LIBERO training covers four suites with 50 demonstrations per task.The LIBERO suites assess spatial reasoning, object-centric manipulation, goal-conditioned behaviors, and long-horizon execution.Source: §B.1 · p. 18, §B.2 · p. 18
  • RoboTwin evaluates the same clean-trained checkpoint on 50 tasks under Easy and Hard, while LIBERO-Plus evaluates the LIBERO-trained checkpoint without adaptation across seven shift categories.RoboTwin Hard introduces domain randomization; LIBERO-Plus shifts include viewpoint, initial state, instruction, lighting, texture, sensor noise, and object layout.Source: §B.1 · p. 18, §B.2 · p. 18

B.2 LIBERO and LIBERO-Plus

  • On LIBERO-Plus, PINT succeeds in 7,974 of 10,030 trials (79.5%), while PINT-VA succeeds in 8,129 of 10,030 trials (81.0%).The Total score is computed over all evaluation trials, not as the arithmetic mean of category-level success rates.Source: §B.2 · p. 18

B.3 Real-World ALOHA Tasks

The real-world evaluation uses five bimanual tasks on an ALOHA platform, with a 700-demonstration training set. Final checkpoints are tested under ID and OOD conditions.

B.3 Real-World ALOHA Tasks

  • PINT’s real-world ALOHA suite covers switch operation, stacking, placement, articulated-object manipulation, and deformable-object manipulation, using 700 training demonstrations.The per-task demonstration counts are summarized in Table 9.Source: §B.3 · p. 19, Table 9 · p. 20
  • Final checkpoints are evaluated with 20 rollouts per task and condition; OOD settings vary lighting, background appearance, and initial object locations while preserving task semantics and embodiment.Scenes are manually reset before each rollout.Source: §B.3 · p. 20

C.1 Main-Table Result Sources

The appendix documents sources and adjustments for external benchmark results, distinguishing them from the authors’ own evaluations.

C.1 Main-Table Result Sources

  • External baseline results use original-paper reports when available and matching public evaluations otherwise; the appendix identifies the authors’ own evaluations separately.The provenance tables identify the source for each reported LIBERO, LIBERO-Plus, and RoboTwin result.Source: §C.1 · p. 20, Table 12 · p. 21, Table 13 · p. 21
  • MemoryVLA++’s standard LIBERO value is 98.5%, calculated by averaging Spatial, Object, Goal, and Long-10 while excluding Long-90; its LIBERO-Plus value is 73.1%.The LIBERO-Plus value is taken directly from the original paper’s zero-shot evaluation.Source: §C.1 · p. 20
  • Motus LIBERO and LIBERO-Plus results come from MoWAM, while StarVLA-OFT results use the matched Qwen3-VL-OFT variant and evaluations.The StarVLA-OFT LIBERO-Plus evaluation is reported by 3DThinkVLA.Source: §C.1 · p. 21
  • RoboTwin leaderboard values are rounded to displayed precision; Evo-1 comes from a released clean-only evaluation with 100 rollouts per task under both settings.The cited leaderboard reports Easy and Hard as clean-to-clean and clean-to-randomized evaluation, respectively.Source: §C.1 · p. 21

C.2 Our Reproductions and Evaluations

The section clarifies how its reproduced baselines were trained and evaluated, and identifies the concrete Fast-WAM implementation behind the WAM (Baseline) label in Fig. 3.

C.2 Our Reproductions and Evaluations

  • Fast-WAM is the reproduced RoboTwin baseline, trained on the same 50-task clean demonstrations as PINT and evaluated under the same Easy and Hard protocol.The official leaderboard lists a separate Fast-WAM entry, so the paper marks its reproduction with †.Source: §C.2 · p. 22
  • The real-world π0.5 and Motus comparisons use official pretrained checkpoints fine-tuned on the same 700 demonstrations as PINT and evaluated under the same ID/OOD protocol.The Motus numbers in Table 1 instead come from external public evaluations reported by MoWAM [52].Source: §C.2 · p. 22

C.3 Future-Video Baseline Provenance

  • “WAM (Baseline)” in Fig. 3 denotes the baseline WAM configuration, specifically the VGM from the Fast-WAM reproduction reported in Table 2.It uses the same RoboTwin clean demonstrations and ID/OOD inputs as PINT.Source: §C.3 · p. 22

D.1 LIBERO and LIBERO-Plus Ablations and OOD Analysis

The ablations compare Predictive Intent conditioning and Stage 1 future-video supervision across LIBERO and LIBERO-Plus, including its seven distribution shifts.

D.1 LIBERO and LIBERO-Plus Ablations and OOD Analysis

  • The evaluation varies two design choices: original VGM prompt conditioning instead of Predictive Intent, and removal of Stage 1 future-video supervision.The reported results include standard LIBERO and category-level LIBERO-Plus outcomes.Source: §D.1 · p. 22
  • Table 14 reports LIBERO ID and seven LIBERO-Plus shifts; its Total pools all 10,030 trials rather than averaging category success rates.Source: Table 14 · p. 22
  • Replacing Predictive Intent with prompt conditioning reduces LIBERO-Plus success by 16.1 points, versus 2.1 points on LIBERO; removing Stage 1 video supervision reduces LIBERO-Plus success by 4.3 points.Prompt replacement produces the largest category drops under background-texture and robot-initial-state shifts (45.7 and 34.6 points); removing video supervision has its largest drops under robot initial states (8.4), object layout (6.1), camera viewpoint (5.7), and lighting (5.3).Source: §D.1 · p. 22

D.2 Repeated Evaluation and Uncertainty

The appendix reports repeated-evaluation variability, inference efficiency and per-task RoboTwin results, with latency measured for full 32-step action chunks.

D.2 Repeated Evaluation and Uncertainty

  • Fixed-checkpoint results vary little across evaluation seeds: standard deviation is 0.39 points on RoboTwin 2.0 Hard and 0.06 points on LIBERO-Plus.Each benchmark checkpoint was evaluated three times using different seeds.Source: §D.2 · pp. 22–23, Table 15 · p. 23

D.3 Inference Efficiency

  • PINT’s inference latency rises from 266 to 331 ms relative to Fast-WAM while improving Hard and average success; PINT-VA improves manipulation further at higher inference cost.PINT-VA synchronously denoises future-video and action latents, while default PINT uses one current-frame VGM pass and caches its features across AE denoising steps.Source: Table 16 · p. 23, §D.3 · p. 23

D.4 Per-Task Results on RoboTwin 2.0

  • Table 17 gives PINT and PINT-VA Easy and Hard success rates for all 50 RoboTwin 2.0 tasks, alongside public RDT, π0, and Evo-1 results.PINT results are averaged over three repeated evaluations.Source: §D.4 · p. 23, Table 17 · p. 24

E.1 Additional Future-Video Predictions

The appendix adds qualitative future-video comparisons, describes how Predictive Intent visualizations are constructed, and shows representative real-world rollouts under ID and OOD conditions.

E.1 Additional Future-Video Predictions

  • Open-loop predictions use the same frame and instruction and compare future frames at t+8, t+16, t+24, and t+32 against ground truth from the same demonstration.The examples cover fixed demonstrations from the first five RoboTwin 2.0 tasks; ID and OOD examples use clean and randomized demonstrations and need not share scene layouts.Source: §E.1 · p. 25, Figure 7 · p. 26, Figure 8 · p. 27
  • Under OOD shifts, PINT better preserves scene layout than the Fast-WAM-trained VGM in examples including Beat Block Hammer and Blocks Ranking; Stage 2 generally refines arm and object motion.All methods’ predictions degrade at the longest horizon, with visible blurring in some t+32 frames.Source: §E.1 · p. 27

E.2 Predictive Intent Visualization Details

  • The Predictive Intent visualization pairs each representation with its ground-truth action chunk from the same episode and timestep.Action chunks are standardized, flattened, reduced to 50 dimensions with PCA, and projected by t-SNE; colors come from pooled Intent Token states projected to three PCA dimensions. Highlighted regions are qualitative inspection aids, not behavior labels or discovered semantic clusters; Stage 2 keeps the plotted representation fixed.Source: §E.2 · pp. 27–28

E.3 Real-World Qualitative Rollouts

  • Real-world rollout figures show temporally ordered representative executions of Turn On Light, Stack Cups, Place Pens, Store Block, and Fold Towel under ID and OOD conditions.For each task, the top row is ID and the bottom row is OOD.Source: Figure 9 · p. 29, §E.3 · p. 29

Related papers

Loading 2610.16355v1…