PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynin-Robotics: Omnimodal Unified Diffusion Vision-Language-Action Model

Hoeun Lee, Jaeik Kim, Jusang Oh, Jinhyeok Kim, Geon Choi, Hyeonggeun Kim, Jaeyoung Do

arXiv:2609.13053v1cs.ROcs.AIcs.LG

TL;DR

Language-conditioned robot policies need to connect task instructions and visual observations with both intended outcomes and action-dependent scene changes. Dynin-Robotics models these variables in a shared masked-diffusion trajectory framework and evaluates its objectives and inference compositions across simulation and a physical robot. It reports improved VLABench results under specified comparisons, competitive benchmark performance, and a 78.4% average success rate across four Franka Research 3 manipulation conditions.

  • Problem

    Language-conditioned manipulation requires connecting instructions and visual observations to actions amid target identification, object interaction, contact, clutter, and occlusion.

    Source: §1 · p. 1

  • Method

    Dynin-Robotics continually pretrains a shared masked-diffusion trajectory model to predict actions, action-conditioned next observations, terminal goals, and trajectory-derived instructions.

    Source: §4.2 · p. 14, §6 · p. 30

  • Results

    Dynin-Robotics reports improved VLABench adaptation and shifted-instruction success under stated comparisons, competitive LIBERO and zero-shot LIBERO-Plus performance, and 78.4% average success across four Franka Research 3 manipulation conditions.

    Source: §1 · p. 3, §6 · p. 30

  • Takeaways & Limitations

    The shared trajectory interface supports learning complementary robot objectives and composing visual predictions during action generation and selection.

    Source: §6 · p. 30

    The visual-dynamics evaluation covers adjacent-future prediction and terminal goal prediction; Task Understanding is qualitative, and the physical study covers one robot platform and workspace.

    Source: §6 · p. 30

Abstract

from arXiv · show

Visual goal and dynamics prediction can provide language-conditioned robot policies with both a target outcome and a representation of action-dependent scene changes. We bring these predictions into action generation and selection through a shared trajectory model. Dynin-Robotics implements this formulation on Dynin-Omni, an omnimodal masked-diffusion backbone, representing language, visual observations, goals, and actions as discrete tokens. By varying conditioning and target spans, the same model learns action prediction, action-conditioned next-observation prediction, terminal goal-state prediction, and trajectory-to-instruction reconstruction. These interfaces support test-time scaling through goal prediction, action-candidate evaluation, and joint refinement of action and future-state predictions. We continually pretrain the model on approximately 1.33 million trajectories from 48 Open X-Embodiment datasets and adapt it separately to downstream domains. On two VLABench tasks, robot pretraining improves adaptation within a fixed Stage-2 step budget, and the full objective mixture improves shifted-instruction success over Policy-only post-training under the same coupled decoder. Combining goal guidance with joint action-next-state denoising further improves shifted-instruction success over action-only decoding; the benefit depends on how the predictions are composed. Dynin-Robotics achieves competitive performance on LIBERO and zero-shot LIBERO-Plus, together with a 78.4% average success rate across four manipulation conditions on a Franka Research 3 robot. An optimized block-parallel implementation accelerates model-side action decoding by up to 29.2x relative to the base implementation under the reported profiling setup. These results support shared trajectory modeling as a common interface for learning complementary robot objectives and composing their predictions during control.

1 Introduction

Dynin-Robotics brings visual goal and dynamics prediction into language-conditioned action inference through a shared trajectory model. Its evaluation reports improved adaptation and shifted-instruction performance in VLABench, competitive policy results across simulation and robot settings, and faster model-side decoding.

1 Introduction

  • Visual goals and action-dependent scene changes offer complementary representations for connecting language-level task semantics with physical execution.The paper frames language-conditioned manipulation as involving target identification, object interaction, and continuous control amid contact, clutter, and occlusion.Source: §1 · p. 1
  • A two-task VLABench diagnostic finds different task-performance profiles and linguistic-input sensitivity for π0.5 and Mimic-Video, motivating joint study of language conditioning and visual prediction.π0.5 performs better on SelectFruit; Mimic-Video is competitive on indirect InsertFlower instructions, and random strings reduce π0.5 success more.Source: §1 · pp. 2–3
  • The model extends Dynin-Omni with robot-action tokens, representing language, visual observations, goals, and actions as typed spans in one shared trajectory sequence.Policy, World Modeling, Task Understanding, and Goal-State Prediction are conditional denoising queries over that representation.Source: §1 · pp. 2–3
  • Dynin-Robotics reports improved VLABench adaptation and shifted-instruction success, competitive LIBERO and zero-shot LIBERO-Plus performance, and 78.4% average success across four Franka Research 3 manipulation conditions.Robot pretraining improves adaptation within a fixed Stage-2 budget; the full objective mixture and selected inference compositions improve shifted-instruction success against their stated comparisons.Source: §1 · p. 3
  • Goal prediction, joint action–future-state denoising, and world-model scoring or reranking reuse one model to guide action generation and selection at test time.Block-parallel decoding with dInfer improves model-side action-decoding throughput by up to 29.2× under the reported profiling setup.Source: §1 · p. 3

2 Related Work

Related work spans masked-diffusion language models, shared-token multimodal models, and robot policies that connect visual prediction with action. Dynin-Robotics places these ideas in a shared trajectory formulation with multiple prediction and inference interfaces.

2.1 Large Diffusion Language Models

  • Masked diffusion reconstructs masked token positions in parallel from visible context and can iteratively revise uncertain predictions under bidirectional context.Prior work ranges from diffusion over token embeddings to discrete-vocabulary methods including absorbing-state masking and reverse-process ratio learning.Source: §2.1 · pp. 3–4
  • For robot trajectories, quantized observations and controls support conditional denoising queries for control, visual prediction, and trajectory understanding.This builds on masked-diffusion language models and policies that apply masked diffusion to action decoding or jointly model actions with visual states and instructions.Source: §2.1 · p. 4

2.2 Unified Multimodal Models

  • Shared-token multimodal models use a common representation for conditioning and iterative prediction, with Dynin-Robotics extending that interface to instructions, visual states, goals, and actions.Other approaches combine shared language backbones with modality-specific generators or model multimodal token sequences using autoregressive, diffusion, or flow-based methods.Source: §2.2 · p. 4

2.3 Modeling Paradigms in Robot Learning

  • Robot-policy work includes VLM-based policies, video-generative policies, and unified approaches combining action learning with visual prediction or understanding.These approaches use differing conditioning structures, including action experts conditioned by VLMs or video generators and joint denoising of visual states and actions.Source: §2.3 · pp. 5–6
  • Dynin-Robotics combines action-conditioned next-observation prediction, terminal goal prediction, policy, and trajectory-to-instruction reconstruction in one masked-diffusion backbone.Changing visible context and prediction targets supports goal conditioning, joint action–next-state denoising, and action-candidate evaluation.Source: §2.3 · p. 6

3 Dynin-Robotics

Dynin-Robotics models robot trajectories as typed multimodal sequences and trains one masked-diffusion backbone for complementary policy, prediction, and instruction-reconstruction queries. Its shared interface also supports inference compositions that allocate computation to goal prediction, joint action–state refinement, and candidate evaluation.

3 Dynin-Robotics

  • The shared trajectory interface represents visual observations, instructions, actions, goals, and optional context as conditioning or target spans, supporting six inference compositions.The default path predicts a goal-state context before denoising an action chunk.Source: §3 · p. 6

3.1 Backbone

  • Dynin-Omni uses a single bidirectional Transformer and shared discrete-token interface to reconstruct masked text, image, video, and speech tokens from visible context.Robot trajectories extend this interface to conditional predictions such as actions from observations and instructions, or visual outcomes from actions.Source: §3.1 · p. 6

3.2 Robotic Token Space Extension

  • Robot examples use typed token spans, with non-overlapping vocabulary blocks for visual and robot-action tokens and reserved blocks for configured sensor types.Text and visual targets use random masking, while action targets use block masking; inverse quantization maps decoded action tokens to continuous controls.Source: §3.2 · pp. 6–7, Figure 3 · p. 7

3.2.1 Actions

  • Actions use deterministic per-dimension uniform binning into reserved action tokens, with inverse quantization returning predicted bin centers to continuous controls.Stage 1 uses 256 active bins; Stage 2 selects a task-specific bin count within the same reserved action block.Source: §3.2.1 · pp. 7–8

3.2.2 Optional Sensor Token Interface

  • Optional sensor spans synchronize streams to observation timestamps and encode calibrated channels with typed delimiters and fixed channel order; sensor-token training and sensor-conditioned policy decoding are inactive in reported runs.The interface can accommodate force/torque, proprioception, and tactile measurements.Source: §3.2.2 · p. 8

3.3 Unified Objective Training

  • Four objective-specific queries predict action chunks, action-conditioned next visual states, terminal goal states, and instructions reconstructed from trajectory frames.Training masks only target spans, using random masking for text and visual targets and contiguous block masking for action targets.Source: §3.3 · pp. 9–10, Figure 4 · p. 9

3.4 Unified Multi-Stage Inference

  • Inference supports action-only decoding, goal-guided policy decoding, joint action–next-state denoising, and world-model-based candidate reranking, with goal guidance combinable with joint denoising or filtering.These compositions allocate additional inference computation to visual prediction and candidate evaluation to trade computation for policy performance.Source: Figure 5 · p. 10, §3.4 · pp. 10–12

3.5 Acceleration

  • Block-parallel decoding commits confident action tokens and refines others; acceleration reuses fixed context approximately and batches candidates, while online reinforcement learning replaces the commitment schedule with learned remasking.The learned remasking controller changes the schedule while keeping the shared token predictor and modality tokenizers frozen.Source: §3.5 · pp. 12–13

4 Training Recipe

The training recipe continually pretrains Dynin-Omni on a shared robot-trajectory corpus, then adapts separate checkpoints to downstream domains. Stage 2 combines supervised objectives and optional metadata or online remasking control while retaining the shared token interface.

4.1 Training Data

  • 1,332,985 trajectories from 48 Open X-Embodiment datasets are mapped to a common 7D end-effector action interface for Stage 1.The runtime training split contains 65,217,081 transitions; the released Stage-1 run uses the OXE mixture without active force/torque or force-policy batches.Source: §4.1 · p. 13

4.2 Continual Pretraining

  • Continual pretraining starts from Dynin-Omni with frozen text and visual tokenizers and runs for 21.8k updates, then 5.8k more under a policy-centered schedule.The second segment loads model weights and reinitializes the optimizer and scheduler.Source: §4.2 · p. 14

4.3 Post-Training

  • Stage 2 adapts the continual-pretraining checkpoint separately to each domain while preserving the shared vocabulary, prompt layouts, and masked-denoising interface.Action statistics, active action-bin count, objective schedules, and loss weights are selected for the downstream setting.Source: §4.3 · p. 14

4.3.1 Trajectory Metadata Conditioning

  • Metadata-conditioned rollouts use fixed Action-Chunk Jerk and Trajectory Steps placeholders of 300 and 200; Jchunk is a normalized-coordinate local proxy, not physical Cartesian jerk.The proxy uses only two third-difference vectors and excludes the discontinuous gripper channel.Source: §4.3.1 · p. 15

4.3.2 Supervised Fine-Tuning

  • Stage-2 supervised fine-tuning keeps the backbone and denoising interface fixed while Policy-only or unified runs vary context, target modality, schedule exposure, and loss weights.Policy-only runs schedule only the Policy objective; unified runs retain the same backbone and interface.Source: §4.3.2 · p. 15

4.3.3 Online Reinforcement-Learning Fine-Tuning

  • Online reinforcement learning trains a lightweight remasking controller and value estimator with the backbone and tokenizers frozen, trading environment return against decoding cost and episode length.The controller changes the refinement schedule, not the shared token predictor; reported ablations do not isolate its contribution from other choices.Source: §4.3.3 · pp. 16–17

5.1 Evaluation Setup

The evaluation measures policy success across standard benchmarks, instruction shifts, and one physical setup, alongside direct prediction outputs and decoding trade-offs. It specifies separate transfer and task conditions for LIBERO, LIBERO-Plus, VLABench, and Franka experiments.

5.1 Evaluation Setup

  • Evaluation covers policy success, direct visual-state and task-description predictions, objective-mixture ablations, and accuracy–latency trade-offs.Task success rate is the primary policy metric; cross-model table differences are treated descriptively because results use model-specific training recipes.Source: §5.1 · p. 17

LIBERO.

  • LIBERO evaluation reports four suite success rates and their macro-average using the two-stage goal-guided inference path.The suites cover spatial relations, object-centric manipulation, goal-conditioned interaction, and long-horizon composition.Source: LIBERO · p. 17

LIBERO-Plus.

  • LIBERO-Plus tests seven perturbation families with the LIBERO-trained checkpoint and no additional training or adaptation.The reported family success rates and macro-average characterize transfer from LIBERO to perturbed evaluation conditions.Source: LIBERO-Plus · p. 17

VLABench.

  • VLABench tests SelectFruit and InsertFlower under direct, commonsense, and indirect-paraphrase instructions, with instruction-shift studies focused on these two tasks.The 5k–30k checkpoints characterize learning behavior, while ablations and sensitivity studies compare 70k-step Stage-2 endpoints; Figure 8 depicts the tasks and tracks.Source: VLABench · p. 17, Figure 8 · p. 18
  • The physical evaluation uses a Franka Research 3 in one workspace across four conditions drawn from three task families, with success macro-averaged across conditions.The conditions are Fruit PnP, Cube Sort, Cube Stack, and Color-Ordered Stack.Source: VLABench · pp. 17–18

5.2 Policy Evaluation

Dynin-Robotics performs strongly on LIBERO and on the Franka robot, while zero-shot transfer to perturbed LIBERO-Plus conditions is uneven. VLABench results show that task performance and instruction sensitivity vary across policies and tasks.

5.2.1 LIBERO

  • 98.1% macro-average on LIBERO matches X-VLA and is 0.5 percentage points below ABot-M0.Scores are highest on Object (99.8%) and Spatial (98.9%), and lowest on Long (95.8%).Source: §5.2.1 · p. 18, Table 2 · p. 19

5.2.2 LIBERO-Plus

  • 73.0% macro-average on zero-shot LIBERO-Plus is above the listed OpenVLA-OFT and RIPT-VLA results but below ABot-M0 at 81.6%.Success is 85.0% under Language perturbations, versus 59.8% for Camera and 48.2% for Robot; gaps to ABot-M0 are larger for Robot and Light than Camera.Source: §5.2.2 · p. 18, Table 3 · p. 20

5.2.3 VLABench

  • Dynin-Robotics substantially exceeds Mimic-Video on SelectFruit and remains competitive on InsertFlower, with relative performance varying across checkpoints and instruction tracks.The checkpoint curves cover 5k–30k post-training steps; the cited 70k-step ablations are distinct.Source: §5.2.3 · p. 19, Figure 9 · p. 20
  • Random-string instructions reduce π0.5 success by 16 and 30 percentage points on InsertFlower and SelectFruit, while Mimic-Video changes by +10 and +2 points.The comparison motivates considering absolute task success alongside sensitivity to instruction randomization.Source: §5.2.3 · p. 19, Table 4 · p. 20

5.2.4 Real-World Manipulation

  • 78.4% is Dynin-Robotics’ highest reported macro-average across the four Franka Research 3 conditions, with 68.5% success on Color-Ordered Stack.Its differences from π0.5 are +1.0, +2.0, −1.5, and +4.5 percentage points for Fruit PnP, Cube Sort, Cube Stack, and Color-Ordered Stack, respectively.Source: §5.2.4 · pp. 21–22, Table 5 · p. 21

5.3 World Modeling and Goal-State Prediction

The Stage-1 checkpoint predicts action-conditioned adjacent observations and terminal goal states on DROID without additional DROID-specific post-training. Results vary by prediction task and metric, and whole-image similarity metrics can favor simple frame persistence.

5.3 World Modeling and Goal-State Prediction

  • The DROID evaluation uses the Stage-1 checkpoint without additional DROID-specific post-training, and DROID is included in the Stage-1 mixture.World Modeling predicts the next visual state from current context and action; Goal-State Prediction generates a terminal state from the initial observation and instruction.Source: §5.3 · p. 22

5.3.1 Comparison with Unified Policies

  • 31.07 PSNR, 0.953 SSIM, and 0.013 LPIPS for copying the last frame exceed Dynin-Robotics’ 20.79, 0.573, and 0.299 on adjacent-future prediction.Whole-image scores aggregate static content and changing regions, so action-dependent changes remain a distinct evaluation requirement.Source: §5.3.1 · p. 22, §5.3 · p. 22, Table 7 · p. 24
  • Among learned predictors, Dynin-Robotics has the highest PSNR and SSIM for World Modeling, while MMaDA-VLA has the lowest LPIPS.The same checkpoint also has the highest reported offline action accuracy at tolerance 0.1 (0.3327) and lowest normalized action MAE (0.2109).Source: §5.3.1 · p. 22

5.3 World Modeling and Goal-State Prediction

  • Figure 11 shows selected action-conditioned next-observation predictions, while Figure 12 shows generated terminal states for object-placement and scene-rearrangement tasks.The figures provide qualitative examples; the section’s quantitative comparisons measure image agreement.Source: §5.3.1 · p. 22, Figure 11 · p. 23, §5.3.2 · p. 23

5.3.2 Comparison with Image-Generation Models

  • For World Modeling, Dynin-Robotics has the highest PSNR, while BAGEL has higher SSIM and lower LPIPS; for Goal-State Prediction, BAGEL leads all three metrics.Dynin-Robotics ranks second among listed models for terminal goal-state prediction.Source: §5.3.2 · p. 23, Table 7 · p. 24

5.4 Qualitative Task Understanding

Task Understanding uses sampled trajectory frames to generate task descriptions, including action verbs, object attributes, and spatial relations. The outputs vary in specificity, and quantitative evaluation against reference instructions remains a next step.

5.4 Qualitative Task Understanding

  • Generated descriptions include actions such as picking and placing, object attributes such as color, and spatial relations involving bowls, drawers, and counters.One description combines retrieving an object from a drawer with placing it on a counter.Source: §5.4 · p. 24
  • Output specificity ranges from referring to a purple object to naming a can and its source and destination.The examples use the Stage-1 checkpoint on DROID validation trajectories without additional DROID-specific post-training.Source: §5.4 · p. 24
  • Quantitative evaluation against reference instructions remains a next step for assessing trajectory-to-language accuracy.Adding Task Understanding to the Stage-2 mixture accompanies higher downstream success under the coupled decoder.Source: §5.4 · p. 24

5.5 Acceleration

The accelerated block-parallel decoder improves model-side action-output throughput under a specified profiling setup. Larger configured blocks bring higher throughput, alongside small changes in reconstruction fidelity.

5.5 Acceleration

  • The throughput figures measure model-side action decoding, excluding goal generation, sensing, communication, and robot execution.Query latency, denoising work, and LIBERO-Goal rollout success are examined separately.Source: §5.5 · pp. 24–25
  • 29.15× the base throughput is reached by BL35 at 268.834 TPS, while BL7 reaches 91.236 effective TPS for a 9.89× speedup.Profiling uses one NVIDIA B200, batch size 1, a 2,048-token context, and a 35-token action chunk.Source: §5.5 · p. 24, Table 8 · p. 25
  • BL7 token accuracy is 99.8214% versus 99.7545% for the base decoder, while action MAE is slightly higher at 0.021383 versus 0.021317.BL35 lowers token accuracy to 99.4643% and raises MAE to 0.021468.Source: §5.5 · p. 24

5.6 Ablation Study

The ablations compare training recipes under a common decoder and inference compositions with a fixed checkpoint. The strongest reported shifted-instruction result comes from combining goal guidance with joint action–next-state denoising, while training and inference gains depend on the chosen recipe.

5.6 Ablation Study

  • Training comparisons share a 70k-step VLABench budget, while inference comparisons hold the checkpoint fixed; ID is Track 1 and OOD averages Tracks 3 and 4.The objective-mixture comparison fixes inference variant (e).Source: §5.6 · p. 27

5.6.1 Continual-Pretraining Ablation

  • With the same 70k-step Stage-2 budget, OXE-pretrained initialization reaches 49.61% ID and 47.28% OOD success, versus 3.26% and 0.70% without OXE continual pretraining.Both pipelines start from Dynin-Omni; the comparison varies the additional robot-pretraining stage.Source: §5.6.1 · p. 27, Table 10 · p. 27

5.6.2 Training-Objective Ablation

  • After 70k Stage-2 steps, the full objective mixture reaches 49.61% ID and 47.28% OOD success, versus 47.15% and 33.88% for Policy-only training.The OOD improvement is larger, but objective exposure changes alongside the active targets, so cumulative increments do not isolate target effects.Source: §5.6.2 · pp. 27–28, Table 11 · p. 27

5.6.3 Inference-Strategy Ablation

  • 49.6% ID and 47.3% OOD success with goal-guided joint denoising exceed action-only decoding by 3.8 and 5.9 percentage points, respectively.The inference comparison holds the post-trained checkpoint fixed.Source: §5.6.3 · p. 28, Table 12 · p. 28
  • World-model filtering (f) reaches 47.7% OOD success at 4.828 TPS; versus joint denoising (e), that is a 0.4-point OOD gain at approximately 44.6% lower throughput.Joint denoising (e) runs at 8.709 TPS, compared with 9.238 TPS for action-only decoding.Source: §5.6.3 · p. 28, Table 12 · p. 28

5.7 Hyperparameter Sensitivity

The sensitivity studies vary action discretization, denoising granularity, and auxiliary-loss strength. Their best observed settings differ across measures, with faster decoding trading off against small changes in rollout success.

5.7 Hyperparameter Sensitivity

  • The studies vary action-bin resolution, token-commitment granularity, and auxiliary-loss strength under the 70k-step VLABench protocol, with decoding evaluated on LIBERO-Goal.They identify operating points and sensitivity only within the tested settings.Source: §5.7 · pp. 28–29

5.7.1 Action Token Binning

  • 32 action bins yield the lowest normalized action MAE (0.02) and highest rollout success, 49.61% ID and 47.28% OOD, among the tested bin counts.Both 16 and 64 bins increase MAE and reduce rollout success.Source: §5.7.1 · p. 29, Table 13 · p. 29

5.7.2 Denoising and Block-Parallel Decoding

  • BL35 reduces mean backbone evaluations per chunk to 1.73 and query latency to 130.2 ms, with 97.2% mean success versus 97.8% for the base decoder.BL7 records 5.14 evaluations and 383.6 ms at 97.7% success; amortized latency assumes all five predicted actions are executed before replanning.Source: §5.7.2 · p. 29, Table 14 · p. 29

5.7.3 Auxiliary-Objective Loss Scale

  • Equal auxiliary-loss weights (α = 1) yield the highest tested success, 49.61% ID and 47.28% OOD; doubling them lowers success to 38.05% and 36.70%.The exposure ratio is fixed at 0.65 : 0.20 : 0.05 : 0.10, with Policy loss weight held at one.Source: §5.7.3 · p. 30, Table 15 · p. 30

6 Conclusion

Dynin-Robotics uses a shared masked-diffusion trajectory representation for robot control, visual prediction, and trajectory understanding. Its reported evaluations include improved shifted-instruction results from selected training and inference choices, alongside scope limits in visual-dynamics and physical evaluation.

6 Conclusion

  • A shared masked-diffusion trajectory sequence represents language, visual states, goals, and actions, allowing visual predictions to be reused during action generation and selection.The interface supports goal prediction, joint refinement, and candidate evaluation.Source: §6 · p. 30
  • On two VLABench tasks, continual pretraining improves fixed-budget adaptation; the full objective mixture raises shifted-instruction success over Policy-only post-training under the same coupled decoder, while goal-guided joint denoising beats action-only decoding with the checkpoint fixed.Evaluations on LIBERO, zero-shot LIBERO-Plus, and a Franka Research 3 platform show competitive policy performance across reported settings. An optimized block-parallel implementation provides up to a 29.2× speedup in model-side action decoding over the base implementation under the profiling protocol.Source: §6 · p. 30
  • Visual dynamics evaluation covers adjacent-future prediction and terminal goal-state prediction; the physical study covers one robot platform and workspace.The authors identify longer-horizon dynamics, trajectory-to-language accuracy, and additional embodiments and sensor modalities as next steps.Source: §6 · p. 30
Loading 2609.13053v1…