PaperlayerView exact revision on arXiv

Source-linked AI summary

LM-X: Explainable Action Modeling with Progress, Event, and Uncertainty Prediction for Generalist Robot Manipulation

Jin Lou, Jingxuan Zhu, Andong Chen, Xupeng Wang, Yuan Xu, Yuexuan Li, Xingdong Zhu, Zhijie Zhu, Yingwei Ji, Wenpeng Nie, Jingyi Li, Liangliang Chen, Jinyan Liu, Zhiqi Song, Jidong Zhang, Hongming Li, Yuchen Zhu

arXiv:2608.25757v1cs.ROcs.LG

TL;DR

Generalist VLA policies conceal progress, intent, and local reliability when they learn long-horizon behavior through short-horizon action targets. LM-X introduces explicitly supervised RTG, ETG, and variance predictions that condition action generation, achieving higher success than the action-only backbone and GR00T N1.7 across simulation and real-robot evaluations. The authors also report task-level reversals and limited statistical conclusiveness in some real-world comparisons.

  • Problem

    Generalist VLA policies lack dedicated targets for task progress, intermediate event intent, and local action reliability, leaving these control states hidden during execution.

  • Method

    LM-X jointly predicts RTG, ETG, and heteroscedastic action variance across task, event, and motor scales, with these signals conditioning the action pathway.

  • Results

    LM-X improves mean success across 50 RoboTwin2.0 tasks and seven real-robot tasks, reaching 74.1% versus 55.4% and 68.6% versus 50.7% for GR00T N1.7, respectively.

  • Takeaways & Limitations

    Explicit multi-timescale predictive state strengthens control while exposing interpretable internal estimates tied to progress, transitions, and motor reliability.

  • Takeaways & Limitations

    Real-world differences are descriptive rather than statistically conclusive because each trial represents 5 pp and only one checkpoint per method is evaluated.

Abstract

from arXiv · show

Generalist vision--language--action (VLA) policies learn long-horizon behavior mainly through short-horizon action prediction and reveal little beyond sampled commands. This creates two coupled bottlenecks: a single action target must implicitly absorb task progress, intermediate intent, and local reliability, while these control states remain hidden during execution. Inspired by functional principles of biological sensorimotor control, we introduce LM-X , which organizes prediction across task, event, and motor scales without claiming anatomical correspondence. Three explicitly supervised signals are emitted online and directly condition action generation: return-to-go (RTG) measures visible task progress, event-to-go (ETG) identifies the next semantic transition, and heteroscedastic action flow estimates local reliability through propagated variance. Explanation is therefore intrinsic to control rather than generated post hoc. Before a costly 20-day pretraining run on 64 NVIDIA B200 GPUs, a controlled five-task pretraining gate verifies the design: the complete model improves success by 16.0 points over the action-only backbone and by 10.8 points over the strongest single-head variant. We then train LM-X on more than 20,000 hours of real-robot trajectories, including over 1,000 hours of failed policy rollouts. LM-X achieves 74.1\% across 50 randomized-hard RoboTwin2.0 tasks versus 55.4\% for GR00T N1.7, and 68.6\% versus 50.7\% across seven real-robot tasks. RTG tracks semantic progress and visible regression, while variance rises during hesitation and oscillatory control. These results show that explicit multi-timescale predictive state can strengthen control while exposing interpretable internal estimates.

1 Introduction

Generalist VLA policies hide task progress, event intent, and local reliability behind short-horizon action prediction. LM-X addresses these bottlenecks with explicitly supervised, action-conditioning predictions across task, event, and motor scales.

  • 1 Introduction: Single-target action supervision leaves terminal outcome, intermediate event intent, and local action reliability without dedicated learning targets.These distinctions must emerge implicitly from the same action-prediction loss.
  • 1 Introduction: Long-horizon manipulation makes identical local motions compatible with different task states, intentions, and execution histories.Action chunking improves short-horizon coherence but does not require predictions to represent progress, the next transition, or command reliability.
  • 1 Introduction: LM-X predicts progress, event transition, and action reliability online, conditioning fine-grained control rather than explaining commands after execution.The design commits to semantically distinct targets, conditional information flow, and measurable online signals.
  • 1 Introduction: More than 20,000 hours of real-robot trajectories, including over 1,000 hours of failed policy rollouts, provide outcome, progress, and event labels.Failures expose missed grasps, stalls, target switches, and other states needed to distinguish progress from regression and commitment from ambiguity.
  • 1 Introduction: 79.6% mean success versus 63.6% for the action-only backbone in the five-task pretraining gate supported retaining all three modules.After pretraining, LM-X reached 74.1% versus 55.4% for GR00T N1.7 across 50 RoboTwin2.0 tasks and 68.6% versus 50.7% across seven real-world tasks.
  • 1 Introduction: The framework separates single-target supervision from control-state observability while providing simulation and real-robot evidence for improved control and temporally meaningful diagnostics.Its contributions include explicit RTG, ETG, and variance, a cost-aware pretraining gate, and scalable outcome and event annotation.

2 Method

LM-X predicts task progress, event-level intention, and local action reliability alongside fine-grained actions, using these signals in a directed conditioning hierarchy. Its RTG, ETG, and variance outputs are supervised online and expose interpretable control variables across timescales.

  • Problem Formulation: LM-X emits fine-grained actions, task progress, an event-level transition, and action uncertainty from visual observations, language, and proprioceptive state.The action chunk has horizon H = 30; ETG reaches the next semantic boundary, and variance is propagated to terminal action uncertainty.
  • Information Flow: RTG, ETG, and variance serve as supervised proxies for latent task phase and event intention while heteroscedastic variance models conditional motor residual dispersion.The decomposition organizes evidence already available from the shared inputs and introduces no external inference-time information.
  • Information Flow: Task-scale RTG conditions event prediction, whose representation and RTG then condition motor generation; uncertainty is estimated inside the action expert.This directed factorization encodes the inductive bias that task evaluation constrains event prediction, which constrains motor control.
  • Architecture: The architecture separates observation-level task assessment from embodiment-specific control: RTG uses the vision-language representation, while event and action experts also receive proprioception.The event expert exposes its hidden state to the action expert, and predicted signals replace ground-truth targets online.
  • Return-to-Go: RTG combines trajectory outcome with remaining duration, approaching zero near successful completion while assigning failed endpoints a distinct negative margin.Its bounded 128-bin representation is assessed from instruction-conditioned scene state without joint-state input, and supervision converts sparse outcomes into dense timestep targets.
  • Event-to-Go: ETG spans 60 steps versus 30 for fine-grained actions, represents the next semantic transition, and covers it in more than 86% of training samples.Terminal event-action repetition provides explicit stopping structure and prevents supervision from leaking into the subsequent subgoal.

3 Dataset and Annotation Pipeline

LM-X builds a common labeled trajectory schema from heterogeneous real-robot data, including failed rollouts, and verifies semantic event boundaries before training. The pipeline converts terminal outcomes into dense progress targets and represents events as action chunks ending at verified transitions.

  • Data schema: More than 20,000 hours of real-robot trajectories, including over 1,000 hours of failed policy rollouts, are converted to a common labeled schema.The mixture spans trajectories generated by ACT, Diffusion Policy, π-series, and GR00T-based agents.
  • Embodiment handling: The pipeline supports single-arm, dual-arm, and humanoid embodiments, while withholding joint state from the progress head to encourage cross-embodiment transfer.Event and action heads retain proprioception for kinematically feasible motion generation.
  • Progress annotation: RTG is computed backward from terminal success or failure labels, converting a single outcome into a dense target that preserves temporal ordering.
  • Data schema: Failed rollouts expose missed grasps, target switches, hesitation, occlusion, and other off-nominal states while discouraging elapsed time as a proxy for progress.
  • Event annotation: Events are semantically meaningful intermediate states that partition trajectories into interpretable subgoals, with candidate boundaries proposed automatically and verified or corrected by human annotators.This avoids frame-by-frame labeling while preserving task-specific semantics.
  • Event annotation: Every verified event becomes an action chunk ending at a semantic boundary, regardless of which task-dependent signals proposed it.At inference, the policy uses observations, language, and robot state rather than annotation heuristics or manual inputs.

4 Experiments

Experiments show that LM-X’s structured objectives provide complementary, task-dependent benefits, improving performance across simulation and real-robot evaluations while exposing progress and uncertainty patterns during execution.

  • 4.2 Component Ablations as a Pretraining Gate: No single component dominates across tasks: RTG gains most on microwave opening, event prediction on microphone handover, and uncertainty on bimanual pot lifting.The reported gains are 14, +16, and +10 points, respectively.
  • 4.2 Component Ablations as a Pretraining Gate: Event-only training reduces open-microwave success from 54% to 16%, showing that isolated auxiliary objectives can be brittle on contact-intensive sequences.The adopted design drops event conditioning stochastically during pretraining to mitigate over-reliance on event priors.
  • 4.2 Component Ablations as a Pretraining Gate: The complete model improves mean success from 63.6% to 79.6% (+16.0 points) across five held-out tasks and exceeds the strongest single-component variant by 10.8 points.It improves all five tasks, with the largest gain on block ranking.
  • 4.3 RoboTwin2.0 Benchmark: LM-X improves the 50-task RoboTwin2.0 mean from 55.4% to 74.1% against GR00T N1.7, outperforming it on eight of ten representative tasks.The largest representative gain is on stamp_seal: 22% to 74% (+52 points).
  • 4.4 Real-World Evaluation: On seven real-world tasks across four embodiments, LM-X reaches 68.6% mean success versus 50.7% for GR00T N1.7 (+17.9 pp), improving five tasks, tying one, and regressing on one.Excluding tableware organization, LM-X still leads by +10.8 pp.
  • 4.5 RTG Tracks Progress and Local Regressions: RTG rises near semantic progress, falls during visible execution regression, and resumes upward as placements complete, while failed rollouts show recurring variance spikes.Variance peaks also coincide with oscillation or delayed commitment, but the analysis establishes temporal correspondence rather than detection accuracy.

5 Related Work

LM-X is positioned against work on generalist VLA control, brain-inspired temporal organization, progress estimation, hierarchical manipulation, and flow-matching uncertainty. Its related foundations motivate predictive state across multiple timescales without prescribing a biological robot architecture.

  • Brain-inspired sensorimotor organization: Brain-inspired accounts motivate maintaining predictive state across slowly evolving context, behaviorally meaningful events, and faster motor trajectories.LM-X adopts this as a functional computational prior rather than a biological replica.
  • Value and progress estimation: RTG differs from optimal value functions and safety certificates by serving as a supervised progress-and-quality signal on successful and failed trajectories.The target is constructed from terminal outcomes and remaining duration, extending value-style ordering to imitation data.
  • Hierarchical and subgoal-conditioned manipulation: Hierarchical manipulation methods condition actions on skills, language events, keyframes, or visual subgoals, while LM-X uses event-conditioned action structure.The surveyed methods include SayCan, RT-H, COT-VLA, π0.5, and π0.7.
  • Uncertainty in flow matching: Prior VLA uncertainty methods largely rely on diffusion perturbations, Bayesian approximations, pixel uncertainty, feature likelihoods, or injected stochasticity.LM-X’s uncertainty treatment is situated within flow-matching policy research.

6 Conclusion

LM-X addresses hidden progress, intent, and reliability in action-centric VLA policies by coupling supervised multi-timescale predictions to action generation. The reported results support stronger control and intrinsic process visibility, while leaving causal faithfulness and safety guarantees unresolved.

  • Conclusion: LM-X exposes progress, event intent, and local reliability through RTG, ETG, and heteroscedastic flow variance coupled to the action pathway.The three signals answer whether the state is progressing, which transition is pursued, and how reliably the command is generated.
  • Conclusion: 74.1% versus 55.4%: LM-X achieves higher mean success than GR00T N1.7 across 50 randomized-hard RoboTwin2.0 tasks.Real-robot mean success also rises from 50.7% to 68.6%.
  • Conclusion: The authors explain the gains as relieving a single-target supervision bottleneck through dense progress ordering, semantic transition prediction, and residual-scale modeling.They explicitly present this as a central explanation rather than a guarantee of causal reasoning.
  • Conclusion: The explanatory interface is functional rather than mechanistic: biological correspondence is not claimed, and explicit predictions do not guarantee causal faithfulness.The model may rely on correlates such as visibility, appearance, camera motion, or episode timing.
  • Conclusion: Deployment still requires independent safety mechanisms because low variance is distribution-relative and RTG can fall from occlusion rather than physical regression.Suggested safeguards include state constraints, contact monitoring, collision checking, and anomaly detection.

A.5 Limitations and Future Evaluation

The paper frames uncertainty and multi-signal diagnostics as qualitative hypotheses rather than validated intervention policies. Future evaluation must establish calibration, robustness, causal responsiveness, and whether these predictions improve closed-loop recovery.

  • Limitations and Future Evaluation: Uncertainty analysis lacks failure-detection AUROC, precision–recall under imbalance, calibration error, and lead-time measurements.Calibration should be tested across tasks and embodiments because a single threshold may not transfer.
  • Limitations and Future Evaluation: ETG lacks dedicated evaluation of boundary accuracy, event-horizon coverage at test time, and semantic consistency.Controlled perturbations could test responses to occlusion, object displacement, unexpected contact, and instruction changes.
  • Limitations and Future Evaluation: Low uncertainty can coexist with poor ETG, so action uncertainty alone is insufficient as a safety signal.This pattern describes confident pursuit of the wrong intermediate objective.
  • Limitations and Future Evaluation: The diagnostic patterns remain recovery hypotheses, not evaluated intervention rules linking signal combinations to replanning, resampling, rollback, or handoff.The proposed patterns include stable ETG with falling RTG and rising uncertainty, or low uncertainty with poor ETG.

B Dataset Details

The dataset combines heterogeneous robot embodiments, sensing configurations, and manipulation domains with task-specific event annotation. Its coverage spans rigid-object transfer, insertion, deformable objects, and wiping, using manual or sensor-based verification to mark semantic transitions.

  • Dataset composition: The dataset uses heterogeneous embodiments and sensing configurations, with most platforms providing three to six camera streams and trajectories downsampled to 30 Hz.Tasks retain complete language instructions rather than short verb–noun templates.
  • Robot embodiments: Most platforms use 1-DoF parallel-jaw grippers, while selected humanoid configurations use 6-DoF dexterous hands.Figure 7 illustrates 10 representative embodiments.
  • Robot embodiments: The collection includes bimanual UR5e, Franka, and ARX systems, AgileX Cobot Magic, and multiple humanoid platforms.The listed humanoids include AgiBot, Fourier, Galaxea, Qloong, Leju, Tienkung, Tianji Marvin, and Dwheel embodiments.
  • Robot embodiments: AgiBot A2, Fourier GR2, Galaxea R1, Qloong, Leju KUAVO, Tienkung, and Tianji Marvin variants span substantially different degrees of freedom and camera layouts.These specifications document the embodiment diversity represented in the dataset.
  • Dataset composition: The dataset additionally samples from OXE and AgiBot World public data with probability 10%.The embodiment distribution is summarized in Table 5 and Figure 8.
  • Event annotation: Event annotation uses task-specific cues: gripper transitions for pick-and-place, operator-marked poses for insertion, stabilization or pose boundaries for deformables, and contact or trajectory turning points for wiping.Human verification removes incidental transitions and aligns retained events with recorded observations and motion.

C.1 Pretraining-Gate Ablation Details

The pretraining gate reserves five RoboTwin2.0 tasks for downstream evaluation and tests the proposed modules before the costly large-scale run. Failed episodes provide RTG supervision, while post-training uses successful episodes only.

  • Gate setup: Five RoboTwin2.0 tasks are excluded from backbone training and reserved for downstream post-training and evaluation.The held-out tasks are microphone handover, pot lifting, microwave opening, color-based block ranking, and hammering a target block.
  • Gate setup: The gate is run before large-scale pretraining to identify ineffective components or harmful interactions before committing approximately 20 days on 64 B200 GPUs.The study serves as a lower-cost validation of the RTG, ETG, and uncertainty modules.
  • Training protocol: Failed episodes update only the RTG branch, with ETG and action-expert gradients disabled during those updates.The protocol samples failures periodically and uses them exclusively for progress supervision.
  • Training protocol: Post-training uses successful episodes only, while component variants share the full model’s applicable architecture, preprocessing, action representation, and loss definitions.This keeps the ablation comparison aligned except for the tested component differences.
  • Training protocol: The complete model is selected for large-scale pretraining only after the pretraining-gate ablation supports the three-module design.The subsequent run uses two epochs, approximately 700,000 gradient steps, and 64 NVIDIA B200 GPUs.

C.3 Real-World Test Details

The real-world evaluation fine-tunes the complete pretrained model on task-specific data and compares methods under shared instructions and success criteria. Tasks span object placement, sorting, cloth folding, battery insertion, and dual- or single-arm coordination.

  • Evaluation protocol: Each real-world task fine-tunes the complete pretrained model for two epochs on task-specific post-training data.Smaller baselines train from scratch, while VLA baselines use official checkpoints.
  • Evaluation protocol: All methods receive the same language instruction and are evaluated under the same task-specific success criterion.This provides a common comparison protocol across the real-world tasks.
  • Tasks: The evaluation includes tape-roll pick-and-place into a marked target area and tableware organization by color across six randomly placed objects.Tableware objects are sampled from 20 objects spanning five colors.
  • Tasks: The evaluation also covers cloth folding, sequential insertion of four batteries, and water-bottle placement using either dual-arm or single-arm coordination.The bottle tasks require grasping, tray repositioning, and placement.
  • Baselines: GR00T N1.7 is initialized from its released checkpoint and fine-tuned on the same post-training mixture with identical instructions and success criteria.The reported comparison controls the downstream data and evaluation conditions.

D.1 Ablation of Event Embeddings during Post-Training

The post-training ablation tests whether the event embedding remains an input to the action expert. Its effect varies by task, so the adopted configuration selects event conditioning using downstream validation performance.

  • Task-dependent effects: Event conditioning helps block ranking and hammering, is neutral for microphone handover, and hurts pot lifting and microwave opening.The ablation isolates whether event conditioning should remain active during post-training.
  • Task-dependent effects: Event embeddings are retained with probability 80% during pretraining because of 20% dropout, while post-training activation is selected per task.The configuration adopted by LM-X uses downstream validation performance to determine the choice.

D.2 RoboTwin2.0 Benchmark

The RoboTwin2.0 benchmark reports full 50-task performance and examines RTG behavior during a held-out pear-sorting episode. RTG decreases for an ambiguous stem grasp, then recovers after later subgoals execute correctly.

  • Benchmark results: Table 8 reports performance across the full 50-task RoboTwin2.0 benchmark.The benchmark comparison is presented as the paper’s broad simulated evaluation.
  • RTG visualization: RTG generally increases during pear sorting but decreases when the right gripper holds only the pear’s stem.The stem grasp appears ambiguous and potentially unstable from the available views.
  • RTG visualization: No comparable RTG decrease occurs for pears grasped around their bodies, suggesting the score captures more than gripper closure or object elevation.The comparison links the local score change to grasp configuration rather than generic object lifting.
  • RTG visualization: Correct execution of later subgoals restores RTG’s upward trend, so a local decrease does not dominate the remainder of the trajectory.The episode-level trend can recover after a locally unfavorable event.
  • Limitations: A vision-only progress head may conflate perceptual ambiguity with physical instability, so the example is interpreted as behavioral state evaluation rather than ground-truth value estimation.Multi-view sensing and contact feedback are suggested as ways to disambiguate these cases.
Loading 2608.25757v1…