GDLAM: Group-Disentangled Latent Action Model for Highly Disentangled Embodied Pretraining

• Prior LAMs use a single unstructured code that entangles camera/base motion, manipulation, contact, and scene dynamics, limiting explicit factor manipulation and downstream factorized control.The introduction argues that LAMs should be evaluated both for future reconstruction and for the usefulness of their latent representations.Source: Introduction · p. 1
• GDLAM factorizes latent actions into N groups with independent variational bottlenecks, spatially gated routing, and four objectives that promote mutual causal distinctness.The objectives are mutual exclusivity, group sparsity, gate sparsity, and static–dynamic orthogonality.Source: Introduction · p. 2
• Single-group intervention leaves other groups essentially unchanged, while GDLAM improves label-free disentanglement and retains more action information than monolithic baselines in- and out-of-distribution.The introduction reports the information comparison across three mutual-information estimators and a linear probe.Source: Figure 1 · p. 1, Introduction · p. 2
• The workflow extracts latent-action labels from actionless video for world-model and VLA pretraining, then post-trains both downstream models using real-action supervision.World models use the labels to condition future-state generation; VLA models predict structured latent actions from observations.Source: Introduction · p. 2
• GDLAM pretraining consistently improves performance on simulation benchmarks and real-world robotic tasks.Figure 1(c) summarizes better downstream transfer.Source: Figure 1 · p. 1, Introduction · p. 2

• Genie learns discrete latent actions with a VQ codebook for interactive video generation, while LAPA uses latent actions to pretrain robot policies from actionless video.Both fit the latent-action-model framing of action-like representations inferred through self-supervised future-frame prediction.Source: Related Work · p. 2
• β-VAE weights KL regularization to encourage factorized posteriors, while FactorVAE and β-TCVAE constrain total correlation and InfoGAN maximizes mutual information between codes and generated outputs.Source: Related Work · p. 2
• Capsule networks and slot-based models structurally partition representations into sub-vectors assigned to different generative factors.Recent UniVLA and MVP-LAM replace pixel-space tokenizers with latent-space representations, but still use monolithic latent-action codes that leave visual-change factors entangled.Source: Related Work · p. 2

Problem Formulation

• GDLAM targets a code whose parts can be intervened on independently, addressing how a monolithic latent action marginalizes heterogeneous visual-change causes.The design uses independent group bottlenecks, spatial specialization, and objectives for causal distinctness; one group is encouraged to carry slow appearance while the others carry change.Source: Problem Formulation · pp. 2–3

Tokenizer as a Stable Perceptual Space

• A frozen vision foundation model supplies patch tokens as a fixed perceptual space, while a trainable Delta Encoder refines them into compact, normalized frame latents.The Delta Encoder begins as the identity on the frozen features and learns to inject change; the compact token code has dimension 32.Source: Tokenizer as a Stable Perceptual Space · p. 3

Grouped Action Encoder with Isolated Bottlenecks

• A grouped encoder reads both frames into N actions using learned queries, with each group's independent variational bottleneck imposing its own rate–distortion cost.Each query has a group-specific prefix embedding, and a factor is encoded only when it provides a genuine reduction in prediction error.Source: Grouped Action Encoder with Isolated… · pp. 3–4

Spatially Gated Routing Predictor

• The predictor routes each group's cross-attention output through a spatial gate, with zero-initialized gates and output projection so training starts at the identity map.Each gate produces a per-token mask that opens as its group proves useful, allowing groups to specialize to distinct image regions.Source: Spatially Gated Routing Predictor · p. 4

Disentanglement Objectives

• Reconstruction and per-group KL are joined by geometric objectives that partition code directions, spatial regions, and activations while separating static from dynamic latents.The sole imposed semantic prior designates one static group for slow appearance, orthogonal to the remaining dynamic groups; those groups are not assigned named factors.Source: Disentanglement Objectives · p. 4

Experiments

• The experiments compare GDLAM with representative LAMs under the same data and configuration, using four groups by default and held-out Bridge V2 transitions for in-distribution evaluation.Pretraining uses Open X-Embodiment, EgoExo4D, and Assembly101; the tokenizer is frozen DINOv2, while the grouped encoder and predictor are trained from scratch.Source: Experiments · p. 5, Disentanglement Without Losing Action… · p. 5

Disentanglement Without Losing Action Information

• GDLAM leads baselines on label-free Modularity, MIG, and DCI while its pooled code is more informative under all three MI estimators and has the lowest linear-probe NMSE.On four unseen LIBERO suites, it has the lowest ID error and smallest ID→OOD NMSE increase, without further probe or LAM training.Source: Disentanglement Without Losing Action… · p. 5

Are the Groups Causally Distinct?

• Single-group interventions yield a near-identity response matrix, and GDLAM’s leakage is an order of magnitude lower than any baseline.Monolithic baselines have no trained groups; their bottlenecks are split in half only as a reference comparison.Source: Table 2 · p. 6, Are the Groups Causally Distinct? · p. 6
• Across training, four groups converge on disjoint image regions without spatial supervision; in two scenes, dynamic-group attention focuses on the end effector and object near contact.MVP-LAM and UniVLA show secondary attention hotspots on task-irrelevant regions, while the designated static group attends diffusely to the background.Source: Figure 6 · p. 6, Figure 5 · p. 6, Are the Groups Causally Distinct? · p. 6

Downstream Transfer

• GDLAM-conditioned world modeling achieves the best overall WorldArena performance across 16 metrics, with the strongest improvements on controllability, especially action following.The reported OOD evaluations also show stronger action consistency and long-horizon prediction stability.Source: Table 3 · p. 7, Downstream Transfer · p. 7

Ablations

• Removing mutual exclusivity or static–dynamic orthogonality causes the largest DCI degradation and leakage increase; group and gate sparsity contribute more modestly.Sharing one bottleneck or using nonzero gate initialization weakens specialization, while action informativeness remains within a narrow range across the ablations.Source: Ablations · p. 6, Table 4 · p. 7

Downstream Transfer

• GDLAM achieves the best results across nine SIMPLER and LIBERO tasks: 66.7% average SIMPLER success (+9.4 points vs MVP-LAM) and 94.4% average LIBERO success.It leads across all four LIBERO task families; the largest gains include LIBERO-Long (+3.2 over MVP-LAM).Source: Figure 7 · p. 7, Downstream Transfer · p. 7

Conclusion

• GDLAM structures pretraining representations with grouped variational bottlenecks, gated routing, and information-geometric objectives.The learned groups are reported to be causally distinct and robust, with improved interpretability, controllability, and transferability to world models and VLA policies.Source: Conclusion · p. 7

Limitations

• The number of groups and coarse static–dynamic decomposition are predefined, so GDLAM cannot adapt factorization granularity by growing, merging, or pruning groups.The limitation concerns adapting to causal factors that differ across environments.Source: Limitations · p. 10
• The frozen vision foundation model passes its biases and perceptual limitations into the representation and may restrict discovery of task-relevant factors absent from its features.Visual representation learning and disentanglement objectives are not jointly optimized.Source: Limitations · p. 10
• GDLAM and its downstream applications are optimized separately, with manually designed routing and action-to-control components that may limit benefits from joint optimization.The potential joint objectives named by the authors are latent representation learning, controllability, and embodied semantic alignment.Source: Limitations · p. 10
• The evaluation does not guarantee that discovered groups match human-interpretable concepts and does not cover the full range of embodied applications.The authors say evaluation primarily focuses on intrinsic representation properties and causal behavior analysis.Source: Limitations · p. 10

A. Label-Free Disentanglement Metrics: Definitions and Protocol

• Modularity, MIG, and DCI are computed post hoc against seven net-relative action factors, using each baseline’s full code as one group and GDLAM’s four groups without factor-to-group labels.The held-out probing set contains 5,000 Bridge V2 transitions disjoint from pretraining and MI-estimator splits.Source: A. Label-Free Disentanglement Metrics:… · p. 10
• Modularity measures whether units concentrate information on one factor, MIG whether one unit stands out for each factor, and DCI whether units’ factor-prediction importance is concentrated.For GDLAM, the reported Modularity score pools mutual information within each group before applying the same formula.Source: A. Label-Free Disentanglement Metrics:… · pp. 10–11

B. Action Normalization and Mutual Information Protocol

• Action information is compared using one horizon-consistent seven-dimensional net-action target, three complementary MI estimators, and a frozen linear probe reduced to 128 PCA components.The three MI estimates are compared by ranking agreement rather than absolute values; the shared-space fidelity evaluation likewise re-embeds pixel predictions with frozen DINOv2.Source: B. Action Normalization and Mutual… · pp. 11–12

C. Fidelity in a Shared Perceptual Space

• GDLAM gives the best next-frame fidelity in the shared DINOv2 feature space, matching or edging out MVP-LAM and DreamDojo, while pixel-decoding baselines incur larger errors.Pixel-decoding predictions are re-embedded with the same frozen DINOv2 encoder before comparison, avoiding a pixel-space metric mismatch.Source: C. Fidelity in a Shared Perceptual Space · p. 12, Table 6 · p. 12
• The auxiliary pixel decoder’s visualizations show more coherent motion predictions and suppress background-related and appearance-irrelevant information, despite not targeting pixel-perfect reconstruction.The latent actions are derived from low-dimensional DINOv2 features, which discard fine-grained appearance details.Source: Figure 8 · p. 11, C. Fidelity in a Shared Perceptual Space · p. 12

D. Architecture and Hyperparameters

• The model uses a frozen DINOv2 ViT-L backbone, a six-block encoder, a twelve-block predictor, and four groups comprising one slow static/scene group and three dynamic groups.Each group has one 32-dimensional latent token; the number of groups was selected from a sweep of 1, 2, 4, and 8 to balance disentanglement and capacity.Source: D. Architecture and Hyperparameters · pp. 12–13
• Training takes approximately 6.5 days on eight H20 GPUs, making compute the practical reason larger group counts and larger-scale data are left for future work.The run lasts approximately 4.3 × 10^5 steps.Source: D. Architecture and Hyperparameters · p. 13

E. Disentanglement Objectives: Details

• Mutual exclusivity encourages distinct group latents, group sparsity keeps groups quiet unless their factors change, gate sparsity compacts spatial masks, and orthogonality separates slow scene content from motion.When enabled, the cross-viewpoint auxiliary is added to reconstruction without adding another disentanglement term.Source: E. Disentanglement Objectives: Details · p. 13

F. World Model: Details

• The experiment substitutes GDLAM into DreamDojo’s unchanged world model, pools its group latents for the existing action interface, and evaluates zero-shot on In-lab, EgoDex, and DreamDojo-HV after GR1_robot post-training.The held-out suites contain unseen objects, scenes, and interactions during GR1_robot post-training.Source: F. World Model: Details · pp. 13, 15
• Across three OOD suites, GDLAM conditioning improves average PSNR by +0.39 over DreamDojo and +0.25 over MVP-LAM, while lowering LPIPS by 0.010 and 0.006, respectively.The DreamDojo generator is held fixed, varying only the latent action model that supplies its conditioning code.Source: F. World Model: Details · p. 15, Table 8 · p. 15
• On DreamDojo-HV, qualitative OOD rollouts show GDLAM produces more distinct right-hand finger configurations and trajectories more consistent with GT future states, while long-horizon bimanual rollouts maintain temporally consistent motion of both hands.Figure 10 samples every 20 steps.Source: Figure 10 · p. 14, F. World Model: Details · p. 15
• Background distortions remain in rollouts, which the authors attribute mainly to the generative world model under long-range occlusions and complex scene transitions.The stated advantage of GDLAM is cleaner motion conditioning for the unchanged generator.Source: F. World Model: Details · p. 15

G. Group-Structured VLA Pretraining

• For simulation, the VLM predicts stop-gradient GDLAM latent targets with one head per group; monolithic baselines use the same backbone, optimizer, pretraining budget, and capacity-matched action decoder.Policies are finetuned with LoRA and evaluated on SIMPLER and four LIBERO suites under a shared protocol.Source: G. Group-Structured VLA Pretraining · p. 16
• Against MVP-LAM, GDLAM raises average success from 57.3 to 66.7 on SIMPLER (+9.4) and from 92.4 to 94.4 on LIBERO (+2.0), and is best on every task across both suites.Source: G. Group-Structured VLA Pretraining · p. 16
• GDLAM’s advantage is largest on SIMPLER finetuned with 100 demonstrations and on LIBERO-Long, where it gains +3.2 over MVP-LAM; near-saturated Spatial and Object suites leave little headroom.Source: G. Group-Structured VLA Pretraining · p. 16

G1. Real-Robot Deployment and Evaluation

• The hardware study uses the same JoyAI-RA-0.1 policy learner across latent-action methods and finetunes on eight desktop manipulation subtasks using an AgiBot G1.Single-task policies are trained and evaluated per subtask; the multi-task regime mixes all eight subtask splits into one policy.Source: G1. Real-Robot Deployment and Evaluation · pp. 16–17
• GDLAM achieves the best real-robot average success in both regimes (53.1% single-task, 41.3% multi-task), about 10 points above baseline (43.1%/29.4%), with an edge over DreamDojo.Improvements concentrate on fine-grained small-object tasks; in the harder multi-task regime, GDLAM retains non-trivial success across every subtask while baseline reaches 0% on pen and correction-fluid insertions.Source: G1. Real-Robot Deployment and Evaluation · p. 17, Table 13 · p. 18

• Protocols use 500 held-out Bridge V2 and LIBERO clip pairs, disjoint from pretraining and MI/probing splits, and average results over four seeds.Source: H. Causal-Intervention Protocols · p. 18
• Single-group intervention perturbs one latent group, decodes and re-encodes the predicted future frame, then measures each group’s response within its own spatial gate.Source: H. Causal-Intervention Protocols · p. 18
• Cross-R2 measures how well each group is predicted from the others; a low value indicates less redundant recoverability and complements intervention-based leakage.Source: H. Causal-Intervention Protocols · p. 19
• On real-content group swaps, GDLAM’s response matrix agrees with synthetic-intervention results within 0.03 on every entry, supporting robustness of its diagonal structure.Source: H. Causal-Intervention Protocols · p. 19

Leave-one-out on the four objectives

• Removing mutual exclusivity drops DCI from 0.88 to 0.61; removing gate sparsity reduces it to 0.74, while static–dynamic orthogonality particularly affects appearance-to-motion confusion.Group sparsity mainly weakens the quiet-unless-active property and mildly lowers DCI.Source: §I · p. 20

Architectural components

• A shared bottleneck yields DCI 0.64, a uniform gate yields DCI 0.71, and removing anti-leak augmentation lowers MI to 1.06.The shared bottleneck permits groups to share capacity; removing anti-leak augmentation allows static appearance to pass through the residual path.Source: §I · p. 20

Number of groups

• DCI peaks at N=4 (0.88) and declines at N=8 (0.85); increasing to N=8 raises per-step cost by approximately 34%.N=1 and N=2 are cheaper but under-factorize the code, while MI(KSG) remains nearly flat; the authors identify N=4 as the accuracy/compute sweet spot.Source: §I · p. 20

Static group and objective-weight sensitivity

• Across ±2× objective-weight settings, DCI stays within ±0.03; removing the static-group designation leaves DCI at 0.86 but roughly doubles static/dynamic leakage.Leakage is measured as dynamic-group gate mass falling on the static background.Source: §I · p. 20

• GDLAM occupies a moderate position in parameter count, memory, FLOPs, and latency, with runtime efficiency comparable to mainstream VFM-tokenized LAMs.The authors describe the group-disentanglement mechanism as incurring modest computational overhead while improving representation disentanglement and downstream controllability and generalization.Source: J. Compute and Training Cost · pp. 20–21
• Profiling uses one GPU, paired 224×224 RGB frames, batch size 1, and FP32, measuring parameters, peak activation memory, inference latency, and forward-pass FLOPs.Source: J. Compute and Training Cost · p. 20
• The tokenizer is a frozen VFM, and grouped cross-attention is batched over all latent groups rather than run as separate attention modules for each group.Trainable components are a delta encoder, grouped action encoder, and grouped predictor with depths of 6, 6, and 12 transformer blocks, respectively.Source: J. Compute and Training Cost · p. 20