PaperlayerView exact revision on arXiv

Source-linked AI summary

Enfold: Folding World Model Imagination into Predictive Representations for Ultra-Efficient Embodied Control

Weili Zeng, Yitong Xing, Fulong Liu, Chengqun Yang, Antao Xiang, Feng Tian, Jingnan Gao, Jisong Cai, Xin Wang, Xiaomin Wu, Yao Mu, Xiaokang Yang, Yichao Yan

arXiv:2607.26657v3cs.ROcs.CV

TL;DR

Robots need representations that anticipate how scenes will change, but world-generator states depend on futures unavailable at deployment. Enfold distills those states into a current-only representation, supporting strong control across simulated and real tasks while running 3.7× faster than Fast–WAM.

  • Problem

    World generators capture long-range interaction, but their future-conditioned internal states are unavailable and variable at deployment, motivating predictable present-only control representations.

  • Method

    Enfold trains a current-only encoder with multi-level generator states, then uses its representation for future generation and detached action readouts without executing the generator during control.

  • Results

    Across LIBERO, RoboTwin2.0, and real-robot tasks, Enfold supports strong control while reducing action latency 3.7× relative to Fast–WAM.

  • Takeaways & Limitations

    Enfold reframes the generator as a training-time source of structured predictive supervision that can be reused efficiently for control without explicit imagination at every step.

  • Takeaways & Limitations

    Generator-state targets depend on realized futures and sampled noise, requiring a trade-off between informativeness, predictability, and non-degeneracy.

Abstract

from arXiv · show

World generative models are typically used through what they produce: a rendered future, a video-conditioned action, or latent context computed by a costly generative branch. We argue that their more reusable asset is the computation that constructs a future. As a generator transforms a corrupted future into a coherent trajectory, its intermediate states organize appearance, spatial layout, and interaction across levels of abstraction. Can this future-generative computation be internalized in a representation inferred from the present alone? We present Enfold, which transfers this computation into a representation predicted from the current visual context and language instruction. During training, multi-level states exposed as the generator processes the observed future supervise a current-only encoder. The learned representation is fed back to condition future generation and is read by task heads without allowing task gradients to reshape the encoder. At deployment, action prediction no longer executes the generator. Across LIBERO, RoboTwin2.0, and real-robot tasks, Enfold supports strong control while reducing action latency by $3.7\times$ relative to Fast--WAM, Enfold-Flash reaches $10.1\times$. Representation analyses show that it suppresses nuisance variation and preferentially captures changes that emerge over longer horizons. When the current scene is altered by human intervention, both the generated continuation and the executed actions adapt, which is inconsistent with fixed trajectory replay. These results recast a world generator as a source of predictive control representations: its future need not be materialized at every step if its internal structure can be enfolded into the present.

1 Introduction

Enfold transfers a world generator’s future-construction computation into a current-only representation that supports both future generation and direct action prediction. It delivers strong control and substantially lower action latency without executing the generator during deployment.

  • Motivation: Effective control requires anticipating scene changes, and world generative models provide long-range motion, object interaction, and multiple plausible future continuations.These capabilities motivate using future modeling as a substrate for embodied control.
  • Problem: Enfold asks whether a world generator’s internal computation can be transferred into a current-only representation supporting both control and future generation.Prior predictive approaches avoid rendered futures but do not establish whether future-constructing computation itself can be internalized.
  • Representation: Generator states organize appearance, spatial layout, and interaction structure across abstraction levels but depend on unavailable futures and vary with generation noise.The framework treats these internal states as structured supervision rather than directly usable control representations.
  • Method: Enfold trains a current-only predictive encoder supervised by multi-level generator states, then uses the representation for future generation and stop-gradient task readouts.The representation is inferred from the present, so deployment need not execute the generator to obtain it.
  • Results: 97.8% success on LIBERO and 91.77% on RoboTwin2.0 are achieved by Enfold, while 134 ms per action chunk is 3.7× faster than Fast–WAM.Enfold-Flash reaches 97.5% on LIBERO, 92.02% on RoboTwin2.0, and 49 ms per action chunk, or 10.1× faster.

2 Related Work

Prior work connects future information to robotic action prediction through future-aware VLA models, generative world-action models, and predictive representation learning. Enfold is positioned as obtaining predictive information from teacher-forced generator states rather than future encoders, executed generative paths, or coupled video-action prediction.

  • Research taxonomy: Related work organizes future-aware control by how future information reaches action prediction: through VLA models, world-action models, or predictive representations.This taxonomy frames the three research lines reviewed in the section.
  • Future-aware VLA models: Future-aware VLA methods extend direct observation-and-instruction-to-action mapping with future queries, latent actions, visual subgoals, or auxiliary prediction.Standard VLA objectives leave environmental dynamics implicit in action supervision.
  • Generative world-action models: World-action models use predicted scene evolution for action generation, either by imagining visual futures before acting or by jointly modeling videos and actions.Other approaches reduce explicit rollout through auxiliary future objectives.
  • Predictive representation learning: Predictive representation learning models temporal structure in feature space, including future target-encoder features, latent transition variables, and pretrained visual dynamics spaces for planning.These representations support VLA pre-training, transfer, and alignment with executable robot actions.
  • Positioning of this work: Enfold differs by training on multi-level states from a teacher-forced generator, rather than future-encoder features, executed generative paths or caches, or coupled video-action prediction.The comparison distinguishes where predictive information is obtained and how it enters control.

3 Motivation: Why Generator States?

Generator states provide a supervision space grounded in realized futures while avoiding the need to preserve every output pixel. Diagnostics show that utility, stability, and selectivity vary across generator depths and corruption levels, motivating a multi-depth target for Enfold.

  • Why generator states: Generator states expose how appearance, spatial configuration, and interaction structure progressively organize into a coherent trajectory without equally weighting every output pixel.They are obtained while processing an observed future across corruption levels and network depths.
  • Target requirements: Generator-state targets must be informative, predictable from the present, and non-degenerate because they depend on the realized future and sampled noise.The paper examines this trade-off before defining Enfold.
  • Diagnostic findings: No generator block dominates across the corruption trajectory: the most discriminative block changes with t, and depths occupy different utility–stability trade-offs.Late states can remain semantically informative while becoming sensitive to sampled noise.
  • Diagnostic findings: Deeper states increasingly suppress illumination changes while remaining responsive to task-relevant layout, but may become more coupled to stochastic generation.Selectivity and predictability do not improve monotonically together, so different levels expose complementary aspects of the transition.
  • Design implication: Enfold therefore constructs its target by selecting and concatenating states from several generator depths, with a timestep-conditioned head accounting for their expression at each t.Concatenation preserves complementary levels before incorporation into the G2R and R2G objectives.

4 Method

Enfold trains a current-only predictive encoder to internalize multi-level future-generative structure, while separating generator conditioning and task readouts through stop-gradient paths. At inference, it predicts actions directly from the encoder representation without executing the video generator.

  • Predictive representation: The predictive encoder maps current video context and language instruction to a representation z supervised by selected generator hidden states from the observed future.The target comes from a teacher-forced forward pass over the real future, with timestep conditioning handled by a separate prediction head.
  • Coupled training, separated optimization: G2R transfers future-conditioned dynamics into z, while R2G tests whether detached z improves future generation; task heads read the same detached representation.The forward paths are coupled computationally but separated during optimization, so task and generation losses do not directly reshape the encoder.
  • Gradient routes: The generator is trained only by future modeling, the predictive encoder and prediction head only by G2R, and each task head only by its readout loss.These gradient routes make downstream performance a readout of generator-mediated learning rather than action-gradient adaptation of the encoder.
  • Inference: At inference, Enfold evaluates the encoder once and generates action chunks directly from z, while future imagination remains available through Gθ but is not executed for control.Enfold-Flash keeps the same model and inference path and adds TensorRT operator-level acceleration.

5 Experiments

Across simulation and real-robot benchmarks, Enfold enables accurate control without video rollout, with latency gains extending through Enfold-Flash. Interventions, ablations, and representation analyses indicate that generator-state supervision produces an adaptive, nuisance-suppressing predictive interface.

  • LIBERO: 97.8% average LIBERO success is achieved at 134 ms per action, improving Fast–WAM by 0.2 points with a 3.7× speedup.Enfold also reaches 100.0% on LIBERO-Object and 97.4% on LIBERO-Long while using half as many parameters.
  • Efficiency: 49 ms latency makes Enfold-Flash 10.1× faster than Fast–WAM, with only a 0.3point reduction in success.Standard Enfold avoids future-video rollout through its representation-level interface; Enfold-Flash additionally accelerates operators.
  • RoboTwin2.0: 92.02% average RoboTwin2.0 success for Enfold-Flash slightly exceeds Enfold’s 91.77% while retaining performance in clean and randomized scenes.The result supports transfer of the representation-level interface to dual-arm manipulation without video rollout at control time.
  • Real-robot tasks: 89.7% average real-robot completion exceeds Fast–WAM by 11.9 points and π0.5 by 3.6 points, with gains concentrated on deformable or evolving configurations.Enfold’s gains over Fast–WAM persist OOD, where Enfold scores 76.6%, 6.6 points higher than Fast–WAM, while π0.5 reaches 83.3%.
  • Counterfactual adaptation: Intervened scenes redirect imagined and executed continuations toward displaced objects or altered cloth geometry, departing from pre-intervention trajectories.The coordinated adaptation provides qualitative evidence against fixed trajectory replay and for constructing a goal-compatible continuation from the alternative present.
  • Ablations: 97.8% average success from multi-level generator-state supervision exceeds single-level supervision by 1.5 points and action-only training by 2.9 points.Single-level supervision reaches 96.3%, compared with 96.0% for future-pixel prediction and 94.9% for action-only training; adding z also improves PSNR, SSIM, and LPIPS.
  • Representation analysis: 7.9–10.4× lower nuisance sensitivity and stronger long-horizon feature alignment show that Enfold suppresses unreliable variation while organizing predictive interaction structure.Sensitivity falls to 0.020, effective rank reaches 31.8 versus at most 10.7 for generator features, and residual cosine error reductions are largest at t = 16 and t = 32.

6 Discussion and Conclusion · A A Predictive-Projection Interpretation of G2R · B Experimental Details

Enfold reframes the generator as a training-time source of structured predictive supervision, transferring future-conditioned internal organization into a current-only representation for efficient generation and control. Its predictive-projection interpretation explains that G2R extracts context-predictable structure while leaving unresolved future and stochastic variation as residual.

  • 6 Discussion and Conclusion: Enfold predicts future-conditioned generator states from current observations and instructions, making the generator’s organization of future change available before deployment-time generation.The representation is not shaped by action gradients and remains useful for future generation and control.
  • 6 Discussion and Conclusion: Human intervention causes coordinated adaptation in imagined continuations and executed actions, suggesting recomputation from the altered scene rather than memorized trajectory replay.The intervention evidence is qualitative.
  • 6 Discussion and Conclusion: Enfold shifts the generator from an action-time simulator to a training-time source of structured predictive supervision while internalizing generative computation for efficient control.The generator remains required during training, but explicit imagination is not required at every action step.
  • A A Predictive-Projection Interpretation of G2R: The projection interpretation is idealized: finite capacity, changing target branches, and SmoothL1 make the learned target a robust conditional center rather than exactly the conditional mean.The quadratic regime approaches mean regression while limiting the influence of large target deviations.
  • A A Predictive-Projection Interpretation of G2R: Timestep conditioning and concatenated generator depths let a shared current-only state support predictions across generative timesteps and abstraction levels.The prediction head handles timestep-specific coordinates, while multiple depths provide complementary structure.
  • A A Predictive-Projection Interpretation of G2R: Because the deployment representation is a deterministic function of current input, R2G gains reflect reorganized accessible predictive structure, not newly acquired sample-specific future information.Detached task heads test whether this organization remains actionable without task gradients defining it.
  • A A Predictive-Projection Interpretation of G2R: At population level, G2R learns the component of generator computation that covaries predictably with the current scene and instruction, rather than copying unconditional features.Unresolved future realizations, corruption noise, and other unpredictable target components remain in the residual.

B.1 Benchmarks and Model Configuration

Enfold is evaluated on LIBERO, RoboTwin2.0, and real-robot tabletop tasks using standardized demonstrations, multiple scene configurations, and embodiment-specific visual inputs. Its policy predicts 32-step action chunks and corresponding video futures with temporally compressed processing and flow-matching action sampling.

  • Benchmarks: LIBERO uses four 10-task suites with 1,712 training demonstrations and 50 trials per task.No-operation transitions are removed before training.
  • Benchmarks: RoboTwin2.0 provides 50 dual-arm tasks, trained with 27,500 demonstrations and evaluated in clean and randomized scenes.The benchmark includes structured domain-randomization axes.
  • Benchmarks: The real-world dataset contains 400 demonstrations from four tabletop tasks collected with an AgileX dual-arm platform.The real-robot policy uses one frontal 256 × 320 visual view.
  • Backbones and feature routing: All embodiments use DINOv3 ViT-H+/16 visual features, with language features from Cosmos-Reason1-7B linearly projected into the DINO stream from block 8.LIBERO uses two views at 224 × 448, while RoboTwin2.0 uses front and bilateral wrist views at 384 × 320.
  • Temporal processing and inference: Each observation conditions a 32-step action chunk and video sequence, reduced by stride-4 subsampling to 1 + 8 frames and by a causal VAE to 1 + 2 latent frames.Actions use continuous flow matching from Gaussian noise and a 10-step Euler solver with noise shift 5.0.

B.2 Control Evaluation Protocols

The protocol evaluates Enfold on a real dual-arm robot with receding-horizon action chunks and staged-completion scoring. It compares methods under matched in-distribution and controlled out-of-distribution settings using repeated rollouts and per-task success reporting.

  • Real robot: On a tabletop AgileX dual-arm platform, Enfold encodes each latest RGB observation and language instruction, predicts 32 actions, executes 10, then replans.The setup uses one fixed frontal camera with images resized to 256 × 320 and includes 400 teleoperated demonstrations across four tasks.
  • Evaluation protocol: 30 independent rollouts are run for every method, task, and evaluation setting under the staged-completion protocol.Each rollout receives a normalized completion score based on the number of ordered stages completed, with s_i ∈ {0, 1, 2, 3}.
  • Evaluation protocol: Full task success corresponds to a score of 3, while lower scores preserve partial progress and terminated rollouts retain their last completed stage.Timeouts and safety-triggered terminations do not cause rollouts to be discarded.
  • Evaluation settings: ID trials sample poses and workspace configurations from training ranges, whereas OOD trials change one factor while holding instructions and completion criteria fixed.Configuration shifts modify towel geometry, object appearance, or workspace background.
  • Evaluation settings: Within each setting, all methods share the initial-state sampler, trial allocation, camera observations, and termination limits.Per-task success rates are reported under clean and randomized RoboTwin2.0 evaluation settings.

B.3 Future-Video Evaluation · B.4 Representation Analyses · B.5 Human-Intervention Visualization

The evaluation covers future-video prediction, representation stability and probing, token-level geometry, and qualitative adaptation after human intervention. These analyses compare Enfold with video-generation features and frozen DINO under controlled clips, real-world episodes, and altered task scenes.

  • B.3 Future-Video Evaluation: Enfold and the video-only Cosmos baseline use matched training and evaluation settings for predicting 32 future frames from the first frame and language instruction.Both models train for 21.7K optimization steps and are evaluated on 100 fixed LIBERO demonstration clips with 35 denoising steps; PSNR, SSIM, and LPIPS are computed over predicted frames.
  • B.4 Representation Analyses: Representation stability compares Enfold’s final predictive encoder with Cosmos layers L7, L15, L23, and L27 at t = 600 across 100 clips and two noise seeds.Text conditioning is disabled; lighting sensitivity is normalized by mean displacement across different input clips, while diversity uses one pooled feature per clip.
  • B.4 Representation Analyses: The representation analysis uses 100 clip-level features for task/scene retrieval and explicitly separates this evaluation from the 20-video generator-state probe in Appendix C.The stated feature construction uses Equation 25 and is bounded by 99.
  • B.4 Representation Analyses: A changed-region diagnostic evaluates both input representations at identical patch locations selected by the largest future DINO displacement.The diagnostic uses k = 64 patches and displacement 1 − cos(zD_0,j, zD_t,j).
  • B.4 Representation Analyses: Token-similarity visualization compares how frozen DINO and the instruction-conditioned Enfold encoder relate a manually selected gripper query patch to every frame token.The comparison uses cosine similarity independently for each encoder and observation.
  • B.5 Human-Intervention Visualization: Human-intervention episodes test qualitative adaptation by relocating the plate or displacing the cloth while keeping the language instruction unchanged.The updated representation is computed at normal control frequency; the middle panel is decoded only for visualization, while physical actions are predicted directly from the representation, and the examples are qualitative rather than evaluation data.
  • B.5 Human-Intervention Visualization: PCA visualization contrasts frozen DINO and Enfold token geometry across rollout observations by mapping the first three principal components to RGB.Frozen DINO separates the robot arm and task objects into visually distinct feature regions, whereas Enfold makes the gripper and the object involved in the current manipulation more similar in the representation.

B.6 Additional Qualitative Results … C.2 Feature Extraction

Enfold’s qualitative analyses show that generator-mediated training reorganizes representations around manipulation-relevant interactions and improves prediction of future features at changing contact regions. The generator-state probe evaluates these representations across controlled tasks, illumination variants, noise seeds, corruption levels, and block depths using pooled teacher features and cosine displacement.

  • B.6 Additional Qualitative Results: Enfold reorganizes token geometry around gripper–object interaction rather than only improving appearance or object-identity discrimination.PCA maps show the gripper and manipulated object becoming more similar near manipulation, complementing gripper-query similarity overlays.
  • B.6 Additional Qualitative Results: At future horizons t = 16 and t = 32, predicted features retain coarse spatial structure around the moving gripper, cloth, and their interaction.The prediction-head visualization directly shows alignment with teacher representations at both displayed horizons.
  • B.6 Additional Qualitative Results: Enfold’s future-feature gains concentrate on moving arms, manipulated cloth, and contact-relevant regions, with advantages increasing at longer horizons.Most selected tokens have positive future-feature cosine-score differences over frozen DINO, indicating more accurate future-feature prediction from Enfold representations.
  • B.6 Additional Qualitative Results: FastWAM predicts false object-in-gripper states in LIBERO and Fold Towel, causing grasping or folding actions to proceed before contact is established.These artifacts can make the policy act on states that have not occurred, preventing successful pickup or folding.
  • C Generator-State Probe: The generator-state probe compares candidate supervision targets across generator depth and corruption level without evaluating complete video rollouts.The probe is designed to characterize internal generator states rather than end-to-end future-video quality.
  • C.1 Setup: The probe uses 20 videos from four tabletop manipulation tasks, three illumination variants, two noise seeds, and six timesteps, totaling 720 teacher forward passes.All samples use the neutral prompt “A robot arm operating on a tabletop.” and contain 33 frames resized to 256 × 320.
  • C.1 Setup: Videos are evaluated at six corruption levels using VAE latents, while the first latent temporal slice remains the clean conditioning frame.Each probe point uses one teacher evaluation at a fixed timestep rather than an iterative denoising trajectory.
  • C.2 Feature Extraction: Teacher passes expose every generator block, while global geometry uses token-mean features and Enfold’s G2R target retains spatio-temporal tokens.Feature displacement is measured with cosine distance; pooling is used only to characterize global feature geometry.

C.3 Metrics

The metrics quantify representation sensitivity to generation noise, illumination, and layout, alongside task/scene retrieval and effective rank. Interpretations account for input dispersion, probe size, stochastic noise, and descriptive aggregation choices.

  • Sensitivity metrics: Noise sensitivity measures feature displacement from changing only diffusion noise, with lower ρseed indicating greater stability to sampled generation noise.Distances at t = 0 are clipped to zero.
  • Sensitivity metrics: Illumination sensitivity uses seed-matched original/dark and original/bright pairs, where lower values indicate stronger invariance to brightness perturbation.The comparison uses S = 2 seeds.
  • Sensitivity metrics: Layout sensitivity captures differences between desktop episodes with distinct object arrangements and is not minimized because arrangement may matter for manipulation.Figure 3 compares ρlayout with ρlight across depth.
  • Retrieval: Task/scene retrieval reports mean average precision when each feature retrieves 19 other videos, with the four same-task videos treated as positives.The metric is not called action recognition because task identity correlates with objects and scene layout.
  • Generator-state probe: Effective rank measures how broadly variation spreads across feature directions, reaching at most 19 in the 20-video probe and 99 in the separate 100-clip comparison.Raw values across the two sample sets are not compared, and effective rank is interpreted jointly with ρseed because stochastic noise can increase rank.
  • Figure aggregation: Figures sweep retrieval and ρseed over depth and timestep, show their trade-off at t = 600, and summarize sensitivity responses across t ∈{300, 600, 800, 900} descriptively.Figure 3 uses interquartile ranges across the four timesteps; these statistics are not used for formal significance testing.

C.4 Scope and Limitations

The representation probes reveal depth- and timestep-dependent trends but are limited by small-scale evaluation, confounds, and simplified interventions. Their measurements require cautious interpretation because pooling discards spatial structure, effective rank is sample- and noise-sensitive, and fixed corruption levels do not form a complete denoising trajectory.

  • Probe scope: The probe uses 20 videos, exposing depth- and timestep-dependent trends without establishing a precise layer ranking.Task retrieval is confounded with object and scene identity.
  • Intervention limits: The illumination intervention changes global intensity and contrast but omits shadows, color temperature, and view-dependent reflectance.The layout response is estimated from one episode pair.
  • Probe interpretation: Token averaging suppresses local spatial structure, so weak pooled responses do not show that a layer lacks useful patch-level information.The probe therefore cannot directly assess all spatially localized information.
  • Probe interpretation: Effective rank depends on sample count and can be inflated by noise, so comparisons are restricted to a shared evaluation set and interpreted with seed sensitivity.These measurements motivate cautious interpretation of representation differences.
  • Probe interpretation: The six fixed corruption levels are single forward passes rather than a complete denoising trajectory.These probes motivate the multi-level target construction.
Loading 2607.26657v3…