Source-linked AI summary
Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model
Dongwon Kim, Gawon Seo, Jinsung Lee, Minsu Cho, Suha Kwak
TL;DR
World-model planning is hindered by the hundreds of tokens used to represent each observation, especially when attention-based models repeatedly roll out futures. CompACT compresses observations into 8 or 16 discrete semantic tokens and uses generative decoding for perceptual detail, achieving competitive planning with 40× faster planning. Its compression assumes that planning needs semantic and spatial information more than fine-grained visual detail.
Problem
Conventional world-model tokenizers encode observations into hundreds of tokens, making decision-time planning computationally expensive for real-time control.
Method
CompACT uses a frozen vision foundation model to produce 8 or 16 discrete semantic tokens and a generative decoder to synthesize omitted perceptual details.
Results
40× planning speedup is achieved while maintaining comparable navigation accuracy to a model using 784 continuous tokens, and reconstruction quality remains competitive with recent tokenizers.
Takeaways & Limitations
World models trained in CompACT’s compact latent space can preserve planning-relevant information while substantially reducing planning computation.
Takeaways & Limitations
CompACT necessarily discards fine-grained visual details because its 128-bit representation cannot preserve all information in an image.
Abstract
from arXiv · showhide
World models provide a powerful framework for simulating environment dynamics conditioned on actions or instructions, enabling downstream tasks such as action planning or policy learning. Recent approaches leverage world models as learned simulators, but its application to decision-time planning remains computationally prohibitive for real-time control. A key bottleneck lies in latent representations: conventional tokenizers encode each observation into hundreds of tokens, making planning both slow and resource-intensive. To address this, we propose CompACT, a discrete tokenizer that compresses each observation into as few as 8 tokens, drastically reducing computational cost while preserving essential information for planning. An action-conditioned world model that occupies CompACT tokenizer achieves competitive planning performance with orders-of-magnitude faster planning, offering a practical step toward real-world deployment of world models.
1. Introduction
World models support internal simulation and planning, but conventional perceptual representations make decision-time planning too expensive. CompACT addresses this bottleneck by compressing images into a few discrete tokens while preserving planning-relevant information.
- World models predict future states to reduce environment interactions and support planning through model-predictive control.
- Photorealistic generative models encode images into hundreds of tokens, increasing attention-based planning cost quadratically.
- CompACT encodes each image into as few as 8 discrete tokens, approximately 128 bits per image.
- CompACT preserves high-level semantics in compact tokens while synthesizing perceptual details during generative decoding.
- 40× planning speedup is achieved on RECON navigation while maintaining comparable accuracy to a model using 784 continuous tokens.
- On RoboNet, CompACT supports action regression comparable to tokenizers using 16× more tokens and maintains strong action consistency.
2. Related Work
Prior work uses tokenizers and world models for visual generation, prediction, and planning, but real-time deployment remains limited by computational cost. CompACT targets this limitation with unconditional extreme compression for planning across navigation and video prediction.
- Image tokenization reduces the difficulty of modeling high-dimensional visual distributions.
- Conventional patch-grid tokenizers tie token count to resolution, while 1D approaches allow flexible lengths but may not explicitly preserve spatial structure.
- Foundation-model-based tokenizers use pretrained features primarily to improve downstream generative tractability, whereas CompACT targets extreme compression.
- Some discrete world models reduce per-frame tokens by conditioning on previous frames, limiting long-horizon planning or viewpoint-change scenarios.
- Masked generative models reconstruct multiple masked tokens per step, reducing sampling steps relative to autoregressive generation.
- CompACT is evaluated in goal-conditioned visual navigation and action-conditioned video prediction to address computational challenges across both planning paradigms.
3. Method
CompACT builds a latent world model around extremely compact discrete representations, preserving semantic information for planning and generating perceptual detail only when needed. The resulting model rolls out efficiently and supports goal-directed action optimization.
- Latent world model: The latent world model predicts stochastic future latent states conditioned on current latent tokens and actions.
- Latent world model: Keeping the token count N small reduces quadratic attention cost and enables efficient decision-time planning.
- Decision-time planning: Planning rolls out candidate action sequences, evaluates final predicted states against a goal, and optimizes actions using a distance-based cost.
- CompACT tokenizer: CompACT encodes images into 16 or 8 discrete tokens and avoids iterative denoising through a discrete latent space.
- CompACT tokenizer: A frozen DINOv3 encoder and latent resampler distill semantic features into discrete tokens while discarding low-level visual details.
- Generative decoding: Generative decoding predicts hundreds of target-tokenizer latent tokens from compact tokens, replacing deterministic pixel reconstruction with conditional generation.
- Generative decoding: The tokenizer preserves high-level semantics in discrete tokens and synthesizes plausible high-frequency details during decoding.
4. Experiment
CompACT is evaluated for reconstruction, latent-token structure, inverse dynamics, planning, and action-conditioned video prediction. The experiments test whether extreme compression preserves planning-relevant information while reducing computational cost.
- Tokenizer evaluation: CompACT achieves reconstruction performance comparable to recent state-of-the-art tokenizers despite extreme compression.MaskGIT-VQGAN serves as the reconstruction-fidelity baseline, and CompACT also outperforms it in IS.
- Tokenizer ablations: Frozen semantic features and generative decoding are key design choices for maintaining reconstruction quality with compact tokens.Full DINOv3 finetuning produces rFID of 5.22, while replacing the generative decoder with a single-step feedforward decoder severely degrades reconstruction quality.
- Latent-token characterization: Each compact latent token attends to coherent, semantically meaningful regions, capturing modular object-level elements across ImageNet, RECON, and RoboNet.The attention structure emerges from frozen DINO features and focuses on objects or structural elements rather than fixed spatial regions.
- Latent-token characterization: CompACT latents yield superior inverse-dynamics performance to MaskGIT-VQGAN while using 16× fewer tokens.The IDM evaluates whether consecutive latent representations preserve dynamics-relevant state changes for manipulation control.
- Planning performance: CompACT reduces RECON planning latency by approximately 40× while maintaining comparable accuracy to the 784-token SD-VAE baseline.NWM models using 16 or 8 CompACT tokens also outperform FlexTok models using 16 or 64 tokens.
- Action-conditioned video generation: CompACT achieves 3× lower action prediction error and 5.2× faster generation than the 256-token baseline on RoboNet video prediction.Generated videos preserve action-driven end-effector movements more consistently than the target tokenizer.
5. Conclusion
CompACT represents images with 16 or 8 discrete tokens while preserving planning-critical information, enabling faster planning than larger-token baselines.
- CompACT represents images with only 16 or 8 discrete tokens while preserving planning-critical information.
- World models trained in CompACT’s compact latent space outperform larger-token baselines while achieving a 40× planning speedup.
Supplementary Material
The supplementary material provides implementation details and additional experiments covering theory, backbone ablations, latency, robotic manipulation, and tokenizer training.
- The supplementary material includes a theoretical framework for the planning sufficiency of compact tokens.
- It presents cross-backbone ablations and a planning latency breakdown for CompACT.
- It evaluates closed-loop robot arm manipulation on Robomimic and describes the complete tokenizer architecture and training procedure.
A. Planning sufficiency of compact tokens
The paper formalizes planning sufficiency as preserving the information needed to select optimal actions and derives an entropy lower bound for such representations. CompACT connects this theory to compact discrete tokens that retain action-relevant information empirically.
- Planning sufficiency requires latent-space planning to produce the same optimal action as observation-space planning.
- For deterministic optimal actions, a planning-sufficient representation has minimum entropy bounded by the mutual information between observations and optimal actions.
- The deterministic assumption concerns a unique optimal action for each observation, while CompACT’s CEM uses stochastic sampling only in the optimizer.
- CompACT encodes frames into 8–16 discrete tokens totaling 128–256 bits, while action spaces in the targeted tasks are low-dimensional.
- An inverse dynamics model using 16 CompACT tokens achieves R2 = 0.716, exceeding the baseline’s R2 = 0.684 with 256 tokens.
B. Cross-backbone ablation for CompACT
CompACT’s semantic tokenizer remains effective across distinct vision foundation model backbones, while its compact tokenization substantially reduces world-model rollout cost during planning.
- Cross-backbone ablation: CompACT replaces DINOv3 with MAE and SigLIP-2 while keeping all other components unchanged.
- Planning latency breakdown: The latency breakdown compares world-model rollout and decoding on a single RTX 6000 ADA GPU, using a log-scaled x-axis.
- Cross-backbone ablation: All three backbones yield competitive rFID, with SigLIP-2 surpassing DINOv3.
- Planning latency breakdown: SD-VAE’s rollout dominates its 177.70s planning time, consuming 177.05s across 1,920 forward passes over 784 tokens per frame.
- Planning latency breakdown: CompACT reduces the rollout bottleneck by 99.2%, from 177.05s to 1.34s.
D. Closed-loop robot arm manipulation
CompACT supports closed-loop manipulation by planning in compact latent space and decoding inter-frame actions for execution. On RoboMimic Lift, it matches the target tokenizer’s success rate with fewer action steps, while the world-model designs support efficient action-conditioned rollouts.
- Closed-loop manipulation: CompACT plans in latent space and uses an inverse dynamics model to decode inter-frame actions for closed-loop execution on RoboMimic Lift.The experiment uses a hierarchical formulation in which the world model plans and the IDM produces executable actions.
- Closed-loop manipulation: 17% fewer action steps are required by CompACT (16 tokens) than by the target tokenizer (256 tokens) while matching its success rate.The result is reported as preliminary evidence that planning-sufficient compression extends to contact-rich manipulation.
- Inverse dynamics model: The inverse dynamics model encodes consecutive frame tokens and conditions an action denoiser to predict the action applied between them.For the target tokenizer, the conditioning signal contains 512 tokens; CompACT uses 32, a 16× reduction.
- World-model architecture: The manipulation world model predicts multiple future frames in parallel with a block-causal transformer conditioned on observations and action sequences.Block-causal masking preserves causal structure while allowing simultaneous processing of future frames.
- World-model architecture: CompACT processes 16 tokens per frame versus 256 for the target tokenizer, resulting in significantly faster action-conditioned video generation.The same compact representation is used in the manipulation world model’s sequence.
H. Planning with cross-entropy method
The planning procedure uses a trained world model to optimize goal-conditioned navigation trajectories with the Cross-Entropy Method. Candidate trajectories can be evaluated in pixel or discrete latent space, with latent-space evaluation avoiding decoding and retaining meaningful semantic similarity with marginal degradation.
- Cross-entropy planning: Cross-Entropy Method planning optimizes goal-conditioned navigation trajectories using stochastic world-model rollouts and candidate action distributions.The procedure follows the NWM protocol and evaluates sampled candidate trajectories against a goal observation.
- Goal evaluation: Planning costs can be computed in discrete latent space instead of pixel space, avoiding the decoding step.Pixel-space evaluation decodes tokens and uses LPIPS, whereas latent-space evaluation computes distances directly between discrete representations.
- Goal evaluation: The latent-space distance uses FSQ^-1 representations and provides meaningful semantic similarity with faster planning and marginal degradation compared with pixel-space metrics.The distance is defined as the L1 norm between the corresponding level-based radix representations.
I. More qualitative results
Additional qualitative results examine reconstruction, planning, and action-conditioned video generation. These materials emphasize how compact latent tokens support dynamics-relevant information for manipulation while the planning procedure selects low-cost action sequences.
- Qualitative reconstruction: CompACT reconstruction examples span ImageNet, RECON, and RoboNet, preserving semantic content and spatial structure while discarding fine-grained textures and lighting.Attention visualizations show compact tokens attending to semantically coherent regions.
- Navigation planning: The navigation planning procedure encodes initial and goal frames, samples candidate action trajectories, evaluates stochastic rollouts, and returns the minimum-cost action.The algorithm uses a population of candidates, iterative selection, and cost-based ranking.
- Navigation planning: The Cross-Entropy Method retains the top-K candidates with the lowest costs during iterative navigation planning.Candidate selection is based on the computed trajectory costs.
- Manipulation qualitative results: Compact latent tokens capture dynamics-relevant information for manipulation tasks.This conclusion accompanies the qualitative action-conditioned video and manipulation results.
J. Planning efficiency analysis
CompACT improves planning efficiency across accuracy, latency, memory, and model scaling analyses. Its compact representation enables substantial speedups while retaining comparable or improved navigation accuracy, including for a larger world model.
- Efficiency trade-offs: Up to 80× speedup over SD-VAE is achieved by CompACT while maintaining comparable planning accuracy on RECON.The analysis jointly compares ATE, planning latency, model size, and peak VRAM usage across tokenizers.
- Scaling up: The 750M-parameter CompACT world model achieves ATE of 1.305 and RPE of 0.370 on RECON, improving over the base 16-token model’s ATE=1.330 and RPE=0.390.The larger model increases depth to 24 layers and hidden dimension to 1024.
- Scaling up: 24.7 seconds per trajectory is required by the scaled model, remaining 7× faster than the SD-VAE baseline at 178.78 seconds.The comparison concerns planning latency on RECON.
- Evaluation setup: Planning latency and peak memory usage are measured for single-trajectory optimization on one RTX 6000 ADA GPU.Figure 14 reports the ATE, latency, and memory comparison under this setup.