Source-linked AI summary
ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Training
Haian Jin, Rundi Wu, Tianyuan Zhang, Ruiqi Gao, Jonathan T. Barron, Noah Snavely, Aleksander Holynski
TL;DR
Large-image 3D reconstruction is hindered by quadratic attention costs, while sequential approaches can lose quality. ZipMap uses Test-Time Training to compress an image collection into a compact state for linear-time bidirectional reconstruction, matching or surpassing quadratic-time methods and reconstructing over 700 images in under 10 seconds.
Problem
Quadratic attention makes state-of-the-art feed-forward reconstruction inefficient for large image collections, while sequential approaches reduce this cost at the expense of reconstruction quality.
Method
ZipMap uses Test-Time Training layers to compress all image tokens into compact fast weights in one forward pass for linear-time bidirectional reconstruction.
Results
Over 700 images are reconstructed in under 10 seconds at 75FPS, over 20× faster than VGGT with comparable or superior accuracy.
Takeaways & Limitations
The queryable scene state supports real-time novel-view geometry and appearance prediction and extends to sequential streaming reconstruction.
Takeaways & Limitations
ZipMap’s deterministic scene representation cannot hallucinate rich high-frequency details or entirely unseen objects, though it can extrapolate basic structures beyond observed views.
Abstract
from arXiv · showhide
Feed-forward transformer models have driven rapid progress in 3D vision, but state-of-the-art methods such as VGGT and $π^3$ have a computational cost that scales quadratically with the number of input images, making them inefficient when applied to large image collections. Sequential-reconstruction approaches reduce this cost but sacrifice reconstruction quality. We introduce ZipMap, a stateful feed-forward model that achieves linear-time, bidirectional 3D reconstruction while matching or surpassing the accuracy of quadratic-time methods. ZipMap employs test-time training layers to zip an entire image collection into a compact hidden scene state in a single forward pass, enabling reconstruction of over 700 frames in under 10 seconds on a single H100 GPU, more than $20\times$ faster than state-of-the-art methods such as VGGT. Moreover, we demonstrate the benefits of having a stateful representation in real-time scene-state querying and its extension to sequential streaming reconstruction.
1. Introduction
ZipMap targets the quadratic scaling and quality trade-offs of existing 3D reconstruction methods with a linear-time, bidirectional, stateful feed-forward model. It compresses image collections into a queryable hidden scene state while retaining competitive reconstruction fidelity and substantially improving efficiency.
- Quadratic global attention makes VGGT-like reconstruction computationally prohibitive for long image sequences, while sequential or local methods often reduce reconstruction quality.
- ZipMap combines feed-forward transformer principles with Test-Time Training layers to reconstruct large image sets using a bidirectional architecture.
- Linear complexity lets ZipMap process large image collections in seconds while matching or exceeding the fidelity of quadratic-time state-of-the-art systems.
- Test-Time Training compresses an entire image collection into compact MLP fast-weights in a single forward pass instead of using global attention across all tokens.
- The hidden scene state supports efficient globally coherent aggregation, real-time novel-view geometry and appearance queries, and sequential streaming reconstruction.
- Over 700 images can be reconstructed in under 10 seconds at 75FPS, more than 20× faster than VGGT while delivering comparable or superior accuracy.
2. Related Work
Prior large-scale 3D reconstruction methods include classical SfM pipelines, global methods, and linear-scaling sequence models, each with practical limitations. ZipMap uses a stateful, non-recurrent design based on Test-Time Training to combine linear scaling with a compact queryable scene representation.
- Classical SfM can support city-scale reconstruction and high accuracy, but typically produces sparse outputs, requires substantial image overlap, and includes time-consuming multi-view stereo.
- Sequential image processing achieves linear scaling but often sacrifices reconstruction quality, motivating a linearly scaling stateful model without recurrent processing.
- Test-Time Training treats model parameters as fast-weight memory updated by gradient descent to capture in-context information.
- Building on LaCT, ZipMap updates nonlinear fast weights over large token chunks to compress large image inputs into a compact, queryable scene representation.
3. Method
ZipMap combines local attention with large-chunk test-time training to reconstruct scenes bidirectionally in linear time while maintaining a compact, queryable scene state. Its fast weights support both novel-view queries and online streaming updates.
- Architecture: ZipMap processes N images in one feed-forward pass to predict camera poses, depth maps, and point maps while forming an implicit scene representation.The representation is stored in adapted fast weights and can be queried using target ray maps.
- Architecture: Local window attention captures within-view relationships, while a global large-chunk TTT layer aggregates information across all input views.The backbone replaces global attention with TTT to preserve global context at linear cost.
- Test-Time Training: TTT adapts SwiGLU-MLP fast weights once over key-value pairs from all image tokens, creating an in-context associative memory unrelated to the reconstruction loss.A single gradient update is followed by orthonormalization and L2 normalization before the updated weights process image tokens.
- Scene Queries: Applying the adapted fast-weight MLP to tokens provides a linear-complexity alternative to querying all key-value pairs through self-attention.The same weights can process target ray tokens with constant runtime per token, independent of the number of input views.
- Streaming Reconstruction: ZipMap extends to streaming reconstruction by updating fast weights online one view at a time, although the main paper focuses on bidirectional reconstruction.The streaming update uses the same virtual key-value objective computed from the current view’s visual tokens.
- Prediction Heads and Training: The model uses camera, point, depth, and query heads, with confidence-weighted depth training and scale-invariant local point reconstruction losses.The depth head also produces a confidence map, while query losses are enabled during finetuning against target images and depths.
4. Experiments
ZipMap matches or surpasses quadratic-time models across camera pose, point-map, and depth tasks while scaling linearly and reconstructing long sequences efficiently. Its stateful representation also supports real-time querying and plausible inference of unseen common scene structure.
- Benchmark Evaluation: ZipMap matches or surpasses state-of-the-art quadratic-time models across camera pose, point-map, and depth estimation while retaining linear computational complexity.It is comparable to VGGT and π3 for camera pose, matches or exceeds them for point maps, and generally exceeds VGGT for video depth.
- Efficiency and Scalability: Over 700 frames are reconstructed in 10 seconds on a single H100 GPU, while VGGT requires over 200 seconds.ZipMap is also about 3× faster than CUT3R and TTT3R, which process frames sequentially.
- Efficiency and Scalability: ZipMap maintains low camera-pose error as scene scale or view density increases, matching π3 and VGGT while other linear-time methods degrade.The DL3DV evaluation varies either the first N frames or uniformly subsampled frames along a fixed trajectory.
- Ablation Studies: Newton–Schulz normalization, the gated unit, and dynamic per-token learning rates are important TTT components because removing or replacing them degrades performance.Dynamic per-token learning rates outperform fixed global rates of 0.1 or 1.0.
- Ablation Studies: Removing the reference view improves accuracy and generalization on long input sequences but provides no clear or consistent standard-benchmark advantage.The final training stage uses an affine-invariant loss instead of explicit reference-view selection.
- Implicit Scene Representation: The hidden scene state can be queried at approximately 100 FPS to produce novel-view RGB, depth, and colored point clouds closely matching input-based reconstructions.It can infer common unseen structures such as walls, floors, and ground, but not rich high-frequency details or entirely unseen objects.
5. Conclusion
The paper presents ZipMap as a linear-time stateful architecture for scalable 3D reconstruction. Across benchmarks, it combines competitive accuracy with substantially faster processing and enables real-time scene-state querying and streaming reconstruction.
- ZipMap is a stateful bidirectional feed-forward architecture that scales linearly for 3D reconstruction.
- Across benchmarks, ZipMap matches or surpasses quadratic-time models while being substantially faster.
- Its learned scene state supports real-time novel-view point-map prediction and extension to streaming reconstruction.
Outline
The supplementary material organizes evaluation, training, implicit-scene-state results, and further analyses, while documenting direct baseline evaluation and input-resolution settings.
- Appendix A provides evaluation details, including baseline, runtime, and long-sequence evaluation protocols.
- Appendix B describes training datasets, the complete loss function, and implementation details for scene-state querying and streaming reconstruction.
- Appendix C provides visualizations of querying the learned implicit scene state.
- Evaluation Details: The authors directly evaluate CUT3R, TTT3R, VGGT, π3, and ZipMap, while using π3-reported results for other baselines.For the directly evaluated methods, input widths are 512 pixels for CUT3R and TTT3R and 518 pixels for VGGT, π3, and ZipMap.
A.2. Runtime Evaluation Details
Runtime evaluation measures inference on one H100 GPU across sequences up to 750 frames, using standardized implementations and warm-up procedures. ZipMap is slightly slower on sparse five-frame inputs but is faster than quadratic methods as input density grows, while scene-state queries reach about 100 FPS.
- Hardware and setup: All methods were evaluated on a single H100 SXM5 GPU with PyTorch 2.7.1 and CUDA 12.8.Input resolutions matched the ScanNet-v2 pose-estimation setting, with method-specific patch sizes.
- Evaluation range: 750 frames was the maximum evaluated sequence length, near the 80GB memory limit of both ZipMap and the baseline.Runtime was averaged over 10 iterations after two warm-up iterations.
- Runtime behavior: 5 frames: ZipMap is slightly slower than VGGT and π3 on very sparse inputs.The reported explanation is standard PyTorch TTT code and Newton–Schulz orthonormalization versus fused FlashAttention kernels in the quadratic baselines.
- Scene-state queries: 100 FPS: scene-state querying is faster than reconstruction because it applies TTT blocks without an update step.The query operation uses the learned state directly rather than performing another state update.
- Protocols: Long-sequence tests used the first N frames for ScanNet-v2 camera and depth evaluation, with up to N = 750 frames.The 3D point-estimation protocol used either the first N frames or uniformly subsampled frames on 7-Scenes.
B.1. Full Training Datasets
The supplementary material combines broad multi-dataset training and implementation details with runtime context and procedures for scene-state querying and streaming reconstruction. Training spans static and dynamic datasets, while later adaptations impose explicit context and data-selection conditions.
- Training datasets: 29 publicly available datasets support training, including 23 static-scene datasets and six dynamic-scene datasets.The listed collection includes datasets such as ScanNet, Matterport3D, Waymo, BEDLAM, Dynamic Replica, and Kubric.
- Runtime evaluation: Table 7 compares inference time versus input-image count, with VGGT and π3 scaling quadratically and ZipMap scaling linearly.The table caption states that ZipMap is fastest for dense input frames.
- Training losses: Training adds normal supervision for local point maps and gradient regularization for locally smooth predicted depths.The normal target is computed from adjacent edges in predicted and ground-truth point maps.
- Implementation: Training uses FSDP and torch.compile for the TTT block, with color and grayscale augmentations plus geometric normalization.Ground-truth cameras, depths, and local points are normalized using global point-cloud scale.
- Scene-state querying: Scene-state-query finetuning uses a reference-view coordinate system and fixes target-camera scale for training stability.The procedure adds query losses while modifying data selection and color augmentation for photometric consistency.
- Streaming reconstruction: Streaming reconstruction replaces the transformer camera head with a lightweight two-layer MLP and finetunes on 24-view contexts.The setup notes that streaming baselines use contexts up to 64 views, motivating further context scaling.
C. More Results for the Implicit Scene State
Additional results show that ZipMap’s implicit scene state supports novel-view RGB and depth queries whose reconstructed point clouds closely match those from input images. The supplementary evaluations also cover monocular depth, qualitative reconstruction, reference-view ablations, and streaming variants.
- Unseen structure: The scene state can infer basic structures beyond observed views, but deterministic predictions do not recover rich high-frequency details or entirely unseen objects.Walls, floors, and ground can be extrapolated beyond observed regions, whereas an unseen sofa may be missing.
- Implicit scene state: Novel-view RGB and depth queries produce point clouds closely resembling those reconstructed directly from input images.This visual match indicates that the learned state captures scene geometry and appearance.
- Monocular depth estimation: Monocular depth: ZipMap consistently outperforms VGGT and π3 across four benchmarks and performs comparably to MoGe.The model was not trained with purely monocular input.
- Qualitative comparison: Qualitative comparisons cover DTU and ETH3D, with corresponding quantitative results reported in the main paper.The supplementary figure provides visual reconstruction comparisons on both datasets.
- Reference-view ablation: Removing the reference view improves accuracy for long input sequences after switching to an affine-invariant camera loss.The improvement is shown for long-sequence camera estimation, while its effect on standard benchmarks is limited.
- Streaming reconstruction: Streaming reconstruction updates the TTT-based scene state one view at a time and generally outperforms CUT3R and TTT3R across three task types.The reported tasks are point-map reconstruction, video depth, and camera pose estimation.
D.6. More Long-Sequence Evaluation
The supplementary long-sequence evaluation extends video-depth and 3D-point experiments as sequence length, scene scale, or view density increases. The protocols use first-frame prefixes or uniform subsampling to vary the input regime.
- Video depth estimation: Long-sequence video-depth evaluation uses the first N frames of each ScanNet-v2 test sequence as N increases.Figure 9 reports the protocol for progressively longer inputs.
- 3D point estimation: Long-sequence 3D point estimation on 7-Scenes varies scene scale with first-N-frame prefixes and view density with uniform subsampling.The two cases distinguish extending the trajectory from increasing views over a fixed trajectory.
E. Limitations
ZipMap has two reported limitations: performance degrades on very long sequences beyond the training distribution, and novel-view RGB rendering can be blurry in high-frequency regions. The authors position novel-view querying primarily as a colored-point-cloud capability rather than state-of-the-art unposed view synthesis.
- E. Limitations: Very long sequences whose scene scale extends beyond the training distribution cause noticeable performance degradation.The authors note that this limitation is shared by existing feed-forward methods and suggest longer-context training or global alignment as possible directions.
- E. Limitations: ZipMap does not claim state-of-the-art unposed novel-view synthesis because its queries are primarily designed to generate colored point clouds.Novel-view synthesis is explicitly described as outside the method’s main focus, particularly relative to explicit 3DGS-based methods.
- E. Limitations: Novel-view RGB queries can exhibit blurry artifacts in high-frequency regions.The authors attribute this partly to a mismatch between ground-truth pose conditioning and implicit geometry learned from unposed images, which produces noisy RGB supervision.