PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast-SAM3D: 3Dfy Anything in Images but Faster

Weilun Feng, Mingqiang Wu, Zhiliang Chen, Chuanguang Yang, Haotong Qin, Yuqi Li, Xiaokun Liu, Guoxin Fan, Libo Huang, Yulun Zhang, Michele Magno, Yongjun Xu, Zhulin An

arXiv:2602.05293v2cs.CV

TL;DR

SAM3D offers high-quality open-world reconstruction but faces substantial computation overhead that hinders deployment. Fast-SAM3D profiles its heterogeneous inference dynamics and applies adaptive caching, token carving, and aggregation, achieving up to 2.67× speedup with negligible fidelity degradation.

  • Problem

    SAM3D’s strong single-image, open-world reconstruction quality is accompanied by substantial computation overhead, while its latency dynamics lack systematic characterization.

  • Method

    Fast-SAM3D is a unified, training-free framework that accelerates geometry, texture, and decoding through modality-aware caching, spatiotemporal token carving, and spectral-aware aggregation.

  • Results

    2.01× and 2.67× speedups are achieved for scene and object generation, respectively, while maintaining or exceeding the base model’s geometric fidelity.

  • Takeaways & Limitations

    Fast-SAM3D provides substantial acceleration across diverse objects and scenes with negligible degradation in reconstruction fidelity.

  • Takeaways & Limitations

    Dynamic step caching assumes stable regimes in which the denoising trajectory is well approximated by its tangent.

Abstract

from arXiv · show

SAM3D enables scalable, open-world 3D reconstruction from complex scenes, yet its deployment is hindered by prohibitive inference latency. In this work, we conduct the \textbf{first systematic investigation} into its inference dynamics, revealing that generic acceleration strategies are brittle in this context. We demonstrate that these failures stem from neglecting the pipeline's inherent multi-level \textbf{heterogeneity}: the kinematic distinctiveness between shape and layout, the intrinsic sparsity of texture refinement, and the spectral variance across geometries. To address this, we present \textbf{Fast-SAM3D}, a training-free framework that dynamically aligns computation with instantaneous generation complexity. Our approach integrates three heterogeneity-aware mechanisms: (1) \textit{Modality-Aware Step Caching} to decouple structural evolution from sensitive layout updates; (2) \textit{Joint Spatiotemporal Token Carving} to concentrate refinement on high-entropy regions; and (3) \textit{Spectral-Aware Token Aggregation} to adapt decoding resolution. Extensive experiments demonstrate that Fast-SAM3D delivers up to \textbf{2.67$\times$} end-to-end speedup with negligible fidelity loss, establishing a new Pareto frontier for efficient single-view 3D generation. Our code is released in https://github.com/wlfeng0509/Fast-SAM3D.

1. Introduction

Fast-SAM3D presents the first systematic study of SAM3D inference, showing that latency and acceleration failures arise from heterogeneous denoising and decoding dynamics. Its training-free modules adapt caching, token refinement, and decoding resolution to these redundancies, achieving substantial speedups with negligible fidelity degradation.

  • Motivation: SAM3D performs mask-conditioned, open-world multi-object 3D reconstruction directly from a single scene image.This enables reconstruction of arbitrary objects from minimal user input.
  • Profiling: The first systematic profiling identifies latency bottlenecks in dual-stage iterative denoising and long-token-sequence decoding.Latency is dominated by coupled structure and texture generation together with combinatorial decoding complexity.
  • Acceleration challenge: Generic uniform step skipping and random token pruning are brittle because SAM3D combines distinct shape-layout dynamics with sparse texture refinement and spectral geometric variation.Uniform skipping can induce pose drift, while the pipeline’s multi-level heterogeneity makes generic acceleration unreliable.
  • Fast-SAM3D: Fast-SAM3D’s modules cache structure steps, refine dynamically selected active regions, and compress simple shapes while preserving details for complex geometries.Spectral-aware aggregation uses geometric spectral entropy to adapt decoding compression to shape complexity.
  • Fast-SAM3D: Fast-SAM3D is a unified, training-free framework that accelerates geometry, texture, and decoding by exploiting stage-specific redundancies.The framework integrates three lightweight components: modality-aware step caching, spatiotemporal token carving, and spectral-aware token aggregation.
  • Results: Extensive experiments show substantial end-to-end speedups across diverse objects and scenes with negligible degradation in reconstruction fidelity.The reported gains hold across diverse objects and scenes while preserving reconstruction quality.

2. Related Works

Prior 3D reconstruction evolved from deterministic representations and feed-forward transformers toward diffusion models, while efficient-generation methods remain poorly matched to single-view 3D structure. Existing 3D accelerators are limited by representation scope or unavailable multi-view redundancy, motivating a distinct acceleration approach.

  • 3D Reconstruction and Generation: Early 3D reconstruction methods deterministically regressed voxels, point clouds, or meshes, whereas recent feed-forward transformers improved inference speed but often struggled with high-fidelity generation.The field has notably shifted toward diffusion models.
  • Efficient Generative Models: Efficient generative-model acceleration includes training-based distillation and quantization, alongside training-free step optimization and token pruning.These strategies span both learned and inference-only approaches.
  • Efficient Generative Models: 2D-oriented acceleration methods exploit spatial smoothness but neglect intrinsic 3D structural sparsity and geometric sensitivity.This mismatch limits direct transfer of generic acceleration strategies to 3D generation.
  • Efficient Generative Models: Existing 3D accelerators are restricted to implicit fields or regression transformers, while Fast3DCache depends on multi-view redundancy unavailable in single-view tasks.These limitations expose a critical gap for single-view 3D acceleration.

3. Preliminaries

SAM3D reconstructs 3D shape, texture, and layout from an image and object mask through a two-stage coarse-to-fine pipeline. Its inference cost is concentrated in iterative generators and mesh decoding, motivating holistic acceleration.

  • SAM3D takes an image I and object mask M to reconstruct 3D shape S, texture T, and layout parameters (R, t, s).
  • Its pipeline first predicts coarse structure and global layout, then refines geometric details and synthesizes texture.The SS generator predicts coarse structure and layout, while SLaT refines appearance-related signals and fine-grained geometry.
  • Inference overhead is dominated by the SS and SLaT generators’ iterative denoising steps and the mesh decoder’s processing of long structured 3D token sequences.Dense-grid 3D convolution makes mesh decoding expensive when decoding dense voxel tokens.
  • These bottlenecks motivate Fast-SAM3D, which jointly reduces sampling costs in both diffusion stages and decoding costs in 3D heads while preserving reconstruction fidelity.

4. Methods

Fast-SAM3D aligns computation with heterogeneous denoising and decoding behavior across modalities, spatial regions, time, and object complexity. Its methods combine modality-specific caching, saliency-guided refinement reuse, and spectral-aware token aggregation.

  • Modality-Aware Step Caching: Shape tokens evolve smoothly, whereas volatile layout tokens make uniform caching either unstable or overly lagged.Shape updates support extrapolation, while layout updates require safeguards against pose drift.
  • Modality-Aware Step Caching: Modality-Aware Step Caching extrapolates structural tokens from finite differences and smooths layout predictions with a momentum-corrected anchor.The anchor is the latest full backbone layout evaluation, and smoothing suppresses high-frequency jitter and mitigates pose drift.
  • Joint Spatiotemporal Token Carving: Refinement changes are spatially non-uniform: smooth regions change slowly, while edges, seams, and thin structures receive larger persistent updates.Temporal activity uses first- and second-order variations, while spatial complexity uses FFT-based frequency statistics.
  • Joint Spatiotemporal Token Carving: Joint Spatiotemporal Token Carving reuses tangent-like updates during stable regimes and refreshes the backbone when accumulated relative change exceeds a threshold.The active-token mask identifies carved Top-K tokens, while error-bounded switching prevents uncontrolled drift.
  • Spectral-Aware Token Aggregation: Spectral-Aware Token Aggregation adapts token reduction to geometric complexity because complex shapes contain more high-frequency energy and uniform downsampling can destroy details.Complexity is inferred from spectral energy, combining boundary and volumetric cues before selecting an instance-adaptive downsampling factor.
  • Spectral-Aware Token Aggregation: Quantized coordinates group tokens into voxel-aligned bins, and max pooling reduces token count by approximately S3 while preserving salient local features.The resulting token set adapts to the selected aggregation factor S.

5. Experiments

Experiments on Toys4K and Aria Digital Twin show that Fast-SAM3D achieves strong efficiency–quality trade-offs against caching, step-skipping, and naive saliency baselines. Ablations identify complementary gains from its three modules and establish stable settings for caching, token carving, and spectral aggregation.

  • Datasets and Metrics: Fast-SAM3D is evaluated on Toys4K and Aria Digital Twin using Chamfer Distance, F-Score, and Volumetric IoU with ICP alignment against caching, step-caching, and naive variants.Baselines include Fast3Dcache, TaylorSeer, EasyCache, Random Drop, and Uniform Merge.
  • Performance and Efficiency: Fast-SAM3D achieves 2.01× and 2.67× speedups for scene and object generation, respectively, while maintaining or exceeding baseline geometric fidelity.F-Score reaches 92.59 versus 92.34 for the base model.
  • Baseline Analysis: Fast3Dcache provides only 1.03× acceleration, whereas Random Drop causes 3D-IoU to fall to 0.094, exposing the brittleness of generic single-view acceleration.The experiments attribute these failures to missing multi-view redundancy and non-uniform 3D structural information.
  • Impact of Acceleration Components: Combining all three modules reduces inference time to 230s, while standalone Mesh reduces latency from 462s to 320s and SLaT reduces it to 366s.SLaT also increases F1 from 92.34 to 92.50, supporting saliency-based removal of noisy, low-confidence tokens.
  • Cache Stride k (Anchor Update Frequency): Setting cache stride k = 3 achieves the best vIoU of 0.5521, while k = 4 reduces 3D-IoU from 0.3750 to 0.2408 through accumulated pose drift.The selected stride balances layout alignment and acceleration.
  • Hyperparameter Ablations: The selected SLaT and Mesh settings balance speed and fidelity: top-10% carving reduces object time from 12.15s to 11.60s, while thresholds {0.5, 0.7} cut inference time by ∼8.5% and reach vIoU 0.5521.More aggressive top-20% carving drops F1 from 92.585 to 90.254; β = 0.5 yields F1 92.59 and vIoU 0.552.

6. Conclusion · A. Detailed Experimental Settings

The paper identifies latency as a critical obstacle to interactive open-world 3D reconstruction and attributes generic accelerator inefficiency to SAM3D’s multi-level generation heterogeneity. It presents Fast-SAM3D as a training-free framework designed to dynamically harmonize computation.

  • 6. Conclusion: Fast-SAM3D targets the critical latency bottleneck limiting interactive deployment of open-world 3D reconstruction frameworks.The framework is presented as a response to the deployment challenge identified in the conclusion.
  • 6. Conclusion: The first systematic analysis of SAM3D’s inference dynamics links generic accelerator inefficiency to intrinsic multi-level heterogeneity in 3D generation.The conclusion identifies this heterogeneity as the reason generic acceleration strategies are inadequate.
  • 6. Conclusion: Fast-SAM3D is introduced as a training-free acceleration framework that dynamically harmonizes computational demands.The supplied passage ends after describing this computational harmonization objective.

A.1. Data Preparation and Evaluation Protocols … B.1. Transferability to TRELLIS

The paper evaluates Fast-SAM3D across curated geometry, scene-layout, and perceptual-fidelity protocols, using standardized metrics and matched implementation conditions. Extended validation transfers the acceleration principle to TRELLIS as Fast-TRELLIS through architecture-specific adaptation.

  • A.1. Data Preparation and Evaluation Protocols: Toys4K provides 600 unique views, one per object, for geometry evaluation after viewpoint rendering, background removal, and low-quality filtering.Reconstruction quality is assessed with Chamfer Distance (CD), F-Score (F1@0.05), and Volumetric IoU (vIoU).
  • A.1. Data Preparation and Evaluation Protocols: ADT evaluation uses 16 views from four distinct scene-type sequences after filtering similar frames, reporting 3D IoU and rotation error (ICP-rot).These metrics measure alignment between reconstructed scenes and ground truth.
  • A.1. Data Preparation and Evaluation Protocols: ISO3D supplies 101 synthetic objects with 101 unique views, and Uni3D computes perceptual similarity from cosine similarity between input-image and generated-point-cloud embeddings.Each generated mesh is uniformly sampled into 8,192 surface points.
  • A.3. Implementation Details: All baselines share the same SAM3D backbone, and experiments run on a single NVIDIA-A800 for fair comparison.Fast-SAM3D accelerates the SS Generator, SLaT Generator, and Mesh Decoder.
  • A.3. Implementation Details: Fast3Dcache, TaylorSeer, EasyCache, Random Drop, and Uniform Merge provide matched comparison baselines for caching, carving, and token aggregation.Fast3Dcache is adapted to single-view inputs; TaylorSeer and EasyCache use warmup periods; naive baselines match the proposed sparsity or downsampling settings.
  • B. Extended Validation: Extended validation examines Fast-SAM3D’s transferability beyond SAM3D and its stage-wise memory usage.These experiments clarify the framework’s deployment properties.
  • B.1. Transferability to TRELLIS: Fast-TRELLIS replaces MASC’s layout-aware branch with TaylorSeer-style step caching, while transferring the remaining heterogeneity-aware designs to TRELLIS refinement and decoding.The substitution is required because TRELLIS’s first-stage sparse-structure generator lacks explicit spatial-layout tokens; transferability is evaluated on Toys4K.

B.2. GPU Memory Analysis … D. More Analysis of Joint Spatiotemporal Token Carving

Fast-SAM3D lowers peak inference memory by combining diffusion-state caching, token carving, and spectral-aware aggregation. Analysis shows smooth Shape Token trajectories support extrapolation, whereas volatile Layout Token trajectories motivate modality-aware treatment, alongside a saliency and carving-factor study.

  • B.2. GPU Memory Analysis: 1.18GB lower peak memory reduces Fast-SAM3D’s end-to-end inference footprint from 19.07GB to 17.89GB.The reduction is attributed to adaptive aggregation directly reducing mesh-stage workload, which dominates the original peak.
  • B.2. GPU Memory Analysis: Fast-SAM3D combines diffusion-stage intermediate-state caching with SLaT token carving and spectral-aware aggregation to reduce inference workload.These mechanisms shorten effective sequence length and reduce mesh-decoder computation.
  • C. More Analysis of Modality-Aware Step Caching: The analysis investigates the motivation for Modality-Aware Step Caching through the assumed heterogeneity of shape and layout modalities.The investigation visualizes denoising trajectories of different token types in latent space.
  • C.1. Visualizing Trajectory Heterogeneity: Shape Tokens follow smooth, quasi-linear trajectories whose gradual adjacent-step gradients make local linear extrapolation reliable.The trajectory forms a predictable arc, supporting prediction of earlier structural states from the local trend at step t.
  • C.1. Visualizing Trajectory Heterogeneity: Layout Tokens exhibit volatile, non-monotonic fluctuations, contrasting with the temporal coherence of Shape Tokens.This contrast demonstrates that layout updates are more sensitive to step skipping than structural-token evolution.
  • D. More Analysis of Joint Spatiotemporal Token Carving: The token-carving analysis evaluates whether the saliency mechanism predicts important regions accurately.It also examines how the carving factor γ affects carving behavior.

D.1. Visualizing Saliency Effectiveness … E.1. Visualizing Spectral Heterogeneity

The analyses validate saliency-based token carving, identify γ = 0.7 as the best balance for capturing transient changes, and motivate instance-adaptive aggregation from spectral differences between simple and complex geometries.

  • D.1. Visualizing Saliency Effectiveness: Unified Saliency correctly localizes regions that subsequently undergo significant token updates, including edges and complex topological structures.The predicted map aligns with actual next-step update magnitudes and captures the diffusion process’s active set.
  • D.1. Visualizing Saliency Effectiveness: Shape tokens evolve smoothly and predictably, whereas layout tokens exhibit erratic, high-frequency oscillations that require modality-aware caching.The contrasting latent trajectories provide empirical support for decoupling cache treatment by modality.
  • D.1. Visualizing Saliency Effectiveness: Combining first- and second-order temporal dynamics with spatial frequency accurately predicts future variation without random pruning.This selective preservation is intended to maintain geometric fidelity under high sparsity.
  • D.2. Sensitivity to Carving Factor γ: γ = 0.7 achieves the best balance between update magnitude and abruptness in the saliency potential.γ = 0.0 misses critical turning points, while γ = 1.0 increases noise sensitivity.
  • E. More Analysis of Spectral-Aware Dynamic Token Aggregation: The spectral-aware aggregation analysis examines how different spectral cues contribute to adaptive mesh decoding.Its premise is that geometric information is non-uniformly distributed across object instances.
  • E.1. Visualizing Spectral Heterogeneity: Simple objects concentrate spectral energy at low frequencies, so coarse token grids can accelerate decoding with negligible information loss.Their geometry is dominated by smooth surfaces with minimal high-frequency detail.
  • E.1. Visualizing Spectral Heterogeneity: Complex objects retain substantial high-frequency energy, making uniform downsampling prone to aliasing and fine-geometry loss.Their long-tail spectra reflect intricate topological details and sharp edges.
  • E.1. Visualizing Spectral Heterogeneity: The High-Frequency Energy Ratio dynamically allocates grid resolution where geometric structure requires greater computational budget.This instance-adaptive design directly responds to spectral heterogeneity across objects.

E.2. Sensitivity to Complexity Weight w … F.2. Robustness to Degraded Mask Inputs

Fast-SAM3D performs best when its complexity metric emphasizes high-resolution 2D cues while retaining 3D topology, and remains robust across broader views, degraded masks, and coefficient perturbations. These studies preserve near-lossless quality and the reported acceleration or latency under the tested conditions.

  • E.2. Sensitivity to Complexity Weight w: w = 0.0 produces the lowest perceptual score, Uni3D: 0.3194, because the coarse voxel grid lacks fine surface details.
  • E.2. Sensitivity to Complexity Weight w: w = 1.0 lowers geometric precision, with F1 decreases to 92.103, by neglecting internal volumetric topology.
  • E.2. Sensitivity to Complexity Weight w: w = 0.9 achieves the best balance, with Uni3D: 0.3503 and F1: 92.585.The 2D-dominant fusion uses high-resolution detail cues while the 3D voxel provides topological regularization.
  • F. Robustness Analysis: The robustness studies test Fast-SAM3D under broader ADT coverage, degraded masks, and repeated regeneration/evaluation.These experiments stress the method from three complementary perspectives.
  • F.1. Broader ADT View Coverage: 128 views preserve near-lossless quality versus SAM3D, match TaylorSeer’s 3D-IoU, achieve lower ICP-rot error, and retain 2.67× object-level acceleration.
  • F.2. Robustness to Degraded Mask Inputs: 50% masking of the major object region on Toys4K leaves Fast-SAM3D stable across β = 0.4/0.5/0.6, close to SAM3D in quality, better than TaylorSeer, and at 11.60s latency.The test also examines whether low-quality masks and anchoring-coefficient perturbations cause instability.

F.3. Reproducibility Across Runs · H. More visual comparison

Fast-SAM3D’s reported speed-quality trade-off is reproducible across five regeneration and evaluation runs, with small variance on object-level geometry and scene-level layout metrics. Additional visual comparisons cover existing methods, original SAM3D, and transfer to TRELLIS, where Fast-TRELLIS achieves a 2.26× speedup while preserving visual quality.

  • F.3. Reproducibility Across Runs: Five regeneration and evaluation rounds were used to verify that the reported gains were not incidental.Table 14 reports the mean and standard deviation across runs.
  • F.3. Reproducibility Across Runs: Fast-SAM3D exhibits small variance on both object-level geometry and scene-level layout metrics.This supports reproducibility of its speed-quality trade-off.
  • F.3. Reproducibility Across Runs: Table 13 reports an ablation study under degraded masks on Toys4K, with 50% of the major object region masked out.The degraded-mask setting evaluates robustness when half of the major object region is unavailable.
  • F.3. Reproducibility Across Runs: Table 14 presents an ablation study of statistical variation across five regeneration and evaluation runs.The table reports mean and standard deviation across runs.
  • H. More visual comparison: Figure 11 compares Fast-SAM3D with SAM3D, EasyCache, TaylorSeer, and Random Drop.These methods are listed as the compared approaches.
  • H. More visual comparison: Figure 12 provides a visual comparison between Fast-SAM3D and original SAM3D.The original method is identified as SAM3D (Chen et al., 2025).
  • H. More visual comparison: Fast-TRELLIS transfers Fast-SAM3D’s heterogeneity-aware acceleration principle to TRELLIS.The qualitative comparison covers single-view and multi-view reconstruction examples.
  • H. More visual comparison: 2.26× speedup is achieved by Fast-TRELLIS while preserving visual quality across single-view and multi-view reconstruction examples.This result is reported in the TRELLIS qualitative comparison.
Loading 2602.05293v2…