Source-linked AI summary
Fast 4D Mesh Generation by Spatio-Temporal Attention Chains
Dvir Samuel, Yuval Atzmon, Gal Chechik, Yoni Kasten
TL;DR
Existing 4D mesh generation is slow and difficult to scale to longer videos. The paper introduces a training-free Spatio-Temporal Attention Chain that extracts correspondences from a frozen 4D backbone to animate an anchor mesh, achieving faster generation, longer rollouts, and additional tracking and camera-recovery capabilities.
Problem
Existing 4D mesh generation is slow, computationally expensive, and prone to drift when extended to longer videos.
Method
A training-free Spatio-Temporal Attention Chain composes frozen-backbone attention from anchor vertices to latent tokens, across time, and back to target-frame vertices.
Results
The method achieves 9 s generation with best performance on three of four geometric ActionBench metrics, while supporting 16× longer sequences and competitive zero-shot 2D and 4D tracking.
Takeaways & Limitations
The same correspondence mechanism supports topology-consistent meshes, longer rollouts, zero-shot tracking, and camera recovery without additional training.
Takeaways & Limitations
Mesh quality inherits limitations of the frozen models, while sparse deformation can damp fine motion and multi-minute rollouts may degrade as errors accumulate.
Abstract
from arXiv · showhide
4D mesh generation has recently emerged as a powerful paradigm for recovering dynamic 3D structure from videos, but existing methods remain slow, computationally expensive, and difficult to scale to longer sequences. We introduce a training-free approach that accelerates 4D mesh generation while improving temporal correspondence quality. Our key observation is that temporal correspondences emerge inside a 4D backbone long before its generated meshes become visually accurate. We exploit this with a general framework we call Spatio-Temporal Attention Chain which propagates information across space and time. Starting from vertices on an anchor mesh, the chain maps vertices to latent tokens. It then follows temporal correspondences in latent space, and recovers frame-specific vertices through latent-to-vertex attention. This design avoids expensive explicit matching while preserving anchor mesh details and thereby improving dynamic mesh geometry and temporal consistency. Compared to state-of-the-art, our method generates a 4D mesh in 9 seconds, achieving a $13\times$ speedup while producing higher-quality results. Moreover, our approach scales to videos up to $16\times$ longer without degrading mesh quality. Beyond generation, the improved correspondences enable competitive zero-shot performance on two downstream tasks: 2D object tracking and 4D tracking. We further show that our framework enables reliable camera estimation, a capability not supported by prior 4D mesh generation methods.
1 Introduction
The paper targets scalable video-to-dynamic-mesh reconstruction by replacing costly staged animation with correspondences extracted from a frozen 4D backbone. Its attention-chain framework accelerates generation, preserves topology, supports longer rollouts, and extends to tracking and camera recovery.
- Motivation: Video-to-dynamic-mesh reconstruction seeks temporally coherent 3D mesh sequences from ordinary videos, addressing scarce and expensive high-quality 4D data.Dynamic reconstruction must infer detailed geometry in every frame while preserving surface identity across time.
- Motivation: Existing staged pipelines use a costly 4D generative stage followed by a separate network that animates an anchor mesh with shared connectivity.The additional animation stage motivates applying the backbone’s motion prior directly to the anchor mesh.
- Core idea: Useful temporal correspondences emerge after as few as four denoising steps, linking anchor-frame 3D tokens to matching tokens in later frames.The method exposes this early correspondence signal through a Spatio-Temporal Attention Chain.
- Core idea: The chain maps anchor vertices to latent tokens, transports them across time, and projects them back to target-frame 3D points.It replaces explicit animation with correspondence propagation through the frozen backbone.
- Contributions: 9 seconds reduces generation time from 120 seconds while preserving topology and improving 3D accuracy; the framework also supports coherent 16× longer sequences and zero-shot tracking.The same chains enable 2D and 4D point tracking and camera recovery from 2D–3D matches.
2 Related Work
Prior work spans image-to-3D backbones, video-to-4D generation, emergent diffusion correspondences, attention control, and point tracking. The paper distinguishes its approach by composing attention maps across modalities and modules to tie correspondences to an anchor mesh surface.
- Image-to-3D generative backbones: Image-to-3D generators reconstruct high-fidelity meshes from images using latent tokens and decoder cross-attention, while related backbones share compatible structures.The paper instantiates its chain on TripoSG and notes possible adaptation to other generators with the same decoder form.
- Video-to-4D generation: Video-to-4D methods include optimization pipelines, novel-view diffusion, topology-free feed-forward representations, and mesh-animation systems based on skeletons or deformation fields.These approaches differ in whether they optimize scenes, decode frames independently, or assume clean input assets.
- Emergent correspondences: Recent correspondence methods either match frozen diffusion features as descriptors or read attention weights directly for tracking and 3D alignment.The paper positions its method within the attention-reading line while using a distinct multi-stage composition.
- Emergent correspondences: The proposed chain composes vertex-to-token, temporal token-to-token, and token-to-surface attention into a Va →Za →Zf →Vf correspondence tied to an anchor mesh.This composition connects spatial and temporal representations inside a frozen 4D generator.
3 4D Mesh Generation: Preliminaries and Notations
Video-to-dynamic-mesh pipelines represent a sequence with shared topology and typically separate anchor reconstruction, temporal generation, and topology-consistent decoding. The paper’s overview and method instead recover anchor motion through chained attention correspondences.
- Representation: A dynamic mesh sequence contains per-frame vertices Vf with shared faces F, fixed topology, and consistent vertex identities across F frames.This representation explicitly encodes temporal correspondence through shared connectivity.
- Attention notation: Attention between query sequence X and context sequence Y is represented by a softmax-based map whose projections use WQ, WK, and WV.The notation treats attention as the mechanism connecting representations in the pipeline.
- Staged pipelines: Prior pipelines use Stage 0 for image-to-3D anchor reconstruction, Stage I for per-frame temporal generation, and Stage II for topology-preserving alignment.Stage II may use learned decoders or test-time optimization to predict framewise displacements.
- Attention components: The anchor mesh decoder maps vertices and image patches to shared latent tokens, while temporal self-attention links anchor tokens to target-frame tokens.These maps provide the spatial and temporal components needed for correspondence chaining.
- Method overview: Figure 1 traces an anchor mesh vertex through latent tokens and time before returning it to a target-frame vertex; image-patch endpoints support related tasks.The same overview includes 2D tracking, camera pose estimation, and 4D tracking without additional training.
4 Method
The method extracts spatio-temporal correspondences from frozen 4D backbone attention, maps anchor vertices through latent tokens to target surfaces, and deforms the anchor mesh with topology-preserving operations. Correspondence reinforcement further stabilizes long-sequence rollouts and supports 2D/4D tracking extensions.
- Attention Chain: The attention chain maps anchor vertices to latent tokens, transports them across frames, and projects them onto target surface points.It composes vertex-to-token, temporal token-to-token, and token-to-surface attention maps.
- Attention Chain: Anchor-to-target correspondences are computed by sequentially multiplying internal spatial and temporal attention maps into a dense correspondence map.Top-scoring surface samples constrain the correspondence to the target surface and reduce drift to unrelated regions.
- Topology-Preserving Animation: Topology-preserving animation tracks sparse control landmarks, smooths their trajectories, and lifts them to the full mesh with geodesic rigid skinning.Local rigid transformations based on geodesic distances help prevent motion leakage between disconnected parts and preserve volume.
- Long-Sequence Rollout: Correspondence reinforcement addresses long-sequence drift by preserving latent correlations across autoregressive windows.Naive rollouts accumulate errors as matched latent-point correlations weaken, whereas reinforcement stabilizes mesh quality over time.
- Extensions: Attention chaining also composes image-patch, latent-token, and mesh-vertex representations for 2D tracking, 4D tracking, and pixel-to-mesh correspondence.The framework reuses temporal attention for 2D point tracking and creates a training-free correspondence layer connecting input pixels to canonical mesh geometry.
5 Experiments
The experiments evaluate fast 4D mesh generation and zero-shot 2D and 4D tracking against established baselines. The method combines strong geometric quality, camera-aware alignment, and competitive tracking without task-specific training.
- 4D Mesh Generation: 9 s per 16-frame clip is roughly 14× faster than ActionMesh and 100× faster than ShapeGen4D.The method achieves 9 s generation time versus 2 min for ActionMesh and 15 min for ShapeGen4D.
- 4D Mesh Generation: CD-3D 0.048, CD-4D 0.077, and Normal Consistency 0.97 are the best results on three of four ActionBench geometric metrics.ActionMesh remains slightly better on CD-M, with 0.148 versus 0.163.
- 4D Mesh Generation: With camera pose estimation, the method achieves the best LPIPS, CLIP, and DreamSim results among 4D mesh generation methods on Consistent4D.The camera-pose-enhanced variant surpasses the aligned L4GM baseline on all reported metrics.
- 4D Mesh Generation: Ours is preferred in 75% of 2,000 pairwise comparisons with ActionMesh for appearance and motion consistency.The study used 100 raters.
- 4D Mesh Generation: The method produces sharp, temporally consistent, camera-aligned meshes with fewer mismatch regions than object-centric baselines.Dense attention-chain correspondences support PnP camera recovery and image alignment.
- 2D Point Tracking: The method is the strongest zero-shot tracker on DAVIS-foreground and achieves the best overall Occlusion Accuracy of 90.41.On BADJA, it remains close to supervised trackers and outperforms several despite using no tracking supervision.
- 4D Point Tracking: +28.4 APD3D on PointOdyssey and +23.7 on Dynamic Replica improve over ActionMesh Stage II for zero-shot 4D tracking.The method is also competitive with supervised trackers, outperforming TraceAnything and approaching Any4D and 4RC.
6 Conclusion
The paper presents a training-free attention-chain framework for fast video-to-4D mesh reconstruction and downstream tracking with camera recovery. Its quality and scalability remain bounded by frozen-model accuracy, deformation simplification, and accumulated rollout errors.
- Conclusion: The framework exposes spatio-temporal attention chains in a frozen 4D backbone to animate an anchor mesh without a learned deformation network.It yields topology-consistent meshes, faster inference, longer rollouts, and 2D/4D tracking with camera recovery.
- Limitations: Mesh quality depends on the image-to-3D model and denoiser, while sparse smoothing and local-rigid deformation can damp fine motion.These are limitations inherited from the frozen models and the animation procedure.
- Limitations: Multi-minute rollouts may degrade as errors accumulate and attention over generated anchors becomes increasingly diffuse.The limitation concerns long autoregressive sequences.
A Additional Qualitative Results
Additional qualitative results compare mesh generation, long-sequence rollout, tracking, and scene placement. They emphasize sharper geometry, stable long-horizon structure, smoother tracking, and camera-enabled world alignment at low inference cost.
- 4D mesh generation: 9 s per clip versus 120 s for ActionMesh accompanies comparable-to-better geometry with fewer temporal artifacts.The comparison covers ActionBench and in-the-wild videos.
- Long-sequence generation: Correspondence reinforcement maintains global pose, fine detail, and stable mesh quality through frame 240.ActionMesh instead accumulates drift and progressively loses recognizable structure.
- 2D point tracking: Attention-chain correspondences produce smoother, more accurate 2D trajectories, especially on articulated body parts.The matches are geometrically grounded rather than purely appearance-based.
- 4D mesh placement: PnP camera recovery places the animated mesh in a reconstructed scene and supports arbitrary novel viewpoints without pose supervision.The mesh occupies the subject’s physical space across views.
B Ablation Study
The ablations examine denoising-step quality, inference cost, and long-sequence behavior. The method reaches strong geometric quality with few steps, reduces latency, and maintains stability over extended rollouts, while ActionMesh retains an advantage in motion smoothness.
- Denoising steps vs. quality: 4 denoising steps yield roughly 2× better CD-3D and CD-4D than ActionMesh, with scores of 0.048 vs. 0.095 and 0.077 vs. 0.125, respectively.The advantage remains at higher step counts, while ours plateaus by step 4.
- Long-sequence generation: Across 240-frame rollouts, correspondence reinforcement maintains stable mesh quality, whereas ActionMesh accumulates drift and loses recognizable geometry.The reinforced method preserves global pose and fine detail through frame 240.
- Denoising steps vs. quality: ActionMesh achieves slightly better CD-Motion, scoring 0.148 vs. 0.152 at 30 steps and 0.161 vs. 0.163 at 4 steps.The gap grows at very few steps, where its learned Stage II better smooths noisy latents.
- Inference time breakdown: ∼9 s per clip replaces ∼110 s by combining 4 denoising steps with removal of ActionMesh’s learned Stage II network.The speedup comes from shrinking Stage I and using lightweight closed-form operations instead of an additional network.
- Component contribution on long videos: The attention-chain pipeline improves long-video alignment and motion behavior relative to unaligned ActionMesh predictions in the 240-frame ablation.The ablation evaluates variants without ICP so the metrics reflect intrinsic alignment.
C Implementation Details
The implementation uses a frozen two-stage ActionMesh backbone and a single configuration across experiments. Generation replaces learned animation with attention reading, closed-form correspondence operations, and geodesic-rigid skinning, while camera estimation assumes fixed canonical intrinsics.
- Backbone and pipeline: Stage 0 produces an anchor mesh and latent code, while Stage I jointly generates per-frame latents over 16-frame windows.The backbone is inherited from ActionMesh and remains frozen.
- Backbone and pipeline: The pipeline reads decoder cross-attention and temporal-denoiser self-attention, then applies closed-form correspondence and skinning operations without training or fine-tuning.The same attention chain supports mesh generation, 2D tracking, 4D tracking, and camera estimation.
- Inference cost: ∼9 s replaces ∼110 s per 16-frame clip by using 4 rather than 30 denoising steps and replacing learned Stage II with four cheap closed-form operations.The two changes jointly reduce wall-clock latency.
- Hyperparameters: A single configuration is used across all experiments, including mesh generation, point tracking, and long-video tests, with no per-scene tuning.The configuration includes 4 Stage I steps and specified landmark, rejection, and smoothing settings.
- Camera estimation: Camera estimation assumes focal length 2.1875 in canonical NDC, corresponding to a roughly 49.2° full vertical field of view.Camera poses use RANSAC with EPnP as the minimal solver.
- Synthetic long sequences: Extended ActionBench sequences are created by repeatedly playing the original 16 frames forward and backward in a ping-pong ordering.This synthetic construction lengthens the temporal duration for long-sequence tests.
D Topology-Preserving Animation Details
Topology-preserving animation smooths attention-derived landmark trajectories and transfers them to the anchor mesh through geodesic local-rigid deformation. The procedure rejects implausible motion, preserves the anchor frame, and avoids cross-part motion leakage and volume shrinkage.
- Landmark processing: Outlier filtering rejects landmark displacements exceeding relative or absolute thresholds before temporal smoothing.A relative example is 5× the median displacement magnitude in the frame.
- 1D Gaussian temporal smoothing: Confidence-weighted 1D Gaussian smoothing fills rejected gaps while smoothing displacements from the anchor pose rather than absolute positions.Anchor-relative smoothing prevents stationary landmarks from drifting over time.
- 1D Gaussian temporal smoothing: The Gaussian kernel uses standard deviation σ in frames and a small constant ϵ for numerical stability, while the anchor frame remains pinned to its original geometry.Pinning prevents smoothing leakage into the anchor.
- Mesh deformation: Local rigid transformations preserve mesh volume and avoid the shrinkage artifacts associated with linear blend skinning.Applying these transformations produces the final animated mesh with shared anchor connectivity.
- Local rigid deformation: Each free vertex receives geodesic Gaussian weights from nearby landmarks, so deformation respects surface topology and articulated part boundaries.Geodesic neighborhoods prevent torso landmarks from influencing an arm inappropriately.
- Local rigid deformation: Weighted Procrustes alignment estimates a local rotation and translation from anchor landmarks to their smoothed target positions, then applies the transformation to each vertex.The rotation is obtained through SVD.
D.3 User Study Details
The user study directly compares rendered meshes from the proposed method and ActionMesh against reference videos. Across 2,000 pairwise judgments, raters preferred the proposed method in most comparisons.
- Study design and outcome: 2,000 pairwise comparisons from 100 raters judging 20 clips yielded 85% preference for the proposed method over ActionMesh.Each rater evaluated every clip, with method ordering randomized per trial.
- Study design and outcome: The study asked which rendering better matched the reference in appearance and motion consistency, including fewer temporal mesh distortions.Both methods used default settings on the same input frames and were shown beside the reference video.