Source-linked AI summary
Helix4D: Complex 4D Mesh Generation
Jiraphon Yenphraphai, Jianqi Chen, Jian Wang, Gordon Qian, Sergey Tulyakov, Rameen Abdal, Raymond A. Yeh, Peter Wonka, Chaoyang Wang
TL;DR
Video-to-4D methods struggle with complex topology changes, transparent materials, thin structures, and inner surfaces. Helix4D adapts Trellis2 into a video-conditioned dynamic mesh generator using anchor-based cross-frame attention and parameter-free temporal encoding, and it reports strong results on ActionBench and a challenging 52-video benchmark.
Problem
Existing video-to-4D methods struggle with complex topology changes, transparent materials, thin structures, and inner-surface reconstruction.
Method
Helix4D adapts Trellis2 to video-conditioned 4D generation with sliding-window cross-frame attention, first-frame conditioning, and temporal encoding that repurposes low-frequency spatial RoPE bands.
Results
Helix4D improves CD-3D by 3.8% over ActionMesh on ActionBench and outperforms all baselines on every metric on its 52-video benchmark.
Takeaways & Limitations
The approach shows that a pretrained static 3D foundation model can be lifted to 4D while retaining strong geometry and material fidelity and adding temporal consistency.
Takeaways & Limitations
The method inherits Trellis2 limitations, including occasional mesh holes and color-shifting textures that can produce metallic surfaces for transparent objects.
Abstract
from arXiv · showhide
Current video-to-4D methods struggle with complex topology changes, transparent materials, thin structures, and inner surfaces. We present Helix4D, a dynamic mesh generation framework by inheriting the expressive representation of Trellis2, adapting it from image-to-3D to video-conditioned 4D generation. Our design arises from two key questions: (a) how to enable Trellis2's frame-local attention to share information across frames while preserving its pretrained quality on rare cases such as transparent objects and inner surfaces, and (b) how to inject temporal information into a purely 3D positional encoding without breaking pretrained capabilities. We address (a) with a sliding-window cross-frame attention and anchor on the first frame. The first frame is generated by the base Trellis2 model and injected into our model, letting it inherit Trellis2's quality in rare cases through cross-frame attention. We address (b) with a 4D temporal encoding that repurposes redundant low-frequency spatial RoPE bands for time, extending the encoding from 3D with no additional parameters. Extensive experiments show the effectiveness of Helix4D for high-quality dynamic mesh generation on ActionBench and our own challenging complex dynamics set.
1 Introduction
Helix4D extends image-to-3D generation to video-conditioned 4D mesh generation for dynamic scenes. It targets complex topology, materials, transparency, and inner surfaces while preserving pretrained 3D capabilities.
- Video-to-4D generation must model geometry, materials, motion, topology changes, and temporal coherence.
- Existing methods perform strongly on rigid and simple objects but struggle with complex topology variations, transparency, materials, and inner-surface reconstruction.
- Helix4D extends Trellis2 to dynamic mesh generation while preserving its pretrained strengths for transparent objects, complex materials, topology changes, and inner surfaces.
- Sliding-window cross-frame attention with an anchor frame shares temporal information efficiently while retaining geometry and material quality similar to full attention.
- On ActionBench, Helix4D improves CD-3D by 3.8% over ActionMesh, while on a 52-video benchmark it outperforms all baselines on every metric.
- Its contributions include advanced 4D geometric and material capabilities, anchor-based cross-frame modeling, and parameter-free temporal RoPE through frequency repurposing.
2 Related Work
Prior video-to-4D approaches use optimization, reconstruction, deformation, or end-to-end spacetime generation, each with speed, topology, or quality limitations. Helix4D builds on a 3D representation intended to support more expressive dynamic meshes.
- Optimization-based 4D methods optimize per-instance representations against pretrained diffusion priors and motivate feed-forward generation because they are slow and produce artifacts.
- Feed-forward 3D generators use voxel-based or vecset-based latent representations, but signed-distance and occupancy decoding requires watertight, manifold training data.
- Feed-forward 4D generation: Deformation-centric 4D methods produce smooth motion but inherit the canonical asset’s topology, preventing topology changes.
- Feed-forward 4D generation: End-to-end spacetime-latent generators avoid the topology constraint, but their per-frame geometry and material quality is limited.
- Feed-forward 4D generation: Helix4D uses the O-Voxel representation to support non-manifold geometry, topology changes, and transparent materials at high quality.
3 Background
Trellis2 generates textured meshes from images using sparse voxel features, latent compression, and three flow-matching stages. Its sparse representation supports open, thin, and interior surfaces without requiring watertight geometry.
- Trellis2 takes an input image and predicts a textured mesh through an O-Voxel representation, a Sparse Compression VAE, and three flow-matching models.
- O-Voxel converts a 3D asset into sparse voxel features, retaining local geometry and material information while discarding empty voxels.
- Unlike SDF-based representations, O-Voxels do not require watertight geometry and can represent open surfaces, thin structures, and interior surfaces.
- Trellis2 generates sparse structure, geometry, and material latents through three flow-matching stages built on the same DiT-style backbone.
4 Dynamic Mesh Generation
Helix4D converts Trellis2 from image-to-3D into video-conditioned 4D generation using cross-frame attention and a parameter-free spatiotemporal RoPE extension. Its design combines local temporal interaction with first-frame conditioning while preserving Trellis2’s pretrained spatial and appearance capabilities.
- Helix4D converts Trellis2’s image-to-3D architecture into a dynamic mesh generator conditioned on object-centric video.
- Cross-frame attention with first-frame anchor: Full cross-frame attention is too costly for sequences containing up to 10^5 tokens, motivating sparse temporal attention.
- Cross-frame attention with first-frame anchor: Sliding-window attention lets each token attend within a temporal window and to all first-frame tokens, combining local motion with a global shape reference.
- Cross-frame attention with first-frame anchor: Sliding-window attention with an anchor matches full-attention quality at 2× lower computation cost.
- Cross-frame attention with first-frame anchor: First-frame conditioning supplies a clean Trellis2-generated reference that later noisy frames can attend to, helping retain quality for transparent objects and inner surfaces.
- Repurposing Spatial RoPE for Time: Helix4D repurposes low-frequency spatial RoPE bands for time while retaining high-frequency spatial bands, avoiding additional parameters and preserving relative-position attention.
- Repurposing Spatial RoPE for Time: Removing low-frequency bands preserves Trellis2 quality for α ≥0.4, whereas α < 0.4 reduces quality; the method uses α = 0.75.
5 Experiments
Helix4D is evaluated on ActionBench and the challenging Helix4DBench using qualitative, quantitative, and ablation studies. It outperforms prior methods while its component analyses support the contributions of 4D rotary encoding and sliding-window-plus-anchor attention.
- Qualitative comparisons: Qualitative examples show better preservation of fine geometry, topology changes, emerging structures, transparent objects, and inner surfaces than competing methods.The fish-in-bottle example is reconstructed correctly only by Helix4D among the compared methods.
- Helix4DBench results: Across Helix4DBench, Helix4D outperforms all baselines on 3D alignment, video quality, temporal consistency, and pairwise user preference.The benchmark compares rendered geometry, appearance, temporal behavior, and 1-on-1 user-study preference.
- Quantitative comparisons: On ActionBench, Helix4D improves CD-3D by 3.8% over ActionMesh, while on the harder TexVerse set it achieves the best CD-3D and CD-4D results.CD-3D measures per-frame shape accuracy; CD-4D also measures temporal consistency through a shared alignment.
- Ablation studies: Ablations show that all three components contribute to quality, with the largest drop after removing first-frame conditioning and visible geometric degradation without 4D rotary embedding.Removing 4D rotary produces rough back surfaces and extra arms, while the full model maintains clean geometry and correct articulation.
- Attention analysis: Sliding-window-plus-anchor attention achieves the best score on all four quality metrics while remaining competitive in cost.Full attention is more expensive and underperforms, whereas plain sliding-window attention permits drift from the initial geometry.
6 Conclusion
Helix4D adapts a pretrained image-to-3D generator into a video-to-4D mesh generator using temporal ReRoPE, sliding-window cross-frame attention, and first-frame conditioning. The design preserves strong geometry and material priors while adding temporal consistency, and experiments show gains over recent baselines.
- Helix4D adapts a pretrained image-to-3D generator into a video-to-4D mesh generator.
- Temporal ReRoPE, sliding-window cross-frame attention with an anchor frame, and first-frame conditioning provide the lightweight conversion to dynamic generation.
- The design preserves strong geometry and material priors while adding temporal consistency for topology changes, transparent objects, thin structures, and inner surfaces.
- Helix4D shows strong geometric accuracy on ActionBench and consistent gains over recent video-to-4D baselines.
- Future work targets longer videos, scene-level interactions, stronger camera motion, and more accurate physical dynamics.
Appendix
The appendix ablation evaluates how the RoPE ratio affects generated meshes relative to a full-spatial-RoPE reference on held-out TexVerse objects.
- The ablation evaluates 32 held-out TexVerse objects using CD-3D between each ratio's generated mesh and the full spatial RoPE reference.
- A spatial RoPE ratio of 1.0 serves as the reference setting.
A1 Limitations or failure cases
Helix4D inherits mesh and texture limitations from Trellis2 and also sometimes lacks temporal consistency in high-frequency geometric regions.
- The Trellis2 backbone does not enforce watertight meshes, so generated surfaces may occasionally contain holes.
- Trellis2 textures exhibit color shifting that could produce metallic-looking surfaces for transparent objects.
- Because Helix4D outputs mesh sequences, temporal consistency can sometimes be lacking in regions with high-frequency geometric details.
A2 User Study Details
The user study measures perceived quality by having participants compare rendered dynamic 3D reconstructions pairwise against prior methods.
- Participants compare pairwise rendered videos and select which result looks better overall.
- The comparison considers visual quality, temporal consistency, and preservation of details.
A2.1 Instructions Shown to Participants
Participants compared pairs of rendered 4D reconstructions and judged which result looked better overall. They were instructed to consider visual quality, temporal consistency, and detail preservation.
- Each question presented an input video and two generated results, A and B, for an overall quality comparison.
- Participants assessed realism and visual appeal, smoothness and coherence of motion, and preservation of fine details.
- The available responses were that Result A was better, Result B was better, or there was no clear difference.
- Participants were told the study contained 25 questions, allowed answer changes, and would take approximately 10 minutes.
A2.2 Participants and Comparisons
The user study used 25 pairwise comparisons per participant across RGB and normal-map renderings against several 4D reconstruction baselines. Results were summarized using a win rate based on wins, losses, and ties.
- Each participant answered 25 pairwise comparisons: 15 on RGB appearance and 10 on normal-map renderings emphasizing surface geometry.
- RGB comparisons used Motion 3-to-4, Mesh4D, and SS4D, while normal-map comparisons used ActionMesh and ShapeGen4D.
- Question order was randomized independently for each participant, and responses were collected anonymously.
- The win rate was computed from the numbers of wins, losses, and ties for Helix4D.
A3 Helix4DBench Construction
Helix4DBench combines generated motion sequences from Trellis2 example images with a fixed held-out TexVerse test set. Its examples target challenging dynamic phenomena, while the study reports 32 held-out TexVerse-1K assets.
- The benchmark uses Trellis2 repository example images as static-object inputs and generates videos with Wan2.2 using corresponding text prompts.
- All generated videos share a fixed-camera, pure-white-background prompt with no camera movement or rotation.
- The motion prompts cover diverse dynamic behaviors, including mechanical actions, expressive character motion, fire, deformation, and energetic effects.
- A fixed TexVerse test set contains 32 held-out assets from the TexVerse-1K dataset.
- The benchmark is intended to reduce barriers to high-quality 4D generation for animation, gaming, and VR/AR applications.
- Qualitative comparisons on TexVerse show baselines struggling with topology changes, while Helix4D follows ground-truth motion and preserves fine geometry.