PaperlayerView exact revision on arXiv

Source-linked AI summary

JanusMesh: Fast and Zero-Shot 3D Visual Illusion Generation via Cross-Space Denoising

Siang-Ling Zhang, Huai-Hsun Cheng, Tsung-Ju Yang, Yu-Lun Liu

arXiv:2606.20563v1cs.CV

TL;DR

Generating one 3D mesh with different semantics from different viewing angles remains challenging. JanusMesh uses zero-shot cross-space denoising and view-conditioned texturing to create coherent 3D illusions, achieving 84% GPT Accuracy and completing generation in 3–5 minutes.

  • Problem

    Generating a single 3D object that presents entirely different semantics from different viewing angles remains a formidable challenge.

  • Method

    JanusMesh combines cross-space dual-branch denoising with CLIP-guided orientation alignment, SDF blending, and view-conditioned texturing in a training-free two-stage framework.

  • Results

    84% GPT Accuracy, lowest FID (185.555), and best Object Detection scores are reported across 50 randomly sampled prompt pairs.

  • Takeaways & Limitations

    The framework generates coherent, dual-semantic 3D meshes from text prompts in 3–5 minutes without per-shape optimization.

  • Takeaways & Limitations

    The method inherits TRELLIS failure cases and struggles with automated alignment for three-object illusions, for which angles are currently fixed.

Abstract

from arXiv · show

Creating 3D visual illusions, a single 3D mesh that reveals entirely different semantics from various viewing angles, is a fascinating but tough challenge. Existing optimization-based methods are slow and can produce oversaturated colors. In contrast, naive stitching approaches fail to produce geometrically coherent objects. This results in visible unnatural seams and semantic leaks. In this paper, we present a fast and training-free framework for generating text-driven 3D visual illusions. Our approach decouples the generation into two stages. First, we propose a cross-space dual-branch denoising process. This process dynamically decodes 3D latents into voxel space for CLIP-guided orientation alignment and Signed Distance Field (SDF) blending, which ensures seamless geometric fusion. Second, we introduce a view-conditioned texture synthesis module that projects and aggregates view-specific 2D diffusion priors onto the fused geometry. Extensive experiments demonstrate that our method generates highly realistic, dual-semantic 3D illusions in just 3-5 minutes. It significantly outperforms existing methods in geometric integrity, semantic recognizability, and efficiency. Project page: https://siang1105.github.io/JanusMesh.github.io/

1 Introduction

JanusMesh targets zero-shot 3D visual illusions: one coherent, fully textured mesh presenting different semantics from different viewpoints. It addresses the inefficiency and artifacts of prior approaches with a training-free, two-stage framework combining cross-space geometric fusion and view-conditioned texturing.

  • Motivation: Zero-shot 3D visual illusion generation seeks one mesh with entirely different semantics from different viewing angles, extending computational illusions beyond 2D.The task remains challenging despite diffusion models enabling computational optical illusions in 2D.
  • Prior Work: Prior 3D methods optimize representations with SDS so renders match different prompts at different viewpoints.Earlier approaches also included projection- or surface-pattern-based illusions such as shadow art, wireframe silhouettes, and height fields.
  • Limitations: ∼40 minutes per shape is required by optimization-based methods, which also suffer severe color over-saturation, while direct concatenation creates seams and backside leakage.These artifacts break the intended perceptual illusion and reveal the limitations of separately generated objects.
  • Method: 3–5 minutes is the generation time of JanusMesh’s zero-shot two-stage framework for coherent 3D visual illusions.Stage 1 uses TRELLIS dual-branch denoising, voxel-space decoding, CLIP-guided orientation search, and SDF blending; Stage 2 projects Stable Diffusion predictions onto the fused mesh.
  • Contributions: The framework extends generative multi-view illusions to fully textured 3D meshes through training-free dual-branch denoising, SDF blending, CLIP alignment, and view-conditioned texturing.Its evaluation incorporates CLIP, GPT-4.1-mini, FID/KID, and a novel Object Detection metric, including scalability experiments with three-object illusions.

2 Related Work

Prior work spans appearance-varying 3D illusions, zero-shot diffusion-based synthesis, optimization and feed-forward 3D generation, native 3D generative models, and coordinated multi-view denoising. JanusMesh builds on these directions, including TRELLIS’s structured latent representation for dual-semantic mesh generation.

  • Appearance-varying 3D illusions: Appearance-varying 3D methods include shadow art, wire art, view-dependent heightfields, SDF-based anamorphic packing, and spatially ambiguous geometry.Related 2D approaches include spatial-frequency decomposition and progressive vector sketching, which alter perceived content or semantics.
  • Diffusion-based illusion synthesis: Zero-shot illusion synthesis averages per-view noise, while related methods extend this principle through frequency decomposition, phase transfer, audio-visual spectrograms, and 3D lifting.SDS-based multi-prompt methods optimize slowly; Illusion3D and LookingGlass lift diffusion priors into NeRF and anamorphic images, respectively.
  • 3D generation methods: Optimization-based methods distill 2D diffusion priors with SDS and its variants, whereas feed-forward methods combine multi-view diffusion with reconstruction networks.The cited improvements include interval score matching, rectified-flow distillation, and Gaussian acceleration.
  • Native 3D generative models: Native 3D generative models learn latent representations directly from 3D data, while TRELLIS encodes geometry and appearance in a sparse structured latent space.The paper repurposes TRELLIS’s representation for dual-semantic mesh generation.
  • Denoising trajectory coordination: Merging denoising trajectories supports compositional generation, seamless panoramas, and perceptual synchronization, while multi-view methods coordinate denoising for sparse-view reconstruction.Recent work also averages clean-image predictions, applies spatial guidance in 3D latents, and uses geometry-aware strategies for multi-view outpainting.

3D Texture Synthesis.

The method synthesizes textures by assigning different prompts to angular sectors and blending viewpoint-specific images, while CLIP-guided orientation search aligns silhouettes before SDF fusion.

  • View-Conditioned Texture Synthesis: View-conditioned synthesis assigns distinct prompts to angular sectors and back-projects viewpoint-specific clean images using cosine-weighted blending.This contrasts with prior diffusion-based mesh texturing methods that apply a single prompt uniformly.
  • Orientation Alignment: CLIP-guided Orientation Search selects the relative rotation that maximizes silhouette alignment between the two objects.The alignment resolves geometric mismatches that could otherwise cause SDF fusion failures.

3 Method

JanusMesh generates dual-semantic 3D illusions through two stages: cross-space dual-branch geometric denoising followed by view-conditioned texture synthesis. It further improves fusion by injecting spatial guidance and adaptively aligning object orientations with CLIP.

  • Stage 1: Dual-Branch Geometric Denoising: Stage 1 decodes two prompt-conditioned latent branches into voxel space, rotation-aligns them, blends their SDFs, and re-encodes the fused geometry during denoising.The SDFs are averaged element-wise and thresholded to produce geometrically continuous blended occupancy.
  • Stage 2: View-Conditioned Texturing: Stage 2 renders the fused mesh from target viewpoints, predicts depth-conditioned textures with Stable Diffusion, un-projects them, and aggregates them using cosine-weighted blending.This stage is separated from geometry generation because direct TRELLIS texturing fails on the unnatural fused Stage 1 mesh.
  • Noise Guidance: Two noise-guidance strategies inject a spatial prior by mixing a guidance latent with Gaussian noise or interpolating it at timestep t0 before denoising.Noise Blending Guidance uses α to balance structural prior and generation diversity, while Space Control Guidance varies structural constraint through t0; in a 25-step setting, t0 = 10 guides the first 10 steps.
  • Adaptive Orientation Search: CLIP-guided orientation search selects a representative anchor view and evaluates 28 rotations of the second object to maximize image-image similarity before SDF blending.The anchor is selected from four orthogonal renders using CLIP text-image similarity, while the second object is rotated for silhouette alignment.
  • Multi-Object Extension: The framework extends to three-object illusions by adding a third Stage 1 denoising branch while sharing the initial noise.This extension is described for three prompts and a shared noise zt.

4 Experiments … Implementation Details.

Experiments compare JanusMesh with four baselines using 60 objects across five categories, while evaluating three fusion cases and implementation settings on an RTX 4090. The method completes Cases 1–2 in approximately 3 minutes and Case 3 in approximately 5 minutes, with CLIP-guided search accounting for the difference.

  • 4.1 Experimental Setup: The study compares Shape from Semantics, Direct Concatenation, TRELLIS, and DreamBeast as four baselines for view-dependent 3D illusion generation.Shape from Semantics uses SDS and requires ∼40 minutes per object; Direct Concatenation stitches independently generated TRELLIS objects.
  • Data: The dataset contains 60 distinct objects spanning 16 birds, 19 mammals, 5 reptiles and aquatic animals, 9 plants, and 11 man-made artifacts.Two objects are randomly sampled per experiment to form each prompt pair.
  • Data: All experiments run on a single NVIDIA RTX 4090, with Stage 1 using 25 denoising steps and Stage 2 using 30 denoising steps.SDF blending uses clip_s = 12 and τ = 0.8, while Interval CFG uses ω = 7.5 within t ∈[0.5, 0.95].
  • Implementation Details: Case 3 automatically selects object B’s rotation angle through CLIP-guided Orientation Search, typically placing the objects approximately 180° apart.Case 3 requires no additional guidance because its fusion angle is adapted through CLIP.
  • Implementation Details: ∼3 minutes is the total runtime for Cases 1 and 2, comprising ∼1 minute for Stage 1 and ∼2 minutes for Stage 2.For Case 3, Stage 1 takes ∼3 minutes because of the additional CLIP-guided Orientation Search, while Stage 2 remains ∼2 minutes, totaling ∼5 minutes.

Metrics.

The paper evaluates generation quality and illusion effects using six quantitative metrics plus a user study. These metrics assess semantic alignment, realism, geometric fusion, view-conditional contrast, and seam visibility.

  • Evaluation Metrics: Six quantitative metrics and a user study evaluate generation quality and illusion effects.The metrics include CLIP Similarity, GPT Accuracy, FID, KID, Object Detection Score, View-Conditional CLIP Contrast, and Impact Factor.
  • Semantic Evaluation: GPT Accuracy (%) measures semantic clarity by comparing GPT-4.1-mini’s two-way interpretation with the ground-truth prompt assignment.Accuracy is the proportion of responses matching the assigned left-right semantics.
  • Realism Evaluation: FID and KID measure visual realism against 1,000 reference images from Objaverse 1.0.The comparison uses 1,000 renders of the results and 1,000 reference images comprising 20 views of 50 objects.
  • Geometric Evaluation: Object Detection Score measures geometric fusion using midpoint-angle renders, average object count, and multi-object rate.The ideal average object count is 1, while multi-object rate measures renders with more than one detection.
  • Seam Evaluation: Impact Factor quantifies seam visibility: values near 1 indicate geometrically indistinguishable boundaries, while larger values indicate abrupt seams.Direct Concatenation has high Impact Factor scores because of sharp geometric discontinuities at stitching boundaries.

User Study. · 4.2 Results and Analysis

The user study collects perceptual evaluations from 50 participants who compared rendered results from three methods. Participants assessed semantic recognizability, semantic alignment, and the naturalness of adaptive versus fixed viewing angles.

  • User Study.: 50 participants evaluated rendered results from all three methods in a perceptual user study.Each participant assessed the methods using the study’s listed questions.
  • User Study.: Participants rated how recognizable the intended semantics were at each target viewpoint.The scale ranged from 1, unrecognizable, to 3, clearly recognizable.
  • 4.2 Results and Analysis: The recognizability scale used three levels: unrecognizable, partially recognizable, and clearly recognizable.Scores were defined as 1, 2, and 3, respectively.
  • User Study.: Qualitative comparisons examined View 1, the blended mesh, and View 2 across the evaluated methods.The figure presents these views from left to right.
  • 4.2 Results and Analysis: The qualitative comparison contrasts Shape From Semantics, Direct Concatenation, Ours, and TRELLIS.The figure describes method-specific visual outcomes, including saturation, leakage, seams, and view-dependent semantics.
  • 4.2 Results and Analysis: Participants identified which result better aligned with the intended semantics.This was the second evaluation question in the user study.
  • 4.2 Results and Analysis: Participants compared CLIP-adaptive orientation, labeled Case 3, against fixed 0°/180° angles.The comparison addressed which orientation strategy produced a more natural illusion effect.
  • 4.2 Results and Analysis: The study asked which orientation strategy produced a more natural illusion effect.This question specifically contrasted adaptive orientation with fixed viewing angles.

Quantitative Comparison.

Across 50 randomly sampled prompt pairs, the method outperforms or matches both baselines on most metrics and achieves the highest GPT Accuracy. It also completes in 3–5 minutes, matching Direct Concatenation and substantially improving on Shape from Semantics’ approximately 40-minute runtime.

  • Metric comparison: 84% GPT Accuracy is highest, exceeding Direct Concatenation at 76% and Shape from Semantics at 70%.The comparison uses 50 randomly sampled prompt pairs.
  • Metric comparison: 29.030 CLIP score is highest for Direct Concatenation, compared with 28.170 for the method.The paper attributes this advantage to naive preservation of per-viewpoint appearance rather than genuine illusion quality.
  • Qualitative comparison: CLIP-guided search adaptively aligns compatible silhouettes, producing clearer semantics at both viewpoints than methods lacking adaptive rotation.Examples include pairing a sideways fish with a pineapple and a flying owl with a plane; the baselines show silhouette misalignment and degraded fusion.
  • Efficiency: 3–5 minutes is the method’s completion time, matching Direct Concatenation and improving on Shape from Semantics at approximately 40 minutes.The reported comparison highlights efficiency alongside recognizability, visual realism, and geometric coherence.

Qualitative Comparison. … Geometry Blending Strategy.

Qualitative comparisons and user studies show that the method produces coherent, recognizable, and preferred 3D illusions, with CLIP-guided orientation yielding more natural results. Ablations identify SDF averaging, geometry-dependent noise guidance, and view-conditioned texturing as key components.

  • Qualitative Comparison.: The method produces geometrically coherent results with clear per-viewpoint semantics, unlike Shape From Semantics, which suffers from over-saturation, and Direct Concatenation, which exposes junction seams.
  • User Studies.: 78.5% of participants rated the results as clearly recognizable, while 71% preferred the method over Shape From Semantics [40] and Direct Concatenation.
  • User Studies.: 91% of participants found CLIP-guided orientation more natural than fixed 0°/180° angles, supporting adaptive orientation search.
  • Geometry Blending Strategy.: Space Control suits pairs with large silhouette discrepancies, while Noise Blending benefits similar silhouettes with distinct semantics; compatible pairs require less intervention.
  • Geometry Blending Strategy.: SDF Average best balances geometric integrity and semantic preservation, whereas Union creates conflicting junctions, Blur Average loses details, Minkowski expands geometry, and Polar Coord distorts asymmetric objects.
  • Geometry Blending Strategy.: Without view-conditioned texturing, TRELLIS produces semantically incoherent textures at both viewpoints; the proposed synthesis assigns each viewpoint its own texture prediction.

4.4 Applications

JanusMesh performs well across diverse semantic pairings and extends naturally from two-object to three-object 3D illusions without architectural changes. Its three-object results produce one coherent mesh with distinct semantics from three viewpoints.

  • Semantic combinations: The framework performs strongly across semantic combinations ranging from structurally similar to geometrically distinct object pairs.These results are shown in Fig. 9 and Fig. 10.
  • Three-object illusions: A single coherent mesh presents three distinct semantics at 0°, 120°, and 240° viewpoints.Each qualitative result includes the fused mesh and renders for a prompt triplet.
  • Three-object illusions: The method scales to three-object illusion generation without architectural modifications.This extension is demonstrated in Fig. 15.

5 Conclusion

JanusMesh generates coherent 3D meshes with distinct viewpoint-dependent semantics in under 5 minutes using zero-shot text prompts, but remains limited by inherited category failures and unresolved automated alignment for three-object illusions.

  • Contributions: Under 5 minutes, the zero-shot framework generates a single coherent mesh with distinct semantics from different viewpoints without per-shape optimization.It extends visual illusions to true 3D geometry from text prompts.
  • Contributions: CLIP-guided Orientation Search aligns silhouettes, while Noise Guidance resolves geometric conflicts during fusion.The framework scales to three-object illusions without modifying its core fusion procedure.
  • Limitations: The method inherits TRELLIS’s failure cases for specific categories, including pigs and bats.These failures are documented in the supplementary material.
  • Limitations: Three-object illusions remain challenging because averaging three silhouettes produces ambiguous shapes, so angles are fixed to 0°/120°/240° as automated alignment remains future work.CLIP-guided Orientation Search currently struggles in this setting.
Loading 2606.20563v1…