PaperlayerView exact revision on arXiv

Source-linked AI summary

Map2World: Segment Map Conditioned Text to 3D World Generation

Jaeyoung Chung, Suyoung Lee, Jianfeng Xiang, Jiaolong Yang, Kyoung Mu Lee

arXiv:2605.00781v1cs.CV

TL;DR

Existing 3D world-generation methods face limited world-scale data, domain restrictions, view-dependent inconsistencies, and constraints from grid layouts or diffusion-model memory. Map2World uses text and arbitrary-shaped segment-map conditioning, global structured latents, latent fusion, and a detail enhancer built on TRELLIS; the framework is designed for global consistency, flexible control, and domain-generalized generation with limited data.

  • Problem

    World-scale 3D generation remains challenging because high-quality world-level datasets are scarce, while existing approaches can be domain-limited or suffer from 3D and view-dependent inconsistencies.

  • Method

    Map2World builds on TRELLIS to generate a globally structured world latent from text and arbitrary-shaped segment maps, then enhances details using adjacent-cube information and global structure.

  • Results

    Map2World is designed to provide global consistency, flexible user control, and robust generation across domains with limited scene-generation data.

  • Takeaways & Limitations

    The framework supports text-guided 3D world generation from user-defined segment maps while adding detail without compromising overall scene coherence.

  • Takeaways & Limitations

    The detail enhancer is fine-tuned only on scene-level cropped data, and its generalizability and photorealism could improve with broader and more realistic-textured training data.

Abstract

from arXiv · show

3D world generation is essential for applications such as immersive content creation or autonomous driving simulation. Recent advances in 3D world generation have shown promising results; however, these methods are constrained by grid layouts and suffer from inconsistencies in object scale throughout the entire world. In this work, we introduce a novel framework, Map2World, that first enables 3D world generation conditioned on user-defined segment maps of arbitrary shapes and scales, ensuring global-scale consistency and flexibility across expansive environments. To further enhance the quality, we propose a detail enhancer network that generates fine details of the world. The detail enhancer enables the addition of fine-grained details without compromising overall scene coherence by incorporating global structure information. We design the entire pipeline to leverage strong priors from asset generators, achieving robust generalization across diverse domains, even under limited training data for scene generation. Extensive experiments demonstrate that our method significantly outperforms existing approaches in user-controllability, scale consistency, and content coherence, enabling users to generate 3D worlds under more complex conditions.

1 Introduction

Existing 3D world generators are limited by scarce world-level data, restricted domains, grid layouts, and weak global consistency. Map2World addresses these constraints with arbitrary segment-map conditioning, latent-space multi-diffusion, detail enhancement, and asset-generator priors.

  • World-scale generation remains difficult because high-quality world-level datasets are scarce and harder to construct than object-centric collections.
  • Pre-trained image and video diffusion pipelines suffer from view-dependent inconsistencies, incomplete reconstructions, weak 3D consistency, or limited memory spans.
  • Asset-generator-based methods leverage expressive 3D priors but can fail to model relationships among independently generated regions.
  • Map2World coordinates overlapping diffusion windows in structured latent space to preserve priors, connect regions beyond cube boundaries, and support arbitrary-resolution generation.
  • Map2World conditions generation on user-defined segment maps rather than grid-only layouts, adds fine details while preserving structure, and targets domain-generalized generation.

2 Related work

Prior 3D world-generation methods either reconstruct scenes from generated views or directly generate explicit 3D representations. Their applicability is constrained by consistency problems, limited scene datasets, missing textures, or narrow training domains.

  • View-based methods generate images or videos and lift or reconstruct them into 3D scenes, but their pipelines can produce inconsistencies during expansion and stitching.
  • Direct explicit 3D methods achieve 3D consistency but often train from limited scene datasets, restricting domains and sometimes producing geometry without textures.
  • SCube and InfiniCube improve color estimation or global consistency but remain constrained to driving scenes.
  • Off-the-shelf asset generators broaden world-generation scope and provide strong asset quality, while overall scene quality remains dependent on the available dataset.

3 Preliminary: Structured Latent and Generation Pipeline

TRELLIS represents geometry and appearance with structured local latents and generates them in two stages, progressing from sparse structure to a decodable 3D representation. Map2World’s pipeline builds on this representation and adds map-conditioned world generation followed by detail enhancement.

  • Structured Latent: TRELLIS structured latent, or SLAT, encodes geometry and appearance as local latent vectors paired with positions on a 3D grid.
  • Structured Latent: Each positional index identifies an active voxel in an N^3 grid, while its latent vector encodes geometry and appearance at that position.
  • Map2World Pipeline: Map2World’s overall pipeline first estimates a large-world structured latent using segmentation-map and per-segment text conditions, then upscales scene resolution with a detail enhancer.
  • Generation Pipeline: The pipeline first denoises a sparse structure conditioned on text, decodes it into a signed scalar field, and records the positive-valued active voxel positions.
  • Generation Pipeline: A second flow-Transformer stage estimates latent features that can be decoded into 3D Gaussian splats, radiance fields, or meshes.

4 Proposed Method

Map2World expands asset-generation priors into large, controllable worlds by fusing overlapping latent windows and segment-conditioned velocities. It further targets scale-aware generation and detail enhancement through optimized initialization and coarse-to-fine latent learning.

  • Pipeline: The overall pipeline combines latent fusion, detail enhancement, and decoder fine-tuning to exploit asset-generator knowledge while preserving input-world content and producing higher-quality scenes.The decoder fine-tuning stage is introduced as the final component of the generation process.
  • Latent fusion: Map2World uses overlapping 64-sized 3D windows and Gaussian-weighted velocity fusion to expand generation beyond the original asset generator’s active voxel cube.The fused velocity updates each spatial position during rectified-flow denoising.
  • Segment-map conditioning: Multiple text prompts and arbitrary-shaped segment maps condition local velocities through mask-weighted fusion with diffusion-time-dependent Gaussian smoothing.The smoothing changes from soft boundaries to sharper masks as denoising proceeds, improving stability and supporting arbitrary segment shapes.
  • Scale-aware generation: Scale-aware initialization exploits a scale-dependent sparse-structure manifold and optimizes the initial noisy latent using an approximate denoising trajectory.A stop-gradient approximation avoids backpropagating through the full trajectory, while 3D FFT parameterization stabilizes optimization under large learning rates.
  • Detail enhancement: The detail enhancer learns coarse-to-fine relationships in latent space by training on large scene cubes and their eight axis-split smaller cubes.It is designed to increase world resolution while leveraging TRELLIS priors rather than directly adding details in 3D space.

Network architecture.

The detail enhancer conditions each small-cube prediction on global and neighboring structured latents, then integrates these conditions through shared MLP layers and the original flow Transformer. Initialization preserves the pretrained TRELLIS behavior before fine-tuning.

  • Conditioning: The enhancer predicts each small cube’s structured latent from noise, a spatially truncated large-cube latent, and adjacent-cube latents.The large-cube latent supplies coarse target information, while adjacent latents address cross-cube geometry and texture continuity.
  • Condition fusion: Noise and conditioning latents are concatenated channel-wise and mixed by MLP layers whose input dimension is c + C and output dimension is c.The MLP layers for large-cube and adjacent-cube conditions share parameters.
  • Flow prediction: The mixed features are passed into the original GS and GL flow Transformers, with expanded-region predictions cropped to retain the target cube.This lets neighboring context influence the target while preserving the target-cube output region.
  • Initialization and training: The MLP is identity-initialized so its output initially matches the noise regardless of conditioning, while the pretrained flow Transformer parameters are retained.Only the MLP parameters are updated during fine-tuning.
  • Sampling: After fine-tuning, small-cube latents are sampled autoregressively from index 0 to 7, using already generated adjacent cubes as additional conditions.The eight estimated structured latents are merged into a detailed scene representation.
  • Qualitative comparison: Map2World’s qualitative comparisons show support for arbitrary-shaped maps and seamless world filling, whereas SynCity is limited to square tiles and lacks contextual continuity.The arbitrary-map and grid-map comparisons respectively highlight flexible map support and connected scene structure.

5 Experiments

Experiments evaluate Map2World’s controllability, world-level coherence, scale optimization, and detail-enhancer design against baselines and ablations. The method produces arbitrary-map worlds with seamless region connections, higher WQ/GPTscore quality, and faster scale-constrained optimization.

  • Comparison on World Generation: Map2World aligns generated regions with arbitrary segment-map shapes and prompts while seamlessly connecting adjacent regions.Top-down views match the input map, and neighboring structures transition smoothly.
  • Comparison on World Generation: Map2World generates denser, larger, and more structurally complex worlds than SynCity, avoiding tile gaps and disconnected grid-like assets.It can create structures spanning multiple same-label tiles and connect contents across adjacent segments.
  • Comparison on World Generation: 7.93/10 GPTscore exceeds SynCity’s 7.48/10 across sharpness, world completeness, coherence, and realism.The comparison uses scenes synthesized from captions for 35 NuiScene43 meshes.
  • Comparison on World Generation: WQ combines sharpness, world completeness, coherence, and realism as WQ = 0.15S + 0.45W + 0.25C + 0.15R, emphasizing world completeness.The metric is designed to assess coherent world structure rather than individual-object fidelity.
  • Spectral parameterization for stable initial latent optimization: With spectral-domain parameterization and learning rate 9.0, IoU and Dice reach approximately 0.9 within five optimization steps.Direct sparse-structure optimization is unstable at the same rate, while learning rate 1.0 avoids divergence but requires more steps and computation.
  • Design choices for the detail enhancer: The detail-enhancer ablations compare architectures, classifier-free guidance, decoder fine-tuning, and removal of the enhancer under fixed conditions.IP-Adapter produces disconnected boundaries, CFG causes distortions and saturated colors, and decoder fine-tuning sharpens geometry and textures.

6 Conclusion

Map2World addresses the shortage of suitable world-scale data with a text-guided, TRELLIS-based pipeline for scalable, flexible, and coherent 3D world generation. It combines globally consistent structured-latent generation, arbitrary-shaped segment-map conditioning, and detail enhancement while retaining TRELLIS’s generalization capacity.

  • 6 Conclusion: Map2World targets the lack of appropriate world-scale data that limits flexible and scalable 3D world generation.The paper identifies world-level datasets as difficult to construct compared with object-centric collections.
  • 6 Conclusion: The two-stage pipeline generates a world-level structured latent, then applies a detail enhancer conditioned on global structure.Latent fusion shares features across the world, while the enhancer adds details through a parameter-efficient module before the frozen TRELLIS generator.
  • 6 Conclusion: Map2World supports segment-map-guided generation with arbitrary-shaped regions and produces worlds aligned with the supplied map and prompts.This capability is described as unavailable in existing works.

Supplementary Materials for Map2World: Segment Map Conditioned

The paper frames text-to-3D world generation as a distinct topic within 3D world-generation research.

  • Text to 3D World Generation is identified as the supplementary material’s topic.

S1 Implementation Details

The network architecture handles sparse active-voxel features by interpolating missing conditional values and directly incorporating adjacent-cube latents before self-attention.

  • S1.1 Details on Network Architectures: Trilinear interpolation estimates conditional latent features at target active-voxel positions when those features are absent.The interpolation is fully parallelized for fast execution.
  • S1.1 Details on Network Architectures: Expanding the target cube preserves adjacent-cube latent positions, allowing direct concatenation before self-attention.This incorporates structured latent information from cubes adjacent to the target.

Feature interpolation for structured latent flow Transformer (GL).

The structured latent flow Transformer (GL) uses interpolated and adjacent-cube features to support latent processing across expanded spatial regions. Training and inference accommodate varying neighboring-cube conditions, while implementation follows TRELLIS configurations with limited parameter fine-tuning.

  • Feature interpolation: Target-cube features are estimated by trilinear interpolation from the original condition-latent positions, while expanded adjacent-cube regions directly reuse neighboring features and positions.This feature concatenation process is illustrated in Fig. S1.
  • Fine-tuning: Approximately 4% of total parameters are fine-tuned, a ratio similar to typical LoRA fine-tuning.The parameter counts are reported in Table S1.
  • Training configuration: Fine-tuning follows the original TRELLIS configurations for the flow Transformers and 3DGS decoder, using 100k iterations for all networks.Each network is trained on a single NVIDIA A100 80GB GPU and takes approximately 60 hours.
  • Adjacent-cube conditioning: The model handles between 0 and 3 adjacent cubes during autoregressive inference within one detail-enhancer architecture.Training varies the number of adjacent cubes used as input conditions to support robust sampling across these cases.
  • Loss computation: During loss computation, regions containing noise from adjacent-cube conditions are excluded, leaving only positions corresponding to the target cube.This isolates the target-cube contribution during training.

S2 Additional Experiment Results

Additional experiments evaluate segment-text alignment, arbitrary-shaped conditioning, recursive detail enhancement, and qualitative world generation. The results indicate stable semantic alignment, faithful adherence to free-form maps, and sharper details with surrounding-scene harmony.

  • Segment-text alignment: The CLIP-Score evaluation uses grid-aligned generation for controlled comparison with SynCity and reports qualitative and quantitative results.The metric measures correspondence between each segmented region and its associated textual prompt.
  • Segment-text alignment: Consistent region-text alignment is maintained across alternative CLIP backbones and softmax-normalized variants.These extended quantitative results are presented in Fig. S4.
  • Arbitrary-shaped conditioning: Free-form segmentation masks with textual descriptions produce 3D scenes whose spatial structure and semantic content reflect the user inputs.Quantitative results are aggregated over 50 randomly seeded generations across CLIP backbones and scene configurations.
  • Detail enhancement: Recursive detail enhancement further upscales scene resolution by passing latent tensors through the detail enhancer multiple times.The ×1 condition uses an initial structured latent without latent fusion for evaluating the enhancer alone.
  • Detail enhancement: The detail enhancer sharpens blurred features such as window boundaries while maintaining harmony with surrounding elements.It slightly transforms original contents, and the paper accepts such modifications when they improve text-aligned world quality.
  • Qualitative results: Latent fusion and detail enhancement produce high-quality outputs for irregular segment maps that SynCity cannot handle.The video demonstration likewise shows rendered worlds satisfying image segment-map conditions.

S3 Discussion

The discussion identifies limitations inherited from TRELLIS and from the detail enhancer’s training data. It also bounds the work’s scope to controllable 3D scene generation rather than personal-data, identity, or downstream decision-making applications.

  • Inherited positional limitation: Merging small cubes can alter TRELLIS’s absolute positional information, potentially changing the decoded 3D structure.Relative positional encoding or adaptation training is suggested as a mitigation.
  • Detail-enhancer limitation: The detail enhancer is currently fine-tuned only on scene-level cropped data, limiting its generalizability.The paper proposes object-level and world-level training data to improve quality.
  • Detail-enhancer limitation: More complex and realistic texture datasets could move the detail enhancer toward more photo-realistic results because Objaverse mainly contains simple meshes.This is presented as a direction for improving the enhancer’s quality.
  • Scope boundary: The work focuses on controllable 3D scene generation and does not directly address personal data, identity modeling, or downstream decision-making tasks.The stated application scope is primarily creative or industrial, including simulation, virtual environments, and content generation.
  • Design trade-off: Recursive detail enhancement can introduce content changes, reflecting a quality-oriented rather than exact-reconstruction objective.The paper considers minor modifications acceptable when they enhance overall text-aligned world quality.
Loading 2605.00781v1…