PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneFrom3D: Geometry-Conditioned Outdoor 3D Scene Generation via View Scheduling with Object-Level Control

Geonung Kim, Jeongeun Park, Nuri Ryu, Di Liu, Sunghyun Cho

arXiv:2607.04540v1cs.GRcs.CV

TL;DR

Outdoor 3D scene generation lacks reliable view schedules for arbitrary, large, unstructured geometry. SceneFrom3D automatically schedules views and adds object-level appearance and geometry-adherence control, producing high-quality scenes that follow input layouts and transfer object appearances.

  • Problem

    Existing outdoor 3D scene generation methods lack general view scheduling for arbitrary geometry, limiting reliable coverage and suitable views for reconstruction and generation.

  • Method

    SceneFrom3D builds a directed graph of geometry-covering anchor views and interpolation trajectories, while conditioning each object with an identity image and region-wise geometry adherence.

  • Results

    SceneFrom3D generates high-quality outdoor 3D scenes that preserve input layouts and transfer appearance cues from identity images across diverse geometry layouts.

  • Takeaways & Limitations

    The framework supports outdoor 3DGS generation without explicit camera trajectories while providing object-level control over appearance and geometry adherence.

  • Takeaways & Limitations

    Anchor-view generation may fail beyond eight distinct object identities, and missing global-illumination priors can cause inconsistent shadow direction and size.

Abstract

from arXiv · show

Geometry-conditioned 3D scene generation enables the creation of 3D environments from user-provided geometry, offering direct control over scene structure and object layout. To generate such 3D scenes, current methods commonly adopt a three-stage design that first defines a view schedule, then synthesizes multi-view observations along the scheduled views, and finally reconstructs a 3D representation from the generated images. However, defining the view schedule becomes a major bottleneck for outdoor scenes, where large, unstructured, and unbounded geometry makes it difficult to obtain views that provide sufficient coverage while supporting stable generation. To address this bottleneck, we present SceneFrom3D, a framework that automatically schedules views from outdoor input geometries. SceneFrom3D constructs a directed generation graph whose nodes represent anchor views and whose edges represent interpolation trajectories, defining which views to synthesize, which view pairs to interpolate, and in which order generation should proceed. Beyond automatic view scheduling, SceneFrom3D further improves controllability through object-level conditioning, assigning each object an identity image for appearance guidance and a geometry-adherence parameter for region-wise control over the input geometry. Experiments demonstrate that SceneFrom3D achieves state-of-the-art geometry-conditioned outdoor 3D scene generation, producing high-quality scenes with controllable object appearance and geometry adherence.

1 Introduction · 2 Related Work · 3 Methods

SceneFrom3D addresses geometry-conditioned outdoor 3D scene generation by automatically scheduling generation views and adding object-level appearance and geometry-adherence control. Its pipeline constructs a directed generation graph, synthesizes anchor and interpolated observations, and optimizes a 3DGS scene aligned with the input geometry.

  • 1 Introduction: The method follows a three-stage design: view scheduling, diffusion-based multi-view synthesis, and reconstruction of a NeRF or 3DGS representation from generated observations.This decomposition exploits image or video diffusion priors while avoiding direct geometry-to-complete-scene training.
  • 1 Introduction: Object-level conditioning assigns each object an identity image for appearance guidance and a geometry-adherence parameter for region-wise structural control.During anchor-view synthesis, larger geometry-adherence values enforce stronger adherence to the corresponding input geometry.
  • 1 Introduction: SceneFrom3D automatically selects anchor views, interpolation trajectories, and generation order from arbitrary outdoor geometry, removing the need for explicit camera trajectories.The directed generation graph uses nodes for anchor views and edges for interpolation trajectories, organizing the three-stage generation pipeline.
  • 2 Related Work: Related work progresses from object-level geometry-guided generation and forward-facing multi-view synthesis toward scene-level generation with meshes, layouts, semantic maps, and coarse geometry.View scheduling is also connected to next-best-view planning and aerial path planning, while object-level conditioning follows localized guidance trends in 2D and 3D generation.
  • 3.3.1 Anchor-view generation.: Multi-view generation synthesizes anchor images from parent-view warps, rendered depth, object masks, identity images, adherence-conditioned corruption, and text prompts, then densifies observations with video diffusion along graph edges.The resulting posed RGB observations and mesh-rendered depth maps supervise subsequent 3DGS optimization.
  • 3.4 3DGS optimization: The final 3DGS optimization combines RGB reconstruction, DSSIM, LPIPS, and depth losses so generated appearance is matched while the reconstruction remains aligned with the input meshes.LPIPS reduces perceptual inconsistencies inherited from the generative model, whereas the depth term preserves geometric alignment.
  • 3.2.1 Node construction.: View scheduling constructs a compact, coverage-oriented anchor set using visibility-guided densification, camera refinement, and losses for coverage, collision avoidance, and camera tilt.The visibility score combines field-of-view inclusion, preferred distance, front-facing orientation, and non-occlusion; redundant views are removed or merged during refinement.
  • 3.2.2 Edge construction.: The scheduler connects anchor views with shared visibility and collision-free trajectories, adds distance-based links for disconnected or leaf nodes, and orients edges by generation order to form an acyclic graph.Topological sorting yields a valid anchor-view generation order, with earlier views serving as parent conditioning views.

4 Experiments

Experiments show that SceneFrom3D faithfully generates outdoor scenes from diverse layouts, transfers object-level appearances, and produces high-quality renderings from unseen viewpoints. It outperforms the evaluated baselines across all reported metrics, while ablations validate the roles of its losses, interpolation, refinement loop, and geometry-adherence control.

  • Qualitative results: Across diverse layouts, generated scenes preserve input geometry, transfer identity-image appearances, and render consistently from unseen viewpoints.The scenes follow object arrangements and shapes, transfer cues to elements such as sky, ground, trees, buildings, and walls, and maintain quality from cameras A–D not used for generation.
  • Baseline comparison: SceneFrom3D outperforms UrbanArchitect, YoNoSplat, and Zhang et al.’s method across all reported metrics.Table 1 compares visual-quality and structural-fidelity metrics against the baselines and ablation variants.
  • Ablation studies: Removing tilt or repulsion losses causes incomplete coverage or invalid camera configurations that degrade reconstruction quality.Without tilt loss, cameras bias toward the ground; without repulsion loss, cameras can enter the mesh, producing invalid depth maps and path collisions.
  • Ablation studies: Removing video interpolation leaves insufficient views for stable 3DGS training and degrades rendering quality despite a potentially favorable visual-quality metric.The reported metric advantage is attributed to high-frequency artifacts favored by the metric.
  • Refinement analysis: The refinement loop improves coverage by removing low-contribution cameras, merging similar-visibility cameras, and adding cameras for under-covered samples.Without refinement, occlusions during initialization create redundant nearby cameras with overlapping observations.
  • Geometry-adherence analysis: Increasing geometry-adherence parameter α makes generated objects follow detailed input structure more closely, while decreasing α relaxes the constraint.With α near one, details such as a roof dormer, doors, and windows are accurately reflected in the generated result.

5 Conclusion

SceneFrom3D introduces a geometry-conditioned framework for outdoor 3D scene generation that combines automatic view scheduling with anchor-view interpolation, enabling high-quality 3DGS generation without explicit camera trajectories. It also provides object-level control over appearance and geometry adherence, while remaining limited by crowded anchor views and inconsistent global illumination.

  • SceneFrom3D combines automatic view scheduling with an anchor-view-and-interpolation pipeline for high-quality 3DGS generation in large, unstructured outdoor scenes without explicit camera trajectories.
  • The framework provides object-level control over object appearance and geometry adherence.The paper presents this as a first step toward object-level controllable 3D scene generation.
  • Limitations: Anchor-view generation may fail when a single view contains more than eight distinct object identities, and missing global-illumination priors can cause inconsistent shadows across anchor views and the 3DGS output.The limitation is attributed to the pretrained model's limited number of effectively incorporated input images; shadow inconsistencies are shown in Fig. 12.

Supplemental Document

The supplemental document provides a detailed inspection of SceneFrom3D through a video and additional implementation details, statistics, and analyses. It covers geometry, view scheduling and generation, 3DGS optimization, latency, comparisons, and ablations.

  • Supplemental Document: The supplement provides a video for detailed inspection and adds implementation details, statistics, and analyses for SceneFrom3D.The authors state that code and datasets will be released upon acceptance.
  • Supplemental Document: It documents input geometry, ground meshes, view scheduling, anchor-view generation and interpolation, and 3DGS optimization.
  • Supplemental Document: It reports scene statistics and latency, compares view-scheduling methods, and includes ablations of the soft visibility score and 3DGS training loss.

S1 Input Geometry and Ground Mesh

The input geometry combines controllable object meshes with an oversized ground mesh, whose expanded extent prevents boundary-induced artifacts during view generation.

  • Input Geometry and Ground Mesh: The ground mesh covers roughly ten times the objects’ occupied area, preventing its boundary from appearing in depth maps and causing floating ground patches in the sky.This design avoids artifacts during view generation caused by a ground mesh that is too small.

S2 View Scheduling Details

This section summarizes the numerical parameters used in the main-paper view-scheduling formulation and the supplementary scheduling procedure. Table S1 provides these parameter settings.

  • Table S1 summarizes the numerical parameters used by the main-paper view-scheduling formulation.
  • The supplementary scheduling procedure also has its numerical parameters summarized in Table S1.
  • The section’s scheduling details are presented as a parameter summary rather than a description of additional scheduling results.

S2.1 Surface Samples and Filtering

Surface samples are drawn from object meshes and an expanded object-centered region of the ground mesh, then filtered for approximately uniform spacing and usable surface visibility. Samples with immediately occluded outward normals are removed because contact configurations make them uninformative for camera placement.

  • Sampling region: Ground sampling is restricted to the object meshes’ XY bounding box expanded by a fixed factor, preventing the larger ground mesh’s empty areas from dominating.Object and ground meshes both contribute samples, but ground samples are limited to this region of interest.
  • Surface sampling: Area-weighted barycentric triangle sampling followed by 3D Poisson-disk filtering produces approximately uniform surface samples.For surface area A and spacing h, approximately ⌈A/h^2⌉ candidates are drawn, with minimum Poisson-disk distance h.
  • Filtering: A normal-clearance filter removes samples whose outward normals immediately intersect nearby geometry, excluding occluded contact-region samples from camera placement.This addresses arbitrary user-specified mesh contacts, such as a house floor touching the ground mesh.

S2.2 Node Initialization

Node initialization builds a coverage-oriented anchor-view set from filtered surface samples, then refines it for compactness before continuous pose optimization. Candidate cameras are restricted by geometric visibility and camera-orientation criteria.

  • S2.2 Node Initialization: After initialization and refinement, continuous pose optimization proceeds with the camera count fixed under the objective defined in the main paper.Algorithm 1 summarizes the complete initialization procedure from filtered surface samples to the anchor-view set.
  • S2.2 Node Initialization: Candidate cameras are placed along outward surface normals at a preferred distance, shortened when geometry blocks the ray, and retained only when pitch and visibility thresholds are satisfied.Samples failing these criteria remain available for coverage evaluation but cannot initialize cameras.
  • S2.2 Node Initialization: The progressive initialization repeatedly adds cameras from under-covered eligible samples, orienting each new camera toward its selected surface sample until coverage cannot be expanded.Under-coverage is determined using the initialization visibility threshold δvis.
  • S2.2 Node Initialization: A refinement loop improves compactness by adding cameras for remaining under-covered samples, removing low-contribution cameras, and merging cameras with similar visibility vectors.Merged poses use a visibility-weighted mean, with thresholds δdense, δremove, and δmerge; refinement stops when the camera set stabilizes.

S2.3 Edge Construction · S2.4 Direction Construction and Generation Order

The method represents scheduled connections with smooth camera trajectories and converts the resulting view graph into a directed acyclic graph with an order for anchor-view generation. Each target anchor is generated after its parent views under a canonical edge orientation.

  • S2.3 Edge Construction: Each selected edge stores a smooth camera trajectory instead of only its endpoint cameras.This design aims to mimic natural in-domain camera motion during video interpolation, where cameras follow gentle paths around the scene.
  • S2.3 Edge Construction: For an accepted edge (v_i, v_k), the trajectory is defined from endpoint camera centers and forward directions using a cubic Bezier curve.
  • S2.4 Direction Construction and Generation Order: The scheduler begins with an undirected edge set over the anchor views.
  • S2.4 Direction Construction and Generation Order: A canonical order index rho(v_i) in {1, ..., N_v} orients every edge from the lower-order endpoint to the higher-order endpoint.
  • S2.4 Direction Construction and Generation Order: Because rho strictly increases along directed edges, the resulting generation graph G_gen = (V, E→) is a directed acyclic graph.
  • S2.4 Direction Construction and Generation Order: Anchor views are generated in any topological ordering, with each target view generated after its parent views.For target anchor v_k, its parents are P_a(v_k) = {v_i | (v_i, v_k) ∈ E→}.

S3 Anchor-view Generation Details · S3.1 Dataset Synthesis · S3.2 Training

SceneFrom3D trains anchor-view generation with synthetic paired conditions combining sparse observations, corrupted geometric guidance, and object-specific identity-region controls. The dataset uses reusable identity images, architecture-specific grid conditioning, semantic masks, and rule-based prompts, while LoRA fine-tuning adapts FLUX.2 Klein with the text encoder and VAE frozen.

  • S3 Anchor-view Generation Details: The synthetic training dataset pairs each target anchor image with sparse RGB evidence, corrupted depth and boundary cues, and identity-region conditions for visible objects.These paired inputs mirror the anchor-view generation conditioning used by SceneFrom3D and provide appearance, partial-observation, and coarse-geometry guidance.
  • S3.1.1 Identity Images & Anchor Image.: Architecture components use a 3-by-3 grid of multi-angle identity variants because single building references induce strong pose bias toward the reference viewpoint.Qwen-Image-Edit-2511 with a multi-angle LoRA generates the posed variants, and this grid condition is restricted to architecture components.
  • S3.1.1 Identity Images & Anchor Image.: Grounded-SAM25 estimates sky, ground, and object masks, pairing each visible object identity image with its corresponding image region.The identity-region condition is {(I_o, M_j,o)} for visible objects, so appearance guidance is tied to the region where it should be expressed.
  • S3.1.2 Semantic Mask.: Sparse RGB conditions are synthesized by applying stripe-like or block-shaped masks and Bernoulli pixel dropping to anchor images, with zeroed inputs representing no observation.The retained pixels form P_j, while O_j records the observed regions.
  • S3.1.3 Partial Observation.: Geometric conditions combine MoGe-2 depth estimates corrupted by per-pixel strength maps with HED boundary cues, training coarse-structure adherence under imperfect guidance.The resulting pair (D̂_j, S_j) supplies approximate geometry without requiring exact target appearance.
  • S3.1.4 Structure Condition: A rule-based template specifies the fixed image-slot layout and appends object-specific clauses so the model receives consistent instructions for observations, geometry, and identity regions.The prompt explicitly asks for one coherent scene that follows all conditions, remains faithful to the depth input, and avoids extra objects.
  • S3.2 Training: Training fine-tunes FLUX.2 Klein with rank-128 LoRA adapters on transformer attention projections while freezing the text encoder and VAE.Conditioning images are encoded as image tokens after the noisy target latent, with separate coordinates distinguishing the target from paired conditions; alpha is 128 and dropout is 0.05.

S3.3 Anchor-view Verification · S4 Anchor-view Interpolation Details · S5 3DGS Optimization Details

The supplementary sections specify safeguards for anchor-view generation, the configuration of anchor-view interpolation, and the losses and view-dependent LPIPS weighting used for 3DGS optimization.

  • S3.3 Anchor-view Verification: Anchor-view candidates are checked with a semantic overflow ratio comparing predicted visible-object masks against target object regions, excluding sky and ground.This detects stochastic object-content placement outside intended semantic regions when multiple object references are conditioned together.
  • S3.3 Anchor-view Verification: Candidates exceeding an overflow ratio of 0.01 are resampled with new random seeds, with at most six attempts before retaining the lowest-overflow candidate.The first candidate below the threshold is accepted; otherwise, the candidate with the lowest ratio is kept.
  • S3.3 Anchor-view Verification: Verification only filters stochastic anchor-view failures and leaves the scheduled view order and conditioning inputs unchanged.
  • S4 Anchor-view Interpolation Details: Each scheduled edge is interpolated with Wan2.1-VACE-14B using 49 frames, fixed endpoint anchor images, 47 normalized-depth control frames, and 30 denoising steps.The same fixed prompt is used for every edge.
  • S4 Anchor-view Interpolation Details: The interpolation prompt enforces temporal coherence, endpoint appearance fidelity, trajectory-following geometry and camera motion, consistent layout and identity, and artifact avoidance.It explicitly asks the model to avoid flicker, deformation, new objects, and abrupt transitions.
  • S5 3DGS Optimization Details: 3DGS optimization combines RGB reconstruction, DSSIM, metric-depth, and LPIPS losses, with numerical parameters summarized in Table S2.The implementation follows CAT3D [Gao et al. 2024] for view-dependent LPIPS weighting.
  • S5 3DGS Optimization Details: LPIPS weighting is zero on anchor views and maximal at the midpoint of each interpolated edge, using midpoint-normalized edge weights.For edge frames, the weighting depends on their position along the 49-frame trajectory.
  • S5 3DGS Optimization Details: Supplementary summaries report per-layout object counts, graph nodes and edges, generated views, and runtimes covering scheduling, generation, 3DGS training, and the full pipeline.A qualitative figure compares view scheduling with Zhang et al. [2021], using yellow anchor-view frustums and green interpolation edges.

S6 Scene Statistics & Latency … S9 Ablation on 3DGS Training Loss

The supplementary analyses characterize SceneFrom3D’s scene-scale runtime and view scheduling, then show how visibility terms and 3DGS losses affect coverage, geometry alignment, and appearance quality. Together, these studies identify failure modes from poor camera placement or omitted training objectives.

  • S6 Scene Statistics & Latency: SceneFrom3D reports per-layout object, graph, observation, stage-latency, and total-runtime statistics measured on NVIDIA A100-80G GPUs.View scheduling and 3DGS training used one GPU each, while multi-view generation used two GPUs.
  • S7 View Scheduling Comparison: Across four layouts, SceneFrom3D schedules sparse anchor views and interpolation trajectories that cover scene structure, unlike Zhang et al. [2021]’s denser, overhead-biased placements.The denser placements produce redundant observations compared with SceneFrom3D’s scheduled views.
  • S8 Analysis on Soft Visibility Score: Removing the FOV term makes view optimization nearly random with respect to actual coverage, producing cameras that observe little informative scene content or redundant nearby views.Without Vfov, samples outside the camera view can receive visibility scores similar to visible samples.
  • S8 Analysis on Soft Visibility Score: Removing the distance term permits cameras whose target regions are blocked by nearby foreground geometry when background samples dominate visibility scoring.The resulting views can be largely occluded by close objects.
  • S8 Analysis on Soft Visibility Score: The front-facing term favors frontal rather than oblique observations, yielding clearer and less distorted conditioning views for image generation.Without Vfront, cameras may observe surfaces at slanted angles and degrade generated-view quality.
  • S9 Ablation on 3DGS Training Loss: Removing the depth loss allows Gaussians to drift from the input geometry and form floaters that can occlude scene elements, whereas depth consistency suppresses these artifacts.The depth loss keeps the optimized 3DGS better aligned with the input geometry.
  • S9 Ablation on 3DGS Training Loss: Removing the LPIPS loss overly smooths high-frequency appearance details and produces blurry textures, while perceptual supervision preserves sharper, more detailed textures.The difference is particularly visible in the highlighted ground region.
Loading 2607.04540v1…