PaperlayerView exact revision on arXiv

Source-linked AI summary

FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flow

Zhifei Yang, Guangyao Zhai, Keyang Lu, YuYang Yin, Chao Zhang, Zhen Xiao, Jieyi Long, Nassir Navab, Yikai Wang

arXiv:2603.19598v1cs.CV

TL;DR

Existing language-driven methods offer limited object control and style coherence, while graph-based methods struggle with high-fidelity textured scenes. FlowScene uses a tri-branch multimodal-graph rectified-flow generator, and experiments report gains in realism, controllability, style consistency, efficiency, and human preference.

  • Problem

    Language-driven retrieval overlooks object-level control and scene-level style coherence, while graph-conditioned methods struggle with high-fidelity end-to-end textured-scene generation.

  • Method

    FlowScene uses three coupled branches conditioned on multimodal graphs to generate layouts, object shapes, and textures while exchanging node information during sampling.

  • Results

    FlowScene reports stronger realism, style consistency, and human preference alignment than language-driven and graph-conditioned baselines, with improvements in object and holistic-scene quality.

  • Takeaways & Limitations

    FlowScene provides fine-grained control over object appearance and relations while preserving scene-level style consistency across structure and texture.

  • Takeaways & Limitations

    Experiments use a single dataset focused primarily on synthetic indoor environments, and outdoor or larger, more diverse scenes remain unevaluated.

Abstract

from arXiv · show

Scene generation has extensive industrial applications, demanding both high realism and precise control over geometry and appearance. Language-driven retrieval methods compose plausible scenes from a large object database, but overlook object-level control and often fail to enforce scene-level style coherence. Graph-based formulations offer higher controllability over objects and inform holistic consistency by explicitly modeling relations, yet existing methods struggle to produce high-fidelity textured results, thereby limiting their practical utility. We present FlowScene, a tri-branch scene generative model conditioned on multimodal graphs that collaboratively generates scene layouts, object shapes, and object textures. At its core lies a tight-coupled rectified flow model that exchanges object information during generation, enabling collaborative reasoning across the graph. This enables fine-grained control of objects' shapes, textures, and relations while enforcing scene-level style coherence across structure and appearance. Extensive experiments show that FlowScene outperforms both language-conditioned and graph-conditioned baselines in terms of generation realism, style consistency, and alignment with human preferences.

1 Introduction

FlowScene targets controllable, realistic indoor scene generation while addressing weak object-level control, relation handling, and style coherence in prior approaches. Its tri-branch multimodal-graph design jointly generates layouts, shapes, and textures through tightly coupled node exchanges.

  • Scene generators must control object categories, geometry, spatial relations, and appearance while preserving style consistency across structure and visual appearance.
  • Language-driven retrieval methods provide limited per-object control, overlook inter-object relations, and often produce mismatched scale, topology, and appearance.
  • Graph-based methods improve object and relation controllability but struggle to generate textured scenes end-to-end, limiting fidelity and downstream utility.
  • FlowScene uses a multimodal scene graph and three branches to generate scene layouts, object shapes, and object textures.Nodes can fuse textual and visual object information.
  • Multimodal Graph Rectified Flow exchanges node information during denoising, coupling object states across graph edges for fine-grained appearance and relation control.
  • FlowScene outperforms language-driven and graph-conditioned competitors in realism, style consistency, and human preference alignment.It also accelerates graph-conditioned generation while improving individual-object and holistic-scene quality.

2 Related Work

Prior 3D scene synthesis methods use text, graphs, or images, but each modality and modeling family leaves gaps in holistic 3D reasoning, unified multimodal conditioning, or combined geometry-appearance control. FlowScene is positioned to unify multimodal inputs with geometric, appearance, and relation control.

  • Scene Graph and Applications: Scene graphs represent objects and directed inter-object relations and support retrieval, reasoning, synthesis, manipulation planning, and instruction-conditioned navigation.
  • Scene Graph and Applications: Multimodal scene graphs integrate textual and visual information at the object-node level.
  • Rectified Flow and Applications: Rectified flow offers deterministic ODE sampling and straight-line supervision as an alternative to diffusion-based generation, supporting few-step generation.
  • 3D Scene Synthesis: Text-conditioned methods can produce incomplete or ambiguous spatial structure, while image-based methods face fixed-viewpoint limits and cross-view inconsistencies.
  • 3D Scene Synthesis: Existing systems do not unify multimodal inputs with geometric and appearance control and reliable relation compliance, motivating FlowScene.

3 Preliminary

The preliminary framework defines rectified flow as straight-line transport between data and a prior, then introduces multimodal scene graphs and graph message passing for relational conditioning. These components provide the basis for FlowScene’s denoising-time information exchange.

  • Rectified Flow: Rectified flow learns a time-dependent velocity field that transports data from the data distribution to a simple prior through an ODE.
  • Rectified Flow: Training interpolates linearly between data d0 and prior d1, using the constant target velocity d1 − d0 for least-squares regression.
  • Rectified Flow: During sampling, rectified flow draws from the prior and integrates the reverse-time ODE from t=1 to t=0, optionally conditioning on side information.
  • Multimodal Graph: A multimodal scene graph represents object nodes and relations, with nodes combining learnable embeddings and textual or visual foundation features.
  • Multimodal Graph: Missing modalities are zero-padded, allowing text-only, image-only, and multimodal nodes within one graph representation.
  • Multimodal Graph: A triplet-GCN performs message passing between connected nodes and mean-pooling aggregation for each node.
  • Training: The training algorithm defines node-wise data samples and a prior for the multimodal graph rectified-flow process.

4 Method

FlowScene uses graph-conditioned rectified flow to generate layouts, shapes, and textures through coordinated branches that exchange object information. Its design couples individual object quality with inter-object and scene-level consistency.

  • Multimodal Graph Rectified Flow: Graph rectified flow jointly models multiple contents so object information is exchanged during generation for individual quality and inter-content consistency.The model extends single-content rectified flow with tightly coupled conditions and graph-based feature aggregation.
  • Multimodal Graph Rectified Flow: The method trains a denoiser on linear interpolations between target data and a known prior, using graph conditions to approximate the constant target vector field.Inference starts from Gaussian noise and integrates the learned field backward toward the target distribution.
  • FlowScene: FlowScene has three branches that generate scene layouts, object shapes, and object textures before synthesizing the final scene from textured shapes.The layout branch generates 3D bounding boxes, the shape branch decodes latent shape codes into voxelized shapes, and the texture branch follows the generated geometry.
  • FlowScene: The layout branch represents each object with location, size, and rotation, while LayoutExchangeUnit extracts global layout constraints for consistent generation.Rotation is encoded using sine and cosine, and the branch is trained independently with the rectified flow objective.
  • FlowScene: The texture branch preserves geometry while denoising texture features and exchanges information across nodes to produce style-consistent object appearances.Multiview DINOv2 features and a texture VQ-VAE provide structured texture latents, with exchanged information especially guiding text-only nodes.

5 Experiments

Experiments evaluate FlowScene across realism, controllability, style consistency, efficiency, and qualitative fidelity using SG-FRONT and 3D-FRONT. Across these evaluations, FlowScene generally outperforms language-driven and graph-conditioned baselines while maintaining robust performance across modality ratios.

  • Experimental setup: Experiments use SG-FRONT and 3D-FRONT, with SG-FRONT covering about 45K object instances, 15 relationship categories, and bedroom, dining-room, and living-room scenes.Each multimodal graph selectively includes textual and visual modalities at the node level.
  • Experimental setup: FlowScene is compared with training-free language-based retrieval methods and graph-conditioned generative models under a shared evaluation protocol.Language-based baselines include Holodeck and LayoutVLM; graph-conditioned comparisons include CommonScenes and other graph-based methods.
  • Quantitative results: FlowScene consistently outperforms graph-based baselines in scene-level fidelity, including bedroom improvements over MMGDreamer of 7.37 lower FID, 1.34 lower FIDCLIP, and 0.20 lower KID.It also achieves the lowest FID and KID in the full generation setting, while LC and VQ indicate the best human-preference alignment.
  • Quantitative results: FlowScene improves object-level realism across most categories, reducing nightstand and lamp MMD by 43.90% and 41.76% and increasing COV by 45.43% and 40.30% over MMGDreamer.Its 1-NNA results also indicate closer matching of the target shape distribution.
  • Quantitative results: FlowScene attains the highest CLIPScore across room types and outperforms all baselines on FPVScore prompt adherence and perceptual-study prompt adherence.These evaluations assess alignment with textual descriptions and instruction-following quality.
  • Quantitative results: FlowScene outperforms all baselines on visual quality and style consistency, while parallel layout-and-shape generation takes 6.83 seconds, 84.93% faster than MMGDreamer.With texture generation enabled, total inference time remains fastest at 37.38 seconds; style consistency remains stable at SC 3.79 with variance 0.03 across modality ratios.

6 Discussion

FlowScene is demonstrated as a robust multimodal-input scene generator, but its scalability beyond the single evaluated dataset remains unexplored.

  • Experiments show FlowScene serves as a robust scene generator driven by multimodal input.
  • Current experiments use a single dataset, leaving scalability to larger and more diverse environments for future study.Future directions include larger and outdoor scenes and deeper integration with interactive design and planning tools.

7 Conclusion

FlowScene jointly generates layouts, shapes, and textures from multimodal graphs while preserving scene-level style consistency. Its graph rectified flow exchanges object information during sampling and improves efficiency, with experiments showing gains across key evaluation dimensions.

  • FlowScene is a tri-branch multimodal-graph generator for scene layouts, object shapes, and object textures.
  • Multimodal Graph Rectified Flow exchanges object information during sampling to control per-object appearance and relations.The module is used as the backbone of each generation branch.
  • The graph rectified flow module improves efficiency compared with diffusion-based pipelines.
  • Experiments show consistent gains in realism, controllability, style consistency, and human preference.

Supplementary Material Overview.

The supplementary material covers preprocessing, experiments, implementation, evaluation protocols, applications, and limitations.

  • Supplementary materials include the object preprocessing pipeline, additional experimental results, and qualitative comparisons.
  • Implementation details and FPVScore criteria and perceptual study protocols are provided in the appendices.
  • The appendices describe language-driven and GUI-driven application interfaces, along with limitations and failure cases.
  • Object preprocessing voxelizes each object into x_i and trains Φ(E,D) to obtain a latent shape code s_i.Multi-view DINOv2 features are rendered, reprojected onto the voxel representation, and averaged across views.
  • The processed representation is decoded back to the original object o_i.

B Additional Results

Additional evaluations report strong graph-constraint preservation, inter-shape consistency, robustness to multi-view inputs, and improved fidelity with denser relations.

  • Scene Graph Constraints: FlowScene achieves the best performance on most scene-graph constraints while preserving spatial consistency.The evaluated modes are Relationship Change, Node Addition, and Generation Only.
  • Inter-Shape Consistency: FlowScene consistently yields the lowest Chamfer Distance across room types and object categories.Lower Chamfer Distance denotes higher consistency between identical object shapes.
  • Ablation on Different Input Views: Training only on single-view images causes a 7.97 FID increase when testing on multi-view inputs.Multi-view training instead yields greater robustness across input views.
  • Ablation on the Mask Ratio of Relations: As relation mask ratios decrease, performance consistently improves, indicating stronger guidance from denser inter-object relations.FlowScene still outperforms the unmasked baseline at a 50% mask ratio.

C Qualitative Results

FlowScene improves qualitative scene-level consistency and controllability while producing more faithful individual object geometry than graph-conditioned and object-generation baselines.

  • Scene-Level Generation: FlowScene produces more consistent styles and more plausible layouts than graph-conditioned baselines.In the Dining Room example, baseline scenes often show disparate object styles or chaotic placements, whereas FlowScene generates unified chair appearance and texture.
  • Object-Level Generation: FlowScene reconstructs sharper, more precise object geometries than MMGDreamer from single-view inputs.MMGDreamer produces over-smoothed, distorted geometries, while FlowScene captures sharp edges and complex topological structures.

D Implementation Details

The implementation uses multimodal scene-graph data, three Transformer-based rectified-flow branches, graph message passing, and standardized automated and perceptual evaluations.

  • Data Preparation: SG-FRONT extends 3D-FRONT with 15 spatial, support, and style relationship types for modeling object interactions.Examples include left/right, above, standing on, same material as, and same style as.
  • Model Architectures: All three branches use Transformer-based flow-matching backbones with five-layer triplet-GCN InfoExchangeUnits for iterative message passing.Each branch has a modality-specific unit and dedicated projector; graph-convolution parameters are not shared across branches.
  • Model Architectures: Shape and texture branches use VQ-VAE models to encode high-dimensional structured representations into compact discrete latent codes.The shape branch uses a 3D convolutional VQ-VAE for sparse voxelized geometry.
  • Evaluation: FlowScene is evaluated with FPVScore across prompt adherence, layout correctness, visual quality, style consistency, and overall preference.The evaluation uses 12 static multi-view images per method and ranks five methods from 1 to 5 across the criteria.
  • Evaluation: A perceptual study compares five methods across 20 scenes with 25 voluntary participants using randomized output order.Participants rate each method through a custom web-based interface.
  • Applications: The workflow supports language-driven and GUI modes that convert user descriptions, images, selections, and optional relations into multimodal graphs for scene generation.Language inputs may be parsed by an LLM, while GUI users select objects from multi-view images or provide text descriptions.

H Limitations

The reported scope is primarily synthetic indoor environments, and generation quality depends on the accuracy and completeness of upstream multimodal scene graphs.

  • Scope: FlowScene has primarily been evaluated on synthetic indoor environments such as 3D-FRONT, not outdoor scenes.The authors identify broader-domain applicability as future work.
  • Input Dependence: Inaccurate or ambiguous graphs from upstream LLM or VLM constructors may propagate errors into the generated scene structure.The paper links limited or biased input-image viewpoints to imperfect object shapes.
  • Input Dependence: Missing key spatial-relation edges can reduce layout constraints and result in object interpenetration.The authors present such failure cases and plan to improve robustness to noisy graph inputs.
Loading 2603.19598v1…