PaperlayerView exact revision on arXiv

Source-linked AI summary

The Scene Language: Representing Scenes with Programs, Words, and Embeddings

Yunzhi Zhang, Zizhang Li, Matt Zhou, Shangzhe Wu, Jiajun Wu

arXiv:2410.16770v2cs.CVcs.AI

TL;DR

Precisely describing a scene requires structure, semantics, and instance identity, while natural language alone can be difficult to use for that task. The Scene Language combines programs, words, and embeddings and infers the representation from text or images using pre-trained language models. It produces high-fidelity 3D and 4D scenes and supports precise editing.

  • Problem

    Precisely describing a scene requires structural, category-level, and instance-specific information, which natural language alone can be difficult to convey.

    Source: §1 · p. 1

  • Method

    The Scene Language combines programs for scene structure, words for entity semantics, and embeddings for entity identity, with training-free inference from text or images.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    The Scene Language produces 3D and 4D scenes with high fidelity and enables easy and precise editing.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Compared with direct 3D scene generation using MVDream, the approach is compositional and adheres more closely to prompts in multi-object scenes.

    Source: §6.1 · p. 6

    The inference pipeline inherits backbone-model limitations: textual variations can change output quality, and image parsing can vary across inference runs.

    Source: E. Limitations · p. 16

Abstract

from arXiv · show

We introduce the Scene Language, a visual scene representation that concisely and precisely describes the structure, semantics, and identity of visual scenes. It represents a scene with three key components: a program that specifies the hierarchical and relational structure of entities in the scene, words in natural language that summarize the semantic class of each entity, and embeddings that capture the visual identity of each entity. This representation can be inferred from pre-trained language models via a training-free inference technique, given text or image inputs. The resulting scene can be rendered into images using traditional, neural, or hybrid graphics renderers. Together, this forms a robust, automated system for high-quality 3D and 4D scene generation. Compared with existing representations like scene graphs, our proposed Scene Language generates complex scenes with higher fidelity, while explicitly modeling the scene structures to enable precise control and editing.

1. Introduction

Natural language alone can be difficult for precisely describing a scene, which needs structural, category-level, and instance-specific information. The Scene Language combines programs, words, and embeddings, with training-free inference from text or images and a rendering module.

1. Introduction

  • A complete scene description needs structure, category-level semantics, and instance-specific attributes, which natural language alone can be difficult to convey precisely.Programs, words, and embeddings respectively capture these complementary kinds of information.Source: §1 · p. 1
  • Existing techniques ground programs, words, and identity embeddings in different ways, but none of these representations alone is sufficient for scene generation and editing.Pre-trained language models infer programs [3], while textual inversion [6] and low-rank adaptation [9] can capture object identity.Source: §1 · p. 2
  • The Scene Language integrates programs for scene organization, words for semantic groups, and embeddings for each entity’s instance-specific attributes.Programs can specify entity organization, including poses and structural regularity such as repetition.Source: §1 · p. 2
  • A training-free module uses pre-trained language models to infer the representation from text or images, and a generic renderer renders it into images.The paper also reports empirical results on text- and image-prompted scene generation and editing.Source: §1 · pp. 2–3

2. Related Work

Related work is organized around program-based representations, semantic graphs, and generative-model latent spaces. The discussion contrasts their structural, semantic, and visual properties with the Scene Language.

2.1. Representing Scenes as Programs

  • Program representations can express hierarchy and repetition; unlike renderer-specific zero-shot methods, the Scene Language permits parameters in high-dimensional embedding spaces.ShapeAssembly [17] shares function abstraction with the Scene Language but uses cuboids and does not model appearance.Source: §2.1 · p. 3

2.2. Representing Scenes with Semantic Graphs

  • Semantic graph representations preserve high-level scene or object semantics but often omit geometric, textural, or relational details that language or hand-crafted rules cannot fully specify.The Scene Language addresses this issue through embeddings.Source: §2.2 · p. 3

2.3. Representing Scenes with Generative Model Latents

  • Generative-model latent spaces can capture exact visual content, including geometry and appearance, but typically do not offer interpretable semantics or explicitly encode hierarchical scene structure.Source: §2.3 · p. 3

3. The Scene Language

The Scene Language represents scene structure, semantics, and visual content with programs, natural-language words, and neural embeddings. Its formal definition and DSL specify how entity functions compose and arrange entities, while a renderer can convert the executed representation into an image.

3. The Scene Language

  • Programs encode entity relations and structure, words denote each entity’s semantic group, and neural embeddings capture its visual details and identity.The representation also supports an expressive input parameter space for visual details.Source: §3 · p. 3, §3.1 · p. 4, §3.2 · p. 5
  • After program execution, a renderer reparameterizes the resulting entity data object into its own parameter space before rendering the image.The renderer examples are listed in Table 2.Source: Figure 3 · p. 4, Table 2 · p. 4

3.1. Formal Definition

  • The formal representation is Φ(s) := (W, P, Z): words W index entity functions P, and ordered embeddings Z specify entity identities and descendant identities.The embedding order follows a depth-first traversal of the computation graph.Source: §3.1 · p. 4
  • Entity functions recursively transform and compose sub-entities, representing hierarchy, repeated classes, and precise relative layouts such as an 8 × 8 grid.During execution, the root function expands recursively into the scene’s entities.Source: §3.1 · p. 4

3.2. Scene Language as Programming Language

  • The DSL defines entity functions using call, union, union-loop, and transform, with bind and retrieve associating functions with words.An unbound word denotes a primitive entity with no children.Source: §3.2 · p. 5

4. Rendering

Rendering first executes the Scene Language program to produce an entity data object, then maps it into a graphics renderer’s parameter space. The framework specifies this mapping and describes renderer instantiations spanning differentiable 3D, primitive, asset-based, and feed-forward approaches.

4. Rendering

  • A graphics renderer maps the executed Entity object into its parameter space, then applies its rendering operation to produce an image.The renderer specification includes a primitive parameter space, rendering operation, and reparameterization function.Source: §4 · p. 5
  • Primitive poses are computed by multiplying the transformations along the entity hierarchy from each primitive to the scene root.The reparameterization function maps each word-and-embedding tuple into the renderer’s primitive parameter space.Source: §4 · p. 5
  • The Gaussians renderer uses SDS guidance to optimize a differentiable 3D representation from CLIP embeddings, and editing embeddings can personalize output scenes.The implementation uses 3D Gaussian Splatting; other differentiable representations such as neural fields are also suitable.Source: §4 · p. 5
  • Other proposed renderer options include primitive-based Mitsuba, the asset-based MineCraft engine, and feed-forward layout-conditioned diffusion models such as MIGC.These alternatives use different graphics or image-generation approaches to render the representation.Source: §4 · p. 5

5. Inference via Pre-Trained Language Models

The inference method uses a pre-trained language model to generate program and word components, then obtains embeddings from text or image inputs. The supplied evaluation caption identifies prompt-alignment preferences, CLIP similarity, and numeric-prompt counting accuracy as metrics for text-prompted 3D generation.

5. Inference via Pre-Trained Language Models

  • A training-free method infers the Scene Language’s words and program with a pre-trained language model, then obtains embeddings from text or image inputs.The text path uses a CLIP text encoder, while the image path uses a pre-trained text-to-image diffusion model.Source: §5 · p. 5
  • The language model receives the input condition, DSL helper functions, and an example script, and generates Python scripts using those functions.Claude 3.5 Sonnet is used for the method and language-model-dependent baselines.Source: §5 · p. 5
  • Text-prompted 3D generation is evaluated using prompt-alignment preference percentages, CLIP similarity, and counting accuracy on numeric prompts.The caption states the metrics are averaged across 9 numeric and 8 generic prompts and reports standard deviations across scenes.Source: Table 3 · p. 6
  • For image-conditioned tasks, GroundingSAM segments each primitive entity region and Textual Inversion optimizes an embedding to reconstruct its crop.The diffusion-model training objective is used for the embedding optimization.Source: §5 · p. 6

6. Experiments

The experiments evaluate text- and image-prompted scene generation and editing, including 4D generation and rendering across graphics engines. Results report stronger text-prompt alignment and counting accuracy than baselines, while image prompts and program edits support additional control.

6.1. Text-Prompted Generation and Editing

  • The method achieves more favorable text-prompt alignment than baselines in user-study and CLIP-similarity measures, with a clear advantage in counting accuracy.The user study asked 103 users to choose the best-aligned of three videos; CLIP similarity was averaged over 120 viewpoints per scene.Source: §6.1 · p. 6
  • Different graphics renderers produce visually aligned outputs from the same representation while exhibiting different imaging styles.The figure shows text inputs beneath the renderings.Source: Figure 5 · p. 6, §6.1 · p. 7
  • 4D-fy has higher per-frame prompt alignment, but its generated scenes show small dynamic motions compared with the proposed method.Motion magnitude is measured using dynamic degrees computed from RAFT optical flow over neighboring frames.Source: Table 4 · p. 6, §6.1 · p. 8
  • Language-instructed edits use an LM to infer changes to the scene program, enabling precise parameter edits that preserve scene structure and localized changes to relevant parts.Minor program changes can affect multiple primitives, such as all staircases, while preserving their circular arrangement.Source: Figure 7 · p. 7, §6.1 · p. 8

6.2. Image-Prompted Generation and Editing

  • Image-prompted generation better aligns outputs with input images than GraphDreamer, while embeddings preserve visual details and can be updated for partial or full-scene style edits.The reported comparison averages LPIPS over three scenes and 20 viewpoints each; style-transfer edits update embeddings for the corresponding entities.Source: Table 4 · p. 6, §6.2 · p. 8, Figure 8 · p. 8

7. Conclusion

The Scene Language combines programs, words, and neural embeddings to represent scene structure, semantics, and entity identity. The conclusion presents it as supporting high-fidelity 3D and 4D scene generation and precise editing.

7. Conclusion

  • The Scene Language encodes hierarchy and repetition with programs, component semantics with words, and component identity with neural embeddings, supporting high-fidelity 3D and 4D generation and precise editing.It can be inferred from text and image inputs using pre-trained language models, then rendered with different graphics engines.Source: §7 · p. 8

A. Overview

  • The appendices collect representation and experiment details, extended results, limitations, and language-model prompts and outputs.Source: A. Overview · p. 11

B.1. Domain-Specific Language

  • Four DSL macros define entity functions: call applies function calls, union aggregates entities, union-loop aggregates entities in a loop, and transform pairs an entity with its pose.The DSL binds entity functions to words and retrieves them when those words occur in programs.Source: §B.1 · p. 11

B.2.1. SDS-Based Renderer

  • The SDS-based renderer represents primitives with 3D Gaussians and renders transformed Gaussians through projection and blending.Gaussian parameters include center, opacity, scale, quaternion rotation, and color; the renderer also uses regularization and layout losses, and can adjust computation-graph granularity when fine parts are difficult to model.Source: §B.2.1 · pp. 11–12

B.2.2. Mitsuba Renderer

  • The Mitsuba renderer uses cubes, spheres, and cylinders with affine transformations, and renders them using a path tracer with maximum depth 8.Its generated primitive parameters are used directly rather than encoded into CLIP embeddings.Source: §B.2.2 · pp. 12–13

B.2.3. Minecraft Renderer

  • The Minecraft renderer uses asset parameters and similarity transformations but prevents rotations because they could produce shapes Minecraft cannot render correctly.The model chooses block types, with semantic-similarity checks used when a chosen block is unavailable in the Minecraft version.Source: §B.2.3 · p. 13

B.2.4. Text-to-Image (T2I) Model Renderer

  • The T2I renderer projects primitive 3D bounding boxes into a 2D layout and conditions a text-to-image model on that layout.Program parameters are encoded with the CLIP text encoder.Source: §B.2.4 · p. 13

C. Experiment Details

The experiments report per-scene computation costs and describe textual inversion for obtaining image-conditioned entity embeddings.

C.1. Computation Cost

  • SDS-based rendering takes approximately 30 minutes per object, compared with under 1 minute for LM inference and primitives-based rendering.The reported experiments ran on one A5000 GPU with 48GB memory.Source: §C.1 · p. 13

C.2. Textual Inversion Optimization

  • For image-conditioned embeddings, the method uses a text-guided GroundingSAM mask to crop each entity, then pads the crop to a square and resizes it.The crop background is set to a random grayscale color.Source: §C.2 · p. 13
  • Textual inversion optimizes the class and style token embeddings for 100 iterations at learning rate 1e-2, then uses the resulting entity embedding to guide entity optimization.Other template tokens are frozen during optimization.Source: §C.2 · pp. 13–14

C.3. GraphDreamer Implementation

The GraphDreamer baseline was re-implemented to generate scene graphs from text and optimize object representations from their nodes and edges.

C.3. GraphDreamer Implementation

  • The re-implementation prompts a language model to output scene graphs in JSON, avoiding manual conversion of its outputs into model configurations.The prompt requests scene, nodes, edges, and attributes fields.Source: §C.3 · p. 14
  • The graph format represents objects as nodes and relationships as edges, with attributes listed for each object.Each edge is an object-relationship-object triple.Source: §C.3 · p. 14
  • GraphDreamer optimization represents each node object as an SDF and color field and uses SDS loss for objects and object pairs in the edge list.When a graph exceeds the optimization memory limit, the graph is regenerated with a prompt limiting scenes to three objects.Source: §C.3 · p. 14

C.4. Evaluation Metrics

The user study compares video renderings from the proposed method and two baselines across nine scenes.

C.4. Evaluation Metrics

  • 103 Prolific users chose the best-aligned rendering among three randomized videos for each of nine scenes.Source: §C.4.1 · p. 14
  • The study asked users to judge text alignment by considering visual quality and adherence to the prompt's layout structure.Source: §C.4.1 · p. 14

C.4.2. CLIP-Based Evaluation

The section identifies the OpenCLIP model used for image-text similarity and captions extended qualitative comparisons for generation and editing.

C.4.2. CLIP-Based Evaluation

  • OpenCLIP-ViT-H-14-378-quickgelu measures image-text similarity for Tables 3 and 4.It is the OpenCLIP variant with the highest ImageNet zero-shot classification accuracy.Source: §C.4.2 · p. 14
  • Figure 10 extends Figure 4 with text-prompted 3D generation comparisons.Source: Figure 10 · p. 15
  • Figures 11 and 12 extend the editing examples with 3D language-instruction editing and 4D editing, respectively.Source: Figure 11 · p. 15, Figure 12 · p. 16

D. Extended Experiment Results

Extended experiments examine text-prompted 3D and 4D generation, scene editing, representation ablations, and information encoded by the Scene Language. They report benefits from flexible primitives and hierarchical programs, while showing that removing representation components degrades editing accuracy.

D.1. Text-Prompted 3D Scene Generation.

  • Qualitative examples for numeric and generic text-prompted 3D scenes appear on the website and in Figure 10, respectively.Source: §D.1 · p. 16

D.2. Text-Prompted 4D Scene Generation

  • Flexible primitive entities support 4D scenes at different scales, while grouping entities into composable functions reduces motion-space dimension and improves motion fidelity.With prompt variants, 4D-fy shows small increases in CLIP similarity and dynamic degrees, but its motion remains relatively small compared with ours.Source: §D.2 · p. 16

D.3. Text-Prompted 3D and 4D Scene Editing

  • The same method is applied to text-prompted 4D editing, with additional qualitative editing results shown for 3D and 4D.Source: §D.3 · p. 16

D.4. Roles of Representation Components

  • Removing programs or words degrades 3D editing accuracy; baseline methods that regenerate scenes fail to preserve their original content.For generation, embeddings are necessary for image-prompted editing, and removing programs harms language-model accuracy.Source: §D.4 · p. 16

D.5. Visualization of Discriminative Information

  • The representation exposes semantic maps, instance segmentation, repeated-instance correspondence, and dense temporal correspondence for 4D scenes.Source: §D.5 · p. 16

E. Limitations

The inference pipeline inherits limitations of its pretrained backbones and neural rendering components. These include sensitivity to text variation, variance in image parsing, rendering bias, and entangled texture and shape controls.

E. Limitations

  • Text-conditioned outputs can vary substantially with minor prompt changes, while image parsing can vary across runs on the same input.Source: E. Limitations · p. 16
  • Neural-based engines may introduce text-to-image model errors, such as an umbrella rendered not fully folded, whereas non-neural engines are described as fully controllable and correct.Source: E. Limitations · p. 16
  • For neural engines, texture and shape controls are not disentangled, so embedding changes can alter both.Source: E. Limitations · p. 17
  • The inference pipeline inherits pretrained-model weaknesses; addressing them could improve robustness, complexity, and output quality for downstream tasks.Source: E. Limitations · pp. 16–17

F. Language Model Prompts and Responses

This section presents language-model prompts and example responses for generating and editing structured scene programs from text or images, including Minecraft adaptations and text-conditioned 3D scenes. It also covers text-conditioned 4D animation through programs for a carousel, solar system, and toy wind turbine.

F.1.1. Text- and Image-Conditioned Scene Generation

  • The system prompt defines the DSL’s data types and macro function headers in Python, then restricts the model to functions and the provided helper library.It also specifies camera coordinates, modular registration of shapes or groups, and use of primitives to approximate complex components with correct poses.Source: §F.1.1 · pp. 17–18
  • The user prompt supplies a valid-program example and a task placeholder, replaced by a scene description for text generation or by a reconstruction instruction plus input image for image generation.The prompt also says examples are illustrative and asks the model to write a similar program for the task.Source: §F.1.1 · pp. 18–19
  • The books-on-a-desk example uses a loop to create and transform books, then translates their group so the books sit on the desk.The code computes shape bounds to align the books with the desk surface.Source: §F.1.1 · pp. 18–19

F.1.2. Scene Editing

  • Scene editing uses two LM rounds: first generate a program from a textual scene description, then make a minimal edit to that program to follow an editing instruction.The second round reuses the system prompt and supplies the first-round program and editing task.Source: §F.1.2 · p. 19

F.1.3. 4D Generation

  • For 4D generation, the system prompt adds an animation-registration macro while keeping the rest of the prompt unchanged.The registered animation function is stored for the rendering engine and should not be called elsewhere in the program.Source: §F.1.3 · p. 19
  • The 4D prompt’s example defines a Pacman-and-ghost scene and animates their positions across four frames.The example registers the animation function and updates Pacman and ghost translations frame by frame.Source: §F.1.3 · pp. 19–20

F.1.4. Minecraft Rendering

  • Minecraft prompts remove rotation and reflection matrices, while generated programs can also be converted to Mitsuba cubes colored by each block’s average color.Each Minecraft block is mapped to a Mitsuba cube at its corresponding location.Source: §F.1.4 · p. 20
  • Minecraft primitive calls support cuboid placement and block deletion, with valid Litematic block types and integer-only scale dimensions.The examples also specify block-state properties such as the axis for oak logs.Source: §F.1.4 · p. 20
  • The Minecraft example program combines a red cuboid and blue pyramid, and also defines a forest of trees with varying heights.The forest example places trees in nested loops and randomizes their heights.Source: §F.1.4 · pp. 20–21

F.2.1. Text-Conditioned 3D Generation

  • Text-conditioned examples use prompts for Moai statues at Ahu Akivi and a chessboard at game start.The Moai prompt requests slight variations.Source: §F.2.1 · p. 21
  • The Moai program builds heads, bodies, and a platform, combines them into statues, and distributes seven varied statues across the scene.Its component list includes scale and rotation variation for individual Moai.Source: §F.2.1 · p. 21
  • The Moai scene contains seven statues on a stone platform, with slight size and rotation variations among the statues.The description relates the platform-and-statue arrangement to the real Ahu Akivi site.Source: §F.2.1 · p. 21
  • The chessboard program defines chess squares and a board, alongside separate pawn, rook, and knight constructions.The supplied code fragments show named functions for each of these components.Source: §F.2.1 · p. 22
  • A chessboard program composes a board with white and black pieces placed in their correct starting positions.The listed pieces include pawns, rooks, knights, bishops, queens, and kings.Source: §F.2.1 · p. 23

F.2.2. Text-Conditioned 3D Editing

  • Editing examples use two LM rounds: a scene description specifies the task, followed by an editing instruction.The passage says first-round LM outputs are omitted and rendering outputs appear in Figs. 7 and 11.Source: §F.2.2 · p. 23
  • The Bramante Staircase edit reduces its radius by 80% while retaining its height and step count, and proportionally reduces the central column.The program applies a factor of 0.2 to both radii.Source: §F.2.2 · p. 23
  • The Jenga program makes block removal an optional layer parameter and applies it to the second-top layer.The example specifies removal of the leftmost and rightmost blocks from that layer.Source: §F.2.2 · p. 24
  • The Jenga edit removes two blocks from the second-top layer, leaving the rest of the tower intact and the layer supported by its remaining block.The example identifies the edited layer as index 6 and removes blocks 0 and 2.Source: §F.2.2 · p. 24
  • Another editing example requests a trinary, 3D branching structure for a fractal tree-like scene and presents a recursive program example.The accompanying description characterizes the shape as simple and not a pyramid.Source: §F.2.2 · p. 24
  • A recursive branch function builds a fractal tree from a cylinder and two smaller rotated branches, with depth controlling complexity.The wrapper shifts the tree so its base is at the origin.Source: §F.2.2 · p. 25
  • The Magritte scene edit moves the apple 0.15 units left by changing its x-translation to -0.15 while preserving its height and depth.The y and z coordinates remain unchanged.Source: §F.2.2 · p. 25
  • The castle edit replaces the left and right towers with copies of the middle tower, making all three towers identical.The program also changes the base color to match the center tower and spaces the towers evenly across the base.Source: §F.2.2 · p. 26
  • The nesting-doll edit places seven dolls evenly around a circle of radius 2.0 units, with each facing outward.The program combines translation and rotation so each doll faces outward from its position.Source: §F.2.2 · p. 27

F.2.3. Text-Conditioned 4D Generation

  • For a carousel with a small canopy, the 4D-generation response begins a program with separate functions for the base, platform, pole, and horse.The excerpt identifies the requested carousel and shows these component function declarations.Source: §F.2.3 · p. 27
  • The carousel program combines modular components and rotates the complete carousel, allowing individual parts to be modified independently.Its component list includes a base, platform, pole, four horses, and canopy.Source: §F.2.3 · pp. 27–28
  • The solar-system model places the sun at the center, with Mercury, Venus, Earth, and Mars, and includes an orbit path for each planet.The program describes the sun as a sphere and constructs circular orbit paths at specified radii.Source: §F.2.3 · p. 28
  • The solar-system animation runs for 120 frames, moving planets at different speeds around a stationary central sun.Each planet completes a different fraction of its orbit during the animation.Source: §F.2.3 · p. 28
  • The toy wind turbine program assembles a cylindrical base, a nacelle, and a rotor made of three arranged blades.The complete turbine combines these components.Source: §F.2.3 · p. 29
  • The wind-turbine animation generates 60 frames and rotates the rotor through a full 360 degrees.The described animation rotates the blades while keeping the turbine's other components assembled.Source: §F.2.3 · p. 29

Related papers

Loading 2410.16770v2…