Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract Scene Descriptions

1 INTRODUCTION

• Scene creation is cumbersome because creators must specify detailed language and manually find, select, and restylize assets, potentially across 30–50 objects.Indoor scene datasets lack scene and object diversity, while language-scene labels are limited or prohibitively enumerative.Source: §1 · p. 2
• The system uses LLMs and vision-language models to synthesize asset collections from abstract scene descriptions.It asks how far zero-shot inference using foundation models can facilitate 3D scene creation for artists.Source: §1 · p. 2
• The pipeline semantically expands a description into objects and appearance attributes, retrieves matching 3D assets, then textures them with diffusion models.LLM in-context prompting produces the semantic shopping list; vision-language models retrieve candidates, and diffusion models texture them.Source: §1 · p. 2
• Natural language connects the stages as an interpretable, editable representation that supports varying levels of specificity and model replacement.Users can inspect and edit intermediate outputs when one foundation model’s output is not an optimal input for another.Source: §1 · p. 2
• The system uses existing 3D asset databases, enabling outdoor and out-of-distribution scenes while allowing specialization to the supplied asset store.The paper presents stylized asset curation as a task, introduces the CLIP-D/S metric for diversity and semantic alignment, and reports human evaluation with 72 evaluators.Source: §1 · p. 2

2 RELATED WORKS

• Language-to-shape systems often produce implicit representations that are difficult to convert into usable, editable meshes, and typically generate what is stated verbatim rather than infer scene composition.Mesh editing guided by CLIP can yield improbable, distorted outputs; diffusion texturing work may require detailed descriptions of individual objects.Source: §2 · pp. 2–3
• Diffusion-based scene pipelines can make assets difficult to extract, while depth-projection methods constrain scene geometry and some compositional methods require fully enumerative language.SceneScape is described as biased toward long, tunnel-like scenes, and Text2Room as limited to closed, star-convex meshes.Source: §2 · p. 3
• Indoor scene datasets and generation methods lack open vocabularies, and the cited generation methods cannot retexture elements to match the input description.The paper identifies retexturing scene elements as a focus of its own system.Source: §2 · p. 3
• GPT-3 prompt templates share a structure between template and query segments until the next-token prediction prompt, and are used to generate anchor objects.The hierarchy then expands objects around each current-level object, usually adding smaller, more peripheral elements.Source: Figure 2 · p. 3, Figure 3 · p. 3

3.1 Semantic Upsampling

• Hierarchical prompting generates a small set of anchor objects, then recursively adds objects and attributes around them to form a semantic shopping list.The authors report that requesting all objects at once can cause semantic drift; anchor sets typically contain 6–8 objects.Source: §3.1 · p. 4

3.2 Object retrieval & retexturing

• The semantic shopping list guides CLIP-based retrieval of template geometries using visual renderings and textual annotations.The system linearly weights language- and image-based cosine similarities when annotations are available.Source: §3.2 · p. 4
• Retrieval uses the original asset’s appearance as well as its category because attributes such as “old” can affect geometry, and CLIP relies on color properties.A textureless candidate rendering can hurt retrieval performance; stylistic inconsistency is addressed by including the abstract scene description in every retrieval and texturing query.Source: §3.2 · p. 4
• The retrieval database combines Future3D’s indoor objects with a 30K subset of Objaverse for more diverse categories, including outdoor objects.The paper describes Objaverse as a source of distinctive scene elements that can better match a description’s “vibe.”Source: §3.2 · p. 4
• The current implementation derives CLIP image embeddings from asset thumbnails, while more complex object renderings remain a future direction for retrieval accuracy.The authors use thumbnails because they are readily available and artists already use them to judge asset suitability.Source: §3.2 · p. 4
• After retrieval, depth-guided and language-guided diffusion generates object-view images that differentiable rendering uses to optimize mesh textures across views.Depth and language conditioning encourage 3D consistency.Source: §3.2 · p. 4

4 EXPERIMENTS

• The usability demonstration used 20 scene-description prompts sourced from eight people with no prior 3D-design or system experience, spanning plausibility, emotional valence, and complexity.The pipeline ran fully automatically in a single pass, without cherry-picking among runs.Source: §4 · pp. 4–5

4.2 Evaluating stylized asset collections

• CLIP-S can favor prompt-matching assets that do not compose a scene, motivating CLIP-D/S, which combines semantic diversity and similarity as a heuristic for collection fidelity and usefulness.CLIP-D measures semantic variation; CLIP-D/S combines CLIP-D and CLIP-S with equal weighting.Source: §4.2 · p. 5

4.3 System outputs

• System outputs add scene “character” through wider object categories, while stochastic GPT-3 generation yields different semantic shopping lists across runs that can produce richer scenes.The paper reports 32.69% top-1 accuracy when CLIP-S classifies 572 assets across 20 scenes, versus 5% for random guessing.Source: §4.3 · p. 5

4.4 The importance of semantic upsampling

• Semantic upsampling supplies object attributes and appearances for retrieval; removing it can yield narrow, non-composable selections or incorrect template shapes that impair retexturing.The baseline retrieves the top-K assets by CLIP similarity to the abstract scene description instead of using the semantic shopping list.Source: §4.4 · pp. 5, 8

4.5 The importance of retexturing

• Retexturing with semantic-shopping-list prompts generally increases visual similarity to the abstract scene description compared with retaining original textures.Table 2 reports mean CLIP-S scores and the percentage of assets whose scores increased after retexturing, but the supplied passage gives no values.Source: §4.5 · p. 8, Table 2 · p. 9

4.6 User study

• Across 11 scenes, evaluators found assets from the system with semantic upsampling better matched abstract scene-description semantics than the no-semantic-upsampling baseline in 10 scenes, and preferred them for creating similar scenes in 9.Source: §4.6 · p. 9

• The system combines LLM, vision-language, and diffusion models to stylize 3D assets from abstract scene descriptions.Semantic upsampling with in-context learning produces an intermediate representation described as human-readable and editable, and conducive to higher-quality retrieval and texturing.Source: §5 · p. 9
• The framework transfers foundation-model knowledge to identify, texture, and compose meshes, with outputs expected to improve as the underlying models improve.Source: §5 · p. 9
• The authors showcase diverse language inputs and evaluate framework components using quantitative metrics and user studies, reporting zero-shot robustness and power for the task.Source: §5 · pp. 9–10
• Open challenges include generalizable open-vocabulary scene layouts, more 3D-consistent texture maps and material properties, and appropriate backgrounds for asset collections.Source: §5 · p. 10

A TEMPLATES FOR SEMANTIC UPSAMPLING

• Four manually created templates, drawn from a single indoor scene, support GPT-3 in-context semantic upsampling and are combined as the process moves down the scene hierarchy.The templates extract anchor objects, then add related peripheral objects and appearance attributes; one example limits each step to eight assets.Source: A TEMPLATES FOR SEMANTIC UPSAMPLING · p. 12
• A small template set based on one indoor scene may limit semantic upsampling for outdoor scenes, motivating more diverse templates to improve generalizability.Source: A TEMPLATES FOR SEMANTIC UPSAMPLING · p. 12

B HIERARCHY OF SEMANTIC SHOPPING LISTS

• GPT-3 often captures natural groupings for anchors with objects placed on, inside, or around them, but its peripheral-object groupings vary by anchor class.The figure illustrates one scene at a maximum hierarchy depth of one; more meaningful groupings remain a challenge and could support placement from coarse positional relations.Source: B HIERARCHY OF SEMANTIC SHOPPING LISTS · p. 12

C INPUT SCENE DESCRIPTIONS, SEMANTIC SHOPPING LISTS & SCENE RENDERINGS

• The appendix presents shopping lists and scene renderings for 20 scenes, with 11 selected randomly for a user study and baseline visualizations used in human evaluation.Source: C INPUT SCENE DESCRIPTIONS, SEMANTIC… · p. 12

• The shopping list gives Poseidon’s room ocean-themed assets, including a throne, fireplace, trident, mermaid statues, fish statues, and coral, with visual attributes and condition details.Source: §C.1 · pp. 12–13
• The hierarchy groups plausible children beneath anchors such as the table and fireplace, while stand-alone anchors such as chairs receive plausible nearby object categories.Source: Figure 12 · p. 13
• Other listed furnishings include navy couches, pillows, blankets, and an ocean-blue ottoman, with descriptions emphasizing plushness and pristine condition.Source: §C.1 · p. 13

C.2 (★) A romantic french restaurant (romantic restaurant)– Figures 15 and 16

• The romantic restaurant list combines oak tables, ivory-upholstered chairs, polished tableware, and gold-accented bar furnishings with appearance and wear descriptions.Source: §C.2 · p. 15

C.3 (★) A retro arcade in the style of the 1980s (retro arcade) – Figures 17 and 18

• The retro arcade list describes pinball machines, colorful bumpers, arcade controls, neon lighting, and retro music equipment with period-styled visual details.Source: §C.3 · pp. 15, 17

C.4 (★) Anne frank’s room during world war II (anne frank room)– Figures 19 and 20

• Anne Frank’s room list describes worn furnishings, aged books and papers, clothing, photographs, and other objects with age-related condition details.Source: §C.4 · p. 17

C.5 (★) The interior of the smurf house (smurf house)– Figures 21 and 22

• The Smurf-house list specifies a blue-and-white mushroom house, colorful mushrooms and flowers, and clean, well-maintained furnishings and decorations.Listed items include a fireplace, table, chairs, plates, silverware, Smurf figures, and cupcakes.Source: §C.5 · pp. 17, 20
• Figures pair system and baseline outputs for the retro-arcade and Anne Frank’s room prompts.Their captions identify the prompts and which output is ours or the baseline, but do not state a comparative outcome.Source: Figure 17 · p. 18, Figure 18 · p. 18, Figure 19 · p. 19

C.6 (★) A murder in an abandoned living room (murder room)– Figures 23 and 24

• The murder-room list combines worn furnishings with crime-scene details such as broken glass, a bullet hole, blood stains, a knife, and a bullet casing.It also specifies damage and aging across objects including the couch, desk, papers, and lamp.Source: §C.6 · p. 20

C.7 (★) Bedroom in Shrek’s home in the swamp (shrek home)– Figures 25 and 26

• The bedroom list specifies a dark-wood bed with green accents, green-and-gold furniture, and a green-and-brown glazed set of dishes and bowls.The bed and dishes are described as being in good condition, while several furnishings show minor wear.Source: §C.7 · pp. 20, 23
• Soft furnishings and room accessories include pillows, blankets, a plush rug, a bookshelf, and a green-glass lamp with bronze accents.The descriptions give varying wear conditions, from slightly worn blanket edges to pristine books and pillows.Source: §C.7 · pp. 20, 23
• Whimsical themed details include Shrek-motif books and notepad, character art, and stuffed Donkey and Puss in Boots.The lists also specify green leather books, gold lettering, and swamp-themed book titles.Source: §C.7 · p. 23

C.8 (★) Confucius’s bedroom (confucius bedroom)– Figures 27 and 27

• The Confucius-bedroom list centers on calligraphy and writing materials alongside dark-wood furniture, scrolls, ink tools, and Chinese-character decorations.It specifies materials and motifs such as bamboo, horsehair, black slate, carved dragons, and lacquer, with condition ranging from aged books to well-maintained furnishings.Source: §C.8 · p. 23

C.9 (★) A Marvel-themed bedroom of a five-year old toddler (marvel bedroom)– Figures 29 and 30

• The Marvel-themed toddler bedroom list specifies red-and-blue furnishings, Avengers bedding and posters, character toys, and cartoonish accessories.Items include Iron Man, Captain America, and Thor toys, wall stickers, action figures, and a toy chest; the descriptions include both new and lightly used items.Source: §C.9 · p. 25

C.10 (★) A futuristic teahouse from the future in silicon valley (futuristic teahouse)– Figures 31 and 32

• The futuristic teahouse combines pristine metallic furnishings and holographic tea service with voice-activated menus, interactive displays, and AI bots.The list specifies plasma-topped tables, holographic accents on tea ware, and smooth-gliding bots with natural-sounding speech.Source: §C.10 · pp. 25, 28

C.11 (★) A saloon from an old western– Figures 33 and 34

• The old-west saloon list specifies a walnut-and-brass bar, worn wooden and leather seating, and aged gaming, drinking, and entertainment objects.Examples include dusty bottles, creased cards, worn poker chips, and a brightly colored jukebox with minor wear.Source: §C.11 · p. 28

C.12 An alien tea garden on Mars (alien teagarden)– Figure 35

• The alien tea garden features glossy metallic and glazed-ceramic tea ware in vivid colors, with holographic imagery and alien-shaped decorative details.The list includes a purple metallic table, blue-and-green cups, holographic saucers, and kettles with alien-shaped handles.Source: §C.12 · pp. 28, 31

C.13 Antichrist in the bedroom of the pope (antichrist vatican)– Figure 36

• The pope’s bedroom pairs a dark-wood, royal-blue-velvet bed with black-and-gold religious objects, polished desk furnishings, and aged books and ritual items.The list includes a gold-detailed crucifix, a mahogany desk with a green marble top, and a dragon-motif incense burner.Source: §C.13 · p. 31
• Descriptions pair a polished frame, glossy mirror, intact gold-leather bible, and black-iron crucifix with signs of age or oxidation.Source: §C.13 · p. 31
• Candles are described with gold accents and signs of use, as well as slight melting and charring.Source: §C.13 · p. 31
• An incense holder has gold detailing, while the chandelier is crystal clear with gold accents and a slightly faded finish.Source: §C.13 · p. 31
• A chain, bells, and crosses are described with rust, dents, tarnish, weathering, or scratches.Source: §C.13 · p. 31

C.14 Hades’s man cave (hades cave)– Figure 37

• A throne, velvet cover, skeletons, crown, chains, sword, and scepter establish a dark, ornate centerpiece with varied wear and finishes.Source: §C.14 · p. 33
• The list adds a kraken statue, flexible tentacles, sea shells, rocks, and pearls, mixing marine motifs with contrasting textures and finishes.Source: §C.14 · p. 33
• Bookshelves, leather-bound books, rune-embossed scrolls, potions, and oils extend the setting with aged or carefully preserved details.Source: §C.14 · p. 33
• Fireplace tools, a grate, poker, bellows, cauldron, and firewood are specified with black iron, sturdy construction, or ready-to-use condition.Source: §C.14 · p. 33
• Additional furnishings and decor include a velvet-cushioned chair, black-and-red goatskin rug, skulls, crossbones, and a brass-finished torch.Source: §C.14 · p. 33
• An anvil, hammer, tongs, iron rods, ingots, and forge add a blacksmithing cluster with condition details ranging from slight rust to well-maintained.Source: §C.14 · p. 33

C.15 A michelin star restaurant opened by a mad scientist (mad scientist restaurant)– Figure 38

• The mad-scientist restaurant list combines polished fine-dining furnishings with laboratory glassware, equipment, and a steampunk-design kitchen.It specifies clean, glossy tables and tableware alongside petri dishes, test tubes, flasks, chemical containers, and a Bunsen burner.Source: §C.15 · pp. 33–34

C.16 A North Korean classroom (north korean classroom) – Figure 39

• The North Korean classroom list pairs desks, books, pencils, and uniforms with political symbols and state-doctrine materials.It includes a globe highlighting North Korea and a blackboard portrait of Kim Il Sung.Source: §C.16 · pp. 34–35

C.17 Headquarters of an occult cult (occult cult) – Figure 40

• The occult-cult headquarters list specifies ritual furnishings and objects, including candles, incense, a dagger, robes, carved symbols, books, scrolls, and a chalice.Materials and finishes include black iron, velvet, bronze, silver, and onyx with gold accents.Source: §C.17 · p. 35

C.18 A rustic backyard in the countryside (rustic backyard) – Figure 41

• The rustic countryside backyard list combines outdoor dining and grilling supplies with a hammock, lantern, birdhouses, trees, and wildflowers.Many objects are described as slightly faded, weathered, or worn, while retaining a usable or intact condition.Source: §C.18 · p. 37

C.19 A church for strawberries (strawberry church) – Figure 42

• The strawberry church list adapts church furnishings and devotional objects with strawberry motifs on cushions, hymnals, candles, an incense burner, and a Bible.It also describes conventional elements such as an altar, pews, a pulpit, flowers, and a chandelier.Source: §C.19 · p. 37
• For the strawberry church, the asset descriptions specify stained-glass windows, candles, banners, and flowers with particular colors and condition details.The descriptions include bright stained glass, candles with red petals, gold-lettered banners, and vibrant flowers.Source: §C.19 · p. 38

C.20 A busy street in downtown New York (busy New York street)– Figure 43

• The busy New York street example is considered a failure, highlighting the system’s limitations on outdoor scenes and the need for future work on busy urban settings.The paper says this failure is reflected in the scene’s CLIP-D/S score; the figure caption also calls for improvement on busy urban outdoor scenes.Source: §C.20 · p. 38, Figure 43 · p. 39