PaperlayerView exact revision on arXiv

Source-linked AI summary

MAGIC: Transition-Aware Generation of Navigable Multi-Scene Game Worlds with Large Language Models

Tsz Hei Fan, Choi Wing Fung, Yuxuan Wan, Shuqing Li, Michael R. Lyu

arXiv:2607.11594v1cs.AIcs.GR

TL;DR

Single-scene generators do not ensure consistency, navigability, or working transitions across connected game scenes. MAGIC addresses these gaps with a transition-aware pipeline and achieves executable projects for all 100 cases, with 0.96 F1 for end-to-end transition identification.

  • Problem

    Single-scene generators leave cross-scene consistency, in-scene navigability, and working-transition evaluation unresolved in multi-scene game-world generation.

  • Method

    MAGIC uses a shared transition-aware representation, flood-fill portal-reachability validation, joint scene and transition generation, and project stitching.

  • Results

    0.96 F1, with 0.99 precision and 0.95 recall, identifies end-to-end transitions while MAGIC produces executable projects for all 100 benchmark cases.

  • Takeaways & Limitations

    MAGIC demonstrates that a single natural-language prompt can yield a runnable multi-scene 3D game project with consistent, navigable transitions.

  • Takeaways & Limitations

    MAGIC targets indoor Unity scenes, supports only FadeInOut and IrisWipe transitions, accepts English prompts, and can leave some portals blocked when regeneration is exhausted.

Abstract

from arXiv · show

Multi-scene navigation (clearing an objective in one bounded space and then crossing a portal into the next) is a defining feature of contemporary 3D games, but authoring it is laborious: every portal must have consistent endpoints on both sides, each interior must remain navigable once it is furnished, and the resulting connectivity must be kept consistent across many files. Recent large language model (LLM) and multimodal LLM (MLLM) scene generators have made single-interior synthesis dramatically cheaper, yet they produce one scene at a time and cannot, by naive repetition, yield a connected multi-scene world. We identify three obstacles that single-scene methods leave unsolved: cross-scene consistency, in-scene navigability, and the evaluation of whether a transition actually works. We present MAGIC, a prompt-to-project system that addresses all three. MAGIC is a four-stage pipeline that turns a single natural-language prompt into a runnable multi-scene game project: it plans a shared transition-aware intermediate representation, specifies each scene while enforcing portal reachability with a flood-fill validator, generates the scenes together with their transition scripts, and combines them into one project. Because existing single-scene fidelity metrics never execute a transition, we further introduce a transition-focused evaluation agent that runs each transition in play. On a new benchmark of 100 multi-scene cases, MAGIC produces an executable project for every case and reaches 0.99 precision, 0.95 recall, and 0.96 F1 on end-to-end transition identification; stage by stage, it recovers more ground-truth portals and yields markedly more navigable layouts than an LLM baseline and Holodeck. Our code is available at https://github.com/sereneee1201/MAGIC/.

I. INTRODUCTION … B. 3D Scene Generation

MAGIC extends rapidly advancing single-scene 3D generation into interconnected, navigable multi-scene game worlds. It addresses cross-scene consistency, in-scene navigability, and transition evaluation through a shared representation, reachability validation, and end-to-end testing.

  • I. INTRODUCTION: Multi-scene games require coherent, navigable interiors and portals with matching endpoints, destinations, and transition effects across scenes.These coupled requirements make multi-scene authoring substantially more difficult than generating isolated interiors.
  • I. INTRODUCTION: Single-scene generators such as Holodeck and Scenethesis reduce interior-generation cost but do not jointly generate inter-scene portals and destination scenes.Naive repetition therefore leaves scene connectivity unresolved.
  • I. INTRODUCTION: MAGIC uses four stages—planning, scene specification, scene generation, and combination—to share a transition-aware intermediate representation and validate portal reachability with flood-fill.The pipeline is designed to enforce consistency and navigability while producing a runnable multi-scene project from one prompt.
  • I. INTRODUCTION: 0.99 precision, 0.95 recall, and 0.96 F1 are achieved by MAGIC on end-to-end transition evaluation across 100 multi-scene benchmark cases.The benchmark derives cases from MIT Indoor 67 and MMIS and includes comparisons with an LLM baseline and Holodeck.
  • A. LLMs in Game Development: LLM game-development research spans runtime NPC interaction and pre-play generation of game artifacts, with growing use of language models as automated designers.These applications leverage natural-language understanding for dialogue, contextual responses, and content creation.
  • A. LLMs in Game Development: LLM-based gameworld generation has focused mainly on 2D environments, localized spatial content, individual levels, and standalone scenes rather than explicit structural transitions.This limitation motivates modeling relationships between scenes for playable multi-scene worlds.
  • A. LLMs in Game Development: Representative generators cover only subsets of the capabilities needed for playable multi-scene games: text prompting, 3D synthesis, cross-scene transitions, and interactive game-engine output.Holodeck and the Wonder family generate isolated interiors, while Genie and Narrative-to-Scene target 2D artifacts.
  • B. 3D Scene Generation: Procedural, neural, and LLM-based methods have rapidly advanced 3D scene generation, while MAGIC adds action-triggered connections to expand these systems toward automated multi-scene game generation.PCG is efficient for repetitive rule-based elements, neural methods generate scene parameters, and LLMs contribute semantic understanding and controllable requirement following.

III. METHODOLOGY · A. Stage 1: Planning

MAGIC plans a connected multi-scene project from one natural-language query by producing scene descriptions and a transition-aware graph. The planner preserves scene-specific detail while encoding bidirectional portal connections that downstream stages can query and update.

  • III. METHODOLOGY: MAGIC uses four stages—planning, scene specification, scene generation, and combination—to merge separately generated scenes into one playable multi-scene project.The pipeline automatically builds customized connections between scenes for smooth navigation from a single input prompt.
  • A. Stage 1: Planning: The planning stage accepts only the user query and outputs individual scene prompts together with a transition-aware automaton.The query describes multiple scenes and their transitions; later stages repeat scene specification and generation for each resulting description.
  • A. Stage 1: Planning: The LLM separates the natural-language prompt into detailed, feasible scene descriptions while retaining information from the original input.Scene separation guidelines and multiple example input-output pairs guide the planner’s behavior.
  • A. Stage 1: Planning: Each scene description is processed independently, differentiated in style, and expanded to include 8 to 12 internally consistent objects.This gives the later scene-generation model substantial information even when the initial prompt is complex.
  • A. Stage 1: Planning: The transition-aware automaton stores portal connections and transition effects between scene pairs as a graph data structure.Its transition relation uses textual scene descriptions, portals, effects, and scene endpoints.
  • A. Stage 1: Planning: A transition relation entry (s_i, p, e, s_j) denotes portal p connecting scene s_i to scene s_j with effect e.The representation defines S as scene descriptions, P as portals, E as transition effects, and δ as the transition relation.
  • A. Stage 1: Planning: Because portals work in both directions, each transition is stored once as an undirected, effect-labeled edge in a connected transition graph.The graph is queried and updated by downstream stages and must remain connected so every scene is reachable.
  • A. Stage 1: Planning: The planner counts and verifies the number and types of portals required for each scene so the corresponding portal information appears in scene descriptions.This check links the graph’s connectivity requirements to the textual specifications passed to later stages.

B. Stage 2: Scene Specifications

Stage 2 converts scene descriptions and portal requirements into structured, individually processed specifications for Unity generation. It expands prompts into region-level requirements, injects and verifies portal-bearing objects and connections, validates placements, and iteratively removes blockages using flood-fill connectivity.

  • Scene specification: Stage 2 generates a complete structured scene specification for Unity while processing each scene individually and iteratively to prevent cross-scene interference.The inputs are scene descriptions and portal requirements.
  • Region and object construction: MAGIC subdivides each scene into regions, expands requirements, assigns requirement snippets to region sub-prompts, and injects objects using entity extraction.The resulting region sub-prompts are intended to remain faithful to the original scene prompt.
  • Portal validation: Portal requirements are enforced by tagging object portals, regenerating missing tagged objects, and verifying connection counts for doors and windows.Object portals are regenerated within their assigned regions, while outside-connected doors and tagged windows serve as connection portals.
  • Placement correction: The correction module compares draft placements against DSL constraints and regenerates placements when violations are detected.The DSL lists positional and rotational requirements for retrieved objects.
  • Navigability validation: A flood-fill validator rates portal connectivity and loops object placement until blockage is resolved or the maximum iterations are reached, returning the placement with the highest connectivity score.Connectivity equals 1 when all portals are reachable and the region is fully navigable; lower values indicate partial or complete blockage.

C. Stage 3: Scene Generation · D. Stage 4: Combination · IV. EXPERIMENTS AND EVALUATION

Stages 3 and 4 turn scene specifications into executable Unity scenes, connect portal-triggered transitions, and merge scene projects into one multi-scene project. Evaluation compares MAGIC’s stages and end-to-end performance against an LLM baseline and HOLODECK.

  • C. Stage 3: Scene Generation: Stage 3 generates assets and an executable Unity project for each scene, while leaving transition scripts unconnected to other scenes until combination.Players can control the camera to navigate each generated scene.
  • C. Stage 3: Scene Generation: Portal objects and doors connected to “outside” are matched to transition edges, producing scripts with destination scene, portal name, and transition effect.The preprocessing uses objects or connections tagged true “isPortal” together with outside-connected doors.
  • C. Stage 3: Scene Generation: Meshes are generated for scene geometry, while predefined script templates process scene specifications to produce precise, executable scripts.Meshes are stored by materials, vertices, and faces, with paths recorded in the scene specification.
  • C. Stage 3: Scene Generation: Transition LevelLoader scripts attach to portal objects, check camera collisions, and load the correct scene with the required transition effect.This avoids the insensitive and unpredictable triggering caused by adding a rigid body to the camera.
  • D. Stage 4: Combination: Stages 2 and 3 repeat for every planned scene, after which the combination algorithm discards excess outputs and merges the single-scene projects.Each scene output includes specifications, object meshes, scene meshes, and a complete executable Unity file.
  • IV. EXPERIMENTS AND EVALUATION: The evaluation separates stage-by-stage analysis from end-to-end assessment to measure both MAGIC’s components and overall multi-scene generation performance.The first three stages use an LLM prompted to follow the required output format as the baseline.
  • IV. EXPERIMENTS AND EVALUATION: HOLODECK serves as an additional baseline because it generates a single-scene specification before completing the software project.The comparison covers the first three stages of MAGIC.

A. Research Questions … D. Stage-by-Stage Evaluation

The evaluation examines MAGIC from planning through end-to-end transition functionality, using a 100-case benchmark assembled from MIT 67 and MMIS scene data. Experiments compare MAGIC with an LLM baseline under specified runtime conditions and assess scene-graph generation across one to five scenes.

  • A. Research Questions: MAGIC is evaluated through five questions covering planning, specification, generation, agent agreement, and end-to-end transition accuracy.The questions ask whether stages recover transition graphs, produce reachable portals, generate executable Unity projects with correct scripts, agree with human judgment, and create functional cross-scene transitions.
  • B. Experimental Setup: MAGIC uses GPT-4.1-mini-2025-04-14, while the LLM baseline uses GPT-4.1, with generation temperature set to 0.7.Experiments ran on macOS and Windows without requiring a GPU.
  • B. Experimental Setup: 33.35 minutes is the median generation duration per scene, compared with a 46.36-minute mean.The right-skewed distribution reflects a small number of long-running cases.
  • C. Benchmark Test Cases: 100 test cases combine the MIT 67 Indoor Scenes and Multimodal Interior Scenes datasets.The two datasets provide complementary scene categories, interior imagery, textual annotations, and audio descriptions.
  • C. Benchmark Test Cases: The benchmark varies spatial layout, object composition, and interior style through complementary source datasets and procedural case construction.MIT 67 covers diverse functional indoor categories, while MMIS contributes broad interior-scene variation.
  • C. Benchmark Test Cases: Each benchmark case is a scene graph whose nodes are scene instances and whose edges represent transitions between selected scene pairs.A procedural script integrates data from both datasets to construct the cases.
  • C. Benchmark Test Cases: Each case contains one to five MIT scenes instantiated with characteristic objects and spatial constraints.Cases use a randomly sampled interior style from the MMIS taxonomy.

1) Scene Plan Evaluation: · 2) Scene Specification Evaluation: · 3) Scene Generation:

The three evaluations assess scene-plan accuracy, portal correctness and navigability in scene specifications, and executable transition scripts in generated Unity projects. Together, they measure whether the pipeline preserves scene connectivity from planning through playable generation.

  • 1) Scene Plan Evaluation:: Scene-plan evaluation measures description accuracy for individual scenes and graph accuracy for a transition-aware automaton.The automaton records each transition’s origin, destination, portal type, and transition effect.
  • 1) Scene Plan Evaluation:: Scene accuracy is the fraction of ground-truth requirements mentioned in the generated scene descriptions.An LLM evaluator extracts requirement snippets, assigns them to scenes, and checks whether each appears in the corresponding description.
  • 1) Scene Plan Evaluation:: Transition-automaton scoring compares ground-truth and generated transition edges, portals, and effects, with greater weighting assigned to edges and portals because they enable connectivity and triggering.Edges match when neighbor pairs are identical, with bidirectional origins and destinations treated as interchangeable; portal scores support later transition stages, while effects are used in Stage 3.
  • 2) Scene Specification Evaluation:: Stage 2 evaluates generated portals against ground truth using precision, recall, F1, and per-scene blockage scores.Portal candidates include objects and windows tagged “isPortal” plus doors connected to “outside”; matching, missing, and excess portals are counted.
  • 2) Scene Specification Evaluation:: Flood-fill blockage evaluation tests whether every portal is reachable from all walkable cells, with connectivity computed as visited cells divided by walkable cells.Region occupancy is also measured because sparse layouts ease flood-fill passage but can reduce scene plausibility.
  • 3) Scene Generation:: Stage 3 evaluates script generation for playable single-scene Unity projects by counting LevelLoader scripts and testing script executability.LevelLoader scripts power the transition process, while the LLM baseline substitutes an LLM-based script-generation algorithm.
  • 3) Scene Generation:: The expected LevelLoader count equals the scene’s transition count; missing loaders prevent transitions, whereas excess portals can cause unexpected transitions.Generated scripts are evaluated by executing SceneBuilder, which calls the other generated scripts to create the full scene.

E. End-to-End Evaluation … V. RESULTS AND ANALYSIS

MAGIC evaluates scene transitions with an agent that identifies and executes candidate portals, checks destinations and reachability, and compares generated portal appearance with textual requirements. Against human judgment, the agent achieves the smallest mean absolute difference, while qualitative analysis finds correctly placed wall connections and successful transition effects.

  • 1) Evaluation Agent Design:: Existing pixel- and deep-learning-based metrics compare outputs with references or prompts, but rarely evaluate whether scene transitions work.This gap motivates MAGIC’s transition-focused evaluation agent.
  • 1) Evaluation Agent Design:: The evaluation agent screens every scene for transition candidates, tests collisions, records destinations, and attempts to reach triggered portals.It evaluates transitions through gameplay rather than static output comparison.
  • 1) Evaluation Agent Design:: Portal orbit captures are analyzed by an MLLM to assess whether generated objects match their original textual descriptions.The agent produces a final report summarizing scene scores and generation criteria.
  • 1) Evaluation Agent Design:: 20 test cases spanning scene counts, room types, styles, portal objects, and transition edges are evaluated against human-judgment ground truth.Ground truth manually inspects every object, activates portals, and records destination scenes.
  • 2) Reliability of the Evaluation Agent:: MAGIC’s evaluation agent performs best overall, with most metric values superior to the ablations and the smallest difference from human judgment.The comparison treats smaller differences from human judgment as greater evaluation accuracy.
  • 2) Reliability of the Evaluation Agent:: The evaluation agent has the smallest MeanAbsDiff from human judgment, while Ablation Study 2 shares its precision, recall, F1 score, and approach rate.Ablation Study 1 considers many extra objects, which confuses evaluation and lowers its approach rate and portal-related performance.
  • V. RESULTS AND ANALYSIS: Qualitative analysis finds correctly placed door and window portals on walls without gaps, with both FadeInOut and IrisWipe transition effects successfully implemented.These qualitative checks complement the quantitative stage-by-stage and end-to-end evaluations.

A. Stage-by-Stage Evaluation Results … 3) Scene Generation Results:

Across planning, scene specification, and generation, MAGIC better preserves user requirements, portal completeness, navigability, and executable project structure than the baselines. Its strongest generation advantage is reliable execution, while baseline scripts can make transitions unpredictable.

  • 1) Scene Plan Evaluation Results:: MAGIC wins most test cases against the LLM baseline in scene accuracy and transition-aware automaton performance.Its advantage comes from additional guidelines and predefined user intentions, while still depending on LLM understanding.
  • 1) Scene Plan Evaluation Results:: MAGIC’s planning stage accurately identifies requirements and user intent from a single textual input.The LLM baseline loses quality in scenarios involving transitions and multi-entity relationships.
  • 2) Scene Specification Evaluation Results:: MAGIC achieves the largest matched portion with ground truth and the fewest missing portals, while HOLODECK usually generates fewer portals than required.The comparison indicates weaker transition-logic understanding and more frequent instruction neglect by HOLODECK.
  • 2) Scene Specification Evaluation Results:: MAGIC has the highest average connectivity, providing nearly complete reachability across walkable cells while maintaining occupancy near 30%.This combination prevents blockage and avoids scenes becoming too sparse or too compact.
  • 2) Scene Specification Evaluation Results:: MAGIC achieves the highest portal recall, over 80% precision, and nearly 90% F1, balancing portal correctness and completeness despite sometimes generating extras.Its recall identifies nearly all ground-truth portals with minimal misses.
  • 2) Scene Specification Evaluation Results:: Although the LLM baseline has the highest precision and F1, its lower recall causes more missed ground-truth portals and risks disconnecting scenes.HOLODECK performs poorly across all three metrics, consistent with weaker connectivity.
  • 3) Scene Generation Results:: MAGIC produces executable Unity projects for every test case, whereas none generated directly by the LLM baseline can be executed.The baseline lacks functional Unity project templates, so minor component or file-naming errors can prevent execution.
  • 3) Scene Generation Results:: The LLM baseline generates excess LevelLoader scripts in nearly half the test cases, making transition logic unpredictable despite usually producing the required count.Accurate transition implementation remains necessary for smooth gameplay even when a generated Unity project executes.

B. End-to-End Evaluation Results … VI. CONCLUSIONS

MAGIC achieves high-precision end-to-end transition generation while producing navigable multi-scene Unity projects, but its evaluation remains limited by dataset, engine, model, and proxy-metric constraints. The conclusion emphasizes a shared transition-aware representation, reachability validation, and executed transition evaluation as its core contributions.

  • B. End-to-End Evaluation Results: The evaluation agent deploys agents in the final Unity executable to simulate in-game situations and assess transition accuracy and quality.Performance is determined solely by the completed multi-scene Unity project.
  • B. End-to-End Evaluation Results: Precision, recall, and F1 all exceed 0.9, indicating that most required portals are generated correctly.Portal match rate is lower because MLLM detection requires exact image-ground-truth matches and the dataset may omit portal types.
  • B. End-to-End Evaluation Results: Above-90% precision, recall, and F1 show that most generated transitions align with the ground-truth plan, favoring clean but less complete results.The pipeline is therefore suited to settings prioritizing high transition precision.
  • B. End-to-End Evaluation Results: Performance does not clearly degrade from one to five scenes or with expected portal count, while most portals remain reachable from the character spawn point.The authors refrain from stronger scalability claims beyond the tested range; a few inaccessible portals occur.
  • C. Limitations: Portal quality remains over 75% but is reduced by strict MLLM requirements and limited object-mesh coverage.A plain board can fail the “directory board” match because it lacks evidence of the intended semantic use, and the strictness can be adjusted through the system prompt.
  • C. Limitations: MAGIC currently supports indoor Unity scenes, English prompts, and only FadeInOut and IrisWipe transition effects.Portal appearance is bounded by the object-mesh dataset, and object placement is best-effort.
  • D. Threats to Validity: The benchmark uses synthetic procedural cases and proxy metrics, while single-run stochastic evaluations without variance or significance testing may yield optimistic point estimates.The benchmark contains 100 indoor cases spanning one to five scenes, and generalization beyond the tested settings is untested; the reliability study uses 20 cases and two annotators.
  • VI. CONCLUSIONS: MAGIC converts one natural-language prompt into a runnable multi-scene 3D game project with consistent, navigable transitions.Its core contributions are a shared transition-aware intermediate representation, flood-fill portal-reachability validation on a 2D occupancy grid, and an agent that executes transitions in play.

APPENDIX A EVALUATION PROMPT FOR STAGE 1 OF MAGIC

The Stage 1 evaluation prompt compares generated and ground-truth scene dictionaries, checks required scene and transition attributes, and scores each scene by satisfied requirements. It requires a detailed checklist and valid JSON output containing scene-level accuracy results.

  • Inputs and matching: The evaluator receives ground-truth and generated scene-description dictionaries, then outputs ground-truth scene names matching generated scene indices.It also evaluates how accurately each generated description matches its ground truth.
  • Accuracy criteria: Required checks include scene type, style, transition type, portal position, object injections, and additional spatial or stylistic constraints.A transition type is required for both scenes involved, while transition effects are excluded.
  • Scoring: Each requirement receives 1 when satisfied and 0 when missing or violated, and the prompt computes a success_rate for every scene.The output must include a detailed checklist identifying which requirements were met.
  • Output format: The evaluator must return valid JSON only, without Markdown fences, comments, or text outside the JSON object.The example organizes results under scene nodes and includes transition information in the structured output.
  • Example result: The example reports success_rate 0.8 for the office and success_rate 0.5 for the storage room.The office checklist marks the carpet requirement as 0, while the storage-room checklist marks the sliding metal partition, exposed brick walls, and steel beams as 0.
Loading 2607.11594v1…