LLMR: Real-time Prompting of Interactive Worlds using Large Language Models

• Prior 3D generation work often focused on visual appearance, while creating virtual worlds requires artistic and technical skills and existing rendering methods can be slow.LLMs integrated with Unity may support faster content development and spontaneous user creation, while mixed-reality scenes provide spatial multimodal information.Source: §1 · p. 2
• LLMR creates interactive 3D scenes and objects with visual and behavioral features, and supports bespoke edits to existing environments.Its self-contained interactive tools can be combined into larger systems and transferred across environments.Source: §1 · p. 2
• Specialized GPT modules support scene understanding, skill selection, code generation, and checks for compilation and run-time errors.The Scene Analyzer summarizes scene objects, the Skill Library selects relevant skills, and the Inspector evaluates generated code before execution.Source: §1 · p. 2
• 4x reduction in code errors in both empty and existing scenes versus off-the-shelf GPT-4 is reported for LLMR.The framework was tested on two sets of 150 prompts covering creation and modification tasks.Source: §1 · p. 2
• In an 11-participant study, users found LLMR intuitive and easy to use, while authors note its stochasticity limits precise-control contexts.Participants iteratively reached desired outputs without much manual scripting; generated scenes can serve as starting points for more complex generation.Source: §1 · p. 2
• The framework is designed for adaptation across mixed-reality tools, environments, and devices, and the authors share its installation package, code, and prompts.The authors also report extensive technical evaluations, an ablation study, and a user study.Source: §1 · p. 3

2.1 Generative 3D Assets

• Prior text-guided 3D asset work includes object-generation methods, with Magic3D addressing slow optimization and low-resolution output associated with DreamFusion [35].LLMR’s stated extension is to include interactive and behavioral elements beyond visual appearance.Source: §2.1 · p. 3

2.2 Generative Interactive 3D Environments

• Related interactive-environment research spans generated 3D spaces, NPC interaction, scene dialogue, terrain authoring, and realism-performance trade-offs.The section frames bringing LLM capabilities to a real-time Unity editor for mixed-reality applications as an open gap.Source: §2.2 · p. 3

2.3 Editor Support for Mixed Reality Development

• Prior mixed-reality editor work includes interface adaptation and telepresence; LLMR lets users direct the environment in natural language.Source: §2.3 · p. 3

2.4 LLMs Interpreting Spatial, Non-Linguistic Information

• Related work applies LLMs to spatial and embodied data; Figure 2 shows LLMR routing prompts and scene summaries through planning, building, and inspection.The figure’s pipeline integrates selected skills with the Planner and Scene Analyzer outputs to guide code generation.Source: §2.4 · p. 3, Figure 2 · p. 4

3.1 Planner

• The Planner breaks requests into appropriately scoped subtasks, consults users about scope and detail, and carries out the plan through sequential code generation and updated scene states.The Planner's conversational role helps accommodate prompts whose difficulty or desired detail users may not know how to specify in advance.Source: §3.1 · p. 5

3.2 Scene Analyzer

• The Scene Analyzer summarizes the existing scene in response to a user request, providing relevant context for downstream modules instead of the full scene hierarchy.The raw Unity hierarchy can be distracting and may exceed the LLM's fixed context window, especially for intricate scenes.Source: §3.2 · p. 6

3.3 Builder-Inspector

• The Builder generates code from instructions and scene context, while the Inspector checks for compilation and run-time errors and returns suggestions for another attempt.The Builder can face cognitive overload when it must combine creative work with extensive specific guidelines; inspection and feedback form a self-debugging loop.Source: §3 · p. 4, §3.3 · pp. 6–7

3.4 Compilation, Save and Reload

• After inspection, LLMR compiles and executes scripts at run time; users can save and selectively reload generated scripts or regenerate them from their summaries.Run-time compilation makes the framework a real-time generative tool rather than an offline development tool.Source: §3.4 · p. 7

3.5 Skill Library

• The Skill Library retrieves skills relevant to a prompt, keeping high-level descriptions separate from detailed usage information and examples passed to the Builder.This design addresses limits on prompt token size and Builder attention as available skills expand.Source: §3.5 · pp. 7–8

4 INCORPORATING EXISTING OPEN-SOURCE 3D ASSETS

• Artist-created assets can offer high real-world fidelity, but a query such as “clock” may return many models, few suited to the intended setting.The section gives an office scene with 50 clocks, only three of which are suitable, as an example.Source: §4 · p. 8
• The Object Retriever narrows Sketchfab matches to the top 5 by language similarity, then selects the candidate most visually similar to a DALL·E 2 target image.It repeats the process for each prompted object to assemble a full scene.Source: Figure 7 · p. 9, §4 · p. 9
• The retrieval workflow may be improved by changing the order of its language- and visual-similarity stages; future work will use human feedback to identify the workflow that best matches user intent.The authors note that visual-first selection might yield better results.Source: §4 · p. 9

5 MEMORY MANAGEMENT

• No LLMR module uses full memory, the default GPT paradigm; limited-memory modules retain only the most recent N episodes, typically N=1.An episode is a module’s input and output for one user prompt.Source: Table 1 · p. 9, §5 · pp. 9–10
• A memory-less Scene Analyzer avoids token exhaustion: an AI2-THOR scene hierarchy is around 7k GPT-4 tokens, so an 8k-token full-memory Analyzer could handle only one instruction.The authors argue this would make the framework essentially unusable outside a memory-less setting.Source: §5 · p. 10
• Repeated inspections can make the Inspector more lenient, allowing proposed code to pass before all errors are fixed.The authors report this as an empirical observation supporting reduced memory for some modules.Source: §5 · p. 10
• Limiting memory makes failures easier to attribute to the latest prompt rather than earlier tasks, supporting faster debugging and iteration.With all history retained, the source of a failure at the final step can be difficult to identify.Source: §5 · p. 10

6 CROSS-PLATFORM COMPATIBILITY AND INSTALLATION

• LLMR supports Web, mobile, AR, and VR devices by compiling on a PC server and streaming generated results through existing remoting protocols.Platform-specific packages and namespaces can be added as Skills to enable input and interaction modalities.Source: §6 · p. 11
• Interactive elements can be saved as source code and re-executed in new scenes with different physics or APIs; cross-SDK translation is possible, while adapting obsolete code remains future work.The authors describe this as an application for further exploration.Source: §6 · p. 11

6.1 Installation

• The Unity package and open-source dependencies can be added to existing Unity scenes in a few installation steps.The foundational framework and examples are open-sourced.Source: §6.1 · p. 11
• Trying the framework requires an OpenAI API key and Roslyn compiler; a Sketchfab account is optional for automatic asset loading.The metaprompts for LLMR modules are also provided in the appendix.Source: §6.1 · p. 11

7 EXAMPLE PROMPTED INTERACTIVE WORLDS AND USES

• The examples span prompted objects, tools, and scenes, and the authors state they are achieved without manual intervention.They characterize the framework as modular, real-time, adaptive, interactive, and multimodal.Source: §7 · p. 11

7.1 Game Design and Creativity

• For game design, the Planner can draft asset-heavy scenes and add responsive players or opponents; toy cars can be reloaded into a moon simulation or driven in the physical world by phone.The examples include a VR moon environment and a mobile-phone-controlled physical-world car.Source: §7.1 · p. 12

7.2 Accessibility and Adaptive Interface

• Users can adapt a kitchen scene for red-green color-blind users, add a magnifier for near-sighted users, or assess wheelchair access and child-proofing.These examples apply LLM prior knowledge to a spatial scene at human scale.Source: §7.2 · p. 12

7.3 Remote Assistance and Planning

• An instructor can prompt a training guide from instructions; trainees can ask unanticipated questions and receive context-appropriate answers with motions animated on the fly.The example guide uses an animated hand to point out coffee-machine operating steps.Source: Figure 11 · p. 12, §7.3 · p. 13
• For rescue planning, operators can prompt a helicopter flight-path simulation for target locations and examine how different wind conditions might affect the route.The guide presents an interactable 3D terrain, helicopter, and simulated wind.Source: §7.3 · p. 13, Figure 12 · p. 13

8.1.2 Results and Discussion.

• 20.5% and 25.2% of original GPT-4 error rates remained for empty and existing scenes, respectively; Figure 14 reports 3x few-shot and over 4x zero-shot improvement.The study used five model conditions across 150-prompt datasets for each scene type, with five runs per condition.Source: §8.1.1 · p. 13, §8.1.2 · p. 14, Figure 14 · p. 15
• The Scene Analyzer significantly improves bathroom-scene performance, while the Inspector reduces errors; adding it to few-shot GPT-4 cut empty-scene error from 45.0% to 13.1%.The Skill Library had marginal error-rate impact; the Planner was excluded because its prompt decomposition made results incommensurable.Source: §8.1.2 · p. 14

8.2.2 Results and Discussion.

• LLMR consistently had lower error rates across difficulty levels and both scene types, although errors generally rose with difficulty and task difficulty was difficult to characterize in the bathroom scene.The Easy empty-scene category contained only one prompt, a basic “Hello World” display; bathroom tasks categorized as Easy could be harder in practice than adding objects.Source: §8.2.2 · pp. 14–15

8.3 Task Complexity

• For specialized tasks, an experienced developer can add relevant Skill Library scripts; LLMR’s error rate rises with difficulty but remains below 40%.The Scene Analyzer filters for relevant top-level game objects to manage scene-summary token demands, while prior prompts must compile successfully and remain robust for later tasks.Source: §8.3 · pp. 15–16
• Users can choose visually simple but functional interactive scenes for rapid prototyping instead of waiting for visually complex, compute-intensive scenes that are harder to modify.The Planner can help create a complex scene from primitives, such as a full kitchen.Source: §8.3 · p. 16

8.4.2 Results and Discussion.

• Approximately 2.5x better performance across all sequential-task metrics followed integration of all LLMR modules; memory-efficient design keeps context usage constant for arbitrary prompt sequences.The evaluation used 80 sequences averaging five single prompts; Table 2 presents the numerical results.Source: §8.4.2 · p. 16, §8.4.1 · p. 16, Table 2 · p. 17

8.5.2 Results and Discussion.

• The full LLMR framework takes a little over a minute per prompt, versus around half a minute for off-the-shelf GPT-4, including generation and compilation.The authors considered this acceptable given task complexity and improved completion rates; Table 3 reports average time in seconds.Source: §8.5.2 · p. 17, Table 3 · p. 17
• Sketchfab retrieval time was excluded because it varies with model size; the Inspector nearly doubles code-generation time but can be disabled for simple tasks.User interactions and iteration add time, while saving and reloading prior creations takes less than 10 seconds to recompile.Source: §8.5.2 · p. 17

9.1 Procedure

• Participants generated varied outputs, including cities and Asteroids-like games, and some recreated professional work such as camera rigs and animations.The study recruited 11 participants with varied Unity experience; sessions lasted about two hours, with at least 1.5 hours for experimentation.Source: §9.2 · p. 18, §9.1 · p. 18

9.2 Results and Design Recommendation

• Questionnaire responses were generally positive about achieving goals, intuitiveness, and iterative use, though participants identified room to reduce frustration and improve satisfaction.Beginners rated their experience more positively than experienced Unity users across most categories.Source: §9.2 · p. 18

9.2.1 Approach to Prompting and Instruction Strategies.

• Participants recommended specific prompts, iterative trial and error, and breaking tasks into smaller steps; one participant also prioritized static elements before motion-centric ones.Some participants advised building simple components first and gradually adding complexity.Source: §9.2.1 · p. 19
• Participants valued describing ideas directly, automation such as selecting and placing repository assets, and the ability to manually adjust generated worlds.Some participants said the framework reduced the learning curve for newcomers.Source: §9.2.2 · p. 19

9.2.2 Comparison with Prior Approach to 3D World Creation.

• Participants described outputs as unpredictable; traditional tools may suit projects needing precise control, while manual code editing may be faster for complex or growing projects.Some participants said the choice could depend on a project’s artistic nature and need for creative input.Source: §9.2.3 · p. 19

9.2.3 Considerations and Challenges.

• Participants were surprised by code generation for complex structures, use of game-object hierarchy, integrations with Dall-E 2 and Sketchfab, and understanding of subjective language.Some also reported that LLMR could help resolve errors in Unity scripts or produce unconventional player movement when prompted appropriately.Source: §9.2.4 · p. 19

10 ETHICAL CONSIDERATIONS

• LLMR is presented as supporting human intervention, productivity, and brainstorming rather than proven end-to-end development automation.Participants in the user study described the framework as better at integrating human involvement.Source: §10 · p. 19
• Users may creatively circumvent content moderation and model-alignment safeguards, leaving harmful 3D content a concern.The paper notes that Roslyn checks unsafe code, but further research on moderating 3D content is merited.Source: §10 · p. 19
• The authors identify developer and creator replacement as a concern, while noting that end-to-end development has not been proven.Source: §10 · p. 19

11 LIMITATIONS AND FUTURE WORK

• Scene editing requires a scene graph and exact object names, which can be unreliable for models with random, non-descriptive component names.The framework assumes object and child-component names are correct and unique.Source: §11 · p. 19
• Scene hierarchies omit object affordances and functions, and are not readily available for physical environments containing augmented objects.The authors identify Large Vision Models as a possible next step for visual knowledge and semantic understanding.Source: §11 · p. 19
• Without model orientation and pivot information, the system does not consistently produce desired rotations of imported 3D models.The authors propose incorporating feedback from the virtual world and user to improve world understanding.Source: §11 · pp. 19–20
• Token limits can hide earlier prompt details, so the framework may not know how to modify a specific part of a script generated two prompts earlier.The Scene Analyzer retains script names, summaries, and public fields, but not necessarily the requested earlier detail.Source: §11 · p. 20
• Generated code is stored locally and viewable in Unity, while direct code editing is proposed to give users more agency over complex, precise tasks.Source: §11 · p. 20
• Skill Library skills are currently created by human users; automatic skill generation is proposed to expand task and platform coverage.The examples include loading Sketchfab assets and making objects grabbable; suggested extensions include animation, Quest, and ARKit.Source: §11 · p. 20
• The current implementation assumes Unity, although the framework can be adapted to environments supporting run-time compilation.A web-based approach is proposed to make prompt-based interactive 3D worlds easier to share and collaborate on.Source: §11 · p. 20

12 CONCLUSION

• LLMR orchestrates specialized LLM modules to improve their individual and collective performance on coding and reasoning, alongside engineering aids that add scene content through other AI models.Source: §12 · p. 20
• The authors report that combining specialized components makes the system more robust and significantly better than off-the-shelf LLMs, and that the user study tested output quality and usability.Participants also challenged the framework with prompts beyond the provided examples.Source: §12 · p. 20
• LLMR’s potential lies in generating interactive virtual-world content that may make intuitive sense in human-scale settings; the current implementation assumes Unity.The paper describes the LLM as testing results in a simulation rather than merely incorporating descriptions of the world.Source: §12 · p. 20
• The framework is proposed as a platform for HCI research that uses virtual- or real-world data and code-based action to test LLM reasoning in 3D environments.Source: §12 · p. 20
• The authors conclude that LLMR offers a tool for developers and researchers to use and extend for virtual-world content and experience generation.Source: §12 · p. 20

A NUMERICAL STUDY

• Empty- and bathroom-scene results were averaged over five independent runs; reported standard deviations are across runs, not the 150 examples within each run.The table captions identify Table 4 as the empty-scene results and Table 5 as the bathroom-scene results.Source: Table 4 · p. 23, Table 5 · p. 23, A NUMERICAL STUDY · p. 23
• The configuration with Scene Analyzer and Inspector but no Skill Library performed within LLMR’s error bars, but took longer for scene editing.The authors note that this is compatible with supplying more information in the framework’s context.Source: A NUMERICAL STUDY · p. 23
• Figure 17 organizes architecture error rates by prompt-difficulty level, but the supplied caption states no outcome of that comparison.Source: Figure 17 · p. 24

A.1 Empty Scene Prompts

• Prompts cover a broad range of primitive-built objects with behaviors, from color-changing cubes and bouncing spheres to animals, instruments, and household items.Many requests pair object creation with an explicit interaction, such as clicking, dragging, pressing keys, or using a control.Source: §A.1 · pp. 24–26
• Requests specify user-controlled interactions such as driving with keys, drawing with a mouse, operating switches, and using calculators, pianos, or games.Interaction triggers include clicks, key presses, mouse movement, and controls such as sliders, buttons, and levers.Source: §A.1 · pp. 24–26
• Other prompts ask for behaviors tied to time or conditions, including objects that grow, melt, launch, open, or change when touched or activated.Examples include a cube that explodes after five seconds, a flower that grows when watered, and a snowman that melts when approached or exposed to heat.Source: §A.1 · pp. 24–26
• The set also includes motion and physics tasks, such as bouncing under gravity, swinging pendulums, rolling vehicles, and objects following paths.Some prompts request physical components such as joints and hinges, while others describe environmental motion such as wind, rain, or orbits.Source: §A.1 · pp. 25–26
• Beyond individual objects, prompts include assembled environments and interactive experiences such as an office, terrain, mazes, a platformer, and a roller coaster.Some requests also specify functional tools or systems, including cameras, maps, vending machines, and a script that prints to the console.Source: §A.1 · pp. 25–26

A.2 Bathroom Scene Prompts

• Bathroom prompts request adding, moving, resizing, recoloring, and removing objects, as well as changing the room’s appearance.Examples include placing a sphere on the bathtub, making the toilet gold, adding relaxing objects, and changing the color scheme.Source: §A.2 · pp. 26–28
• Bathroom prompts also request interactive behaviors and effects, including flowing water, opening doors, adjustable controls, and objects that respond to collisions or use.Examples include a temperature slider, a slippery wet floor, a breakable window, and plants that grow or wilt.Source: §A.2 · pp. 27–28
• Some bathroom prompts ask for scene analysis and accessibility considerations, or camera-centered changes and interactions.Requests include identifying objects that may be out of reach for a wheelchair user, moving the camera, and making objects respond to it.Source: §A.2 · pp. 27–28

A.3 Sequential Prompts

• Sequential prompts specify multi-step builds, such as assembling and programming a car, torch, water effect, or cat-and-mouse scene.Some requests chain further behavior onto an initial object or scene, such as making a burning cube stop burning near water.Source: §A.3 · p. 28
• Other sequences request simulations with interacting or adjustable components, including an atom, solar system, pendulum, and cellular automaton.Examples include stochastic electron density, planetary orbits, adjustable pendulum length and mass, and pause-and-resume controls.Source: §A.3 · p. 29
• The sequential examples also build games and interactive objects, from mazes and dice to spaceships, snakes, submarines, and roller coasters.Requests specify player controls, object responses, or gameplay such as reaching a goal, eating food, or riding along rails.Source: §A.3 · p. 29
• Further sequences combine scripts, interfaces, cameras, and multiple entities to add controls, transformations, or coordinated behavior.Examples include a time display with a visibility button, a camera view shown on a mirror, a chess opponent, and a scored ball-and-basket scene.Source: §A.3 · p. 30
• A car-and-road sequence requests forward motion, arrow-key steering and speed control, plus traffic lights and other cars.The prompt also asks the car to steer left or right and speed up or slow down in response to key presses.Source: §A.3 · p. 30
• A pizza-and-knife sequence asks for a delicious-looking pizza, dragging the knife to slice it, falling slices, and click-to-remove behavior.Source: §A.3 · p. 30
• A candle-and-match sequence specifies lighting and burning the match, then igniting the wick by contact and slowly melting the candle.The match should burn out after 5 seconds, and the candle should stand on a table.Source: §A.3 · p. 30
• A snowman sequence combines three differently sized snowballs and a head-top hat with a smiling face, button eyes, carrot nose, and mouse-over waving.Source: §A.3 · p. 30
• A flower-and-bee sequence requests click-triggered blooming, pollen collection on landing, flight around the flower, and departure to a hive when enough pollen is collected.Source: §A.3 · p. 30
• A cube-and-sphere sequence specifies different colors and textures, collision and bouncing, and spinning when the objects bounce.Source: §A.3 · p. 30
• A bird-and-tree sequence asks the bird to fly, land on a branch when tired, sing when happy, and fly away when scared.The tree has green leaves and brown branches.Source: §A.3 · p. 30
• Other sequences pair a terrain map and compass, interactive pen-and-paper behaviors, and a star-and-planet system with orbit, day-night, and click-triggered explosion behaviors.The listing states that individual prompts within a sequence are delimited by semicolons.Source: §A.3 · pp. 30–31

B PLANNER EXAMPLE: BUILDING A VIRTUAL KITCHEN

• The Planner breaks kitchen creation into similar-size subtasks, improving small-object placement and implementing the requested interactive faucet that the one-shot LLM missed.The resulting kitchen includes appliances on the counter, table items, and stove tops on the oven; the faucet uses particle effects.Source: B PLANNER EXAMPLE: BUILDING A VIRTUAL… · p. 32

C METAPROMPTS

• LLMR metaprompts persist across sessions regardless of memory-management protocol and combine a module introduction, output guidelines, and human-tested few-shot examples.The paper describes careful metaprompt design as important for instruction following and learning from demonstrations.Source: C METAPROMPTS · p. 33

C.1 Planner

• The Planner converses with users to form scoped steps for the Unity Builder to execute sequentially, and its instructions encourage committing to one approach.Its examples cover requests ranging from creating objects to adding interactive features.Source: B PLANNER EXAMPLE: BUILDING A VIRTUAL… · p. 32, §C.1 · p. 33

C.2 Scene Analyzer

• The Scene Analyzer interprets Unity scene JSON but summarizes only request-relevant content, listing child names without describing their parent relationship.Its guidelines also say to omit the Main Camera and treat Compiler as a manager containing generated scripts.Source: §C.2 · p. 35

C.3 Skill Library

• The Skill Library selects skills that fit a user instruction and returns “n/a” when no skill is needed.The examples include skills for retrieving object bounds and creating animations, with a caution against using the animation manager for simple Unity primitives.Source: §C.3 · pp. 36–37
• The Skill Library metaprompt pairs requests with reusable helpers, including renderer-bound sizing and guidance to avoid the animation manager for simple primitives; examples cover animation, keyboard controls, and scene queries.Source: §C.3 · p. 37

C.4 Builder

• The Builder prompt requires complete, compilable C# Widgets scripts that the runtime compiler executes, with summaries, reusable public members, and self-contained code.It also provides utilities such as adding physics and displaying messages.Source: §C.4 · p. 37
• Builder examples modify a cat’s behavior and create objects that can be dragged or recolor objects on collision.Source: §C.4 · pp. 38–39
• A further Builder example finds the biggest food item by comparing the products of its render-bound dimensions.Source: §C.4 · pp. 39–40

C.5 Inspector

• The Inspector checks unique GameObject names, safe component creation, editor-independent setup, Widgets inheritance, summary updates, input naming, and avoidance of placeholders, tags, and layers.It also disallows code outside the main class except using statements.Source: §C.5 · p. 40
• Three examples pass: creating a named cube, assigning a public water-bottle variable in code, and finding the largest food item using the available mesh-bounds skill.The water-bottle example does not require editor assignment because the script assigns the cylinder at runtime; the food example searches scene objects and uses GetRenderBounds.Source: §C.5 · pp. 41, 44, 45
• The Inspector rejects a time-display example because its classes are not contained within one overarching class.The suggested correction is to place all classes inside an overarching class.Source: §C.5 · p. 42
• A car-and-person example fails because the generated car body, four wheels, person body, and head lack unique GameObject names.The suggested fix is to assign unique names to each of those objects.Source: §C.5 · p. 43
• A random-cat example fails because adding RandomCatMovement to the created cat in Start triggers an infinite loop of cat creation.The suggested fix is to remove that AddComponent call and find and control the existing Cat object instead.Source: §C.5 · p. 44
• Listing 8 is identified as the Inspector’s metaprompt.Source: §C.5 · p. 45

D USER STUDY QUESTIONNAIRE AND INTERVIEW QUESTIONS

• The questionnaire used 1-to-7 Likert ratings for frustration, satisfaction, iterative improvement, goal attainment, prompting intuitiveness, recommendation, and intended work use.Source: D USER STUDY QUESTIONNAIRE AND… · p. 46
• Semi-structured interviews asked about prompting strategies, prior and typical 3D creation comparisons, surprising or unavailable capabilities, and potential applications.Source: D USER STUDY QUESTIONNAIRE AND… · p. 46