GRUtopia: Dream General Robots in a City at Scale

• Real-world robot data collection is costly and faces hardware generalization challenges, while existing simulation platforms have limited scenario and task diversity and complexity.The authors present simulation as a crucial step toward addressing these issues.Source: §1 · p. 2
• GRScenes extends beyond home-focused agents [42] [48] to service-oriented indoor and outdoor settings, with 100K interactive scenes across 89 functional categories.Its scenes and objects have fine-grained, hierarchical, multi-modal annotations.Source: §1 · p. 2
• GRResidents combines LLM-driven NPCs with scene knowledge to support dynamic dialogue, social interaction, task generation, and task assignment.NPCs can infer object relationships, understand scene semantics, and observe other agents in real time.Source: §1 · p. 2
• GRBench covers Object Loco-Navigation, Social Loco-Navigation, and Loco-Manipulation, with legged robots as primary agents and learning-based control APIs for high-level tasks.The benchmarks progressively increase in difficulty; the paper reports that their metrics have well-defined difficulty granularity and meet research needs across fields and hierarchies.Source: §1 · pp. 2–3
• Integrating realistic actions with high-level planning increases task difficulty, presenting a key challenge for previous embodied algorithms in real-world scenarios.Source: §1 · p. 2
• License issues constrain the initial GRScenes release to 100 scenes across 7 building types, with remaining data planned for progressive release.Source: §1 · p. 2

• 89 scene categories versus Behavior-1K's 8, with GRUtopia building city-scale 3D scenes from its diverse assets.Figure 2 presents the scene-category distribution and annotated-object spectrum.Source: §2 · p. 3, Figure 2 · p. 4
• Earlier benchmarks trade symbolic reasoning, physical realism, and task diversity; GRUtopia supports AI-generated tasks from navigation to mobile manipulation across diverse room layouts.VirtualHome [42] and Alfred [48] abstract physical interactions, while platforms such as Habitat 2.0 and iGibson [47] [30] offer realistic actions and interactions.Source: §2 · p. 3
• Low-level control platforms enable policy training but can be limited to specific terrains; GRUtopia offers diverse physical environments and locomotion and manipulation policy APIs.The APIs are intended to foster research on high-level task planning.Source: §2 · p. 3
• Unlike NPC designs limited to task assignment, GRUtopia's LLM-driven NPCs provide crucial information during task execution and interact with agents using environmental data.Source: §2 · p. 3

3.1 GRScenes: Fully Interactive Environment at Scale

• GRScenes collects about 100k synthetic scene prototypes and curates 100 annotated scenes, including 70 home and 30 commercial scenes, with 2,956 interactive objects.The remaining scenes still need a method to construct object interiors automatically.Source: §3.1 · p. 4
• GRScenes adds region-level and object-region language annotations, using polygons to label regions and VLM-generated object annotations checked manually.These annotations support multimodal interaction and provide a foundation for generating benchmark tasks.Source: §3.1 · p. 5

3.2 GRResidents: Generative NPCs in 3D Environments

• GRResidents uses a World Knowledge Manager to maintain scene knowledge and an LLM planner to query it and generate dialogue-contextual responses.The planner stores chat history, queries scene knowledge through WKM interfaces, and sends the resulting knowledge and history to an LLM speaker.Source: §3.2 · pp. 5–6, Figure 3 · p. 5
• NPCs achieved 95.9%-100% referring and 83.3%-93.2% grounding accuracy across LLMs, while object-centric QA scores indicate precise, helpful navigation assistance.Referring uses a human-in-the-loop evaluation; grounding uses GPT-4o to provide object descriptions.Source: Table 2 · p. 6, §3.2 · p. 6

3.3 Robot Control APIs

• The platform provides RL-based locomotion APIs for humanoids and quadrupeds, allowing researchers to use pre-trained controllers in high-level tasks without handling RL training.The tested robots include Unitree H1 and Aliengo with a Z1 arm.Source: §3.3 · p. 6
• In clutter, navigation success fell from 100% to 58% for H1 and 14% for Aliengo, while manipulation success fell from 100% to 8%.The authors attribute the disparity mainly to path or motion-planning failures despite robust low-level control.Source: §3.3 · pp. 6–7, Table 3 · p. 7

4.1 Benchmark Setups

• Object Loco-Navigation tests language-guided target finding, Social Loco-Navigation adds up to three clarification dialogues, and Loco-Manipulation requires placing a handheld object under specified conditions.Source: §4.1 · pp. 7–8

4.2 Baselines

• VLM baselines select discrete actions from image-and-language inputs, adding Ask for social navigation and Pick/Place for manipulation; the LLM agent separates grounding, memory, decisions, and actions.Source: §4.2 · pp. 8–9

4.3 Evaluation

• Navigation success requires the target within view and under 3 meters; manipulation requires correct placement, and every episode must stop within 14,400 simulation steps.Source: §4.3 · p. 9

4.4 Quantitative Results

• A stronger large-model backend improves performance consistently across all three benchmarks, and the agent framework markedly outperforms direct multimodal decision-making.Source: §4.4 · pp. 9–10
• Loco-Manipulation achieves a success rate (SR) of 0; failures are associated with locomotion resets and arm collisions during manipulation.Source: §4.4 · p. 10
• The three tasks show incremental difficulty, with loco-manipulation harder than navigation because of its larger action space, longer planning horizon, and precision demands.Source: §4.4 · p. 10

4.5 Diagnostic Study

• The diagnostic study finds QwenVL perception superior to GPT-4o and oracle actions substantially improve performance, exposing a gap from less predictable real-world movement.Source: §4.5 · p. 10

4.6 Qualitative Results

• In a Social Loco-Navigation episode, the NPC supplies surrounding goal information after a query, helping the agent identify the target chair through dialogue.Source: §4.6 · p. 10

5 Conclusions

• GRUtopia brings together city-scale functional scenarios, learning-based control APIs for varied embodiments, and multimodal NPCs for social interaction, task generation, and assignment.The platform is presented as a resource for embodied AI research.Source: §5 · p. 10
• The agent successfully finds a target object with help from an NPC.Source: Figure 7 · p. 11
• The initial release partially provides 100 annotated indoor scenes and a city block, while NPC social interaction lacks physically realistic contact.The platform is in active development, with further enhancements planned.Source: §5 · p. 11

A More Details of GRScenes

• GRScenes scene graphs represent objects with textual attributes and relationships, adding “in” and “out of” spatial relations enabled by detailed internal scene structures.Color and shape information is aligned to objects through exact text matching.Source: A More Details of GRScenes · p. 15

B.1 Implementation Details of World Knowledge Manager

• The World Knowledge Manager has three core functions to support knowledge extraction from the environment.Source: §B.1 · p. 15
• The section presents Python-style pseudocode to show how the World Knowledge Manager’s functions are implemented.Source: §B.1 · p. 15
• Algorithms A–C illustrate implementations of the World Knowledge Manager’s functions.Source: §B.1 · p. 15

B.2 More Details of NPC Experiments

• GPT-4o [39] acts as the human annotator in the object-grounding experiment, where NPCs use API definitions and task guidance to compose grounding calls.Source: §B.2 · p. 15
• Grounding utterances were generated from 10 home scenes as 500 template descriptions, then modified by hiding target categories, replacing relationship terms with synonyms, and adjusting sentence openings.Source: §B.2 · p. 15
• The grounding API selects an anchor object from its description, searches nearby scene-graph objects by spatial relation, then matches the target description.Source: §B.2 · p. 15
• The object-centric QA pipeline produced 489 episodes, each with a target object, agent actions, and NPC dialogue, totaling 1,669 interaction turns.Source: §B.2 · pp. 15, 17
• QA answers receive 100 or 0 based on whether their text-embedding-3-large cosine similarity exceeds the manually reviewed 0.6 threshold; the final score averages QA pairs.Source: §B.2 · p. 17
• The object-grounding search prioritizes differences in category, then room, spatial relationship, and appearance to match human description habits.Source: §B.2 · p. 17

C More Details of GRBench

• The appendix covers benchmark episode generation, instruction generation, metric implementation, and simulation setup.Source: C More Details of GRBench · p. 17
• Algorithm B’s get_info function returns target-object information, including category, room, spatial relation, or appearance attributes.Source: C More Details of GRBench · p. 18
• Figure I depicts occupancy-map generation followed by sampling collision-free paths; its grids mark undefined, passable, and obstacle regions.Source: Figure I · p. 18
• Algorithm C filters the original object set against category, room, relation, and appearance conditions.The code compares appearance embeddings using a similarity threshold greater than 0.9.Source: C More Details of GRBench · p. 19

C.1 Episode Generation

• Navigation episodes use collision-free paths of 7–20 m, excluding objects without a qualifying path.Occupancy maps mark objects and areas outside the floor as impassable; collision detection uses a 34 cm radius.Source: §C.1 · p. 19
• Manipulation episodes sample a handheld object and placement conditions drawn from four spatial-relation types.Conditions pair a target object with a relation; “on” targets must be receptacles, and paired nearby objects must be within 1.5 meters.Source: §C.1 · p. 20

C.2 Instruction Generation

• Object-navigation instructions iteratively distinguish the target from candidates, then use the collected attributes to generate a concise description.The search continues until only the target remains.Source: §C.2 · p. 20
• Social-navigation instructions use a coarse target attribute to preserve ambiguity, with multi-round attribute searches adding instruction diversity.The design aims for agents to interact with the NPC to obtain more concrete clues.Source: §C.2 · p. 20
• Manipulation instructions combine the handheld-object navigation goal with up to two placement conditions and descriptions of their target objects.Some room or relation attributes are randomly dropped while retaining a solution through the presence of target objects.Source: §C.2 · pp. 20–21

C.3 More Details of Benchmark Setups

• ECR measures how efficiently dialogue alleviates ambiguous candidates in Social Loco-Navigation.Its definition tracks candidate-set reduction across dialogue rounds.Source: §C.3 · p. 21
• SCR measures the fraction of Loco-Manipulation placement conditions satisfied in an episode.It averages an indicator for whether each condition is satisfied across the episode’s conditions.Source: §C.3 · p. 21
• Rendering runs at 1 Hz, the low-level control policy at 60 Hz, and physical simulation at a 1/240-second timestep.The rendering frequency is set this way because rendering is the main simulation-efficiency bottleneck.Source: §C.3 · p. 21

D.1 Zero-Shot VLM Baseline

• The zero-shot VLM baseline chooses among movement, turning, stopping, and asking actions based on the robot’s view and target information.Its prompt restricts repeated left or right turns to no more than two consecutive choices.Source: §D.1 · p. 23

D.2 LLM Agent Baseline

• The LLM baseline’s grounding module turns egocentric RGB-D images and robot state into segmentation, point clouds, and candidate bounding boxes.It uses SegFormer for initial RGB segmentation.Source: §D.2 · p. 24
• Its memory maintains a real-time BEV occupancy map, action-observation history, and target information obtained through NPC dialogue.The map includes candidate positions and descriptions.Source: §D.2 · p. 24
• The decision module uses a large model to select navigation goals and formulate NPC questions from memory-provided information.The overall agent receives egocentric observations and robot state, with modules collaborating to support physical and linguistic interaction.Source: §D.2 · p. 24
• The action module replans navigation paths with RRT* when the updated occupancy map blocks the current path, and uses inverse kinematics for manipulation.For picking, the target object may be attached directly to the gripper within a distance threshold.Source: §D.2 · p. 24

E Real-world Demo

• The real-world demonstration applies the LLM agent baseline to Object Loco-Navigation on a Unitree H1 robot.The authors say they will release the demo after further polishing.Source: E Real-world Demo · p. 25
• The H1’s locomotion in the demonstration is powered by the same control policy used in simulation.Source: E Real-world Demo · p. 25
• The authors report that the simulation-platform agents transferred smoothly to a real-world robot driven by their sim-to-real-capable controller.Source: E Real-world Demo · p. 25