Source-linked AI summary
FreeAskWorld: An Interactive and Closed-Loop Simulator for Human-Centric Embodied AI
Yuhang Peng, Yizhou Pan, Xinning He, Jihaoyu Yang, Xinyu Yin, Han Wang, Xiaoji Zheng, Chao Gao, Jiangtao Gong
TL;DR
Embodied navigation simulators and LLM agents often lack real-time, socially grounded interaction. FreeAskWorld combines LLM-based planning with inquiry-enabled VLN; fine-tuned models improve in open- and closed-loop settings, though they still struggle with high-level reasoning and socially grounded navigation.
Problem
VLN frameworks and LLM-driven agents often lack real-time, closed-loop interaction that integrates embodied navigation with socially grounded planning.
Source: Introduction · p. 1Method
FreeAskWorld combines LLM-based intention modeling and semantic interaction with an inquiry-based VLN task in which agents seek guidance and adapt in dynamic 3D scenes.
Source: Introduction · p. 1, Direction Inquiry Task · p. 4Results
Fine-tuned VLN models improve in both open-loop and closed-loop settings, but still struggle with high-level reasoning and socially grounded navigation relative to humans.
Source: Conclusion · p. 7Takeaways & Limitations
Structured interaction can provide information that static perception alone cannot, serving as an additional modality for interpreting the physical world.
Source: Conclusion · p. 7The benchmark suite still needs expansion with more comprehensive metrics for evaluating embodied-AI interactions.
Source: Conclusion · p. 8
Abstract
from arXiv · showhide
As embodied intelligence emerges as a core frontier in artificial intelligence research, simulation platforms must evolve beyond low-level physical interactions to capture complex, human-centered social behaviors. We introduce FreeAskWorld, an interactive simulation framework that integrates large language models (LLMs) for high-level behavior planning and semantically grounded interaction, informed by theories of intention and social cognition. Our framework supports scalable, realistic human-agent simulations and includes a modular data generation pipeline tailored for diverse embodied tasks. To validate the framework, we extend the classic Vision-and-Language Navigation (VLN) task into a interaction enriched Direction Inquiry setting, wherein agents can actively seek and interpret navigational guidance. We present and publicly release FreeAskWorld, a large-scale benchmark dataset comprising reconstructed environments, six diverse task types, 16 core object categories, 63,429 annotated sample frames, and more than 17 hours of interaction data to support training and evaluation of embodied AI systems. We benchmark VLN models, and human participants under both open-loop and closed-loop settings. Experimental results demonstrate that models fine-tuned on FreeAskWorld outperform their original counterparts, achieving enhanced semantic understanding and interaction competency. These findings underscore the efficacy of socially grounded simulation frameworks in advancing embodied AI systems toward sophisticated high-level planning and more naturalistic human-agent interaction. Importantly, our work underscores that interaction itself serves as an additional information modality.
Introduction
FreeAskWorld is an LLM-driven, interactive closed-loop simulator for human-centric embodied AI that connects social intention modeling with navigation and introduces a Direction Inquiry task supported by a synthetic dataset.
Introduction
- VLN systems often rely on one-shot instructions, lack social-intention modeling, and use simulators without dynamic social environments; LLM agents also remain disconnected from embodied navigation and closed-loop semantic-spatial interaction.These limitations constrain agents’ ability to handle dynamic goals, interpret socially salient cues, and communicate in context.Source: Introduction · p. 1
- FreeAskWorld uses LLMs for high-level intention modeling, semantic instruction generation, and naturalistic human behavior in photorealistic 3D environments, enabling dynamic, contextually rich interactions.The framework is grounded in theories of social behavior and intention.Source: Introduction · p. 1
- The Direction Inquiry Task lets agents proactively seek help and adapt navigation to new information, evaluating self-assessment, social interaction, and real-time adaptation.The released dataset includes diverse human avatars, reconstructed urban environments, dynamic vehicles, six synthetic data categories, and over six hours of interactive simulation data for open- and closed-loop evaluation.Source: Introduction · p. 2
- The work evaluates representative VLN models and a human baseline to assess the task and simulator and the effectiveness of the generated data.Its contributions include an interactive LLM-driven simulation framework, a controllable benchmark for human-centric social navigation, and a synthetic data generation pipeline.Source: Introduction · p. 2
Related Work
Prior work advances social simulation and embodied navigation, but still leaves gaps in contextual human behavior and adaptive, socially grounded interaction. FreeAskWorld addresses these gaps with LLM-based planning and semantically grounded, interactive simulation.
Simulating Human Behavior in Social Navigation
- Traditional social-navigation simulators model local physical interactions but miss societal-level nuance, while generative-agent systems model higher-level behaviors and open-ended social interactions (Jin et al. 2024; Zhou et al. 2025; Wang et al. 2024a).The proposed framework integrates LLM planning and semantically grounded interaction informed by intention and social-behavior theories, alongside Unity’s animation engine and asset library.Source: Simulating Human Behavior in Social… · p. 2
Human-Centric Language Use in Navigation
- Verbal directions use route perspectives with landmarks and turns or survey perspectives with distances, street names, and cardinal directions; style varies with sociospatial background and gender (Hund, Schmettow, and Noordzij 2012; Sing and Kalingga 2011).Irregular European layouts tend to favor landmarks, while grid-based American cities rely more on street names; women usually provide longer instructions with more spatial references and hedging expressions.Source: Human-Centric Language Use in Navigation · p. 2
Vision Language Navigation
- VLN progresses from R2R’s discrete navigation graphs to REVERIE’s object grounding, R2R-CE’s continuous actions and real-time physics, and Talk2Nav’s long-range following with attention and spatial memory (Anderson et al. 2018; Qi et al. 2020; Krantz et al. 2020; Vasudevan, Dai, and Van Gool 2021).VLN combines computer vision, natural language understanding, and robotics to follow natural-language instructions in visually rich 3D environments.Source: Vision Language Navigation · p. 2
- VLN systems rely on static instructions, weakly integrate social intention, and use simulators that omit dynamic human agents and socially grounded interactions; the proposed framework combines realistic simulation with human-like instruction generation and social reasoning.The framework uses LLMs and behavior priors to unite language understanding, motor control, and social reasoning in a continuous embodied setting.Source: Vision Language Navigation · p. 2
FreeAskWorld: Simulator Design · Motivation and Overview
FreeAskWorld addresses the lack of realistic social behavior, environmental dynamics, and long-term evolution in existing simulators with a high-fidelity platform for modeling human-like societies. It currently focuses on direction inquiries while offering extensible infrastructure for studying sustained interactions and future complex tasks.
Motivation and Overview
- Scene randomization, people simulation, and instruction generation support environmental diversity and the Direction Inquiry Task data-generation pipeline.Source: Figure 1 · p. 3
- Existing simulators often emphasize physical- or task-level interactions, limiting realistic social behavior, environmental dynamics, and long-term temporal evolution.Source: Motivation and Overview · p. 3
- FreeAskWorld models human-like societies through structured social behavior, open interaction interfaces, and integrated multisystem dynamics.Source: Motivation and Overview · p. 3
- Real-world social structures, behavioral norms, transportation, weather, and daily activity cycles introduce multilevel decision-making and environmental variability.Source: Motivation and Overview · p. 3
- Direction inquiry is the current representative interaction task, while negotiation, task coordination, social navigation, and long-term trust building are future targets.Source: Motivation and Overview · p. 3
- The framework supports large-scale multi-agent deployment and behavior tracking through modular interfaces for language models, multimodal perception, and other intelligent components.Source: Motivation and Overview · p. 3
People Simulation
FreeAskWorld’s People Simulation module combines avatar modeling, profile and schedule generation, navigation-style synthesis, animation control, and appearance variation to create human agents and behaviors.
People Simulation
- The module encompasses avatar modeling, profile and schedule generation, navigation-style synthesis, animation control, and appearance variation.Figure 3 presents an overview of the module.Source: People Simulation · p. 3
Profile And Schedule Generation
- Profiles combine demographic and contextual attributes with scene-based daily schedules segmented by activity, time, and location.The two-stage process first synthesizes profiles, then generates schedules with activities assigned to destinations in the environment.Source: Profile And Schedule Generation · p. 3
Navigation Style Generation
- Navigation-style generation uses regional familiarity, personality, and four linguistic features to produce context-grounded labels and instructions.The four features are landmark use, direction type, distance description, and utterance length; an LLM draws on an encoded literature knowledge base and the role profile, task, and geographic situation.Source: Navigation Style Generation · p. 3
Animation Database
- MotionX animations are organized by activity, and a custom SMPL-X driver with blending enables semantically relevant motion selection and smooth action transitions.The driver provides access to the library’s animations, while blending improves animation fidelity and character realism.Source: Animation Database · p. 4
- Appearance generation combines language-guided textures and varied SMPL-X body shapes with Synbody models featuring detailed clothing, footwear, and hair.Semantic profiles guide texture generation, while randomized shape parameters and semantic control over beta shapes vary body types.Source: Apperance Variation · p. 4
Other System Functions
FreeAskWorld combines occupancy mapping, environmental dynamics, socially aware robot navigation, and synchronous closed-loop simulation. Its system functions support navigation, perception, planning, and varied simulation conditions.
Occupancy Map Generation
- Probabilistic voxel occupancy estimates use random sampling and collision checks, then repeated averaging and filtering before projection into top-down 2D heatmaps or binary grids.The occupancy map supports navigation, mapping, and interaction.Source: Occupancy Map Generation · p. 4
Weather System
- Dynamic sky and weather simulate day-night cycles, rain, and fog to increase realism and data diversity and support generalization across lighting and visibility conditions.Source: Weather System · p. 4
Traffic Simulation
- Route-graph traffic simulation models vehicle movement and traffic rules, adding realistic vehicle behavior and dynamic complexity for perception and planning tasks.Source: Traffic Simulation · p. 4
Robot Simulation
- A* handles global planning and SFM local avoidance; during data collection, SFM improves navigation success by avoiding pedestrians and vehicles but lengthens trajectories.Unity articulation components support realistic robot dynamics, and the simulator integrates physical robots through the URDF-Importer.Source: Robot Simulation · p. 4
Synchronous Closed-loop Framework
- A WebSocket-based synchronous architecture connects server-side models and the simulator through NAT traversal or same-device port communication, exchanging sensor data and control commands.Source: Synchronous Closed-loop Framework · p. 4
Direction Inquiry Task
The Direction Inquiry Task extends VLN with active information-seeking and evaluates both high-level decision-making and low-level navigation in dynamic conditions.
Direction Inquiry Task
- The task adds an inquiry phase to VLN, testing self-assessment, information seeking, knowledge-based planning, and low-level motion planning amid dynamic humans and vehicles.The setting lets agents seek external information and plan from what they acquire.Source: Direction Inquiry Task · p. 4
Evaluation Metrics
- Evaluation uses TL for average distance traveled, SPL for success and efficiency, and SR for the proportion of successful episodes.Source: Evaluation Metrics · p. 5
- NE and ONE measure final and minimum-along-trajectory goal distance, OSR captures reaching the success radius at any point, and NDI averages inquiry actions per episode.Source: Evaluation Metrics · p. 5
FreeAskWorld Dataset
FreeAskWorld’s dataset pipeline generates inquiry-based navigation episodes with human-like guidance, while its multimodal annotations support embodied tasks and scene reconstruction. Human evaluators check whether generated instructions align with destinations and are coherent and understandable.
Dataset Overview
- The dataset is compared with existing VLN datasets, whose characteristics are summarized in Table 1.Source: Dataset Overview · p. 5
Dataset Validation
- Human evaluators verify destination alignment and assess whether LLM-generated instructions are coherent, human-like, and easy to understand.The evaluation is described as a human baseline.Source: Dataset Validation · p. 6
Data Generation Pipeline
- The pipeline randomizes weather and time, solicits nearby human agents for LLM-generated guidance, and records episode data after socially compliant navigation.If the agent misses a predefined time threshold, it asks again for updated guidance; successful episodes record dialogue, panoramic and perspective RGB images, and synthetic annotations.Source: Data Generation Pipeline · p. 6
Sensor Configuration
- Six co-located cameras with 90° fields of view capture panoramic images at 1 Hz, while camera-rig position and orientation are recorded.The rig pose supports alignment of synthesized images with global scene context.Source: Sensor Configuration · p. 6
Trajectory Generation
- Random start and target positions, inquiry-based guidance, and socially compliant rule-based path planning produce expert trajectories for imitation learning.The planner avoids static and dynamic obstacles.Source: Trajectory Generation · p. 6
Data Composition
- The dataset combines six synthetic data types and annotations for 16 object categories common in human-centered environments.Its data include visual and geometric annotations, panoramic and perspective images, interaction histories and trajectories, occupancy heatmaps, and environment metadata; annotations include vehicles, pedestrians, and street furniture.Source: Data Composition · p. 6, Figure 4 · p. 6
Scene Reconstruction
- Occupancy heatmaps, dynamic-entity 3D boxes, and world information enable scene reconstruction as a digital twin for evaluation and downstream tasks.Supported tasks include navigation planning, behavior prediction, and human-computer interaction studies.Source: Scene Reconstruction · p. 6
Experiments
Experiments evaluate FreeAskWorld-trained and fine-tuned models in open-loop and closed-loop navigation settings. Fine-tuning reduces open-loop error, while closed-loop results show benefits from human direction inquiries and improvements in model navigation metrics alongside persistent task challenges.
Baselines
- The experiments compare human navigation with ETPNav and BEVBert, evaluated as pretrained baselines and FreeAskWorld-fine-tuned variants, using L2 distance open-loop and Direction Inquiry Task metrics closed-loop.The human baseline is treated as an upper-bound reference; fine-tuned model variants are named ETPNav-FT and BEVBert-FT.Source: Baselines · p. 7
Experimental Setup
- Closed-loop episodes begin at NDI = 0 after the initial instruction, use averaged multiple trials, and end after collisions or 100 steps.Four participants control the human baseline by keyboard; two follow only the initial instruction and two may ask follow-up questions, each counted as one NDI. Models run synchronously at 1 Hz, with models on an RTX 3080 and the simulator on an RTX 3060.Source: Experimental Setup · p. 7
Results
- Fine-tuned ETPNav-FT and BEVBert-FT reduce open-loop L2 error by ∼50% versus their base versions, with BEVBert-FT performing best overall.Source: Results · p. 7
- Human pathfinding accuracy rises from 40.2% to 82.6% when participants can query for additional navigation instructions.The authors attribute the challenge to scene complexity, including identical stores in different directions within 50 meters, which can disorient people with shallow memory of directions.Source: Results · p. 7
- In closed-loop evaluation, fine-tuned models improve Navigation Error and Oracle Navigation Error over their base counterparts, but ETPNav-FT retains a zero Success Rate.ETPNav-FT achieves partial destination success despite the zero Oracle Success Rate of baseline models; its longer trajectories suggest greater scene familiarity and broader exploration. BEVBert consistently outperforms ETPNav across both evaluation metrics.Source: Table 2 · p. 7, Results · p. 7
Conclusion
FreeAskWorld extends VLN with interactive guidance and dataset support; fine-tuned models improve but still trail humans in reasoning and socially grounded navigation. The conclusion presents structured interaction as information for interpreting the physical world and outlines future directions.
Conclusion
- The Direction Inquiry Task extends VLN with self-assessment, social interaction, and real-time adaptation; fine-tuned models improve in open- and closed-loop settings but still lag humans in reasoning and socially grounded navigation.The released FreeAskWorld Dataset features diverse avatars, dynamic scenes, and rich multimodal annotations.Source: Conclusion · p. 7
- Intentional, structured interaction provides information static perception alone cannot, helping agents understand and interpret the physical world and bridge embodied AI with real-world human interaction.The conclusion frames interaction as both a social signal and an additional information modality.Source: Conclusion · pp. 7–8
- Future work targets negotiation and coordination, multimodal memory and perception for adaptive behavior, higher-fidelity generative visuals, a Steam software release, and broader interaction metrics.Source: Conclusion · p. 8