Source-linked AI summary
Ludi${}_{\scriptscriptstyle 0.1}$: An Agentic System for Socially Intelligent Robots
Wooseong Chung, William Cong, Jakub Dworakowski, Ethan Ewer, Tri Wahyu Guntara, Yeonwoo Jeong, Tianchong Jiang, Chaewon Kim, Hyunseo Kim, Jinwoo Kim, Jinyeon Kim, Yea-Seul Kim, Jack Kunde, Kangwook Lee, Sangheon Lee, Robert Nowak, Junha Roh
TL;DR
Natural human–robot collaboration requires more than isolated command execution: robots must maintain context, handle ambiguity, and revise behavior as intent changes. Ludi0.1 addresses this with a fine-tuned VLM, multi-turn interaction traces, a tool-management harness, and specialized physical policies. In evaluation, it completed 20 of 28 closed-loop scenarios, while the authors identify operational integration across separate components as a limitation.
Problem
Natural human–robot collaboration requires robots to recognize ambiguity, maintain context, communicate intentions, and revise ongoing behavior as user intent changes.
Method
Ludi0.1 combines a fine-tuned VLM trained on multi-turn interaction traces with a harness coordinating speech, navigation, memory, and specialized manipulation policies.
Results
20 of 28 Core28 scenarios were completed end-to-end by the fine-tuned 9B model, approaching GPT-5.6 with an optimized prompt.
Takeaways & Limitations
Ludi0.1 demonstrates a practical agentic path for fluid human–robot interaction while generating multimodal traces for future integrated robot foundation models.
Takeaways & Limitations
Speech, memory, navigation, and manipulation remain distributed across components, causing possible context loss, latency, brittle handoffs, and fragmented behavior.
Abstract
from arXiv · showhide
Robot foundation models have substantially advanced perception and control, but natural human-robot collaboration requires more than executing isolated commands. A robot must recognize ambiguity, maintain context across turns, communicate its intentions, and revise ongoing behavior as the user's intent changes. We present $\scriptstyle\mathsf{Ludi}_{\scriptscriptstyle 0.1}$, an agentic system for socially intelligent robots that integrates interactive speech, multimodal reasoning, memory, navigation, and learned manipulation. Its decision-making core is a fine-tuned vision-language model trained on multi-turn interaction traces spanning ambiguous requests, clarifications, corrections, interruptions, mixed social and task dialogue, and multi-step tasks. A purpose-built harness manages the model-tool interaction loop, while specialized navigation and manipulation policies execute physical skills. Ludi${}_{\scriptscriptstyle 0.1}$ demonstrates a practical path toward fluid human-robot collaboration today while producing the multimodal interaction traces needed to develop a more deeply integrated foundation model for robots and people.
1. Introduction
Natural human–robot collaboration requires robots to interpret changing intent and coordinate dialogue, context, navigation, and manipulation during ongoing action. Ludi0.1 illustrates this capability by revising a delivery after the user changes Coke to Pepsi.
- Motivation: Robot foundation models have expanded physical task competence, but natural collaboration also requires handling ambiguity, context, communication, and changing intent.These capabilities remain difficult to make reliable across varied environments and tasks.
- Motivation: When the user changes the request after reaching for Coke begins, Ludi0.1 treats the correction as superseding the earlier choice.The robot abandons the Coke action and switches to Pepsi.
- Demonstration: Ludi0.1 then retrieves Pepsi, navigates to the bedroom, and delivers it to Chloe’s desk.The episode coordinates dialogue, contextual reasoning, navigation, and manipulation in one evolving interaction.
- Motivation: The system targets fluid interaction by integrating speech, perception, memory, reasoning, navigation, and manipulation so new information can reshape ongoing behavior.The paper presents this integration as a step toward responsive human–robot collaboration.
2. Ludi0.1: An Agentic Approach
Ludi0.1 uses an agentic architecture that coordinates specialized capabilities over time rather than relying on one end-to-end model. A VLM reasons over interaction context while a harness and physical tools execute and update the interaction.
- System overview: Ludi0.1 integrates perception, navigation, manipulation, speech, dialogue, memory, and social reasoning in a functioning Unitree G1 system.Its supported behaviors include clarification, indoor navigation, manipulation, context maintenance, and plan revision.
- Reasoning core: A fine-tuned VLM interprets the scene and interaction history, then decides whether to speak, wait, navigate, or invoke manipulation.The VLM serves as the agent’s reasoning core.
- Control loop: The harness assembles context, calls the VLM, validates and executes a selected tool, records its result, and repeats the cycle.This supports multiple reasoning, tool-use, and observation cycles within one request.
- Control loop: New user input can redirect the robot while it is speaking or acting because the harness maintains shared interaction history across turns.An auxiliary model compacts older context in the background without disrupting active turns.
- System overview: The complete embodied agent combines the VLM and harness with speech, navigation, and manipulation tools so communication and physical action unfold coherently.The stated goal is interaction that cannot be reduced to isolated perception, dialogue, navigation, or control tasks.
3. A VLM for Interactive Robotics
Ludi0.1 fine-tunes a VLM on synthetic multi-turn interaction traces and connects it to typed tools through a closed-loop agent interface. Evaluation uses interactive simulation to measure sustained perception, dialogue, navigation, and manipulation rather than isolated predictions.
- VLM and training: A fine-tuned Qwen3.5 VLM serves as the reasoning core, interpreting scenes and interactions, selecting tools, and revising behavior when new information arrives.Training targets evolving interaction and shared-context reasoning beyond visual recognition and instruction following.
- VLM and tools: The model receives ego-view and panoramic images, transcribed speech, interaction history, and prior tool outcomes before producing one action.Available tools include speech, location lookup, navigation, reachability checks, manipulation, and waiting.
- VLM and tools: The harness enforces typed tool contracts, rejects malformed calls, and exposes newly registered manipulation object types without retraining the agent.A pick must be preceded by a successful reachability check.
- VLM and training: The synthetic corpus contains 14,432 multi-turn trajectories covering ambiguity, corrections, interruptions, multi-step errands, and mixed task-social dialogue.Scenarios are serialized as tool-call traces with intermediate reasoning and pass automated and LLM quality checks.
- Evaluation: Closed-loop simulation in AI2-THOR and ProcTHOR evaluates the same tool interface and deployed-system behavior while scoring interaction and final simulator state.Interaction Core28 contains 28 scenarios spanning delivery, clarification, failed searches, unreachable objects, and mid-task interaction.
- Evaluation: 20 of 28 scenarios are completed end-to-end by the fine-tuned 9B model, with remaining failures concentrated in imprecise delivery-end spatial grounding.Fine-tuning roughly doubles success over the same pretrained model and approaches GPT-5.6 with an optimized prompt.
4. Engineering the Harness for Real-Time Interaction
The harness is designed to preserve responsive interaction despite asynchronous model inference, context management, speech, and physical actions. It compacts history without interrupting active turns and uses visual prefill and queued speech to reduce response delays.
- Real-time design: The control harness coordinates components operating at different timescales to keep interaction responsive and reliable.Its design addresses model inference, context management, speech, and physical actions within one loop.
- Real-time design: Non-blocking context compaction summarizes older history in the background and installs it only between active turns.This prevents context management from interrupting or changing a running agent loop.
- Real-time design: Visual prefill refreshes camera input before instructions arrive, while asynchronous FIFO speech lets reasoning and action continue during spoken responses.Together these choices reduce response delay and prevent speech from blocking the agent loop.
5. Humanoid Navigation: Keypoints and Continuous Planning
Ludi0.1 combines named-keypoint navigation for most household tasks with an orientation-aware continuous planner for finer humanoid motion. The continuous planner accounts for the G1’s geometry and supports constrained maneuvers such as turning, strafing, backward motion, and narrow passage traversal.
- Ludi0.1 navigates most household tasks through a static map of named keypoints rather than direct metric-coordinate reasoning.
- Localization combines KISS-ICP LiDAR odometry with an extended Kalman filter, with hyperparameters tuned automatically using Optuna.
- An orientation-aware continuous planner handles local motions requiring finer humanoid-specific control.It represents position and orientation with quintic spline trajectories and plans heading independently of travel direction.
- The planner uses an orientation-aware multi-circle footprint to model body geometry during turning, strafing, backward motion, and narrow-space traversal.
6. Specialized VLA Manipulation Policies
Ludi0.1 delegates physical manipulation to specialized VLA policies invoked by the agent as tools. These policies are fine-tuned for the target home environment, developed with calibrated simulation, and validated through real-robot rollouts.
- GR00T N1.7 was fine-tuned on task-specific demonstrations using pretrained Unitree G1 embodiment components.
- The VLA controls manipulation degrees of freedom, predicting arm and gripper actions while navigation and other body motion are handled separately.
- Specialized VLA policies execute physical manipulation after the agent invokes them as tools.The VLM interprets the broader interaction, decides when manipulation is appropriate, and provides the task instruction; the VLA returns its outcome when complete or interrupted.
- A calibrated Isaac Sim environment provides repeatable comparisons of architectures, training procedures, execution horizons, action representations, and data mixtures before physical rollouts.Calibration covered system timing, camera pose, scene geometry, and motor dynamics against recorded physical-robot behavior.
- The real-robot demonstration shows a Unitree G1 with a Dex-3 hand grasping a tabletop can from egocentric visual observations.
- 0.92 mask IoU and 0.043 rad Dex-3 hand tracking error were achieved after calibration, with fingertip error reduced to at most 13 mm.Simulated performance did not always preserve physical-robot policy rankings, so promising policies were validated through real-world rollouts.
7. Related Work
Related work combines language or vision-language models with grounded robot skills, interactive memory, feedback, and embodied action. Ludi0.1 builds on these directions while emphasizing fine-tuned, sustained human-robot interaction across physical capabilities.
- SayCan combines language-model action selection with learned affordance values reflecting executable robot skills.
- Inner Monologue returns scene descriptions, success signals, and human feedback so the model can revise its plan.
- KnowNo addresses ambiguous requests by calibrating planner uncertainty so the robot can ask for help.
- HELPER-X uses a memory-augmented language model for interactive embodied domains including dialogue-based execution, instruction following, and active question asking.
- Hi Robot uses a hierarchical VLM/VLA architecture to incorporate complex instructions and situated user feedback as tasks unfold.
- Ludi0.1 emphasizes sustained interaction by fine-tuning its VLM on traces containing ambiguity, clarification, social dialogue, interruptions, corrections, and tool outcomes.It reasons over speech, multi-view visuals, interaction history, and prior action outcomes while coordinating speech, navigation, and manipulation.
8. The Limits (and Promise) of Agentic Robotics
Ludi0.1 integrates interaction across speech, memory, navigation, and manipulation, but its integration remains operational rather than fully learned. The authors therefore pursue both near-term agentic improvements and a future foundation model trained on the system’s multimodal traces.
- The execution layer is modular while the interaction layer is integrated around a central VLM and harness.The system coordinates speaking, clarifying, waiting, navigating, and manipulating through specialized components.
- This architecture supports real-time clarification, interruption handling, memory-informed behavior, and coordination between conversation and physical action.
- The integration remains operational rather than fully learned, leaving speech, memory, navigation, and manipulation distributed across separate components.Model calls and tools can introduce context loss, latency, brittle handoffs, and fragmented behavior.
- The authors pursue two paths: improving the agentic system for present-day responsiveness and using its multimodal interaction traces to develop a more integrated robot foundation model.The traces connect perception, speech, human feedback, decisions, and physical action.
9. Toward Ludi 1.0: A Foundation Model for Robots and People
Ludi 1.0 is envisioned as a foundation model that deeply integrates perception, dialogue, memory, reasoning, and control for robots and people. Ludi0.1 serves as both a functioning embodied agent and a testbed whose interaction traces expose capabilities future models must represent.
- Ludi 1.0 aims to integrate perception, dialogue, memory, reasoning, and control in a shared representation for robots and people.Language and physical action would continually inform one another rather than conversation being layered over robot behavior.
- The envisioned representation tracks the robot and environment, user history and intent, and how language and action evolve together over time.It should support deciding what to do next as well as recognizing when to clarify, revise, pause, or stop.
- Ludi0.1 functions both as an embodied agent and as a testbed for developing these integrated capabilities.
- Its interaction traces expose current architectural limits and reveal what a future robot foundation model must learn to represent.The longer-term goal is integrated physical and social intelligence in a coherent ongoing interaction.
10. Authorship
Ludi0.1 was developed by contributors at Ludo Robotics. The listed contributors include affiliations with the University of Wisconsin–Madison, Toyota Technological Institute at Chicago, and KRAFTON.
- Ludi0.1 was developed by contributors at Ludo Robotics.
- The contributor list includes Wooseong Chung, William Cong, Jakub Dworakowski, Ethan Ewer, Tri Wahyu Guntara, and other named collaborators.The passage lists the full contributor roster.
- Contributor affiliations include the University of Wisconsin–Madison, Toyota Technological Institute at Chicago, and KRAFTON.The asterisk indicates interns; numbered affiliations identify additional institutional ties.