MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration

Xiao Wang; Lu Dong; Ifeoma Nwogu; Srirangaraj Setlur; Venu Govindaraju

Source-linked AI summary

MistyPilot: Enabling Social-Robot Control through Multi-Agent LLM Skill Orchestration

Xiao Wang, Lu Dong, Ifeoma Nwogu, Srirangaraj Setlur, Venu Govindaraju

arXiv:2608.15549v1Aug 16, 2026cs.ROcs.AI

TL;DR

Existing social-robot interfaces require developers to manually compose APIs, bind sensors, and manage task states for open-ended physical and social interactions. MistyPilot uses role-specialized LLM agents to orchestrate these tasks and achieves reliable component-level performance with positive user perceptions.

Problem
Existing social-robot interfaces still require developers to compose APIs, configure skills, bind sensor events, and manage task states for diverse interactions.
Method
MistyPilot routes natural-language instructions to specialized agents that separately handle reactive physical control and stateful social interaction at runtime.
Results
MistyPilot demonstrates higher component-level accuracy and lower variance than a single-agent baseline, with positive participant perceptions of usability and interaction quality.
Takeaways & Limitations
The evaluation supports MistyPilot as a practical framework for natural-language skill orchestration across reactive physical and stateful social robot tasks.
Takeaways & Limitations
The evaluation does not cover every social-robot interaction pattern, including composite and longer-horizon tasks, and broader suites remain future work.

Abstract

from arXiv · show

Programming small social robots from natural-language instructions requires more than invoking isolated APIs. Interactive tasks combine reactive physical behaviors with stateful social behaviors, while existing interfaces often require developers to manually compose APIs into skills, configure their parameters, bind sensor events to skills, and manage task states at runtime. We present MistyPilot, a multi-agent LLM framework that interprets high-level natural-language instructions and orchestrates the corresponding skills on the Misty social robot. A Task Router dispatches each instruction to one of two specialized agents: a Physically Interactive Agent for sensor-triggered robot control and direct skill invocation, and a Social Interaction Agent for dialogue-oriented task-state management and context-dependent multimodal response generation. To improve efficiency, the Social Interaction Agent reuses previously generated results when applicable and invokes full generation otherwise. We evaluate MistyPilot on five component-level suites, with sensor bindings and skill invocations executed on the physical Misty robot, and a preliminary user study with 12 participants. MistyPilot attains high accuracy on routing, sensor-skill binding, task-state parsing, result reuse, and skill extension up to 100 skills, and lower variance than an otherwise identical single-agent baseline, while participants report positive perceptions of usability and interaction quality. The code will be made publicly available via the project page.

1 Introduction

MistyPilot addresses the difficulty of programming social robots for diverse, changing user requests by using a multi-agent LLM framework to orchestrate Misty skills at runtime. Its role-specialized agents separate reactive physical interaction from stateful social interaction, with component-level evaluations and a preliminary user study indicating reliable performance and positive perceptions.

Motivation: Everyday social-robot control remains inaccessible to nonprogrammers because developers must compose APIs into skills, configure parameters, bind sensor events, and manage runtime task states.User requests are diverse, open-ended, and may change during interaction.
Technical gaps: Current workflows have separate technical gaps: reactive interaction needs persistent sensor-skill bindings and runtime skill integration, while social interaction needs task-state maintenance across dialogue turns.Existing LLM-generated skill code can reduce individual skill-creation effort but does not discover, invoke, or orchestrate skills at runtime on Misty.
Framework: MistyPilot interprets high-level natural-language instructions and orchestrates corresponding Misty skills at runtime through a role-specialized multi-agent architecture.The framework separates reactive physical interaction from stateful social interaction into dedicated state and skill spaces.
Architecture: A Task Router directs each instruction to either a Physically Interactive Agent or a Social Interaction Agent, avoiding a single context for routing, sensor control, and dialogue management.The Physically Interactive Agent handles direct skill invocation and persistent bindings, while the Social Interaction Agent maintains dialogue task states and generates context-dependent multimodal responses.
Evaluation: The evaluation covers five component-level suites and a preliminary in-person user study, with reliable component performance and positive perceptions of usability and interaction quality.The suites assess routing, sensor-skill binding, task-state parsing, result reuse, and skill extensibility on the Misty robot.

2 Related Work

Prior LLM-agent research has advanced tool use through API calling, interleaved reasoning and acting, multi-tool search, and retrieval over large tool libraries. In social robotics, affect recognition and non-programmer behavior creation have been studied, but runtime orchestration of generated Misty skills remains unaddressed by AutoMisty.

LLM agents and tool use: LLM-agent methods support API calling, reasoning-and-acting interleaving, tree-structured multi-tool search, and vector-store retrieval for large tool libraries.Toolformer, ReAct, ToolLLM, and Tulip Agent exemplify these directions, while API-Bank, APIBench, and ToolAlpaca standardize evaluation.
Social robots: Social-robot research has explored affect recognition in assistive and mental-health settings, while emotionally expressive multimodal response generation remains underexplored.The passage identifies expressive multimodal response generation as a related open challenge.
Social robots: AutoMisty uses a multi-agent LLM system to generate executable Misty skills for non-programmers but does not invoke or orchestrate those skills at runtime.MistyPilot is positioned as complementary to AutoMisty by addressing runtime skill use.

3 PROBLEM FORMULATION

MistyPilot formulates each natural-language instruction as an end-to-end tool-orchestration problem over an open-ended task space. A Task Router selects a specialized agent, which infers and executes an ordered sequence of parameterized function calls satisfying the task requirements.

Task-space formulation: MistyPilot models natural-language instructions Ti ∈ T as tasks requiring an end-to-end tool orchestration pipeline.The task space T covers open-ended interaction scenarios faced by social robots.
Task routing: The Task Router dispatches each instruction to one specialized agent, K* ∈ {PIA, SIA}.The routing relation is defined as K* = R(Ti).
Function selection: The selected agent parses Ti, selects functions fj from tool library F, and infers parameter configurations θj ∈ Θfj.This formulation supports heterogeneous tasks through agent-specific function and parameter selection.
Execution objective: Each task is executed as an ordered sequence of parameterized function calls whose inferred sequence must satisfy the task requirements.The framework’s core objective is to infer a valid function-call sequence for MistyPilot(Ti).

4 METHODOLOGY

MistyPilot uses role-separated multi-agent orchestration to route natural-language tasks between physical robot control and social dialogue. Its agents manage task states, reuse prior results, generate emotion-conditioned responses, and maintain persistent sensor-skill bindings.

Task routing: The Task Router analyzes each high-level instruction and dispatches sensor-triggered or direct-skill tasks to the PIA, while routing dialogue-oriented tasks to the SIA.The PIA performs reactive event-driven control; the SIA produces emotion-conditioned multimodal responses.
Social Interaction Agent: The SIA represents instructions with schema-constrained task states supporting NEW, UPDATE, DELETE, UPGRADE, DOWNGRADE, and MEMORY actions.An explicit editable external memory stores task states, with atomic localized updates that leave unaffected parts unchanged.
Social Interaction Agent: Fast Thinking retrieves and reuses memory by comparing the current main-task embedding with stored results, while Slow Thinking generates scripts from scratch when reuse is unavailable.Stored results are indexed by main-task embeddings and selected using a cosine-distance threshold.
Social Interaction Agent: Slow Thinking assigns each utterance one of eight multimodal emotion categories and coordinates text, expressive motion, and controllable voice generation.The categories are Happiness, Sadness, Anger, Fear, Disgust, Surprise, Contempt, and Neutral.
Physically Interactive Agent: The PIA maps sensor events to skills through BIND, UPDATE, UNBIND, and INVOKE commands, with STM workers and a Central Scheduler maintaining and recovering bindings.STM scans persisted process-table state at startup and automatically remounts sensor-skill bindings on Misty across sessions.

5 Component-Level Evaluation Suites

MistyPilot is evaluated with five component-level suites covering task routing, sensor binding, task-state parsing, retrieval reuse, and skill extensibility. The suites use realistic instruction and dialogue instances to test specialized capabilities across difficulty levels, paraphrases, and dynamically introduced skills.

Evaluation-suite design: Five suites target task routing, sensor binding, task-state parsing, retrieval reuse, and skill extensibility.Their construction combines expert-grounded seeds with a multi-LLM ensemble to synthesize realistic instructions while mitigating single-model bias.
Route100: Route100 contains 100 instructions, including 58 SIA tasks and 42 PIA tasks, with Easy and Hard cases for dialogue, sensor-triggered, and direct-invocation routing.Hard cases involve implicit or composite conditions, whereas Easy cases use direct and straightforward instructions.
SensorBind40: SensorBind40 contains 40 single-turn instances evaluating sensor binding and immediate single-skill invocation across Easy and Hard cases.The suite targets PIA routing accuracy and sensor-binding capability.
TaskParser256: TaskParser256 contains 40 multi-turn dialogues totaling 256 turns that test intent parsing, dynamic task-state management, and system-level controls under Easy and Hard conditions.The dialogues span daily-life assistance, planning, storytelling, and emotion-supportive interaction; 7 Easy dialogues contain 28 turns and 33 Hard dialogues contain 228 turns.
FastThinking230: FastThinking230 contains 230 commands derived from 46 canonical tasks, with five paraphrased variants per task to evaluate retrieval and reuse of prior implementations.The suite tests whether surface-level linguistic variation can be handled while reducing latency and computational cost.
SkillExtension100: SkillExtension100 tests PIA extensibility with dynamically injected skills at scales of 30, 50, 70, and 100 skills using docstrings alone.The system must select the appropriate skill based only on the provided docstring descriptions.

6 Experiments

MistyPilot was evaluated across routing, physical and social interaction, retrieval, skill extensibility, and user perception, using repeated stochastic trials and comparisons with a Single-Agent or Raw Text baseline where applicable. It achieved strong accuracy and stability across component suites, reliable skill selection up to 100 skills, faster retrieval, and positive user ratings.

Experimental Setup: Each configuration was run five times at temperature 0.3, with results reported as mean ± standard deviation and comparisons focused on architectural role separation.The evaluation covered Task Routing, PIA, SIA, fast-thinking retrieval, and skill extensibility.
Task Routing: 100% accuracy was achieved on SIA–Easy, SIA–Hard, and PIA–Easy, while Multi-Agent routing was higher and more stable than the Single-Agent baseline on PIA–Hard.The Multi-Agent design matched or outperformed the Single-Agent baseline across all reported task categories and difficulty levels.
PIA Performance: 100.00%±0% was achieved on SensorBind40-Hard versus 81.00% for the comparison design, while both designs reached 100% on the Easy subset.The PIA evaluation measured dynamic sensor-skill binding and direct skill invocation.
SIA Performance: 99.29%±1.60% versus 91.43%±13.27% was obtained on TaskParser256-Easy, with MistyPilot showing higher accuracy and substantially lower variance across both subsets.The SIA evaluation focused on Task State and System Control correctness for multi-turn dialogue.
Fast Thinking: 100.00% Top-1 accuracy was achieved across all three embedding models versus 67.83%, 72.61%, and 58.70% for Raw Text, while response time fell from 5.088 ± 2.571 s to 2.263 ± 0.627 s.Fast Path retrieval used a disentangled main_task representation rather than the original user input.
User Study: 12 volunteers interacted with MistyPilot for over 6 hours, and all questionnaire mean scores exceeded 4.4/5, with satisfaction, naturalness, and emotional expressiveness rated highest.Participants used SIA for open-domain conversation and PIA for 30 utility skills; occasional failures were mainly attributed to ASR errors on accented speech.

7 Limitations and Future Work

The evaluation suites are not exhaustive, omitting some social-robot interaction patterns, including composite and longer-horizon tasks. Persistent sensor-skill bindings and user-marked results support continuity but may raise privacy concerns on shared robots.

Evaluation coverage: The evaluation suites omit some interaction patterns and do not yet cover composite or longer-horizon tasks.Designing broader evaluation suites remains future work.
Privacy: Sensor-skill bindings and user-marked results persist across sessions, supporting individual-user continuity but potentially raising privacy concerns on shared robots.The paper identifies this persistence as a future-work concern.

8 Conclusion

MistyPilot is a multi-agent LLM framework for natural-language skill orchestration on the Misty social robot. Its role-specialized architecture separates reactive physical interaction from stateful social interaction while supporting several integrated interaction capabilities.

Framework: MistyPilot orchestrates Misty social-robot skills from natural-language instructions using a multi-agent LLM framework.The framework is designed for natural-language skill orchestration on the Misty social robot.
Architecture: Its role-specialized architecture separates reactive physical interaction from stateful social interaction through the PIA and SIA.The architecture assigns physical and social interaction to distinct specialized agents.
Capabilities: MistyPilot supports sensor-triggered skill execution, dialogue-oriented task-state management, result reuse, multimodal response generation, and extensibility to new skills.These capabilities integrate physical control with stateful and multimodal social interaction.