PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-HSI: Personalized human control of a robot swarm on demand by using LLMs for online automatic code generation

Alessandro Nazzari, Nathan Cerisara, Dorian Tonnis, Raina Zakir, Lorenzo Labarile, Weixu Zhu, Marco Dorigo, Mary Katherine Heinrich

arXiv:2609.16346v1cs.ROcs.AIcs.HCcs.MA

TL;DR

Auto-HSI addresses the prevalence of predefined swarm behaviors and interfaces that often assume trained operators by generating personalized gesture interfaces from demonstrations and natural-language descriptions. Its prototype supports gesture control of swarm behaviors, and operators completed all tested trials, although wheel-speed noise affected tasks differently. The current system relies on external motion tracking and a central base station.

  • Problem

    Existing HSI studies usually offer predefined, often limited swarm behaviors and interfaces that assume operators already know which inputs to use and how.

    Source: §I · p. 1

  • Method

    Auto-HSI generates a personalized state-machine interface by linking demonstrated gestures to desired swarm behaviors described in natural language, with an LLM generating or modifying code.

    Source: §I · p. 1

  • Results

    Operators completed all 18 trials, while wheel-speed noise had similar or sometimes better effects in Task-2 and notably worsened performance in Tasks-1 and -3.

    Source: 4) Results of all trials (Tasks-1, -2,… · p. 7

  • Takeaways & Limitations

    Users operate the swarm through gestures alone after personalization, which can also be repeated during operation to update the interface.

    Source: §I · p. 1

    The current system coordinates robots using external motion tracking and a central base station that issues motion commands.

    Source: §V · p. 7

Abstract

from arXiv · show

This paper presents Auto-HSI, a method for generating personalized human-swarm interaction (HSI) interfaces on demand. The objective is to enable untrained operators to use natural language descriptions and gesture demonstrations to explain how they want the robots to collectively behave in response to their gestures. Based on these inputs, the code should automatically be generated for personalized state machines that will control the robots as desired, in response to the desired gesture inputs. In the developed Auto-HSI prototype, the generated code produces a personalized interface for centralized control using one- and two-handed gestures, enabling a user to teleoperate the robots' motion, formation shape, and shape deformation. We test the gesture tracking and code generation components of Auto-HSI against performance benchmarks. We then test the full Auto-HSI prototype in ``live'' operation experiments, in which real human operators centrally control 50 simulated robots in a physics-based simulator, under nominal and noisy conditions. In these experiments, robots are teleoperated to: score a goal, traverse a maze that requires shape deformation, and score two simultaneous goals by splitting into two groups. We also demonstrate a real human operator making live updates to their personalized Auto-HSI interface during operation (in simulation). Finally, we demonstrate live operation of real robots.

I. INTRODUCTION

Auto-HSI addresses interfaces that usually offer predefined behaviors and inputs, often assuming trained operators, by generating a personalized gesture-to-swarm interface from demonstrated gestures and natural-language behavior descriptions. Interaction has a personalization phase followed by gesture-only live operation, and personalization can be repeated during operation to update the interface.

I. INTRODUCTION

  • Existing HSI interfaces usually predefine a limited set of swarm behaviors and inputs, often assuming operators already know how to steer the swarm.Centralized teleoperation is a common approach, using inputs such as gesture or pose tracking, joysticks, or haptic devices.Source: §I · p. 1
  • Auto-HSI generates a personalized interface in which users associate demonstrated gestures with desired multi-robot shape and motion behaviors described in natural language.The prototype uses hand gestures and requires a closed fist to mark each gesture’s beginning and end.Source: §I · p. 1

II. METHODS: USER INTERACTION WITH Auto-HSI

  • Interaction separates into personalization, where the user demonstrates gestures and describes desired behaviors, and live operation, where gestures alone operate the swarm.The LLM generates or edits code during personalization; it does not issue robot commands, and personalization can be repeated during operation.Source: §II · p. 1

A. Personalization phase

During personalization, Auto-HSI uses the user’s natural-language request and current state-machine registry to create or modify behavior, validating each update before commitment. During live operation, gestures trigger transitions without LLM calls, and personalization can be invoked again to revise the interface.

A. Personalization phase

  • The LLM uses the user’s natural-language description and current state-machine registry to modify a relevant existing machine or create a new one, validating updates before committing them.An existing machine may be extended with a node and a transition associated with the newly demonstrated gesture.Source: A. Personalization phase · pp. 1–2

B. Live operation phase

  • During live operation, recognized gestures are matched to the gesture database with dynamic time warping, and matching identifiers trigger transitions across the robots’ state machines without LLM calls.A match is accepted when its DTW cost falls below a tunable threshold.Source: B. Live operation phase · p. 2

C. Updates during operation: re-triggering personalization

  • Users can re-trigger personalization during operation and describe in natural language how a behavior should change; the LLM produces and applies the corresponding update.The update is applied in the same manner as during initial personalization.Source: C. Updates during operation:… · p. 2

III. METHODS: Auto-HSI SOFTWARE COMPONENTS

  • The software comprises gesture tracking and recognition, code generation, an external LLM, and robot communication; the prototype and experimental software are available in an online repository.Source: §III · p. 2

A. Gesture tracking and recognition

Auto-HSI tracks hand landmarks, segments gestures using fists, and represents each gesture as a sequence of four-dimensional changes from its initial hand configuration. It compares gesture sequences with dynamic time warping to accommodate different performance speeds.

A. Gesture tracking and recognition

  • Each visible hand is represented by 21 landmarks, and non-fist hand configurations yield palm centers combined into a two-handed configuration; an undetected hand is assigned (0,0).The 21 landmarks comprise four points per finger and one wrist point; fist classification compares fingertip–wrist and finger-base–wrist distances.Source: A. Gesture tracking and recognition · p. 2
  • Fists delimit gesture segments, which are represented by successive differences between two-handed configurations and the segment’s initial configuration, producing four-dimensional vectors.A segment contains frames with at least one visible non-fist hand and is preceded and followed by frames containing only fists.Source: A. Gesture tracking and recognition · p. 2
  • Dynamic time warping compares gesture sequences along the temporal axis to account for different performance speeds, using Euclidean distance between four-dimensional vectors.Demonstrated gestures are stored in a database for comparison with gestures during live operation.Source: A. Gesture tracking and recognition · p. 2

B. Code generation

Auto-HSI incrementally builds a registry of state machines from user requests, generating transitions and executable node code alongside shape definitions. The LLM can inspect, create, or modify these elements using registry tools and provided functions, with prompts and examples guiding its code generation.

B. Code generation

  • Each state machine has a neutral base node and behavior nodes with code blocks for entry, each control step, and exit; gesture-triggered or automatic transitions route through the base node.Machines receive general behavior-category labels so later requests in the same category can be applied to them.Source: B. Code generation · p. 2
  • The example registry contains separate machines for movement, rotation, and shape, linking recorded gestures to transitions; a movement node sets forward velocity on entry, maintains it each step, and resets it on exit.The generated executable code can reference a shape-definition database generated alongside the state machines.Source: B. Code generation · p. 2, Figure 1 · p. 3
  • A natural-language request supplies the user’s input, a baseline prompt, and the current registry; the LLM uses tool calls to create or modify the registry, with interaction history supporting incremental changes.The baseline prompt includes tools to inspect and edit machines, nodes, transitions, code blocks, gestures, and shape definitions.Source: B. Code generation · p. 3
  • Generated node code can call functions for formation creation, deformation, scaling, velocity, and rotation, while shape definitions can be based on a user’s drawing or natural-language description.The generated code is free-form Lua using provided functions and generated arguments.Source: B. Code generation · p. 3
  • The baseline prompt gives general interaction and code-structure guidance, and includes an example request that orders registry inspection, gesture and shape handling, machine updates, and verification.The prompt advises a conversational tone and prohibits asking the user for feedback on generated code.Source: B. Code generation · p. 3

C. Interface to an external LLM

The interface connects an external LLM to the state-machine registry and robots. Registry updates are validated and transmitted to robots running local copies of the generated code.

C. Interface to an external LLM

  • The LLM interface accepts typed or dictated natural-language input and supports interchangeable external models through OpenRouter's API.The registry manager validates and compiles Lua fragments, returns rejected fragments for correction, and snapshots the registry.Source: C. Interface to an external LLM · p. 4
  • LLM requests run in a dedicated thread so resolving a query does not block the interface or communication loop.Runtime errors reported by robots trigger an automatic repair request to the server.Source: C. Interface to an external LLM · p. 4

D. Communication with the robots

  • Each robot runs a local copy of the generated code; committed state-machine changes are serialized to JSON and pushed over its TCP connection.Full updates rebuild the state machine while preserving the active-node identifier; partial updates recompile only relevant code blocks.Source: D. Communication with the robots · p. 4

IV. RESULTS

The results section benchmarks gesture recognition and introduces full-prototype operator experiments. Gesture recognition achieves 94.4% overall accuracy, with stronger recall for swipes than for shape gestures.

A. Benchmark: Gesture recognition

  • The benchmark used 41 ground-truth gestures from four operators and 656 test recordings, categorized as one- or two-handed swipes and shapes.Each test sample was matched against all 164 ground-truth recordings; Table I reports recall and precision with 95% confidence intervals.Source: A. Benchmark: Gesture recognition · p. 4, Table I · p. 4

IV. RESULTS

  • Recall measures correct identification of demonstrated gestures, while precision measures whether a selected gesture was actually performed.The results section also states that gesture tracking and code generation are benchmarked before full-prototype operator experiments.Source: A. Benchmark: Gesture recognition · p. 4, §IV · p. 4

A. Benchmark: Gesture recognition

B. Benchmark: Code generation

The code-generation benchmark compares generated state machines with manually designed baselines across behavior categories and prompt variations. Most trials produced sufficiently similar behaviors, though shape creation was more demanding under less detailed prompts.

B. Benchmark: Code generation

  • The benchmark covered 15 baseline state machines across basic motion, shape deformation, shape creation, and state-machine modification.Each baseline had six prompt variations, tested with three external LLMs, yielding 18 trials per baseline and 270 total.Source: B. Benchmark: Code generation · p. 5
  • 91.4% of 270 trials produced behaviors sufficiently similar to their baselines; shape creation succeeded in 68.75% of medium- and low-detail prompt trials.Table II reports success percentages by experimental variation and identifies the three example LLMs.Source: Table II · p. 5, B. Benchmark: Code generation · p. 5
  • Generated and baseline state machines were compared in simulation using motion-tracking error and formation error over target robot positions.Motion-tracking error compares geometric centroids over time; formation error compares sorted pairwise inter-robot distances.Source: B. Benchmark: Code generation · p. 5
  • The benchmark treats generations as successful when their multi-robot behaviors are sufficiently visually similar to those from baseline state machines.Shape creation is identified as more demanding, likely because it involves more degrees of freedom.Source: B. Benchmark: Code generation · p. 5
  • The authors use the imperfect trial success rate to motivate personalization before live operation, with an in-loop simulator and iterative queries for operator feedback.These measures are intended to let operators refine generated behaviors until they match what is desired.Source: B. Benchmark: Code generation · p. 5

C. Real human teleoperation of 50 simulated robots

Real operators used personalized interfaces to teleoperate 50 simulated robots in three tasks, with trials run under nominal and noisy-motion conditions. Operators created distinct interfaces by demonstrating gestures and describing desired behaviors in natural language.

C. Real human teleoperation of 50 simulated robots

  • Three tasks tested teleoperation of 50 differential-drive robots: scoring a goal, traversing a narrow-gap maze, and scoring two goals simultaneously.The robots were modeled after e-puck robots and simulated in ARGoS.Source: C. Real human teleoperation of 50… · p. 5
  • Each task ran with and without injected motion noise, and three operators completed each task-condition combination for 18 trials total.The noise parameter was e = 1.5.Source: C. Real human teleoperation of 50… · p. 5
  • Operators demonstrated different gestures and described different desired meanings, producing three personalized interfaces for completing the same tasks in different ways.The task set demonstrates control of formation shape, motion trajectory, shape deformation, and splitting the swarm into two groups.Source: C. Real human teleoperation of 50… · p. 5

1) Scoring a goal (Task-1):

2) Traversing a maze with narrow gaps (Task-2):

All 18 trials were completed, but wheel-speed noise affected the tasks differently: performance stayed similar or improved slightly in Task-2 and worsened in Tasks-1 and -3. Operators navigated a maze by deforming the swarm, split it to score two goals, and recovered a ball that escaped during a noisy trial.

2) Traversing a maze with narrow gaps (Task-2):

4) Results of all trials (Tasks-1, -2, and -3):

  • In the noisy Task-1 example, the ball slipped through the formation, prompting the operator to turn back, retrieve it, and resume the task with gestures.The robots rotated and moved forward in response to swipe-right and swipe-up gestures during recovery.Source: Figure 3 · p. 6, Figure 2 · p. 6

3) Scoring two goals at the same time (Task-3):

4) Results of all trials (Tasks-1, -2, and -3):

D. Demo: Re-triggering personalization during operation

During live operation, an operator re-triggered personalization to add two deformation behaviors and correct a gesture mapping; the updates took 62 s and 25 s. Separate demonstrations teleoperated real e-puck robots in square and circular formations.

D. Demo: Re-triggering personalization during operation

  • Two operators produced similar square-to-elongated-rectangle deformations using different gestures: swipe right and swipe outwards.Source: Figure 5 · p. 7
  • During operation, the operator added two deformation behaviors, corrected a gesture mapping that produced the same behavior twice, and then completed the task.The first update created one state machine and three nodes in 62 s; the correction modified one line of code in 25 s.Source: D. Demo: Re-triggering personalization… · p. 7

E. Demo: Teleoperation of real robots

  • Real-robot demonstrations teleoperated four e-puck robots in an approximately square formation and six in an approximately circular formation that changed overall size.A base-station computer sent motor commands wirelessly to the robots.Source: E. Demo: Teleoperation of real robots · p. 7

V. DISCUSSION AND FUTURE WORK

Auto-HSI currently depends on external motion tracking and a central base station, while the proposed next steps target more self-organized coordination, onboard operation, and broader real-world use. Field transfer would also require further development of gesture and audio processing and feedback about the swarm’s state and environment.

V. DISCUSSION AND FUTURE WORK

  • Robot coordination currently relies on external motion tracking and a central base station; future work targets self-organized dissemination of global targets while preserving local behaviors such as collision avoidance.The Self-organizing Nervous System (SoNS) architecture is identified as one candidate for this integration.Source: §V · p. 7
  • Real-robot demonstrations include navigating an arena and deforming a shape.Source: Figure 6 · p. 8, Figure 7 · p. 8
  • The authors identify SoNS integration as feasible and propose exploring natural-language descriptions of both individual-robot and swarmwide behaviors.They note that SoNS provides a single behavior-programming access point and that off-board LLM communication from a SoNS-brain has been demonstrated.Source: §V · p. 8
  • The current system runs on a base station computer; future onboard operation would require capable robot platforms, while onboard LLMs remain conditional on sufficient model and processing advances.The proposed onboard-compute path still assumes communication with an off-board LLM.Source: §V · p. 8
  • Transfer to untrained operators and more open-ended real-world scenarios requires further development of gesture tracking, audio processing, emergency intervention, and actuation beyond formation control.The study’s bird’s-eye operator view also makes user feedback about swarm state, performance, and environment a needed development area.Source: §V · p. 8
Loading 2609.16346v1…