Swarm-GPT: Combining Large Language Models with Safe Motion Planning for Robot Choreography Design

1 Introduction

• Choreographic design and safety analysis can quickly become intractable as the number of robots increases.Current applications often rely on expert choreography to balance expressivity and safety.Source: §1 · p. 1
• Swarm-GPT pairs an LLM motion-generation layer with model-based swarm planning to produce feasible, safe drone choreography from language instructions.The planning layer is intended to ensure safe deployment of the LLM-generated choreographies.Source: §1 · p. 1
• Simulation and physical experiments demonstrate LLM-directed drone swarm control as a proof of concept for safe coordination.The system is presented as an intuitive interface for non-expert users to generate complex drone behaviours.Source: §1 · p. 2
• The authors report enabling natural-language choreography modification, integrating a model-based safety filter, and demonstrating real-time drone experiments.They identify the system as, to their knowledge, the first direct use of LLMs for drone choreography.Source: §1 · p. 2

Robot Choreography

• Music-feature-based choreography has been explored, but often still requires manual tuning and domain expertise, with minimal intuitive feedback for non-experts.This work focuses on synchronized spatiotemporal motion generation for drone swarms with natural-language feedback.Source: §2 · p. 2

LLMs for Robotics

• Natural-language robot decision-making has been explored in navigation, affordances, trajectory modification, and high-level task planning; this work applies it to safe real-world drone choreography.Source: §2 · p. 2

Safe Robot Swarm Decision-Making

• The authors adopt a distributed swarm-planning framework as a safety filter, leveraging its scalability and robot knowledge while seeking to preserve LLM-generated motion.Model-based planning explicitly accounts for safety and feasibility constraints.Source: §2 · p. 2

LLM Interface

• The LLM prompt combines users’ language instructions, extracted song features, and preliminary operating-environment constraints to generate timed, beat-synchronized position waypoints for each drone.Song features are extracted with librosa; the system also accepts high-level task specifications such as song selection and swarm behaviour.Source: §3 · pp. 2–3

Safe Trajectory Optimization

• An AMSwarm-based safety filter incorporates speed, actuation, and collision-envelope constraints to compute feasible, collision-free trajectories that best match the LLM’s waypoints.Source: §3 · p. 3

Trajectory Modification (Re-Prompting)

• Users can visualize the waypoints in a drone simulator, deploy the choreography, or re-prompt to update it; updated outputs undergo collision avoidance and processing.A custom prompt such as “go faster, higher” is appended to the original prompt before the LLM updates the waypoints.Source: §3 · p. 3

Evaluation scope

• Evaluation covered four songs of different genres and swarms of up to nine drones in simulation and real-time experiments.The songs were Perfect, Ode to Joy, Mission Impossible Theme, and Here Comes the Sun.Source: §4 · p. 3

Assessment of Sim-to-Real Transfer

• 28.7 mm was the average sim-to-real RMSE across six choreographies, with the worst-case average not exceeding 60 mm.Simulated and real systems received identical control signals; the simulator used the same controllers as the real platform.Source: §4 · p. 3, Figure 3 · p. 4

Response to Safety Constraints

• 100% of trajectories were collision-free after safety filtering, compared with 21.65% of raw LLM-generated trajectories across 97 simulation trials.Collision envelopes accounted for drone dimensions and aerodynamic effects such as downwash; all data points used nine-drone swarms.Source: §4 · p. 3, Figure 4 · p. 4

Prompting and Synchronization to Music

• Beat-timestamped waypoints produced formation changes around musical beats, and reprompting with “fly faster” modified trajectories.Figure 5 marks beat times with large dots; Figure 6 compares mean drone velocity before and after reprompting across 10 choreographies per song.Source: §4 · p. 4, Figure 5 · p. 4, Figure 6 · p. 4

• Experiments demonstrate the efficacy of combining LLMs with a model-based safety filter for safe real-world deployment.The system integrates high-level language-based choreography with low-level safe drone-swarm motion planning.Source: §5 · p. 4

LLM Interface

• GPT-3.5 generates waypoint time series, and the interface supports song selection, prompt viewing, reprompting, and trajectory simulation.The initial prompt was iteratively tuned using prior techniques and empirical experience with different prompts.Source: §A.1 · p. 7
• Beat timestamps guide formation changes, while prompts ask GPT to interpret song mood and use aesthetic descriptors to shape choreography.The prompt assigns GPT the role of an expert swarm choreographer and asks for unique waypoints for each drone at each beat.Source: §A.1 · p. 7
• The prompt includes flight-volume, velocity, acceleration, and waypoint-separation constraints as rudimentary safety guidance.An AMSwarm safety layer is also used, but the prompt itself asks GPT to handle basic collision avoidance and constraint violations.Source: §A.1 · p. 7
• Initial drone positions and a specified output format make the prompt more generalizable for non-expert use and downstream processing.Source: §A.1 · p. 7
• Waypoint sequences give GPT freedom to create choreography without restricting it to motion primitives or requiring domain-specific robotics knowledge.Source: §A.1 · p. 7
• Responses are parsed as JSON, checked for correct formatting, and converted into waypoint arrays sorted by timestamp.Source: §A.1 · p. 7

A.2 AMSwarm-Based Safety Filter Design

• Raw waypoints are normalized to the permitted flying volume, and simultaneously occurring waypoints that are too close are pulled apart before formal collision avoidance.The separation step prevents the formal algorithm from starting in a collision state, where it would not converge.Source: §A.2 · p. 7
• AMSwarm interpolates preprocessed LLM waypoints while satisfying physical constraints and minimizing inter-drone collisions through polar-form constraints and alternating minimization.The approach retains a quadratic-program structure.Source: §A.2 · p. 7
• The trajectory objective penalizes deviation from LLM-specified goal positions over the final κ steps and penalizes q-th derivatives to promote smoothness.The goal and smoothness terms are weighted by ωg and ωs, respectively.Source: §A.2 · p. 8
• Initialization and feasibility constraints set initial positions and higher derivatives, and bound position, velocity, and acceleration.The optimization is described for each drone at each time step over a planning horizon K.Source: §A.2 · pp. 8–9
• Ellipsoidal collision constraints represent neighboring drones or obstacles, while barrier-function constraints can tune how conservatively drones approach safety boundaries.Smaller γ values generally produce more gradual, conservative collision avoidance.Source: §A.2 · p. 9
• AMSwarm optimization is reapplied for each inter-waypoint time window, and its position commands are resampled for simulation and control frequencies before deployment.The simulator provides on-demand visual confirmation of collision-free motion; AMSwarm was selected for scalability and compatibility with waypoint outputs.Source: §A.2 · p. 9

B Additional Evaluation Results

• Figure 9 shows six drones moving in synchronized formations across two additional beat intervals, demonstrating creative and expressive formations synchronized with music features.The intervals shown are 9.58–11.24 s and 11.24–12.91 s; colors identify drones, large dots mark beats, and color saturation increases over time.Source: B Additional Evaluation Results · p. 9, Figure 9 · p. 10
• The additional formation examples extend the music-synchronization results beyond the beat interval shown in Fig. 5.Figure 9 presents two further intervals.Source: B Additional Evaluation Results · p. 9, Figure 9 · p. 10
• Figure 8 shows close agreement between simulated and actual single-drone trajectories given identical reference inputs, a pattern generally observed across drones and choreographies.The passage relates this behavior to the low RMSE values reported in Fig. 3.Source: B Additional Evaluation Results · p. 9, Figure 8 · p. 10