Source-linked AI summary
Freehand Sketching for End-User Programming of Robot Swarms
Riwa Karam, Ian Kuo, Magnus Egerstedt
TL;DR
The paper addresses how non-experts can specify robot-swarm formations without conventional programming by introducing freehand sketching as a sketch-to-swarm interface. In a 20-participant study, the interface achieved a mean SUS score of 84.25, supporting freehand sketching as a usable interaction abstraction for human-swarm collaboration.
Problem
The paper addresses the need for non-expert users to specify collective robot-swarm behaviors and geometries without conventional programming.
Method
The paper integrates freehand spatial specification with autonomous sketch processing, robot assignment, and distributed formation control in an interactive sketch-to-swarm system.
Results
The interface achieved a mean System Usability Scale score of 84.25 among 20 participants, indicating high perceived usability.
Takeaways & Limitations
The study supports freehand sketching as a usable and readily learnable interaction abstraction for accessible human-swarm collaboration without robotics or programming expertise.
Takeaways & Limitations
The evaluation was limited to a primarily university-based participant population and static planar formations.
Abstract
from arXiv · showhide
Robot swarms are increasingly used in applications where accessible interaction with non-expert users is desirable. This paper investigates freehand sketching as an end-user programming interface for specifying robot swarm geometries. Users communicate spatial intent through a drawing, while the swarm autonomously extracts target formation points, constructs a rigid formation graph, assigns robots to formation nodes, and executes distributed formation control with a guarantee against unintended reflected formations. The resulting sketch-to-swarm framework is evaluated through a human study examining the usability of freehand formation specification. Twenty participants generated $42$ geometric shapes, and the interface achieved a mean System Usability Scale score of $84.25$, which conventionally indicates high perceived usability. The results support freehand sketching as an intuitive interaction abstraction for human-swarm collaboration without requiring robotics or programming expertise.
I. INTRODUCTION
The paper frames freehand sketching as an end-user programming abstraction for specifying swarm geometries, addressing the limited evaluation of such interfaces with non-experts. A human study assesses usability and physical realization of participant-generated formations.
- Prior sketch-to-formation work established computational feasibility, but end-user efficacy—especially for people without robotics or control experience—remained insufficiently evaluated.
- The paper integrates human spatial specification with autonomous swarm realization in an interactive sketch-to-swarm system and evaluates it through a human study.
- Freehand sketching lets non-experts communicate formation-level intent without conventional programming or predefined formation shapes.
- 20 participants evaluated the perceived usability and learnability of sketch-based formation specification.
- 42 self-specified participant formations converged to their corresponding formations in Physical Robotarium experiments.
II. SKETCH-TO-SWARM FORMATION FRAMEWORK
The sketch-to-swarm framework converts freehand drawings into rigid formation geometries, assigns those geometries to available robots, and supports distributed realization while accounting for reflection ambiguity.
- II. SKETCH-TO-SWARM FORMATION FRAMEWORK: Freehand input is processed into a geometric formation specification, while graph construction, robot assignment, and robot-level control are handled autonomously.
- A. Sketch Processing and Formation Construction: Laplacian-of-Gaussian edges and Shi-Tomasi features identify sketch locations, which are scaled to the workspace and filtered by minimum separation dmin.
- A. Sketch Processing and Formation Construction: The processed sketch becomes n target points qi in R2, with n constrained by the number of available robots N.
- A. Sketch Processing and Formation Construction: For the letter “a” example, dmin = 0.35m filtering produced 7 target points defining the desired formation geometry.
- A. Sketch Processing and Formation Construction: A formation graph connects target points through desired inter-agent distances and is constructed to be rigid, fixing geometry up to rigid transformations.
- A. Sketch Processing and Formation Construction: Distance constraints preserve shape while permitting global translation and rotation, but can also admit mirrored configurations.
B. Robot-to-Formation Assignment
The framework formulates robot placement as a binary one-to-one assignment that minimizes total distance between physical robots and extracted formation nodes.
- B. Robot-to-Formation Assignment: The extracted formation nodes define geometry but do not determine which physical robot occupies each node.
- B. Robot-to-Formation Assignment: Assignment cost is based on the Euclidean distance between each robot and target point, represented in a cost matrix C.
- B. Robot-to-Formation Assignment: Binary variables xij indicate whether robot j is assigned to formation node i.
- B. Robot-to-Formation Assignment: The binary linear assignment problem assigns exactly one robot to each node and each robot to exactly one node, then uses the Hungarian algorithm to minimize total robot-to-target distance.
C. Reflection-Free Distributed Formation Execution
The framework combines assignment and signed-orientation checks with distributed distance control to preserve the sketch’s intended geometry and orientation while permitting global translation and rotation.
- C. Reflection-Free Distributed Formation Execution: The distributed controller drives neighboring inter-agent distance errors toward zero using relative information, thereby realizing the extracted sketch geometry.Each robot is associated with a formation-graph node after assignment.
- C. Reflection-Free Distributed Formation Execution: Distance-based control preserves formation shape but cannot distinguish translations, rotations, and reflections from pairwise distances alone.Translations and rotations preserve the sketch’s appearance, whereas reflection can alter an asymmetric symbol’s meaning.
- C. Reflection-Free Distributed Formation Execution: Signed orientation angles for three non-collinear formation points provide the condition used to prevent reflected robot configurations.The assigned robots’ orientation-angle signs are compared with those of the target formation.
- C. Reflection-Free Distributed Formation Execution: The Robotarium example realizes the seven-node letter “a” formation with its geometry and orientation preserved under global translation and rotation.Robots R1-R7 occupy target positions obtained from filtered sketch points.
- C. Reflection-Free Distributed Formation Execution: The reflection-free assignment preserves the target orientation and excludes unintended reflected realizations while allowing global translations and rotations.This resolves the ambiguity caused by distance constraints admitting mirrored configurations.
III. USER STUDY
The paper evaluates freehand sketching as an end-user programming interface through a human study conducted with a physical Robotarium system.
- III. USER STUDY: The user study evaluates the usability of freehand sketching as an end-user programming interface for robot-swarm formation specification.It examines interaction with the complete sketch-to-swarm pipeline on a physical multi-robot system.
- III. USER STUDY: Participants used the study interface to draw formations and observe the physical robot team realize the processed result.The setup used a stylus tablet, GUI, and Robotarium testbed.
A. Participants and Study Procedure
Twenty participants completed an open-ended study in which they learned basic controls, selected their own formations, observed physical executions, and completed the SUS questionnaire.
- A. Participants and Study Procedure: Twenty participants completed the study, including 12 undergraduates, 6 graduate students, and 2 postdoctoral researchers, without requiring prior robotics experience.Participants were recruited from the UCI community.
- A. Participants and Study Procedure: Participants received instructions only for drawing, clearing, and submitting sketches, with no example or prescribed formation.The controls were New Line, Clear, and Done.
- A. Participants and Study Procedure: Participants freely chose formation shapes and could submit additional drawings after completing at least one sketch-to-swarm execution.This protocol evaluated self-selected spatial specifications.
- A. Participants and Study Procedure: Each drawing passed through the full pipeline before participants observed the physical robot realization and completed the System Usability Scale questionnaire.Experimental sessions included both system interaction and questionnaire completion.
B. Experiment Setup & Graphical User Interface
The experiment used a physical Robotarium testbed with a Bluetooth Wacom tablet and a GUI displayed on a television, while deployment was limited by the available robots.
- B. Experiment Setup & Graphical User Interface: Participants drew on a Bluetooth-connected Wacom tablet while viewing the GUI on a television positioned in front of the physical Robotarium testbed.This arrangement linked drawing, interface feedback, and physical swarm execution in one setup.
- B. Experiment Setup & Graphical User Interface: The system operated with up to 20 mobile robots, and sketch processing filtered extracted critical points so formation nodes did not exceed available robots.The filtered target configuration was then assigned to robots and realized by distributed formation control.
C. System Usability Scale
The System Usability Scale (SUS) provides a standardized ten-item assessment of perceived system usability, scored from adjusted Likert responses on a 0–100 scale.
- C. System Usability Scale: SUS is a standardized ten-item questionnaire assessing perceived usability across ease of use, complexity, learnability, integration, and confidence.Items use five-point Likert responses and alternate between positively and negatively worded statements.
- C. System Usability Scale: SUS scores are computed by adjusting odd- and even-numbered item responses, summing the contributions, and multiplying by 2.5.The resulting score ranges from 0 to 100 for each participant.
- C. System Usability Scale: A SUS score of approximately 68 is commonly used as a reference for average usability, and SUS scores are not percentages.The score is intended for comparison against established empirical benchmarks.
IV. RESULTS
The sketch-to-swarm interface received favorable usability assessments, while all submitted formations met the convergence criterion. Results also reveal variability across users and a study scope centered on open-ended interaction with a predominantly technical participant pool.
- A. User Study Results and Analysis: 84.25 mean SUS (SD = 15.24) exceeded the reference value of 68, with 17 of 20 participants scoring above 68.Sixteen participants scored at least 80, and scores ranged from 35 to 100.
- A. User Study Results and Analysis: Participants rated learnability, confidence, integration, and ease of use favorably, while generally rejecting descriptions of the system as cumbersome, inconsistent, or unnecessarily complex.The strongest item was rapid learnability at 4.55/5; confidence and integrated functions each received 4.40/5.
- A. User Study Results and Analysis: Participants submitted 42 self-selected drawings, averaging 2.10 ± 1.02 formations each, and 65% completed more than the required single execution.Because extra executions were optional, these counts are descriptive rather than a formal engagement measure.
- A. User Study Results and Analysis: All 42 participant-generated formations satisfied the study’s convergence criterion.The criterion required every prescribed edge to meet the absolute inter-agent distance error threshold.
- A. User Study Results and Analysis: The aggregate usability result was not uniform: technical-assistance responses varied most, and the interface was not equally intuitive for all participants.The authors suggest refining interaction cues and introductory guidance.
- A. User Study Results and Analysis: The study evaluates open-ended usability rather than comparative task performance, and its primarily university-based engineering-oriented participants do not represent the broader non-technical population.Participants selected their own drawings and interaction counts instead of completing standardized formation tasks.
B. Physical Robotarium Formation Results
Physical Robotarium trials translated visually distinct freehand sketches into converged multi-robot formations while preserving intended overall geometry and orientation. The system handled formation-point complexity and robot availability after submission rather than requiring users to manage those constraints.
- B. Physical Robotarium Formation Results: Visually distinct freehand inputs were translated into physical multi-robot configurations while preserving intended geometry and orientation.Figure 7 shows representative sketch/formation pairs from the physical user study.
- B. Physical Robotarium Formation Results: The system absorbed constraints from sketch geometry and available robot counts, leaving users to specify shape while the swarm determined its physical representation.The resulting formation-point number and complexity depended on the input sketch and robots available for execution.
V. CONCLUSION
The study supports freehand sketching as an accessible end-user programming abstraction for human-swarm formation specification. Its evaluation was limited to primarily university-based participants and static planar formations.
- Freehand sketching shows potential as an accessible human-swarm formation specification interface without requiring robotics or programming expertise.