Fleet-DAgger: Interactive Robot Fleet Learning with Scalable Human Supervision

• Interactive Fleet Learning (IFL) is robot-fleet learning through interactive requests for human supervision, with interventions used during execution and continual policy learning.Fleet robots operate independently in parallel environments under the same control policy, and any robot can share intervention data with the fleet.Source: §1 · p. 1
• The open question is how to allocate limited human supervision to maximize fleet throughput, since prior work studies the single-robot, single-human case.Existing interactive-learning methods estimate uncertainty, novelty, risk, or predicted action discrepancy, but their effectiveness in the multi-robot, multi-human case remains unclear.Source: §1 · p. 1
• The paper introduces a formalism for multi-robot, multi-human interactive learning, the Return on Human Effort metric, and an open-source benchmark with three Isaac Gym environments.The toolkit supports development and evaluation of human-to-robot allocation algorithms.Source: §1 · pp. 1–2
• The paper presents Fleet-DAgger and evaluates it in simulation with 100 robots and on four physical robot arms supervised remotely by two humans.The introduction identifies large-scale simulation and physical-robot results as contributions.Source: §1 · p. 2

2.1 Allocating Human Supervisors to Robots at Execution Time

• Execution-time allocation research spans sliding autonomy, HRI, scheduling, and queueing, but most assumes fixed robot policies rather than learning during allocation.This work studies allocation during learning, where allocation affects both human burden and task performance as well as policy-learning efficiency.Source: §2.1 · p. 2

2.2 Single-Robot, Single-Human Interactive Learning

• Human-gated interactive imitation learning requires supervisors to decide when to take or cede control, while robot-gated methods solicit interventions when specified criteria are met.Human-gated approaches are described as difficult to scale to large fleets and continual learning because operators cannot effectively focus on many robots and are prone to fatigue.Source: §2.2 · p. 2

2.3 Multi-Robot Interactive Learning

• The paper identifies multi-robot, multi-human interactive learning as a setting not previously formalized or studied.As the fleet grows relative to the human pool, allocating a supervisor affects both the robot receiving attention and the robots that do not.Source: §2.3 · p. 3

3 Interactive Fleet Learning Problem Formulation

• The fleet is modeled as parallel MDPs with common state and action spaces, transition and reward functions, and discount factor, but potentially different initial-state distributions.The formulation can also represent the fleet as a single MDP with vectorized states and actions.Source: §3 · p. 3
• Robots share a non-stationary policy and constraint indicator, while constraint-violating states are fault states from which a robot cannot progress.The formulation assumes synchronized timesteps across robots.Source: §3 · p. 3
• Each of the M ≪ N supervisors can assist one robot at a time through teleoperation or a hard reset when a robot violates a constraint.Hard resets take tR timesteps; other actions take one timestep.Source: §3 · p. 3
• An allocation policy uses robot states, the current shared policy, prior assignments, and auxiliary intervention information to determine assistance assignments.Teleoperation supplies online data that can be used to update the robot policy.Source: §3 · pp. 3–4
• ROHE defines the IFL objective as expected fleet reward normalized by the total human time spent helping robots.The denominator includes a safeguard for the zero-human-time case and the metric is scaled for fleet and supervisor counts.Source: §3 · p. 4

4.1 Fleet-DAgger

• Fleet-DAgger uses interactive imitation learning with dataset aggregation, and its novel allocation component assigns robots priority scores based on state and the current policy.A minimum teleoperation duration tT is specified to reduce thrashing.Source: §4.1 · p. 4
• The allocation procedure ranks robots, keeps supervisors on ongoing resets or teleoperation until minimum durations elapse, and assigns available supervisors to the highest-priority unassisted robots.Robots with zero priority receive no assistance, even when a supervisor is available.Source: §4.1 · p. 4

4.2 Fleet-DAgger Algorithms

• The Constraint baseline provides hard resets only, while Random assigns thresholded random priorities at each timestep.Constraint prioritization gives positive priority only to robots requiring a hard reset; without it, those robots remain idle.Source: §4.2 · p. 4
• Fleet-EnsembleDAgger prioritizes ensemble-variance uncertainty, or classifier-output entropy for discrete actions, before constraint-violating robots.It adds constraint prioritization because idle robots negatively affect ROHE and the original method was not designed for constraint violations.Source: §4.2 · p. 5
• Fleet-ThriftyDAgger combines normalized uncertainty and goal-failure probability, then prioritizes constraint violations.Its goal critic is pretrained on an offline dataset of human and robot task execution.Source: §4.2 · p. 5
• C.U.R. prioritizes constraint violations, then uncertainty, then risk, while initially assigning zero priority to constraint violations so attention can go to teleoperation data.Risk is defined as the probability of constraint violation; a safety critic estimates it, while uncertainty reflects areas with little reference behavior to imitate.Source: §4.2 · p. 5

5 Interactive Fleet Learning Benchmark

• The IFLB addresses the lack of a unified benchmark with an open-source Python suite and modular architecture for rapid prototyping and evaluation.It is intended to support reproducibility and standardized evaluation of IFL algorithms.Source: §5 · p. 5

5.1 Environments

• The benchmark supports Humanoid locomotion, Anymal locomotion, and AllegroHand cube manipulation environments built on Isaac Gym.Constraint violations are defined by falling for the legged robots and dropping the cube for AllegroHand; users can add custom Isaac Gym environments.Source: §5.1 · p. 5, Figure 2 · p. 6

5.2 Software Architecture

• The software architecture separates IFL algorithms into agent, supervisor, and allocation interfaces that users can implement or customize.The benchmark includes an imitation-learning agent, a trained RL supervisor, and allocations used in the experiments; it can run Fleet-DAgger with configurable experiment parameters.Source: §5.2 · pp. 5–6

6.1 Metrics

• The experiments track cumulative task successes, hard resets, idle time, and ROHE, with success defined differently for locomotion and goal-conditioned tasks.For Humanoid and Anymal, success requires reaching the episode horizon without constraint violation and achieving at least 95% of supervisor-policy reward; goal-conditioned success means reaching the goal state.Source: §6.1 · p. 6

6.2 IFLB Simulation Experiments

• Simulation compares Fleet-DAgger algorithms in Humanoid, Anymal, and AllegroHand with 100 robots and 10 humans over 10,000 timesteps, averaging three seeds.PPO-trained agents supervise, while the initial robot policy is behavior-cloned from 5,000 state-action pairs.Source: §6.2 · p. 6
• C.U.R. achieves higher ROHE and cumulative successes than baselines across all simulated environments; AllegroHand is an exception for hard resets and has a smaller ROHE margin.C-prioritization eliminates idle time, while idle time without it continues after the initial period.Source: §6.2 · pp. 6–7, Figure 3 · p. 7

6.3 Physical Block-Pushing Experiment

• The physical experiment uses four ABB YuMi arms and two humans for image-based block pushing, with physical hard resets and remote keyboard teleoperation.Each robot has 250 timesteps per trial; algorithms use an offline dataset of 3,750 image-action pairs and three random seeds.Source: §6.3 · pp. 7–8, Figure 4 · p. 7
• In the physical block-pushing task, C.U.R. achieves higher ROHE and successes, and lower hard resets and idle time than baselines, by a small margin.The authors suggest image-space safety-critic difficulty and multimodal human actions may explain the smaller gains.Source: Figure 5 · p. 8, §6.3 · p. 8

Limitations

• The formulation assumes homogeneous supervisors, identical robot state and action spaces, independent robots, perfect human situational awareness, no switching latency, constant reset time, and synchronous timesteps.It also assumes no network latency or other communication issues.Source: §7 · p. 8
• The evaluation uses algorithmic rather than human supervision in simulation, while the physical task is relatively straightforward and uses discrete planar actions.Source: §7 · p. 8

Future Work

• Future work proposes learning from nonhomogeneous supervisors, collecting large-fleet data under limited network bandwidth, studying RL algorithms for IFL, and running larger physical experiments.The authors also invite researchers to develop and evaluate IFL algorithms with the benchmark toolkit.Source: §7 · p. 8

8.1 Mathematical Details of the IFL Problem Formulation

• The trajectory distribution depends on the allocation policy and initial robot policy because allocation shapes collected data that updates subsequent robot policies.Source: §8.1 · p. 14
• The joint hybrid policy uses the allocation policy and current human and robot policies to determine each robot’s action.An indicator selects between the robot policy and human policy for each robot.Source: §8.1 · p. 14
• Soft and hard resets can be incorporated into transition dynamics; finite-horizon tasks augment the state with time information to reset at timeout.Source: §8.1 · p. 14

8.2 Fleet-DAgger Algorithm Details

• The paper provides Fleet-DAgger’s detailed algorithmic description in Algorithm 1.Source: §8.2 · p. 14
• Fleet-DAgger uses priority function p̂ and minimum intervention time tT to define an allocation policy ω.The paper characterizes it as a meta-algorithm that outputs an algorithm.Source: §8.2 · p. 14

8.3.1 Simulation Hyperparameters

• Simulation evaluations use 100 robots, 10 humans, minimum intervention and hard-reset times of 5 timesteps, and 10,000 operation timesteps.Table 1 includes state/action dimensions and risk and uncertainty thresholds.Source: §8.3.1 · p. 14, Table 1 · p. 15

8.3.2 Training Critic Q-Functions

• Safety critics are pretrained on behavior-cloning data with constraint violations, while AllegroHand’s goal critic uses expert-policy successes; both continue updating online.The safety-critic datasets contain 376, 63, and 47 violations for Humanoid, Anymal, and AllegroHand, respectively; AllegroHand’s goal-critic dataset contains 489 successes.Source: §8.3.2 · p. 15

8.3.4 Evaluating Trained Simulation Policies

• C.U.R.-learned policies outperform baselines and approach expert-level performance in Humanoid and Anymal, but remain second-best and below expert level in AllegroHand.The authors conclude that 10,000 training timesteps are insufficient for a highly performant AllegroHand policy.Source: Table 2 · p. 16, §8.3.4 · p. 16

8.3.3 Physical Experiment Protocol

• The physical protocol runs three trials of four algorithms on four arms for 250 timesteps per trial, totaling 12,000 pushing actions.Two ABB YuMi robots operate in different locations, with all four arms executing concurrently; image data receive 10× augmentation.Source: §8.3.3 · p. 16

Number of Humans

• With 100 robots, 10 humans achieve the highest ROHE despite fewer cumulative successes; in Humanoid and Anymal, success gains shrink between 25 and 50 humans.With one human, hard resets and idle time are high because robots fail faster than the human can reset them; in AllegroHand, M ≥5 makes little visible difference.Source: §8.4 · p. 17

Minimum Intervention Time

• Both shorter and longer minimum teleoperation times than 5 timesteps reduce ROHE, through premature handoff or superfluous intervention length, respectively.Long interventions also raise idle time and can reduce successes; their labels may increasingly resemble offline data from states visited under the human policy.Source: §8.4 · p. 17

Hard Reset Time

• As hard-reset time increases, ROHE and cumulative successes decrease, while hard resets decrease and idle time increases.The study varies tR across 1, 5, 20, 50, 100, and 500 timesteps.Source: §8.4 · p. 17

Other Parameters

• Batch size and gradient steps per timestep significantly affect policy-learning speed and computation time, with effects varying by fleet size and number of humans.The authors identify backpropagation updates as the simulation’s computational bottleneck.Source: §8.4 · p. 17

Ablations

• C.U.R. beats its ablations in ROHE and cumulative successes across environments, except for low-timestep AllegroHand ROHE, and is competitive on hard resets and idle time.In AllegroHand, C.U.R. and C. incur more hard resets than alternatives; U-only prioritization’s early ROHE is erratic because it requests very little human time while achieving few successes.Source: §8.4 · p. 17, Figure 6 · p. 18