Reset-free Trial-and-Error Learning for Robot Damage Recovery

• Hardware failures can require human rescue, while diagnosis grows more challenging with robot and environmental complexity and may demand costly internal sensors.The paper cites Atlas's reported mean time between failures as hours or, at most, days.Source: §1 · pp. 1–2
• Robotic RL struggles with continuous control and scaling, while reset-dependent methods can prevent mobile robots from learning safely in real environments.A wrong move near an obstacle can further damage a robot, motivating environment-aware action selection.Source: §1 · p. 2
• IT&E searches preselected high-performing intact-robot policies after damage, whereas full-dynamics probabilistic planning is too computationally demanding for typical onboard resources.The paper also draws an analogy to AlphaGo's use of precomputed policies and Monte Carlo Tree Search.Source: §1 · pp. 2–3
• RTE significantly outperforms modified TEXPLORE on the differential-drive task; a physical hexapod recovered most locomotion abilities and completed tasks without human intervention.The introduction also reports that TEXPLORE is not applicable to the hexapod task because of its action-space dimensionality.Source: §1 · p. 3
• RTE adapts IT&E's offline behavior precomputation for Monte Carlo Tree Search, adding a probabilistic model to correct action outcomes and account for prediction uncertainty.The approach exploits intact-robot simulation and is intended to support complex robots, environment-aware learning, and less than one minute between episodes.Source: §1 · pp. 2–3
• The paper frames damage recovery as model-based reinforcement learning and evaluates the approach on damaged differential-drive and hexapod robots in simulation and on a physical hexapod.The target scenario assumes unknown damage without a usable diagnosis and a known environment.Source: §1 · pp. 2–3

2.1. Learning in Robotics

• Most policy-search methods require robot or environment resets, while non-episodic approaches can still need hundreds of samples even for simple reaching tasks.TEXPLORE combines model learning with MCTS, but its deterministic models, discrete actions, declining performance in higher-dimensional action spaces, and data-hungry full-dynamics learning constrain complex-robot use.Source: §2.1 · p. 3

2.2. Fault Tolerance and Recovery in Robotics

• Trial-and-error damage recovery avoids diagnosis, and IT&E uses intact-robot simulations as a prior for searching compensatory behaviors, with reported recovery in under two minutes.Transferability assumes behaviors that avoid damaged parts remain similar; the cited experiments cover a hexapod and an 8-DOF arm.Source: §2.2 · p. 4

2.3. Probabilistic and Sample-Based Planning

• MCTS builds a search tree from random decision-space samples, evaluating states by average rollout outcomes to support sequential decision planning.RRT and PRM are other sample-based planning approaches described in this section.Source: §2.3 · pp. 4–5

• The robot seeks a deterministic state-to-action policy that maximizes expected discounted return in an MDP with state space, action space, transition function, and reward.The transition function is learned, while the reward function is assumed known to the robot.Source: §3 · p. 5
• The mission consists of related tasks G1 through Gn, each formulated as a shortest-path problem.Completing one goal begins execution of the next task.Source: §3 · p. 5
• Reaching a task goal earns Rgoal, a non-goal terminal state earns −Rterm, and other states earn zero.A colliding state is an example of a non-goal terminal state.Source: §3 · p. 5

4.2. Learning the Action Repertoire

• Before deployment, MAP-Elites uses intact-robot simulations to build a repertoire mapping lower-dimensional task-space actions to controller parameters.Its iterative variation-selection process fills task-space cells with behaviors, retaining a better-performing controller or adding one to an empty cell.Source: §4.1 · p. 5, §4.2 · pp. 5–6

4.1. Overview

• RTE replans each episode with MCTS, accounting for model uncertainty, the environment, the robot’s current state, and the target before executing the next action.The loop updates Gaussian processes after each episode and repeats until the task is completed.Source: §4.1 · p. 5

4.3. Learning with Gaussian Processes

• Gaussian processes use the simulator-derived MAP-Elites mapping as a prior and learn corrections to action outcomes on the damaged robot.The mapping is only an approximation because it comes from an intact-robot simulator.Source: §4.3 · p. 6

4.4. Probabilistic Optimal Planning using MCTS

• MCTS uses the probabilistic model for sample-based rollouts and returns its best-performing root action when the computational budget ends.The search iterates through selection, expansion, rollout, and backup; GP predictions provide both outcomes and uncertainty.Source: §4.4 · pp. 6–7

4.5. Reset-free Trial-and-Error Learning Algorithm

• RTE’s mission loop selects an action from the repertoire, executes it, records the outcome, and updates the Gaussian processes before replanning.The algorithm first creates the repertoire and corresponding GP priors, then repeats these steps until stopping criteria are met.Source: §4.5 · p. 7

• Experiments use a simulated differential-drive robot and simulated and physical hexapods to reach 30 target waypoints in an obstacle arena.The damage is unknown to the operator; the setting assumes a known environment and robot position from motion capture.Source: §5 · p. 8
• The comparison includes RTE, MCTS without learning using the original repertoire, and GP-TEXPLORE, which learns action outcomes and plans in controller space.GP-TEXPLORE assumes a known reward and adapts MCTS to continuous actions; it learns relative action outcomes rather than full transition dynamics.Source: §5 · p. 8

6.1. Learning the Action Repertoire

• MAP-Elites selects actions reaching different Cartesian positions while favoring continuous-curvature trajectories that end with the robot facing the trajectory tangent.The implementation represents the task with a 2-D position descriptor and produced 331 actions after 100,000 evaluations.Source: §6.1 · pp. 8–9

6.2. Learning with Gaussian Processes

• Gaussian-process inputs are 2-D action descriptors, and outputs predict relative x, y, and orientation displacements; orientation is represented with cosθ and sinθ to avoid angle discontinuities.Source: §6.2 · p. 9

6.3. Solving the problem with MCTS

• MCTS handles thousands of actions, continuous states, and uncertain transitions using progressive widening, while the planner selects actions using predictions and model uncertainty.The selection and expansion policies use simple and double progressive widening, respectively.Source: §6.3 · p. 9

6.4. Guiding MCTS using A* on a simplified problem

• A* plans on a discretized grid for a virtual point robot, then guides selection among 100 repertoire actions using GP-predicted outcomes, reducing action-selection time to under 40–50 seconds.The mobile-robot arena is 800×800 units, with robot and obstacle radii of 20 units.Source: §6.4 · pp. 9–10, Figure 5 · p. 10

6.5. Experimental results

• 22.28 episodes per target for RTE, versus 32.12 for the re-planning baseline and 26.03 for GP-TEXPLORE.The damaged robot's right-wheel velocity command was halved; the scenario used 30 sequential targets and was replicated 50 times.Source: Figure 6 · p. 10, §6.5 · p. 10
• RTE and GP-TEXPLORE produce faster, safer paths than MCTS; RTE's paths are slightly safer than GP-TEXPLORE's, while MCTS often gets stuck at arena walls.Source: §6.5 · p. 10
• Around 63% of original locomotion capabilities were recovered with RTE, versus around 54% with GP-TEXPLORE and 44% with repertoire-only MCTS planning.The authors report that the repertoire alone is insufficient and that combining prior repertoire information with learning is beneficial relative to learning from scratch.Source: §6.5 · p. 10, Table 1 · p. 11
• RTE and GP-TEXPLORE reduce episodes per target over time, while MCTS alone remains constant; after the first target, RTE corrects its repertoire and outperforms GP-TEXPLORE.GP-TEXPLORE plans in the full action space.Source: §6.5 · p. 10, Figure 7 · p. 11

7.1. Parametric Low-level Controller

• Each hexapod leg has three degrees of freedom, while the path-planning state uses position and yaw; the gait controller represents actions with 36 real-valued parameters.Periodic joint signals are parameterized by amplitude, phase, and duty cycle; MAP-Elites produced repertoires mapping a 2-D task-space descriptor to the 36-D controller space.Source: §7 · p. 11, §7.1 · p. 11, Figure 9 · p. 12

7.2. Simulation results

• The simulation evaluated 30 sequential targets across three damage types, two environments, and two repertoires, with 50 repetitions per scenario.The targets were random, equidistant, and 3.5 m apart.Source: §7.2 · p. 12
• RTE recovered more than 60% of original locomotion capabilities in almost all simulated scenarios, while repertoire-only MCTS recovered less than 60% in half.The reported pattern held in both flat- and rough-terrain evaluations.Source: §7.2 · p. 12, Table 2 · p. 13, Table 3 · p. 13
• Across simulated damage scenarios, RTE reached targets in rough terrain with repertoires learned on flat terrain and required fewer episodes than replanning.GP-TEXPLORE did not solve the task within around 600 episodes; in one comparison, RTE reached a target in about 10 episodes, versus more than 20 for MCTS.Source: §7.2 · p. 12, Figure 10 · p. 13, Figure 11 · p. 14

7.3. Physical robot results

• In the reported results, RTE needed 13.0 episodes in Environment 1 and 18.0 in Environment 2 to reach each target, versus 28.0 and 25.0 episodes for MCTS alone.The reported 25th and 75th percentiles were [12.0, 14.0] and [14.0, 19.0] for RTE, and [24.0, 31.0] and [24.0, 43.0] for MCTS alone, in Environments 1 and 2, respectively.Source: §7.3 · p. 14

8. Discussion and Conclusion

• RTE was tested on damaged mobile and hexapod robots and is presented as enabling autonomous learning without resets while accounting for obstacles.The authors describe physical legged-robot learning without human intervention, including in environments with obstacles.Source: §8 · p. 15
• RTE selects among actions found offline with a different model, so it cannot use potentially better actions elsewhere in the full controller space.The authors report that this approximation sufficed for their experiments, where the robot completed its tasks.Source: §8 · p. 15
• The current Gaussian Process prediction cost grows cubically with sample count, limiting use to a few hundred episodes.Suggested options include time windows, sparse or local GPs, or neural networks with uncertain predictions.Source: §8 · p. 15
• The experiments assumed perfect knowledge of robot position and environment; the authors identify SLAM-based mapping as a next step.They note that treating uncertain maps within an MDP may be insufficient, while a POMDP approach would increase MCTS computation time.Source: §8 · p. 15
• RTE assumes action outcomes are state-independent, although the hexapod handled rough terrain with a repertoire optimized for flat terrain.The authors propose investigating state-dependent priors and learning the full transition model or reward function.Source: §8 · p. 15

Appendix

• A target counted as reached within 20 units for the mobile robot, 0.25 m for simulated hexapods, and 0.2 m for the physical hexapod.The mobile-robot threshold equals the robot radius.Source: Appendix · p. 16