Source-linked AI summary
Robots that can adapt like animals
Antoine Cully, Jeff Clune, Danesh Tarapore, Jean-Baptiste Mouret
TL;DR
The paper addresses how to search for effective robot behaviors after damage when post-damage solutions may fall outside the existing behavior map. It builds a behavior-performance map with MAP-Elites and uses that map to guide map-based Bayesian optimization, achieving rapid target-reaching across most tested robotic-arm damage scenarios while exposing a descriptor-related boundary.
Problem
Post-damage solutions can lie outside the existing behavior map, limiting adaptation when the behavioral descriptor omits important dimensions of variation.
Method
MAP-Elites builds a map retaining the highest-performing solution at each location in a user-defined behavioral space, which M-BOA uses as a prior by modeling map prediction errors on the real robot.
Results
Across robotic-arm damage scenarios, the algorithm reached within 5 cm of the bin center in all runs except two, with median target-reaching times below 11 iterations in the other scenarios.
Takeaways & Limitations
With only final end-effector position as the behavioral descriptor, the approach handled varied target positions and was significantly faster than traditional Bayesian optimization.
Takeaways & Limitations
Adaptation fails when the map contains no behavior able to cope with the damage, a limitation attributed mainly to the simple behavioral descriptor.
Abstract
from arXiv · showhide
As robots leave the controlled environments of factories to autonomously function in more complex, natural environments, they will have to respond to the inevitable fact that they will become damaged. However, while animals can quickly adapt to a wide variety of injuries, current robots cannot "think outside the box" to find a compensatory behavior when damaged: they are limited to their pre-specified self-sensing abilities, can diagnose only anticipated failure modes, and require a pre-programmed contingency plan for every type of potential damage, an impracticality for complex robots. Here we introduce an intelligent trial and error algorithm that allows robots to adapt to damage in less than two minutes, without requiring self-diagnosis or pre-specified contingency plans. Before deployment, a robot exploits a novel algorithm to create a detailed map of the space of high-performing behaviors: This map represents the robot's intuitions about what behaviors it can perform and their value. If the robot is damaged, it uses these intuitions to guide a trial-and-error learning algorithm that conducts intelligent experiments to rapidly discover a compensatory behavior that works in spite of the damage. Experiments reveal successful adaptations for a legged robot injured in five different ways, including damaged, broken, and missing legs, and for a robotic arm with joints broken in 14 different ways. This new technique will enable more robust, effective, autonomous robots, and suggests principles that animals may use to adapt to injury.
1 Methods
The method first builds a behavior-performance map with MAP-Elites, then uses map-based Bayesian optimization to adapt a damaged robot through targeted physical trials.
- Behavior-performance map creation: MAP-Elites stores the highest-performing controller found for each location in a user-defined behavior space.It generates candidates, evaluates their performance and behavior-space location, and replaces each location’s occupant only when the new candidate performs better.
- Behavior-performance map creation: The map is improved by repeatedly selecting, mutating, and evaluating stored solutions, retaining mutations that outperform the current occupant at their behavior-space location.The experiments stopped each MAP-Elites run after 40 million iterations.
- Adaptation step: The algorithm’s adaptation loop continues until a solution exceeds the performance threshold defined for the current search.The threshold is represented as stopping when a solution is above the performance threshold.
- Adaptation step: Bayesian optimization selects behaviors by balancing expected performance against predictive uncertainty, then updates predictions after each physical trial.Gaussian processes provide both a mean prediction and uncertainty for candidate behaviors.
- Adaptation step: M-BOA uses the MAP-Elites output as a prior and models the difference between simulated predictions and actual robot performance.A Gaussian process corrects the map’s predictions using physical observations rather than modeling the objective from scratch.
Initial Map
The robot first builds a behavior-performance map in simulation, then uses map-based Bayesian optimization to test promising behaviors on the physical robot. The procedure balances predicted performance and uncertainty while updating predictions after each test.
- Adaptation: The adaptation step uses the map as a prior and models differences between simulated predictions and real-robot performance.M-BOA uses the MAP-Elites output rather than modeling the real-robot objective from scratch.
- Initial Map: MAP-Elites creates a behavior-performance map by simulating controllers and retaining the highest-performing controller for each behavioral descriptor.The map is generated through repeated controller variation and simulation, with performance and controllers stored together.
- Adaptation: The next physical test maximizes an acquisition function combining expected performance and uncertainty.The UCB formulation makes the exploration–exploitation trade-off explicit through the user-defined parameter κ.
- Hexapod Experiment: The hexapod gait is controlled by 36 parameters and represented in a 6-dimensional space defined by the duty factor of each leg.The physical platform has six legs and three degrees of freedom per leg, with an onboard RGB-D camera estimating forward displacement.
- Robotic Arm Experiment: The robotic-arm experiment uses an 8-joint planar arm, with end-effector position as behavioral dimensions and joint-angle variance as the simulated performance measure.The control objective minimizes variance so that the joints contribute more equally to movement.
2 Supplementary Experiments S1
Across 14 robotic-arm damage scenarios, Intelligent Trial and Error usually found accurate target-reaching behaviors faster and more reliably than traditional Bayesian optimization, while two scenarios exposed a map-coverage limitation.
- Experimental setup: 210 physical-robot runs evaluated 14 damage scenarios with 15 independently generated behavior-performance maps, alongside 210 Bayesian-optimization control runs.Each experiment used 30 evaluations.
- Behavior-performance maps: 11,209 [1,1206; 1,1217] behaviors were contained in each of the 15 generated maps after 20 million MAP-Elites evaluations.The maps formed concentric cardioid-like performance regions covering reachable end-effector positions.
- Adaptation performance: Less than 11 iterations (27.5 seconds) were needed to reach the target in every scenario except scenarios 11 and 12, requiring 31 and 20 iterations.The target was being less than 5 cm from the bin center.
- Adaptation performance: The algorithm reached the target significantly more often than Bayesian optimization across all damage conditions (p < 10−24).It used fewer trials in every condition except scenario 11.
- Adaptation performance: After 31 iterations, accuracy approached 1 cm for all but scenarios 11 and 12, whereas Bayesian optimization’s lowest median accuracy was 2.6 cm.The approximately 1-cm level was never achieved by classic Bayesian optimization.
- Limitation: The two difficult scenarios placed post-damage solutions outside the map, so the robot could not adapt when no mapped behavior could cope with the damage.The authors attribute this boundary partly to the simplicity of the behavioral descriptor.
3 Supplementary Experiments S2
Supplementary ablations show that the algorithm’s map, Bayesian search, and simulation-derived initialization jointly enable rapid discovery of high-performing behaviors, especially compared with direct search in the original 36-dimensional parameter space.
- Ablation design: The algorithm combines MAP-Elites map creation, Bayesian optimization within the map, and initialization from simulation-derived performance predictions.The ablations separately test each component’s contribution.
- Ablation results: After 17 trials, Intelligent Trial and Error achieved 0.26 [0.20; 0.33] m/s and significantly outperformed every tested variant (p < 10−67).The best alternatives were random search in the map at 0.21 [0.16; 0.27] m/s and Bayesian optimization in the map at 0.20 [0.13; 0.25] m/s.
- Component contributions: After 17 trials, policy-gradient performance was limited by one gradient-estimation iteration and strong dependence on its random starting point.The algorithm performs 15 trials per iteration to estimate gradients, and its performance showed high variability.
- Ablation results: Direct searches in the original 36-dimensional parameter space found no working controller, while map-based variants found working behaviors.The compared direct methods were Bayesian optimization and policy gradient.
- Component contributions: MAP-Elites was identified as the most critical component because it reduces the search space and produces high-performing behaviors with useful search-space priors.Map-space variants differed significantly from variants searching the higher-dimensional motor-parameter space (p < 5 × 10−50).
- Component contributions: Bayesian optimization improved the search only when initialized with simulation performance predictions: 0.26 [0.20; 0.33] m/s with initialization versus 0.20 [0.13; 0.25] m/s without.The comparison was significant at p = 10−96.
- Overall result: Additional experiments found that all three components substantially improved performance, with Intelligent Trial and Error outperforming prior damage-recovery and gait-learning algorithms.The authors describe it as state of the art for these comparisons.
4 Supplementary Experiments S3
Intelligent Trial and Error remained effective across simulated slopes, finding fast gaits quickly on moderate terrain variation. Performance degraded beyond ±10°, where controller and sensor constraints limited the behaviors available.
- Fewer than 3 trials were needed to find fast gaits on descending slopes.
- Above 10° ascent, the learned gait still outperformed the hand-designed reference gait on flat ground.
- 10,080 simulated damage-recovery experiments evaluated six damage conditions across slopes from −20° to +20°.
- Between −10° and +10°, all six damage conditions produced gaits above 0.2 m/s in fewer than 15 robot tests.
- Beyond ±10°, adaptation required more trials and performed worse, likely because simple controllers and limited sensors excluded complex slope-specific behaviors.
5 Supplementary Experiments S4
MAP-Elites generated behavior-performance maps that were more diverse and higher-performing than maps produced by random sampling. Its advantage appeared early and persisted for both average and best-solution performance.
- The comparison measured discovered map cells, mean performance, and maximum performance across eight replicated maps.
- After 20 million evaluations, MAP-Elites filled 12,968 cells, about 83% of the map, versus 8,624 cells, about 55%, for random sampling.
- After 20 million evaluations, mean performance was 0.22 [0.22; 0.23] m/s with MAP-Elites versus 0.06 [0.06; 0.06] m/s with random sampling.
- After 20 million evaluations, maximum performance was 0.40 [0.39;0.41] m/s with MAP-Elites versus 0.21 [0.20; 0.22] m/s with random sampling.
6 Supplementary Experiments S5
The study tested whether alternative behavioral descriptors changed damage-recovery performance. All tested descriptors performed well, while informed descriptor choices could provide modest advantages, especially with limited trials.
- After 17 trials, all 11 intentionally chosen descriptors produced median performance within 17% of the duty-factor descriptor.
- After 17 trials, the duty-factor descriptor achieved 0.241 [0.19; 0.29] m/s, while the lowest-performing relative-GRF descriptor achieved 0.204 [0.08; 0.31] m/s.
- After 150 trials, all but three alternative descriptors were within 4% of the duty-factor descriptor.
- After 17 trials, the randomly selected descriptor achieved 0.232 [0.14; 0.30] m/s, 4.2% below the duty-factor descriptor.
- The results indicate that descriptor selection is not critical for good performance, although prior knowledge can further improve results.
7 Caption for Supplementary Videos
The supplementary videos show Intelligent Trial and Error operating on both the hexapod and robotic arm. They also illustrate the variety of walking behaviors produced while constructing the behavior-performance map.
- One video demonstrates Intelligent Trial and Error on the hexapod robot and the 8-degree-of-freedom robotic arm, including multiple damage-recovery behaviors.
- Another video presents walking gaits ranging from classic hexapod locomotion to unexpected forms of movement generated during map creation.