Source-linked AI summary
Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptors
Antoine Cully
TL;DR
Quality-Diversity methods can build diverse robotic repertoires, but require manually chosen behavioral descriptors that shape which behaviors are explored. AURORA learns descriptors by combining QD search with dimensionality reduction trained on the robot’s sensory data. Across two scenarios, its repertoires match or exceed the diversity of handcrafted and existing unsupervised approaches, while questions remain about latent-space size and scaling to more complex settings.
Problem
Quality-Diversity methods require manually defined behavioral descriptors, whose task-dependent choice shapes the solution types discovered and can demand expertise robots may lack in unfamiliar settings.
Source: §1 · pp. 1–2Method
AURORA uses QD-generated behavioral data to train a dimensionality-reduction model, then uses its latent representation as the behavioral descriptor.
Source: §1 · p. 2Results
Across two scenarios, AURORA produced repertoires at least as diverse as manually defined descriptors and significantly better than existing unsupervised approaches.
Source: §1 · p. 2Takeaways & Limitations
AURORA can autonomously define behavioral descriptors and discover repertoires covering a similar range of behaviors as those based on prior knowledge or domain expertise.
Source: §5 · p. 8The influence of latent-representation size remains open, and future work will test scaling to more sophisticated robots and complex environments.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Quality-Diversity optimization is a new family of optimization algorithms that, instead of searching for a single optimal solution to solving a task, searches for a large collection of solutions that all solve the task in a different way. This approach is particularly promising for learning behavioral repertoires in robotics, as such a diversity of behaviors enables robots to be more versatile and resilient. However, these algorithms require the user to manually define behavioral descriptors, which is used to determine whether two solutions are different or similar. The choice of a behavioral descriptor is crucial, as it completely changes the solution types that the algorithm derives. In this paper, we introduce a new method to automatically define this descriptor by combining Quality-Diversity algorithms with unsupervised dimensionality reduction algorithms. This approach enables robots to autonomously discover the range of their capabilities while interacting with their environment. The results from two experimental scenarios demonstrate that robot can autonomously discover a large range of possible behaviors, without any prior knowledge about their morphology and environment. Furthermore, these behaviors are deemed to be similar to handcrafted solutions that uses domain knowledge and significantly more diverse than when using existing unsupervised methods.
1 INTRODUCTION
AURORA combines Quality-Diversity search with dimensionality reduction to learn behavioral descriptors from robots’ own sensory data. Across two scenarios, it produces repertoires at least as diverse as those using manually defined descriptors and significantly better than existing unsupervised approaches.
1 INTRODUCTION
- Robots may face unforeseen situations outside structured settings, while configuring learning algorithms can still require engineers and technical experts.The paper notes that anticipating every situation is impossible, especially in environments such as domestic households.Source: §1 · p. 1
- AURORA combines Quality-Diversity algorithms with dimensionality reduction to generate behavioral repertoires without manually defined descriptors or engineered task and capability insights.The QD-generated solutions provide training data for dimensionality reduction, whose latent representation then serves as the behavioral descriptor.Source: §1 · pp. 1–2
- Across two scenarios, AURORA’s repertoires are at least as diverse as those using manually defined descriptors and significantly better than existing unsupervised approaches.The comparisons include manually defined descriptors informed by prior knowledge of the robot and environment.Source: §1 · p. 2
2 RELATED WORK
The related work describes how Quality-Diversity methods use behavioral descriptors to organize diverse controllers, while automatic characterization approaches seek to reduce manual descriptor design. AURORA differs by learning from its own interactions rather than a task-specific dataset or sampled sensory space.
2.1 Qality-Diversity and Behavioral Repertoires
- Quality-Diversity methods organize diverse controllers by behavioral descriptors, whose task-dependent choice determines which solution families exploration discovers.Descriptors are numerical vectors used to quantify novelty and drive exploration; containers store controllers, with MAP-Elites using a grid and NSLC an unstructured archive.Source: §2.1 · p. 2
2.2 Automatic behavioral characterization
- Meyerson et al. learn descriptors from successful controllers on training tasks, but the approach requires a way to define success that can rely on prior task or robot knowledge.Their learned descriptors are evaluated on unseen test tasks.Source: §2.2 · pp. 2–3
- AURORA builds its own sensory-data dataset through environmental interaction, unlike prior dimensionality-reduction approaches that use manually generated, task-specific data.The related-work discussion describes prior pretraining datasets designed to represent task configurations.Source: §2.2 · p. 3
2.3 Behavioral characterization in high-dimensional spaces
- CVT-MAP-Elites constructs high-dimensional behavior-space centroids from sampled plausible sensory data, enabling discretization of trajectories and other high-dimensional behaviors.The number of centroids controls the number of MAP-Elites cells.Source: §2.3 · p. 3
- CVT-MAP-Elites can outperform direct MAP-Elites in high-dimensional settings, but sampling plausible sensory data may require prior knowledge and constrain discoverable behaviors.A cited 20D maze example reports 0.5% of cells filled by MAP-Elites versus 40% by CVT-MAP-Elites.Source: §2.3 · p. 3
3 METHODS
AURORA alternates between QD exploration and refinement of a dimensionality-reduction representation trained on collected sensory data. It uses learned low-dimensional projections as behavioral descriptors, with PCA or an auto-encoder as the dimensionality-reduction method.
3.1 AURORA
- AURORA trains dimensionality reduction on sensory data from an initial random controller population, then uses each controller’s latent projection as its behavioral descriptor.During QD iterations, new controllers are evaluated and projected through the dimensionality-reduction algorithm; as the collection grows, its sensory data can be used to retrain the representation.Source: §3.1 · pp. 3–4
- AURORA alternates QD exploration with updates that refine the latent space using sensory data from the repertoire.After an update, controllers receive new descriptors and are reinserted into the container; competing similar behaviors are reduced to the best ones, and the repertoire resolution is maintained.Source: §3.1 · p. 4
- The dimensionality-reduction update frequency decreases exponentially over time, with updates at batches 0, 50, 150, 350, 750, 1550, and 3150.The schedule is intended to update more frequently early in learning, when new sensory data is more likely, and less frequently after extended exploration.Source: §3.1 · p. 4
3.2 Dimensionality reduction algorithms
- PCA projects sensory data into a low-dimensional representation used as the behavioral descriptor.The experiments use a 2D representation.Source: §3.2 · p. 4
- An auto-encoder’s encoder supplies the latent behavioral descriptor, while its decoder reconstructs the input; joint training minimizes reconstruction error.Unlike PCA, the auto-encoder’s projection and reconstruction are nonlinear.Source: §3.2 · p. 4
4 EXPERIMENTAL VALIDATION
The validation evaluates AURORA in ballistic and air-hockey scenarios using recorded trajectories as sensory data. The experiments compare learned behavioral spaces with a hand-coded ground-truth space in the ballistic task.
4 EXPERIMENTAL VALIDATION
- Experiments evaluate AURORA in a 2D ballistic task and an air-hockey task, recording planar projectile or puck trajectories over 50 time-steps as 100-dimensional sensory data fed to the dimensionality-reduction algorithm.Source: §4 · p. 4
- All compared QD approaches run without a fitness function, for 5000 batches of 200 evaluations, with 20 replications.The reported quality score is zero because the experiments focus on exploring robot capabilities.Source: §4 · p. 4
- In the ballistic task, Figure 3 compares each approach’s generated 2D behavioral space with repertoires projected into a hand-coded ground-truth space.Point colors encode locations in the ground-truth behavioral space; CVT’s 100-dimensional space is not shown.Source: Figure 3 · p. 5
4.1 First scenario: Ballistic task
The ballistic experiment tests whether AURORA can represent projectile trajectories across the reachable behavior space, using manually described trajectories as a reference and comparing several descriptor-generation approaches.
4.1 First scenario: Ballistic task
- The two-degree-of-freedom ballistic task varies projectile launch direction and force intensity, then evaluates trajectories with ground-truth descriptors based on the highest point’s altitude and distance.The projectile follows a bouncing ballistic trajectory; the setup uses simple iterative functions rather than an expensive physical simulator.Source: §4.1 · p. 5
4.1.1 Performance metrics.
- KLC compares generated and ground-truth behavior distributions after projection into the ground-truth latent space; repertoire size and trajectory reconstruction RMSE are also reported.KLC uses normalized histograms with 30 bins per dimension and is unaffected by repertoire-size differences.Source: §4.1.1 · p. 5
4.1.2 Compared approaches.
- The eight approaches span hand-coded and genotype descriptors, pre-trained and incremental PCA and AE, and CVT-MAP-Elites with or without prior.The hand-coded descriptor uses Xmax and Ymax; CVT-MAP-Elites uses a 100-dimensional descriptor space.Source: §4.1.2 · pp. 5–6
4.1.3 Experimental results:
- KLC medians were 0.051 and 0.053 for AURORA-PCA variants and 0.097 for AURORA-AE incremental; CVT-MAP-Elites without prior scored 3.2, more than 30 times worse.Figure 4 reports the algorithm comparison; hand-coded BD converges to zero.Source: Figure 4 · p. 6, §4.1.3 · p. 6
- All approaches except CVT-MAP-Elites without prior reached every reachable ground-truth region; AURORA repertoires were close to manually defined descriptors without prior knowledge or intelligent pre-sampling.Even 100,000 bins leave only a few feasible trajectories in the 100-dimensional CVT space.Source: §4.1.3 · p. 6
- Genotype over-represented one ground-truth region, while both PCA variants and incremental AE matched its distribution; pre-trained AE was denser in some areas, possibly from overfitting.The paper links genotype skew to action-space regions producing smaller trajectory differences, and suggests the pre-trained AE omitted a training/validation split.Source: §4.1.3 · p. 6
- Genotype, pre-trained AURORA-AE, and CVT-MAP-Elites with prior had higher KLC medians of 0.686, 0.528, and 0.704, respectively.Hand-coded BD eventually converged to zero KLC.Source: §4.1.3 · p. 6
- PCA variants had slightly better trajectory reconstruction errors than AE variants, suggesting these trajectories may be captured by linear feature combinations.Reconstruction error was not defined for the other approaches.Source: §4.1.3 · p. 6
4.2 Second scenario: Air hockey task
In the air-hockey task, AURORA-PCA slightly outperforms AURORA-AE and the hand-coded descriptor on diversity, while both AURORA variants cover the reachable trajectories. The experiment evaluates diversity by measuring trajectory coverage across puck-destination bins.
4.2 Second scenario: Air hockey task
- The task uses a four-degree-of-freedom planar arm whose controllers specify the initial and final angular positions of its four joints.The arm pushes a puck, which may bounce off the arena's four walls.Source: §4.2 · p. 6
4.2.3 Experimental results.
- Both AURORA variants cover the reachable trajectories, while CVT-MAP-Elites without prior knowledge finds few controllers; the hand-coded BD yields more uniform puck endpoints.AURORA solutions are denser near arena walls, where collisions absorb kinetic energy and make diverse stopping trajectories easier to generate.Source: Figure 5 · p. 7, §4.2.3 · p. 8
4.2.1 Performance metrics.
- The diversity score sums trajectory coverage across puck-destination bins, where each bin's coverage is the fraction of arena bins traversed by trajectories reaching it.The arena is discretized into 100 bins, with 10 bins per dimension.Source: §4.2.1 · p. 7
4.2.3 Experimental results.
- AURORA-PCA slightly outperforms AURORA-AE and the hand-coded BD on diversity (p-value < 0.038); AURORA-AE and the hand-coded BD are not significantly different (p-value = 0.561).AURORA-AE has lower reconstruction error than AURORA-PCA, as the trajectories are highly nonlinear.Source: Figure 6 · p. 8, §4.2.3 · p. 8
4.2.2 Compared approaches.
- The comparisons include incremental PCA and auto-encoder variants, CVT-MAP-Elites without prior knowledge, and a hand-coded BD based on the puck's final 2D location.The hand-coded descriptor serves as a reference designed to maximize repertoire coverage.Source: §4.2.2 · p. 8
5 DISCUSSION AND CONCLUSION
The paper concludes that AURORA combines Quality-Diversity and dimensionality reduction to discover behavioral repertoires without prior knowledge, with results comparable to repertoires built using domain expertise. It identifies scaling, latent-space size, and hierarchical repertoires as open directions.
5 DISCUSSION AND CONCLUSION
- Across two experiments, AURORA produced repertoires covering a similar range of behaviors to those created with prior knowledge, without slowing convergence relative to fixed-descriptor approaches.The authors present it as a promising way to scale QD to high-dimensional data without prior knowledge about the robot or environment.Source: §5 · p. 8
- Future work will test scaling to more sophisticated robots and complex environments, fuse multiple sensory streams, and explore more complex dimensionality-reduction algorithms.The authors also propose studying exploration curricula when robots access images of themselves and their environment.Source: §5 · p. 8
- The effect of latent-representation size on behavioral diversity and QD selective pressure remains an open question; hierarchical repertoires are another proposed extension.The authors note that larger latent spaces might increase diversity but could also affect selective pressure.Source: §5 · p. 8