PaperlayerView exact revision on arXiv

Source-linked AI summary

Skew-Fit: State-Covering Self-Supervised Reinforcement Learning

Vitchyr H. Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, Sergey Levine

arXiv:1903.03698v4cs.LGcs.AIcs.ROstat.ML

TL;DR

Manually specifying rewards for broad skill repertoires is burdensome, motivating self-supervised goal setting. The paper proposes Skew-Fit, a mutual-information-based exploration method, and reports improved visual goal reaching plus reward-free real-world door opening, while noting implementation limitations.

  • Problem

    Designing reward functions for many skills is burdensome, while choosing effective self-supervised goal-setting objectives remains difficult, especially with unknown valid image states.

  • Method

    Skew-Fit maximizes mutual information between states and goals by combining goal reaching with learning a maximum-entropy goal distribution over valid states.

  • Results

    Skew-Fit outperforms prior methods on visual goal-reaching tasks and enables a robot to learn door opening from images without a manually designed reward function.

  • Takeaways & Limitations

    Learning diverse goals alongside goal-reaching improves state-space coverage and supports autonomous exploration in simulated and real-world visual tasks.

  • Takeaways & Limitations

    The practical method cannot generally compute the marginal state distribution without accurate models of the environment dynamics and goal-conditioned policy, and entropy improvement is not guaranteed.

Abstract

from arXiv · show

Autonomous agents that must exhibit flexible and broad capabilities will need to be equipped with large repertoires of skills. Defining each skill with a manually-designed reward function limits this repertoire and imposes a manual engineering burden. Self-supervised agents that set their own goals can automate this process, but designing appropriate goal setting objectives can be difficult, and often involves heuristic design decisions. In this paper, we propose a formal exploration objective for goal-reaching policies that maximizes state coverage. We show that this objective is equivalent to maximizing goal reaching performance together with the entropy of the goal distribution, where goals correspond to full state observations. To instantiate this principle, we present an algorithm called Skew-Fit for learning a maximum-entropy goal distributions. We prove that, under regularity conditions, Skew-Fit converges to a uniform distribution over the set of valid states, even when we do not know this set beforehand. Our experiments show that combining Skew-Fit for learning goal distributions with existing goal-reaching methods outperforms a variety of prior methods on open-sourced visual goal-reaching tasks. Moreover, we demonstrate that Skew-Fit enables a real-world robot to learn to open a door, entirely from scratch, from pixels, and without any manually-designed reward function.

1. Introduction

The paper frames unsupervised reinforcement learning as goal-directed exploration that combines broad state coverage with controllable goal reaching. It introduces Skew-Fit to learn diverse goals and reports strong visual goal-reaching results, including reward-free real-world door opening.

  • Manual reward design makes separately learning every potentially useful skill prohibitively time consuming for both agents and users.
  • Maximizing state entropy encourages broad exploration but does not by itself produce policies that can solve new tasks.
  • Goal-directed exploration combines visiting many states with allowing users to specify reachable goal states at test time.
  • The mutual-information objective is equivalently optimized by maximizing goal entropy H(G) while minimizing conditional entropy H(G | S).This formulation jointly encourages diverse goals and reliable goal reaching.
  • Skew-Fit learns a goal distribution that converges to uniform over valid states under regularity conditions, even when the valid state set is unknown.The paper reports state-of-the-art visual goal-reaching results and a robot learning to open a door from images without a manually designed reward.

2. Problem Formulation

The formulation treats goal-conditioned reinforcement learning as maximizing mutual information between terminal states and goals. It separates this into reliable goal reaching and diverse goal setting, while addressing the difficulty of sampling uniformly from an unknown valid state space.

  • 2. Problem Formulation: The formulation maximizes mutual information I(S; G), splitting optimization into minimizing H(G | S) and maximizing H(G).
  • 2.1. Minimizing H(G | S): Goal-Conditioned Reinforcement Learning: The formulation assumes a goal-conditioned Markov decision process with state space S, action space A, unknown dynamics, and goals matching states.The approach can also use a hand-specified goal subset such as a robot’s global XY position.
  • 2.1. Minimizing H(G | S): Goal-Conditioned Reinforcement Learning: Goal-conditioned policies map states and goals to actions intended to make the current state equal the goal.
  • 2.1. Minimizing H(G | S): Goal-Conditioned Reinforcement Learning: Goal reaching is formulated as minimizing H(G | S), which practical goal-conditioned algorithms approximate by deterministically reaching commanded goals.Under deterministic goal reaching, the conditional entropy becomes H(G | S) = 0.
  • 2.2. Maximizing H(G): Setting Diverse Goals: The remaining objective is to maximize the entropy H(G) of the goal distribution, whose maximum over a finite-volume state space is attained by the uniform distribution.
  • 2.2. Maximizing H(G): Setting Diverse Goals: Using a uniform goal distribution directly is difficult because valid states may form an unknown subset of R^n, such as realistic camera images.Sampling uniformly from all pixel arrays is unlikely to produce valid real-world images.
  • 2.2. Maximizing H(G): Setting Diverse Goals: The model samples a goal from q_Gφ, applies a goal-conditioned policy, and obtains terminal states whose marginal distribution p_Sφ integrates over goals and conditional states.
  • 2.2. Maximizing H(G): Setting Diverse Goals: The analysis assumes q_Gφ has full support and that H(p_Sφ) is no less than H(q_Gφ), preventing policies from ignoring diverse goals and remaining in one state.

3. Skew-Fit: Learning a Maximum Entropy Goal Distribution

Skew-Fit learns maximum-entropy goal distributions by repeatedly upweighting rare states and fitting a generative model to the resulting skewed samples. Under regularity conditions, this process converges to a uniform distribution over valid states.

  • Algorithm: The skewing exponent α controls the trade-off between density-estimation reliability and the speed of entropy increase.α = 0 leaves sampling unchanged, while α = −1 recovers exact importance sampling when the estimated density is exact.
  • Variance reduction: SIR produces significantly lower variance than importance sampling while fitting the next generative model to the skewed empirical distribution.Goals may be sampled from either the learned generative model or the skewed empirical distribution; both performed well experimentally.
  • Algorithm: Skew-Fit samples replay-buffer states, upweights rarely visited states, and fits a generative model to the skewed distribution.The procedure can use importance sampling or sampling importance resampling (SIR) before maximum-likelihood fitting.
  • Convergence: Under continuity and related regularity assumptions, iteratively increasing entropy makes the goal distribution converge to the uniform distribution over the valid state space.The general convergence lemma assumes entropy increases with equality only at the uniform distribution; the analysis also proves convergence for α ∈ [−1, 0) in a stated setting.
  • Analysis: Skew-Fit does not require exact generative-model fitting; correlated log densities can suffice for the skewed distribution to have higher entropy.The analysis reports good empirical performance with α values as low as −1.

4. Training Goal-Conditioned Policies with Skew-Fit

The paper jointly trains Skew-Fit with a goal-conditioned policy by maximizing mutual information through goal diversity and predictable goal reaching. It uses a variational reward based on the conditional goal likelihood, implemented with RIG in a learned latent space.

  • Joint objective: Joint training maximizes I(S; G) by combining Skew-Fit with a goal-conditioned policy that minimizes H(G | S).This separates broad goal coverage from predictable reaching of commanded goals.
  • Goal-reaching objective: A variational lower bound replaces direct computation of −H(G | S) with the reward r(S, G) = log q(G | S).The bound uses the Kullback–Leibler divergence between the true and variational conditional distributions.
  • Implementation: The implementation uses RIG, which applies a β-VAE to encode observations and goals into a latent space and compute goal-reaching rewards.RIG is an off-policy goal-conditioned method for vision-based reinforcement learning.
  • Implementation: The implementation trains the generative model on visited states and fixes the β-VAE posterior variance to prevent estimated state likelihoods from collapsing to zero.The model learns only the posterior mean while estimating likelihoods for Skew-Fit.

5. Related Work

Prior work commonly assumes or heuristically constructs goal distributions, or learns diverse but uninterpretable latent skills. Skew-Fit instead provides a principled state-coverage framework for learning reusable policies that reach user-specified goals.

  • Goal selection: Many goal-conditioned RL methods assume an available goal distribution or construct one heuristically from prior experience.The cited heuristics include learning progress, off-policy status, difficulty, and likelihood ranking.
  • Reward-free exploration: Skew-Fit learns without task rewards and acquires a policy reusable for reaching user-specified goals.This differs from exploration methods that modify rewards according to state-visitation frequency.
  • Reusable skills: Latent-variable skill methods produce diverse skills, but their latent variables lack grounded interpretations for immediate goal specification.Skew-Fit policies are described as directly usable for diverse user-specified goals after unsupervised training.
  • Illustrative comparison: In the 2D navigation example, Skew-Fit steadily increases goal-distribution entropy and reaches full coverage over the state space.Entropy is measured by discretizing states onto an 11x11 grid across 9 seeds.

6. Experiments

Experiments show that Skew-Fit improves exploration and goal-reaching across navigation, simulated visual control, and real-world door opening, without task-specific rewards. Its diverse goal distributions avoid the collapse seen in prior methods and improve interaction with relevant objects.

  • Does Skew-Fit Maximize Entropy?: Skew-Fit quickly learns a high-entropy, near-uniform state distribution, unlike uniform sampling from previous experience, which concentrates goals near the initial states.
  • Exploration with Skew-Fit: In ant navigation, Skew-Fit is the only method making significant progress on the challenging labyrinth exploration task.Prior goal samplers choose goals near the start, while #-Exploration’s extrinsic reward dominates goal reaching.
  • Vision-Based Continuous Control Tasks: On simulated visual control tasks, Skew-Fit significantly outperforms prior methods in both task performance and sample complexity.The evaluated tasks include Visual Door, Visual Pusher, and Visual Pickup, using image observations without state-based or task-specific rewards.
  • Vision-Based Continuous Control Tasks: Skew-Fit proposes more varied object, puck, and door configurations, whereas prior goal distributions collapse around narrow configurations.Examples include airborne and ground-level objects, varied puck positions, and changing door angles.
  • Vision-Based Continuous Control Tasks: Skew-Fit is the only method whose pickup rate significantly increases during exploration, indicating goals that encourage interaction with the object.Many other methods maintain a near-constant rate of object lifts throughout training.
  • Real-World Vision-Based Robotic Manipulation: In the real-world visual door task, Skew-Fit improves sample efficiency over RIG and learns door opening from images without user-provided goals during training.The experiment evaluates cumulative successes on five target goals every seven and a half minutes of interaction.
  • Additional Experiments: Across α values from [−1, −0.75, −0.5, −0.25, 0], Skew-Fit works over a broad range, with α = −1 consistently outperforming α = 0.Here α = 0 denotes no Skew-Fit.

7. Conclusion

The paper concludes that Skew-Fit formalizes self-supervised goal-directed exploration and learns a uniform distribution over initially unknown valid states under stated conditions. Using these goals for exploration and relabeling improves state coverage, simulated manipulation, and real-world door opening.

  • Skew-Fit provides a formal objective for self-supervised goal-directed exploration and a way to quantify progress when designing autonomous learning algorithms.
  • Under theoretical conditions, Skew-Fit trains a generative model that approximates a uniform distribution over an initially unknown set of valid states.
  • Using Skew-Fit to choose and relabel goals produces better state-space coverage and effective exploration.
  • The experiments report quantifiable improvements on simulated robotic manipulation and a 95% success rate for real-world door opening without human-provided reward supervision.

A.1. Proof of Lemma 3.1

The proof shows that iteratively entropy-increasing distribution updates converge to the uniform distribution under continuity and regularity conditions. A simplified exponential-family analysis establishes the same conclusion for the specific update sequence.

  • General convergence: If each update is continuous and increases entropy, with equality only at the uniform distribution, the iterates converge to uniformity.The argument assumes a compact state set and uses entropy distance to the uniform distribution.
  • General convergence: The entropy-distance sequence converges because it is bounded below by zero while the entropies increase monotonically.The proof introduces d_t as the entropy-based distance to the uniform distribution and applies monotone convergence.
  • General convergence: A nonzero limiting distance would contradict continuity, because the update would strictly increase entropy away from the uniform distribution.The contradiction uses a distribution q* at the limiting distance and compares q* with F(q*).
  • Exponential-family analysis: The skewed distributions form a one-dimensional exponential family whose entropy initially increases for sufficiently small negative α under the covariance assumption.The sufficient statistic is log q(x), and the derivative at α = 0 is negative under the stated condition.
  • Simplified case: For the simplified update, entropy increases monotonically and is bounded above by uniform-distribution entropy, forcing convergence to the uniform distribution.The proof concludes that a constant limiting log-density is required, which characterizes the uniform distribution.

B.1. Sensitivity Analysis

Sensitivity experiments show that Skew-Fit remains effective across reinforcement-learning algorithms and a range of α values. Its benefits are especially consistent on the more challenging Visual Pusher task.

  • RL algorithm: Skew-Fit performs consistently well with both SAC and TD3 across Visual Door, Visual Pusher, and Visual Pickup.The experiments use SAC in the main Section 6 results and replace it with TD3 for the sensitivity comparison.
  • α hyperparameter: Across tested α values, Skew-Fit is robust, particularly on Visual Pusher, and consistently outperforms α = 0 uniform replay-buffer sampling.The tested values are α ∈ [-1, -0.75, -0.5, -0.25, 0].
  • α hyperparameter: On Visual Pusher, Skew-Fit halves the final distance relative to No Skew-Fit in the α sweep.The figure text notes that α = -1 is not always optimal across tasks.

B.2. Variance Ablation

The variance ablation examines instability from importance sampling and evaluates whether Skew-Fit can train a VAE effectively from imbalanced visual data. Skew-Fit achieves likelihood close to an oracle trained on uniform data while avoiding catastrophic variance.

  • Gradient variance: Importance sampling has catastrophically large gradient variance for α values below −1, whereas Skew-Fit avoids this instability.The comparison measures gradient variance during VAE training on an imbalanced Visual Door dataset.
  • Likelihood evaluation: Skew-Fit considerably outperforms MLE on an unbalanced dataset, achieving a much closer negative log-likelihood to the uniform-data oracle.The oracle is a VAE trained on a uniform dataset of valid door-opening images.
  • Goal-reaching behavior: Only Skew-Fit learns to reach the more difficult goals, although both Skew-Fit and RIG reach goals near the initial position.This comparison is reported through example reached goals in Figure 14.

C.3. Implementation of SAC and Prior Work

The implementation combines SAC-based goal-conditioned learning with VAE goal generation, replay relabeling, and Skew-Fit resampling. It also uses staged VAE training and modifies RIG by replacing prior-based sampling with the learned skewed distribution.

  • SAC implementation: The SAC goal-conditioned policy concatenates target XY goals to states and relabels half of training goals from the goal distribution.The original RIG implementation used TD3, which this implementation replaces with SAC.
  • RIG integration: RIG + Skew-Fit samples exploration and replay-relabeling goals from p_skewed instead of the VAE prior.The VAE is also trained on replay-buffer data sampled using p_skewed rather than uniformly.
  • VAE architecture: The implementation uses 48x48 images and a modified RIG VAE with three convolutional layers followed by a latent fully connected layer and mirrored deconvolutions.The convolutional kernels are 5x5, 3x3, and 3x3, with 16, 32, and 64 output filters.
  • Training schedule: Initially training the VAE with standard maximum likelihood improves stability because early density estimates are changing and inaccurate.The schedules defer Skew-Fit until after an initial MLE phase, commonly the first 5K timesteps.
  • Algorithm procedure: The algorithm samples latent goals, collects trajectories, relabels future states, updates the replay buffer distribution, and fine-tunes the β-VAE.The pseudo-code uses a 0.5 probability for goal replacement and computes reward as negative latent distance.

D. Environment Details

The experiments use simulated and real-world environments for visual and state-based goal-conditioned control, with task-specific spaces, dynamics, rewards, and evaluation metrics. The visual tasks include pushing, door opening, and pickup, while the ant task navigates to target positions.

  • Four Rooms: The Four Rooms environment teleports the agent to its chosen 2D target, applies Gaussian positional noise, and stops movement at walls.The noise standard deviation is 0.0605; the main paper rounds this value to 0.06.
  • Ant: The Ant environment has 29-dimensional observations and 8-dimensional joint actions, with reward defined as negative Euclidean distance to a target 2D position.The observation includes position, velocities, orientation, joint angles, and joint-angle velocities.
  • Visual Pusher: Visual Pusher uses a 7-DoF Sawyer arm to move a puck toward target positions, evaluating distance between goal and final puck positions.The hand occupies a 10x10 cm2 space, while the puck occupies a 30x20 cm2 space.
  • Visual Door: Visual Door uses a 7-DoF Sawyer arm to pull a table-mounted door toward a target angle, with evaluation based on final angular distance.The simulated door angle ranges from 0 to .83 radians, and the hand and door are not reset after each trajectory.
  • Visual Pickup: Visual Pickup constrains a robot to 2D y,z control while lifting or positioning a cube, and evaluates distance between goal and final object positions.For evaluation, 75% of goals place the object in the air and 25% on the ground.
  • Real World Visual Door: Real World Visual Door uses a Sawyer robot, Kinect images, and end-effector velocity control to open a door across a 0-to-45-degree angle range.Actions are commanded at up to 10Hz with magnitudes up to 1cm.
Loading 1903.03698v4…