Source-linked AI summary
Active One-shot Learning
Mark Woodward, Chelsea Finn
TL;DR
Active learning seeks to reduce supervision by selecting which examples to label, but existing selection methods often rely on heuristics. This paper combines one-shot learning with reinforcement learning to train a recurrent action-value model to choose between predicting and requesting labels. The model learns from a handful of requested labels and can trade prediction accuracy for fewer requests through its reward function.
Problem
Active-learning selection often relies on heuristics for estimating the risk of labeling choices, raising the question of whether selection can instead be learned.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
The results suggest that the model learns to reason about its uncertainty when deciding whether to predict or request a label.
Source: §7 · p. 6The experiments need expansion to more complex tasks, and the reinforcement-learning method used is described as quite simplistic.
Source: §7 · p. 6
Abstract
from arXiv · showhide
Recent advances in one-shot learning have produced models that can learn from a handful of labeled examples, for passive classification and regression tasks. This paper combines reinforcement learning with one-shot learning, allowing the model to decide, during classification, which examples are worth labeling. We introduce a classification task in which a stream of images are presented and, on each time step, a decision must be made to either predict a label or pay to receive the correct label. We present a recurrent neural network based action-value function, and demonstrate its ability to learn how and when to request labels. Through the choice of reward function, the model can achieve a higher prediction accuracy than a similar model on a purely supervised task, or trade prediction accuracy for fewer label requests.
1 Introduction
The paper asks whether learning can replace heuristics for selecting labels in active learning, and combines meta-learning with reinforcement learning to train an active learner. Preliminary results show learning from a handful of requested labels and trading prediction accuracy against label requests through the reward function.
1 Introduction
- Active-learning selection often relies on heuristics for estimating the risk of labeling choices, motivating the question of whether selection can instead be learned.The paper frames active learning as reducing supervision by having the model select datapoints for labeling, particularly where mistakes and requests carry different costs.Source: §1 · p. 1
- The method combines meta-learning with reinforcement learning, training a deep recurrent model to choose online whether to label an example or request its true label.Unlike supervised one-shot approaches, it treats the model as a policy with both actions and trains that policy using reinforcement learning [5].Source: §1 · p. 1
- The model learns from a handful of requested labels and trades prediction accuracy against fewer label requests by changing the reward function.The authors describe this as a preliminary result on Omniglot and identify the method as, to their knowledge, the first application of deep recurrent reinforcement learning to active learning.Source: §1 · p. 1
2 Related Work
The paper situates its approach in single-pass active learning and meta-learning: it replaces heuristic selection with a learned policy that chooses online whether to request labels.
2 Related Work
- Single-pass active learning makes labeling decisions as examples arrive in a stream, while prior approaches commonly use similarity or uncertainty heuristics.The paper describes label and incorrect-prediction costs as central and notes that heuristic methods approximate selection risk computationally.Source: §2 · p. 2
- Reinforcement learning explicitly specifies labeling and prediction costs and learns a labeling policy to optimize them, making it a proposed alternative to selection heuristics.The paper calls reinforcement learning a natural fit for active learning and uses a deep recurrent network to represent the action-value function.Source: §2 · p. 2
- Building on Santoro et al.’s one-shot task [5], the method withholds each true label unless requested and uses reinforcement learning to reward correct predictions while penalizing errors and requests.The agent therefore must consider its uncertainty when choosing between predicting and requesting a label.Source: §2 · p. 2
3 Preliminaries
The preliminaries frame reinforcement learning as maximizing discounted future reward and describe learning an action-value function by minimizing Bellman error.
3 Preliminaries
- An optimal policy selects the action with the highest action value, which represents expected discounted future reward when acting optimally thereafter.The objective of reinforcement learning is to maximize the expected sum of discounted future rewards.Source: §3 · p. 2
- The optimal action-value function satisfies a Bellman equation relating its value to immediate reward and discounted future value.The reward is received after taking the current action, and γ discounts future rewards.Source: §3 · p. 2
- The experiments optimize a neural action-value function by minimizing Bellman error using observations, such as images, rather than states.The parameters are trained with stochastic gradient descent, without a separate target network [13].Source: §3 · p. 2
- At each step, the model can predict one of the episode’s class labels or request the image’s label, with different rewards for requests and prediction outcomes.A request costs −0.05; a prediction earns +1 if correct and −1 otherwise, and its true label is not supplied at the next step.Source: Figure 1 · p. 3
4 Task Methodology
The task trains an online active learner to choose between predicting an image label and requesting the true label. It uses short episodes with few examples per class and rewards that define the objective.
4 Task Methodology
- Training uses short episodes with few examples per class, varying classes and randomizing labels between episodes.The stated intuition is to leave episode-specific details to activations while weights fit the general meta-task of learning to label.Source: §4 · p. 3
- At each time step, the agent receives an image and either predicts its label or requests it; a request returns the true label with the next image.After a prediction, the next observation contains a zero-vector instead of the true label.Source: §4 · p. 3
- Each step rewards a label request, a correct prediction, or an incorrect prediction, and the objective is to maximize total episode reward.The reward values determine the trade-off among these outcomes.Source: §4 · p. 3
- The optimal strategy compares a new image with stored class representations, balancing uncertainty and the costs of correct prediction, error, and requesting a label.For a new class, the strategy stores its representation and associates it with the requested label.Source: §4 · p. 3
5 Reinforcement Learning Model
The reinforcement-learning model uses an LSTM-based action-value function with a linear output layer. Its LSTM equations define gates and cell-state computations from observations and hidden state.
5 Reinforcement Learning Model
- The action-value function is an LSTM connected to a linear output layer, with one output value for each action.The passage describes this output structure as similar to the DQN model [13].Source: §5 · p. 3
- The LSTM output is mapped to action-values using weights and an action-value bias.The passage identifies h_t as the LSTM output and W_hq and b_q as the mapping weights and bias.Source: §5 · p. 4
- The LSTM computes its gates and candidate cell state from the observation and previous hidden state using learned weights and a bias.The forget, input, and output gates and candidate cell state are defined in the supplied equations.Source: §5 · p. 4
6 Experimental Results
The experiments test whether reinforcement learning can learn when to request labels and whether it uses uncertainty to guide those decisions. On Omniglot, the model trades label requests against accuracy and outperforms a comparable supervised model in accuracy while using fewer labels.
6.1 Setup
- Experiments used Omniglot’s 1,623 character classes, split into 1,200 training classes and 423 test classes, with images resized to 28x28 pixels.Episodes sampled 30 images from three classes, with random rotations and randomized one-hot label slots; each training step used 50 episodes. The default rewards were +1 for correct predictions, −1 for incorrect predictions, and −0.05 for requests; the Q function used a 200-unit LSTM.Source: §6.1 · p. 4
6.2 Results
- Requests are more frequent for early class instances and decline for later ones, while accuracy rises on later instances.After 100,000 training episode batches, the model is evaluated for 10,000 more batches on held-out test classes.Source: §6.2 · pp. 4–5, Figure 2 · p. 5
- In a two-class test, request rates differed at time step 6 between the 5-example and 10-example sequences and were similar when the second class appeared, consistent with uncertainty-based decisions.The experiment ran 1,000 episodes for each scenario; the authors note that further experiments are needed to confirm a possible explanation for rising requests before the second class instance.Source: §6.2 · p. 5, Figure 3 · p. 5
- The RL model achieves higher prediction accuracy than the supervised model while using 60.6% of test-time labels, versus 100% for supervised learning.The reward for an incorrect prediction controls the accuracy–label-request trade-off; Table 1 reports test-set classification accuracy and label-request percentages.Source: §6.2 · p. 6, Table 1 · p. 6
7 Discussion & Future Work
The paper concludes that reinforcement learning can train an active learner that trades accuracy against fewer label requests and appears to reason about uncertainty. Future work should test more complex tasks and datasets; the reinforcement learning method used here was simplistic.
7 Discussion & Future Work
- The results suggest the model can learn from a handful of requested labels, trade accuracy for fewer requests through reward choice, and reason about uncertainty.The authors identify expanding experiments to more complex tasks, such as episodes with more classes or datasets such as ImageNet, as an important next step.Source: §7 · p. 6
- The experiments used a simplistic reinforcement learning method, and the authors expect stronger strategies or memory augmentation may help scale to more complex tasks.They report that a neural turing machine with least-recently-used addressing overfit this task more than an LSTM.Source: §7 · p. 6