PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Algorithms for Active Learning

Philip Bachman, Alessandro Sordoni, Adam Trischler

arXiv:1708.00088v1cs.LG

TL;DR

The paper addresses how to select labels efficiently when unlabeled data is abundant and labeling is costly. It metalearns representations, selection policies, and prediction construction across related tasks, achieving strong results on Omniglot and outperforming baselines on MovieLens cold-start recommendation. The authors also identify difficulty in the 1-shot, 10-way Omniglot setting and assume full support-set labels during training.

  • Problem

    Active learning seeks data-efficient task performance when labels are scarce or costly, but conventional selection heuristics are engineered rather than learned across related tasks.

  • Method

    The model metalearns a data representation, item-selection strategy, and prediction-function constructor, using queried items to build Matching Network support sets and training end-to-end.

  • Results

    The learned policy approaches an optimistic Omniglot performance estimate and outperforms several MovieLens baselines, improving RMSE by 2.5% after 10 ratings against the best baseline.

  • Takeaways & Limitations

    Learned active policies can share experience across related tasks and show promise for efficient label querying in both one-shot classification and cold-start recommendation.

  • Takeaways & Limitations

    The 1-shot, 10-way Omniglot setting is particularly difficult because the model must query different classes at each step and has limited ability to identify missing classes.

Abstract

from arXiv · show

We introduce a model that learns active learning algorithms via metalearning. For a distribution of related tasks, our model jointly learns: a data representation, an item selection heuristic, and a method for constructing prediction functions from labeled training sets. Our model uses the item selection heuristic to gather labeled training sets from which to construct prediction functions. Using the Omniglot and MovieLens datasets, we test our model in synthetic and practical settings.

1. Introduction

The paper frames active learning as selecting costly labels efficiently and proposes learning that selection process end-to-end across related tasks. The model jointly adapts representation, item selection, and prediction construction, then evaluates learned strategies on Omniglot and MovieLens.

  • Motivation: Active learning selects which unlabeled instances to label to trade labeling cost against task performance and data efficiency.The motivation is that carefully chosen examples can achieve comparable predictions with less labeled data and computation.
  • Applications: Active querying can reduce labeling burdens in applications such as movie recommendation and specialist-dependent medical imaging.In both examples, the system selectively requests labels for the most useful items.
  • Contribution: The proposed model replaces engineered selection heuristics with a metalearned strategy trained across related tasks.It can use existing users’ ratings to guide preference elicitation for new users in a recommender cold-start setting.
  • Contribution: The model jointly co-adapts a data representation, an item-selection strategy, and a prediction-function constructor.These components are trained as an integrated active-learning system rather than as independently engineered pipeline stages.
  • Evaluation: Experiments on active Omniglot tasks show efficient label-querying strategies, while MovieLens tests assess bootstrapping a recommender system.The evaluation spans synthetic one-shot settings and a more practical recommendation setting.

2. Related Work

Prior active-learning work largely designs selection rules around uncertainty, classifier disagreement, information gain, or empirical statistics. This paper situates its approach among end-to-end learned strategies while distinguishing its pool-based setting and anytime objective.

  • Engineered heuristics: Earlier heuristics select examples using classifier confidence, committee disagreement, or expected information gain.These approaches aim to choose labels that reduce uncertainty about predictions or model parameters.
  • Learning active policies: Stream-based meta active learning decides whether to query labels as items arrive in an exogenously determined order.The proposed model instead operates as a pool-based learner with access to a static collection of unlabeled items.
  • Anytime learning: The model is trained to balance prediction performance against labeling cost at every query step.This produces an anytime active learner intended to remain effective when interaction ends early.
  • One-shot learning: Matching Networks provide the one-shot-learning foundation, but the proposed system constructs the labeled support set from a larger unlabeled pool.The paper targets cases where a labeled example for every class is not initially available.
  • Recommendation systems: Recommendation baselines based on popularity or informativeness are computationally cheap but lack adaptation and personalization.The paper motivates learned selection as a way to choose items for a user more adaptively.
  • Training guidance: Training uses samples from an oracle policy that knows all labels to provide guidance during active-policy learning.The approach connects to imitation learning and learning-to-search methods that query an oracle on states visited by the learned policy.

3. Model Description

The model metalearns active-learning algorithms by repeatedly solving tasks sampled from a task distribution. It adaptively selects labeled support items for a Matching Network that predicts labels on evaluation items.

  • Task-level metalearning: The model learns from supervised feedback on sampled tasks so it can perform on new tasks drawn from a similar distribution.Training therefore targets expected performance beyond the episodes encountered during optimization.
  • Active support construction: For each task, the policy adaptively selects items for the labeled support set used by a Matching Network to classify test items.The active learner chooses which labels to acquire while the prediction module uses the resulting support set.
  • Organization: The paper summarizes the model through its architecture, optimization objectives, and training pseudocode.The formal task, model details, and parameter-optimization procedure are presented in separate subsections.

3.1. Task Description

The task treats active learning as sequentially querying labels from a support pool and using the evolving labeled set and control state to predict evaluation items. Training rewards performance throughout the query sequence to promote anytime behavior.

  • Episode structure: Each episode contains a support set with queryable labels and an evaluation set used to measure prediction performance.The model repeatedly interacts with the support set while evaluation predictions provide the reward signal.
  • Sequential querying: After each query, the model updates its control state and the labeled/unlabeled composition of the support set.The control state and current support set jointly determine subsequent predictions and the next label request.
  • Prediction reward: The prediction reward is the evaluation-set log-likelihood, instantiated as negative cross-entropy for Omniglot and negative RMSE for MovieLens.The reward conditions on the test item, current control state, and current labeled/unlabeled support set.
  • Training objective: The idealized training procedure unrolls the policy for T steps and maximizes prediction reward after each step.Intermediate states record the evolving support set and control state throughout the episode.
  • Anytime behavior: Rewarding every query step promotes anytime behavior rather than optimizing only the final queried state.This supports settings such as recommendations, where interaction may stop before the label budget is exhausted.
  • Training approximation: The training approximation uses a fast prediction module for unlabeled support items and a slow prediction module for evaluation items.The fast-module reward assumes full support-set labels are available during training.

3.2. Model Architecture Details

The model repeatedly selects unlabeled support items, updates a controller, and uses the resulting labeled set for fast and slow predictions. Its modules combine context-free and context-sensitive representations, learned selection features, and Matching Network-style prediction.

  • Active learning loop: The active-learning loop encodes items, selects an unlabeled instance, reads its label, updates the controller, and records fast and slow prediction losses.The loop repeats selection and state updates before evaluating support-set and held-out-set predictions.
  • Context-free and context-sensitive encoding: The context-free encoder produces context-independent item embeddings, while a bidirectional LSTM adds support-set context to produce context-sensitive embeddings.The context-sensitive encoder processes support items but is not applied to evaluation items.
  • Controller and reading: The controller receives the selected item and label through the reading module, then updates its recurrent control state.The initial controller state is derived from the context-sensitive encoding of the support set.
  • Selection: The selection module samples unlabeled items from a softmax distribution built from controller-item and item-item similarity features.Item-item features include extrema and averages of cosine similarities to labeled and unlabeled items, while a learned gate combines feature contributions.
  • Fast prediction: The fast prediction module uses sharpened cosine-similarity attention over labeled items, and its sharpening term is necessary for strong performance.The resulting prediction is a convex combination of labeled-item labels; similarities can be precomputed and reused during policy unrolling.
  • Slow prediction: The slow prediction module performs modified Matching Network prediction for held-out items while accounting for labeled and unlabeled support items and conditioning on the controller state.It iteratively updates matching states and returns the final label-attention result, using K = 3 steps in the tests.

3.3. Training the Model

The model optimizes active-learning parameters with both backpropagation and policy gradients, using episode-level rewards and an unbiased gradient estimator. It applies an actor-critic optimization method to improve policy-gradient training.

  • The model parameters are optimized using a combination of backpropagation and policy gradients.
  • The training objective’s gradient combines an expectation over sampled state sequences with the reward gradient from the resulting trajectory.The sequence-generation decisions are treated as constant when computing the trajectory reward gradient.
  • Taking the expectation over episodes from the task distribution yields an unbiased gradient estimator for the training objective.
  • Generalized Advantage Estimation provides an actor-critic approach for approximately optimizing the policy gradients.The updates are applied using ADAM.
  • A rollout visualization shows the policy selecting items with previously unseen labels across active iterations.The support set contains 20 items from 10 classes, with two items per class; rows represent iterations and columns represent classes.

4. Experiments

Experiments evaluate the active learner on Omniglot classification and MovieLens cold-start recommendation. The model approaches optimistic Omniglot performance, generalizes across class counts, and outperforms MovieLens baselines especially when few labels are available.

  • 4.1. Omniglot: The Omniglot experiments use 1,623 characters from 50 alphabets, split into 1,200 training characters and held-out test characters.Test characters were not encountered during training.
  • 4.1. Omniglot: In N-way, K-shot Omniglot classification, the model requests NK labels before making held-out predictions.Support sets contain five sampled items per class, while the held-out set contains one item per class.
  • 4.1. Omniglot: The active policy nearly matches the optimistic balanced Matching Network baseline, with a 2.2% degradation in the 1-shot, 10-way case.Randomly sampled labels perform worst in 1-shot settings because some classes may be unrepresented.
  • 4.2.3. RESULTS: In 20-way classification, the model quickly approaches the optimistic performance estimate after acquiring more labels, and policies trained for 10-way classification generalize to 20-way tasks.
  • 4.1. Omniglot: Removing attention temperature reduces 10-way, 1-shot accuracy from 94.5 to 86.0, while reducing matching steps or removing the context-sensitive encoder has no significant effect.The authors identify architectural streamlining as a topic for future work.
  • 4.2. MovieLens: The MovieLens experiment uses a cold-start collaborative-filtering setup with approximately 1M ratings after selecting 4,000 movies and 6,000 users.The source dataset contains approximately 20M ratings on 27K movies by 138K users.
  • 4.2.3. RESULTS: The active policy outperforms MovieLens baselines in RMSE, with a 2.5% improvement over the best baseline after 10 ratings.The advantage is largest after the first few labels and diminishes as more labels are requested.

5. Conclusion

The paper introduces an end-to-end model that learns active-learning algorithms from related tasks, jointly adapting selection, representation, and prediction. Across Omniglot and MovieLens, it approaches an optimistic benchmark or outperforms baselines, respectively.

  • The model learns active-learning algorithms end-to-end from labeled instances across related tasks.It simultaneously learns a selection strategy, data representation, and prediction function for the task at hand.
  • On active Omniglot tasks, the learned policy approaches an optimistic performance estimate.
  • On a MovieLens cold-start collaborative-filtering task, the model outperforms several baselines and shows promise for more realistic applications.
Loading 1708.00088v1…