Source-linked AI summary
Challenges in Representation Learning: A report on three machine learning contests
Ian J. Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, Yingbo Zhou, Chetan Ramaiah, Fangxiang Feng, Ruifan Li, Xiaojie Wang, Dimitris Athanasakis, John Shawe-Taylor, Maxim Milakov, John Park, Radu Ionescu, Marius Popescu, Cristian Grozea, James Bergstra, Jingjing Xie, Lukasz Romaszko, Bing Xu, Zhang Chuang, Yoshua Bengio
TL;DR
The paper examines how three machine learning contests can test representation learning and what competition results reveal beyond research papers. It describes the contests and datasets, summarizes their outcomes, and draws lessons for future challenge design. The contests showed strong performance gains in some settings, while the multimodal task was too easy to yield interesting results.
Problem
Evidence was needed on current representation learning capabilities and on what machine learning competitions can reveal beyond conventional research papers.
Method
The paper describes three ICML 2013 representation-learning contests, their datasets and rules, competition outcomes, and guidance for organizing future challenges.
Results
The black box contest had 75 teams beat the best 3-layer MLP baseline, while the multimodal contest ended in a three-way tie at 100% test accuracy.
Takeaways & Limitations
Competitions provide a different viewpoint on algorithms, and new test sets can make their results a more realistic evaluation of generalization error.
Takeaways & Limitations
The contests were made too easy to increase participation, making the multimodal contest too easy to be interesting and requiring a redesigned matching task.
Abstract
from arXiv · showhide
The ICML 2013 Workshop on Challenges in Representation Learning focused on three challenges: the black box learning challenge, the facial expression recognition challenge, and the multimodal learning challenge. We describe the datasets created for these challenges and summarize the results of the competitions. We provide suggestions for organizers of future challenges and some comments on what kind of knowledge can be gained from machine learning competitions.
1 Introduction
The workshop organized three machine learning contests to test representation learning capabilities and encourage new developments. Competitors came from academia, industry, and other backgrounds.
- The workshop focused on three contests examining current representation learning algorithms.The contests addressed black box learning, facial expression recognition, and multimodal learning.
- The organizers intended the contests to push representation learning toward new developments.
- The competitions attracted a diverse range of competitors spanning academia and industry.
2 The black box learning challenge
The black box challenge obfuscated a small-label dataset to reduce the usefulness of human-guided algorithm design and test benefits from unlabeled data. Competitors substantially exceeded the organizer baseline, with sparse filtering and semi-supervised methods among the successful approaches.
- Obfuscated data limited human-in-the-loop techniques such as visualizing filters during algorithm development.The contest kept the data domain secret, reducing the usefulness of expert practitioner guidance.
- Only 1,000 labeled examples were provided to test whether algorithms could benefit from extra unsupervised data.
- The BBL-2013 dataset projected 3,072 pixel features to 1,875, removed one class, and withheld the source domain until after the contest.
- 218 teams submitted 1,963 entries, and 75 teams beat the organizer’s best 3-layer MLP baseline.
- 70.22% accuracy won the contest through blending sparse-filtering feature learning, random-forest feature selection, and SVM classification.
- Most top scorers used unlabeled data in some way despite a broader trend toward purely discriminative training.
3 The facial expression recognition challenge
The facial expression challenge introduced a new seven-category face dataset to compare learned and hand-engineered features on a well-studied task. Convolutional networks led the contest, but their advantage over the strongest non-feature-learning system was not extreme.
- The contest compared recognition methods on a new dataset to reduce overfitting to repeatedly used benchmark test sets.
- FER-2013 was built from Google image searches, face detection, human filtering, cropping, resizing to 48x48 pixels, and grayscale conversion.
- Human accuracy was 65±5% on FER-2013, compared with 68±5% on acted facial expressions.
- 56 teams submitted, and the top three teams used discriminatively trained convolutional neural networks with image transformations.
- The winner trained a neural network with a primal SVM objective and an L2-SVM loss that performed strongly on the contest dataset and others.
- The strongest SIFT-and-MKL submission performed close to the third-best convolutional network, suggesting a non-extreme accuracy difference.
4 The multimodal learning challenge
The multimodal challenge sought unified representations for images and text but used a matching-based classification task. That task was too easy to produce informative results, motivating a redesigned evaluation that forces independent image labeling.
- The challenge aimed to develop algorithms that discover unified semantic representations from image and text inputs.
- Training data included 100,000 variably sized images, each tagged with about 14 words from a vocabulary exceeding 4,000 words.
- A new test set contained 1,000 manually labeled images designed to resemble the training labels, including common misspellings.
- The organizers paired each test image with two labels and asked competitors to identify the correct one because standard multimodal metrics were unavailable.
- 100% test accuracy produced a three-way tie after the Hungarian algorithm could obtain optimal matching even with a low-accuracy base classifier.
- Future contests should remove the matching aspect so classifiers must label each image independently.
5 Advice to contest organizers
Successful contests require substantial advance planning, explicit submission rules, and calibrated difficulty. Organizers should also use baselines, leaderboards, and multiple contests to support participation and robust outcomes.
- Organizers must budget time for dataset creation, baseline preparation, rule design, question handling, portability issues, and winner verification.These tasks span the periods before, during, and after the contest.
- Designing rules: Require trained models to be uploaded before test data release so predictions can be verified and leaderboard manipulation is deterred.The organizers can then check that winning predictions were generated by the previously frozen model.
- Designing rules: Contest rules should decide in advance whether transductive methods, public-test labeling, outside data, and web scraping are permitted.The rules should also prevent hyperparameter selection on test data and reserve submission-verification rights.
- Difficulty and participation rate: Making contests too easy can reduce their scientific interest, although Kaggle hosting substantially improves participation rates.The authors report that the multimodal contest was too easy, while even their least popular challenge attracted 26 teams.
- Running multiple contests provides insurance that at least one will produce interesting results, while the marginal cost of additional contests is low.
- Baselines and leaderboards boost participation by reducing boilerplate work for entrants.
6 Discussion and conclusion
The authors argue that competitions provide a complementary view of machine-learning algorithms by evaluating generalization on new test sets under incentives centered on performance. They caution that contests are not controlled experiments, but can draw attention to effective methods that research practice may overlook.
- Competitions let skilled practitioners optimize any method that may win, rather than focusing only on methods they invented or on novelty.
- Using a completely new test set makes contest results a more realistic evaluation of generalization error.
- Contest outcomes should not be interpreted as controlled experiments with statistical analysis.
- The facial-expression challenge applied this complementary viewpoint by comparing feature-learning methods with hand-engineered features on a new dataset.
- The workshop contests highlighted SVM loss functions, sparse filtering, and entropy regularization as methods deserving further attention.