Source-linked AI summary
Deep Learning on Small Datasets without Pre-Training using Cosine Loss
Björn Barz, Joachim Denzler
TL;DR
Learning deep classifiers from small datasets without external pre-training remains difficult, especially where transfer learning is problematic. The paper trains from scratch with cosine loss, evaluates it across limited-data tasks, and finds better performance than cross-entropy, with semantic class hierarchies providing complementary gains. However, pre-training still outperforms the evaluated methods on CUB.
Problem
Learning from small datasets without external pre-training has received limited attention, despite transfer learning being problematic under domain shift or legal restrictions.
Method
The paper trains neural classifiers from scratch by maximizing cosine similarity between network outputs and fixed class embeddings, including one-hot vectors and semantic hierarchies.
Results
Cosine loss outperforms cross-entropy by a large margin on limited-data tasks, while both losses perform similarly with sufficient data or pre-trained initialization.
Takeaways & Limitations
Cosine loss is useful for training deep neural classifiers from scratch on limited data, and semantic class embeddings provide complementary improvements.
Takeaways & Limitations
On CUB, all evaluated methods remain largely outperformed by pre-training on ILSVRC’12.
Abstract
from arXiv · showhide
Two things seem to be indisputable in the contemporary deep learning discourse: 1. The categorical cross-entropy loss after softmax activation is the method of choice for classification. 2. Training a CNN classifier from scratch on small datasets does not work well. In contrast to this, we show that the cosine loss function provides significantly better performance than cross-entropy on datasets with only a handful of samples per class. For example, the accuracy achieved on the CUB-200-2011 dataset without pre-training is by 30% higher than with the cross-entropy loss. Further experiments on other popular datasets confirm our findings. Moreover, we demonstrate that integrating prior knowledge in the form of class hierarchies is straightforward with the cosine loss and improves classification performance further.
1. Introduction
The paper targets classification from small datasets without external pre-training, proposing cosine loss as an alternative to the standard cross-entropy objective. It evaluates the loss across image and text datasets and investigates dataset size and class-hierarchy knowledge.
- Small datasets are defined here as having fewer than 100 training images per class, including CUB with at most 30 images per class.
- Cosine loss maximizes similarity between network outputs and one-hot vectors for the true classes.
- The study evaluates cosine loss on five small image datasets and one text classification dataset.
- Experiments examine how cosine loss behaves across differently sized subsets of CUB, CIFAR-100, and AG News.
- The paper introduces a CUB class taxonomy and tests hierarchy granularity to assess whether semantic class relationships further improve performance.
2. Related Work
Prior work addresses limited-data learning through few-shot methods, data augmentation, domain knowledge, and alternative losses. This paper instead studies cosine loss for fully supervised classification without external data or prior knowledge, while treating semantic embeddings as complementary.
- Learning from Small Data: Few-shot learning commonly uses classes with sufficient training data to improve performance on other classes with few labeled examples.
- Learning from Small Data: The paper differs from typical few-shot learning by training classifiers from scratch on roughly 20–100 samples per class without additional pre-training data.
- Learning from Small Data: Other small-data approaches use domain-specific knowledge to enlarge training data or guide learning.
- Learning from Small Data: This work approaches limited-data classification through the loss function rather than external data or prior knowledge.
- Cosine Loss: Cosine loss has also been used for ranking and cross-modal embeddings, where representations of the same entity are brought together.
- Cosine Loss: Several metric-learning alternatives modify cross-entropy with angular margins or feature normalization, often introducing hyperparameters requiring careful tuning.
- Cosine Loss: For image retrieval, semantic class embeddings produced remarkable NAB results without pre-training, while this paper attributes the main effect to cosine loss and treats semantic embeddings as complementary.
3. Cosine Loss
The cosine loss compares normalized feature vectors with fixed class embeddings by angular similarity, offering bounded, scale-invariant optimization for small-data classification. Semantic class embeddings additionally incorporate relationships among classes, while cosine-loss regularization avoids an extra tuning hyper-parameter.
- Cosine Loss: The cosine loss learns a neural transformation that maps inputs into a feature space and compares features with fixed class embeddings in a common prediction space.
- Cosine Loss: L2 normalization restricts predictions to the unit hypersphere, where cosine similarity becomes equivalent to the dot product.Class embeddings must also lie on the unit hypersphere; one-hot vectors already have unit norm.
- Comparison with Categorical Cross-Entropy and Mean Squared Error: Cross-entropy uses softmax to represent predictions as probability distributions, whereas MSE measures squared Euclidean distance in an untransformed feature space.
- Comparison with Categorical Cross-Entropy and Mean Squared Error: The cosine loss is bounded in [0, 2] and invariant to feature-vector scaling because it depends on direction rather than magnitude.These properties distinguish it from cross-entropy and MSE, whose losses can become arbitrarily large.
- Comparison with Categorical Cross-Entropy and Mean Squared Error: L2 normalization acts as a hyper-parameter-free regularizer and lets optimization focus on feature directions rather than Euclidean distances.The paper motivates this invariance as particularly useful for small datasets.
- Semantic Class Embeddings: Semantic class embeddings place classes on the unit hypersphere so dot products represent ontology-derived semantic similarity.The approach combines cosine loss with prior knowledge about relationships among classes.
- Semantic Class Embeddings: The authors apply cosine loss with and without semantic embeddings across several small datasets to separate the contribution of prior knowledge from that of the loss function.
4. Datasets
The study covers five small image datasets, one larger image dataset used for sample-size analysis, and AG News to test whether cosine-loss benefits extend beyond vision. It also uses class taxonomies where available and introduces a CUB hierarchy for semantic-embedding experiments.
- Dataset coverage: Five small image datasets and one text dataset are used to assess cosine-loss benefits on limited data.The image datasets are CUB, NAB, Stanford Cars, Oxford Flowers, and MIT Indoor Scenes; AG News tests transfer beyond image classification.
- Bird datasets: CUB and NAB are fine-grained bird datasets, with NAB containing more images and classes and distinguishing sex and age within species.NAB comprises four times more images than CUB and almost three times more classes.
- Class hierarchies: A manually constructed CUB taxonomy organizes 200 bird species by scientific taxonomic levels from order through genus.The hierarchy supports experiments with semantic class embeddings.
- Class hierarchies: Two balanced CUB hierarchy variants are derived to study how taxonomy depth affects classification accuracy.The variants include a flat four-level hierarchy and a deeper seven-level hierarchy.
- Sample-size analysis: CIFAR-100 is sub-sampled to vary the number of training samples per class, while its WordNet-derived hierarchy supports semantic-embedding experiments.Although CIFAR-100 has 500 training images per class, sub-sampling allows comparison across smaller and larger regimes.
- Text dataset: AG News contains 120,000 training and 7,600 validation articles across four categories and is sub-sampled for different text-data sizes.The experiments use titles and descriptions from the dataset.
5. Experiments
The experiments compare cosine loss with cross-entropy and MSE across small image datasets, varying dataset size and semantic class information. Cosine loss substantially outperforms cross-entropy in small-data settings, while the difference narrows or disappears with larger datasets, and semantic information can improve results further.
- 5.1. Setup: The cosine-loss advantage is evaluated against categorical cross-entropy, MSE, label smoothing, and fine-tuning from ImageNet-pretrained weights.The study reports the best performance achieved during training rather than the final result after 372 epochs because overfitting begins at different times.
- 5.2. Performance Comparison: Cosine loss outperforms cross-entropy on all small datasets, with largest relative improvements of 30% on CUB and 21% on NAB.On Cars and Flowers-102, the relative improvements are 8% and 6%; on sufficiently large CIFAR-100, the losses perform similarly.
- 5.2. Performance Comparison: The cosine loss outperforms a 57.8% CUB result from a pre-deep-learning method using object-part annotations, without additional information.This comparison concerns a method that uses object-part annotations on the training set.
- 5.2. Performance Comparison: Semantic class embeddings improve cosine-loss accuracy by 1 percentage point on NAB and 3 percentage points on CUB.The CUB gain is smaller than the 17-percentage-point improvement over cross-entropy achieved by cosine loss alone.
- 5.3. Semantic Embeddings: As hierarchy depth increases, cosine loss alone decreases in performance while the cosine-plus-cross-entropy variant improves.With one-hot embeddings, the difference between the two variants is smallest.
- 5.4. Effect of Dataset Size: On CUB, cosine loss remains consistently more accurate than cross-entropy and improves faster as more samples are added.Semantic information is most helpful with very few samples, while the combined cosine-plus-cross-entropy variant performs slightly better in all cases.
- 5.4. Effect of Dataset Size: On CIFAR-100, cross-entropy begins to overtake cosine loss at 150–200 samples per class.Before that range, the gap favoring cosine loss is smaller than on the smallest datasets but remains noticeable and consistent.
- 5.5. Results for text classification: On AG News, cosine loss substantially outperforms cross-entropy through 100 documents per class, with relative improvements of 17% at 10 and 26% at 25.Differences are not significant for larger datasets.
6. Conclusions
The cosine loss is useful for training deep neural classifiers from scratch on limited data, outperforming cross-entropy substantially in that setting. Its advantage diminishes when training data are sufficient or models use pre-trained weights, while L2 normalization and semantic targets offer supported explanations or extensions.
- The cosine loss outperforms cross-entropy by a large margin when deep neural classifiers are trained from scratch on limited data.
- The two loss functions perform similarly when sufficient training data are available or the network starts from pre-trained weights.
- L2 normalization in the cosine loss is hypothesized to act as a strong regularizer.The hypothesis is supported by the poor performance of MSE, which mainly differs by omitting L2 normalization.
- Cosine loss performs better than cross-entropy for datasets with fewer than 200 samples per class.
- Semantic class embeddings improve classification accuracy, but their gain is small compared with the larger improvement from the cosine loss itself.
- The authors hope cosine loss will benefit applications with limited data that cannot use pre-training and encourage research on alternatives to categorical cross-entropy.