Source-linked AI summary
Beyond One-hot Encoding: lower dimensional target embedding
Pau Rodríguez, Miguel A. Bautista, Jordi Gonzàlez, Sergio Escalera
TL;DR
One-hot encoding is simple and prevalent but can inadequately represent correlated labels in large output spaces. The paper integrates ECOCs into CNNs, using random projections and normalized eigenrepresentations for compact target embeddings. Across several image datasets, the approach reports faster convergence with competitive accuracy, while data-dependent codes further reduce error rates.
Problem
One-hot encoding may be inadequate for large, fine-grained label spaces because correlated classes do not span the full orthogonal output space and large outputs slow CNN training.
Method
The paper integrates ECOCs into CNNs, using random label-space projections and normalized Laplacian eigenrepresentations to create compact target embeddings.
Results
Across CIFAR-100, CUB200-2011, ImageNet, and MIT Places, the approach reports faster convergence with competitive accuracy; data-dependent codes achieve lower error rates than random codes.
Takeaways & Limitations
Compact ECOC targets yield less sparse output neurons, more robust gradient estimates, faster convergence, and smaller-minibatch training with lower memory consumption.
Takeaways & Limitations
The preferred embedding depends on the task: prior-information methods need suitable attributes or taxonomies, while data-independent methods suit known equidistant label spaces.
Abstract
from arXiv · showhide
Target encoding plays a central role when learning Convolutional Neural Networks. In this realm, One-hot encoding is the most prevalent strategy due to its simplicity. However, this so widespread encoding schema assumes a flat label space, thus ignoring rich relationships existing among labels that can be exploited during training. In large-scale datasets, data does not span the full label space, but instead lies in a low-dimensional output manifold. Following this observation, we embed the targets into a low-dimensional space, drastically improving convergence speed while preserving accuracy. Our contribution is two fold: (i) We show that random projections of the label space are a valid tool to find such lower dimensional embeddings, boosting dramatically convergence rates at zero computational cost; and (ii) we propose a normalized eigenrepresentation of the class manifold that encodes the targets with minimal information loss, improving the accuracy of random projections encoding while enjoying the same convergence rates. Experiments on CIFAR-100, CUB200-2011, Imagenet, and MIT Places demonstrate that the proposed approach drastically improves convergence speed while reaching very competitive accuracy rates.
1 Introduction
One-hot encoding is prevalent in CNN classification but becomes inadequate for large, correlated label spaces, motivating compact output embeddings. The paper integrates ECOCs into CNNs, using random and data-dependent codes to improve convergence while preserving competitive accuracy.
- Motivation: One-hot encoding can be inadequate for fine-grained classification because correlated labels do not span the full orthogonal output space.Higher-dimensional outputs increase model parameters, while fewer samples per label make training large-output CNNs slower.
- Existing alternatives: Output embeddings capture latent relationships among categories through weight sharing, but existing choices can require expert knowledge, taxonomies, or substantial training data.Data-independent, prior-based, and learned embeddings each impose different assumptions or costs.
- Proposed approach: The paper integrates Error-Correcting Output Codes into CNNs as a generalization of output embedding, providing compact outputs, data-based hierarchies, and error correction.ECOCs also support low-dimensional embeddings and bias-variance error reduction.
- Contributions: Random label-space projections provide useful lower-dimensional embeddings with dramatically faster convergence at zero computational cost.The paper presents this as the first part of a two-fold embedding contribution.
- Contributions: A normalized eigenrepresentation of the class manifold produces more discriminative partitions with minimal information loss, improving random-code accuracy at the same convergence rates.Experiments on CIFAR-100, CUB200-2011, MIT Places, and ImageNet report faster convergence with competitive accuracy.
2 Related work
Related work uses output embeddings and ECOCs to reduce or structure large multi-class output spaces. This paper integrates ECOC decoding with CNN training, targeting faster convergence and error correction without changing the chosen architecture for different code designs.
- Output embeddings: Output embeddings support large-scale multi-class recognition by exploiting hierarchical structure and divide-and-conquer computation.They are useful when testing all class labels or hierarchical structures directly is infeasible.
- Output embeddings: Output embeddings enforce weight sharing, making them useful when classes are numerous, boundaries are unclear, and examples per class decline.WSABIE is one example of learning label embeddings from images.
- Output embeddings: Prior hierarchical approaches and latent-code methods can be difficult to interchange with other embeddings, while learned codes may require finding class-separable representations.This motivates methods that reduce output size without binding the architecture to one embedding choice.
- ECOCs: ECOCs combine binary partitions for multi-class recognition and can correct classifier errors through redundancy while reducing ensemble bias and variance.Examples include exhaustive, BCH, random, circular, and data-learned codes.
- ECOCs: ECOCs assign each class a codeword whose columns represent binary partitions, with decoding based on distances to the predicted code.The code design determines how discriminative the resulting partitions are.
- CNN integration: The proposed CNN integration combines Euclidean decoding with cross-entropy, optimizing intercode distances rather than individual code bits alone.This is intended to provide error correction while improving convergence in large-scale settings.
- CNN integration: Random codes improve CNN convergence, while data-dependent codes achieve lower error rates and halve the required CNN output size without training multiple code-chunk networks.The comparison is made against data-independent Hadamard-code approaches.
3 Low dimensional target embedding
The method replaces the CNN’s n-dimensional output with k encoded partitions, decodes predictions through distances to class codes, and trains with cross-entropy. This reduction improves convergence by sharing parameters and producing more robust gradient estimates, while data-based eigenrepresentations preserve class similarities and reduce error rates.
- 3.1 Embedding output codes in CNNs: The CNN output dimensionality is reduced from n classes to k partitions, with each design-matrix row encoding a class label for distance-based decoding.The predicted class is the row minimizing the distance between the network output and the design matrix.
- 3.1 Embedding output codes in CNNs: The approach trains CNNs with cross-entropy and backpropagation after applying the output-code decoder to the network predictions.Cross-entropy is applied to probability distributions produced by the softmax stage.
- 3.1 Embedding output codes in CNNs: The decoder is inserted between the network output and softmax, using negative normalized Euclidean distances to produce class probabilities.This integrates ECOC decoding into the standard CNN classification pipeline.
- 3.1 Embedding output codes in CNNs: Reducing the output space increases parameter sharing and produces more robust gradient estimates, improving convergence speed without slowing training.The smaller output also reduces the number of network parameters, compensating for the added decoding computation.
- 3.2 Connections with Normalized Cuts: A normalized-Laplacian eigendecomposition yields class partitions ordered by normalized-cut cost, with similar classes receiving similar codes.The eigenvectors are interpreted as likelihoods rather than thresholded into binary codes.
- Experiments: Experiments on CIFAR-100, CUB-200, MIT Places, and ImageNet report faster convergence with comparable or better recognition rates than one-hot CNNs.The comparison also holds with smaller mini-batch sizes.
4 Experiments
Experiments across large-scale and fine-grained datasets evaluate compact output encodings against one-hot baselines. Random codes accelerate convergence, while data-dependent codes improve accuracy and remain robust when code bits are reduced.
- Methods and evaluation: Experiments use ImageNet, MIT Places, CIFAR-100, and CUB-200-2011 with standard CNN models to evaluate target embeddings.The study retrains AlexNet layers on ImageNet and MIT Places, then evaluates data-dependent codes on CIFAR-100 and CUB-200-2011.
- Random codes for faster convergence: Random output codes converge faster than one-hot encoding, especially with small mini-batches, while matching one-hot performance with larger mini-batches.This pattern is reported for AlexNet experiments on ImageNet and MIT Places.
- Random codes for faster convergence: Output encodings require only k = 10log(n) bits under the dense random strategy, compared with one-hot representations whose output activation sparsity worsens as class count grows.For one-hot encoding, the probability assigned to each class is 1/n under a uniform distribution, tending toward zero as n increases.
- Using data-based encodings: Data-dependent codes generated from normalized-Laplacian eigenvectors produce higher accuracy bounds than random codes for fine-grained settings with correlated classes and few examples per class.The method uses class-similarity structure to construct discriminative hierarchical partitions.
- Using data-based encodings: The data-based code matches one-hot accuracy using 25% of the bits, while dense output encodings lose less accuracy than one-hot when code bits are halved.On CIFAR-100, data-based codes outperform other encodings, particularly without thresholding before binarization.
5 Conclusion
Across CIFAR-100, CUB200, Imagenet, and MIT Places, the proposed output codes produce less sparse outputs, more robust gradient estimates, and faster convergence than one-hot encoding. Data-dependent eigenrepresentations further achieve lower error rates than data-independent codes.
- CNNs trained with the proposed approach on four datasets show less sparsity at the output neurons.
- For Boat tailed Grackle and Fish Crow, data-dependent codes group aquatic and non-aquatic birds before classification, unlike direct one-hot labels or random partitions.
- The approach provides more robust gradient estimates and faster convergence than one-hot encoding, while allowing smaller minibatches that reduce memory consumption.These benefits are especially noted for huge label spaces and come at a small cost.
- Data-dependent eigenrepresentations achieve lower error rates than data-independent output codes by producing more efficient hierarchical representations.