Source-linked AI summary
An Overview on Data Representation Learning: From Traditional Feature Learning to Recent Deep Learning
Guoqiang Zhong, Li-Na Wang, Junyu Dong
TL;DR
Learning useful representations is a longstanding problem spanning traditional and deep methods. This paper reviews their historical development, algorithms, models, resources, and research directions, concluding that deep learning improves many domains while remaining concentrated in CNN- and RNN-based approaches.
Problem
The paper addresses how representation learning methods have developed across linear, nonlinear, supervised, unsupervised, traditional, and deep approaches.
Method
The paper reviews traditional feature learning and recent deep learning, including algorithms, models, resources, toolboxes, and research directions.
Results
Deep learning greatly improves progress in artificial intelligence and domains such as finance and bioinformatics.
Takeaways & Limitations
The review connects the development of representation learning with applications across artificial intelligence, finance, and bioinformatics.
Takeaways & Limitations
Most proposed deep learning algorithms are based on deep CNNs or RNNs.
Abstract
from arXiv · showhide
Since about 100 years ago, to learn the intrinsic structure of data, many representation learning approaches have been proposed, including both linear ones and nonlinear ones, supervised ones and unsupervised ones. Particularly, deep architectures are widely applied for representation learning in recent years, and have delivered top results in many tasks, such as image classification, object detection and speech recognition. In this paper, we review the development of data representation learning methods. Specifically, we investigate both traditional feature learning algorithms and state-of-the-art deep learning models. The history of data representation learning is introduced, while available resources (e.g. online course, tutorial and book information) and toolboxes are provided. Finally, we conclude this paper with remarks and some interesting research directions on data representation learning.
1. Introduction
Data representation learning has developed from early linear methods into nonlinear and deep approaches for discovering useful data structure. This paper reviews that progression and the organization, models, resources, and research directions associated with it.
- Representation learning methods have been proposed for roughly 100 years, spanning linear and nonlinear, supervised and unsupervised approaches.
- PCA and LDA represent early linear approaches, while manifold learning introduced locality-based methods for discovering intrinsic structure in high-dimensional data.
- Deep neural networks were applied to dimensionality reduction in 2006, helping establish deep learning for representation learning.
- Backpropagation can generate useful hidden-layer representations, but multilayer networks faced overfitting and gradient diffusion.
- The paper reviews traditional feature learning and recent deep learning, including algorithms, important models, public toolboxes, and future research directions.
2. Traditional feature learning
Traditional feature learning methods are organized by formulation and by whether they preserve global information or local similarity. The section surveys linear, kernel, probabilistic, discriminative, manifold, tensor, and related approaches.
- The survey excludes manual feature engineering methods such as SIFT, LBP, HOG, and TF-IDF from its feature-learning discussion.
- Global feature learning: PCA is linear, unsupervised, generative, and global, while LDA is linear, supervised, discriminative, and global.
- Traditional feature learning is categorized by linearity, supervision, generative or discriminative formulation, and global or local information preservation.
- Global methods preserve global data information, whereas local methods preserve local similarity and are commonly associated with manifold learning.
- Global feature learning: Extensions include kernel PCA, probabilistic PCA, GPLVM, discriminative GPLVM, sparse and robust PCA, and tensor representation methods.
- Global feature learning: LDA brings same-class samples closer and different-class samples farther apart in a learned low-dimensional subspace, and has been used for face recognition.
- Manifold learning: Isomap estimates pairwise distances from local neighborhoods before applying classical MDS, while LLE encodes locality through neighbor reconstruction weights.
3. Deep learning
The section surveys deep learning’s development, models, applications, learning strategies, and supporting resources. It highlights reported effectiveness across recognition, retrieval, and image-classification tasks.
- Research overview: Deep learning research is covered through survey papers on feature learning, model development, and the year-by-year history of artificial neural networks.The cited surveys discuss representation learning, convolutional and recurrent models, and deep learning’s historical development.
- Deep learning models: Greedy layer-wise pre-training and fine-tuning achieved higher performance than state-of-the-art algorithms on MNIST recognition and document retrieval.Stacked autoencoders subsequently supported the greedy layer-wise unsupervised training strategy.
- Deep learning models: Deep architectures include denoising autoencoders, stacked feature-learning modules, deep belief networks, and convolutional networks.Examples include modules built with PCA or SNE and locally trained stacked denoising autoencoders.
- Feature transfer: Deep convolutional activations could be repurposed for novel generic tasks, while later operations improved DeCAF on several image-classification problems.DeCAF was also used to establish baseline results for photographed document-image problems.
4. Conclusion
The paper reviews representation learning from traditional feature learning through recent deep learning, emphasizing that deep learning builds on earlier progress while advancing applications across domains. It identifies continuing gaps between theory and application and points to future work in theory, algorithms, and applications.
- The review covers data representation learning across traditional feature learning and recent deep learning.
- Deep learning is presented as a consequence of progress in feature learning research, large-scale labeled data, and hardware.
- Deep learning has improved progress beyond artificial intelligence, including in finance and bioinformatics.
- Future research directions include fundamental theory, novel algorithms, and applications, alongside more sophisticated models for challenging problems.
- The paper notes a substantial gap between deep-learning theory and application, with many algorithms relying on deep CNNs or RNNs.
- The conclusion stresses that deep learning is not the whole of machine learning or the only way to realize artificial intelligence, and traditional approaches remain necessary.