Source-linked AI summary
EXPANSE: A Deep Continual / Progressive Learning System for Deep Transfer Learning
Mohammadreza Iman, John A. Miller, Khaled Rasheed, Robert M. Branch, Hamid R. Arabnia
TL;DR
Current deep transfer learning methods trade catastrophic forgetting against overly biased frozen models, limiting continual learning and adaptation to distant source-target domains. EXPANSE addresses this with bidirectional model expansion and two-step training, while evaluations report improved transfer performance and continued validity on source data.
Problem
Deep transfer learning methods suffer either catastrophic forgetting or overly biased frozen models, preventing continual learning while adapting to target data.
Method
EXPANSE freezes the pre-trained model while adding nodes to existing layers and, when needed, new layers, and trains with exemplary data before full-data finetuning.
Results
EXPANSE improved the same model’s MNIST accuracy from 98.04% to 98.09% and printed-digit accuracy from 55.56% to 100%, while the final model remained valid on source data.
Takeaways & Limitations
EXPANSE offers a systematic continual-learning approach for deep transfer learning applicable to different problems and deep-learning models.
Takeaways & Limitations
Adding new nodes to frozen layers requires structural changes to the model’s core, including the ability to freeze part of a layer.
Abstract
from arXiv · showhide
Deep transfer learning techniques try to tackle the limitations of deep learning, the dependency on extensive training data and the training costs, by reusing obtained knowledge. However, the current DTL techniques suffer from either catastrophic forgetting dilemma (losing the previously obtained knowledge) or overly biased pre-trained models (harder to adapt to target data) in finetuning pre-trained models or freezing a part of the pre-trained model, respectively. Progressive learning, a sub-category of DTL, reduces the effect of the overly biased model in the case of freezing earlier layers by adding a new layer to the end of a frozen pre-trained model. Even though it has been successful in many cases, it cannot yet handle distant source and target data. We propose a new continual/progressive learning approach for deep transfer learning to tackle these limitations. To avoid both catastrophic forgetting and overly biased-model problems, we expand the pre-trained model by expanding pre-trained layers (adding new nodes to each layer) in the model instead of only adding new layers. Hence the method is named EXPANSE. Our experimental results confirm that we can tackle distant source and target data using this technique. At the same time, the final model is still valid on the source data, achieving a promising deep continual learning approach. Moreover, we offer a new way of training deep learning models inspired by the human education system. We termed this two-step training: learning basics first, then adding complexities and uncertainties. The evaluation implies that the two-step training extracts more meaningful features and a finer basin on the error surface since it can achieve better accuracy in comparison to regular training. EXPANSE (model expansion and two-step training) is a systematic continual learning approach applicable to different problems and DL models.
1 Introduction
Deep transfer learning reuses source-domain knowledge for target-domain learning, but existing approaches can forget prior knowledge or remain biased toward the source. EXPANSE addresses these tensions through model expansion and a two-step training process inspired by human learning.
- Deep transfer learning reuses obtained source-domain knowledge to facilitate learning on target-domain data.
- Existing DTL approaches either risk catastrophic forgetting during finetuning or retain an overly source-biased model when layers are frozen.Catastrophic forgetting can wipe out previously obtained knowledge, while frozen layers may hinder adaptation to target data.
- Progressive learning freezes the source model and adds layers near the output, but frozen earlier layers limit learning of detailed features from distant target data.Its lateral layers support task transfer, yet the approach remains biased toward source data.
- Human learning motivates starting with simplified fundamentals before progressively handling more complex and ambiguous problems.The paper uses this progression as the basis for two-step training.
- EXPANSE expands pre-trained layers by adding nodes, allowing the final model to handle target data while remaining valid on source data.The approach also permits horizontal expansion through additional layers when needed.
2 Related work, problem, and motivation
Related DTL methods trade off preservation of source knowledge against adaptation to target data. The reviewed approaches motivate expanding both existing layers and the network structure to support more distant transfers.
- Learning-rate research has not, according to the paper, examined adjusting the rate to training-data quality in deep transfer learning.Learning rate controls gradient-descent step size and can be too small or too large for effective basin exploration.
- Finetuning can cause catastrophic forgetting because target training may drastically change weights throughout the pre-trained model.This approach is more likely to succeed when source and target data are sufficiently similar.
- Freezing pre-trained CNN layers preserves their feature extraction but relies on broad source training data to contain target-relevant features.Robust models trained on large datasets such as ImageNet are commonly used for this purpose.
- Progressive learning freezes the whole pre-trained model and adds output-side layers, improving transfer for somewhat less-related data but retaining source bias.The approach remains limited when the target contains substantially more features than the source.
- Prior CNN expansion work found that adding units to some layers offered a slight but consistent accuracy benefit over adding new layers.That work did not freeze pre-trained components and was limited to selected lateral CNN layers.
3 EXPANSE
EXPANSE combines two-step training with expansion of frozen pre-trained models in both vertical and horizontal directions. The design increases learning capacity for detailed and abstract target features while preserving source validity and supporting repeated transfers.
- EXPANSE divides training into simpler steps, progressively expanding the model and training samples in a human-education-inspired process.
- Two-step training first uses limited exemplary samples at a higher learning rate, then finetunes on all training data at a lower rate.The intended mechanism is broad movement toward meaningful features followed by finer local-minimum exploration.
- Adding nodes to earlier layers increases capacity for detailed target features that frozen source layers cannot otherwise learn.Adding output-side layers supports more complex abstract knowledge when the transfer objective changes substantially.
- EXPANSE keeps the pre-trained section frozen while adding nodes to existing layers and, when needed, adding new layers.This combination supports target adaptation without replacing the previously trained section.
- The expanded model remains valid on source data and can be expanded again for subsequent target-training steps.
- Expansion size has no exact formula and is selected using experience and analysis of task and data differences.Earlier layers should expand for additional detailed features, while middle layers should expand for greater abstract-knowledge extraction.
- EXPANSE is presented as a methodology intended for different problems and deep-learning models rather than one specific case.
4 Evaluation
Evaluation on MNIST and Fashion MNIST assessed two-step training, model expansion, continual learning, and applicability to distant source and target data. EXPANSE improved accuracy while retaining source knowledge, but its current implementation cannot freeze pre-trained portions within expanded layers.
- Limitations: A major implementation limitation is that adding trainable nodes to frozen layers requires structural changes unsupported by current deep-learning libraries.The current evaluation therefore finetunes the expanded model rather than freezing the pre-trained section.
- Two-step training: Two-step training first uses limited perfected or exemplary samples, then finetunes on the complete training data at a lower learning rate.The evaluation separately compared this procedure with regular training and then assessed the combined EXPANSE methodology.
- Two-step training: 57.22% was the best printed-digit accuracy after training on 60,000 handwritten samples, whereas two-step training found a basin supporting both data types.Adding perfected samples in one-step training improved results but remained less accurate than two-step training.
- EXPANSE evaluation: 98.09% on MNIST test data and 100% on printed digits improved from 98.04% and 55.56%, respectively, for the same model configuration after EXPANSE.These results were reported for the expanded continual-learning process.
- EXPANSE evaluation: EXPANSE handled distant datasets in which digits 0–4 and 5–9 did not share the same visual features.The authors treated these source and target groups as distant data in the evaluation.
- EXPANSE evaluation: Using only 10% of source data during target training produced 96.55% MNIST accuracy without a dramatic loss of previously obtained source knowledge.The authors attribute reduced catastrophic forgetting to vertically expanding the model, while noting that freezing the pre-trained section remains future work.
- EXPANSE evaluation: Fashion MNIST results were consistent with MNIST: both two-step training and model expansion achieved better accuracy, and 10% source-data reuse did not drastically reduce accuracy.This extended the evaluation beyond handwritten-digit transfer.
5 Conclusion
EXPANSE combines model expansion with two-step training to address catastrophic forgetting and overly biased models in deep transfer learning. The evaluation reports effective transfer while preserving source-data validity and finds two-step training effective compared with traditional training.
- Two-step training first uses limited exemplary data, then finetunes with training data to address uncertainty and complexity.
- EXPANSE increases model learning capacity through vertical expansion by adding nodes to pre-trained layers.
- The methodology addresses catastrophic forgetting and overly biased models while keeping the final model valid on source data.
Ethics
The paper reports no direct societal impacts, while emphasizing that AI advances can raise difficult-to-predict ethical issues. It calls for collaboration with other disciplines to consider AI’s effects on society.
- The work reports no direct societal impacts.
- The paper notes that scientific advances can introduce ethical issues that are unavoidable and difficult to predict.
- It urges data scientists and the AI community to partner with social and behavioral sciences when considering AI’s societal effects.