Source-linked AI summary
Deep Co-Training for Semi-Supervised Image Recognition
Siyuan Qiao, Wei Shen, Zhishuai Zhang, Bo Wang, Alan Yuille
TL;DR
Semi-supervised image recognition aims to improve classifiers by using unlabeled images alongside labeled data, while deep-network implementations of Co-Training can collapse into identical views. Deep Co-Training uses multiple networks and adversarial examples to preserve view differences, and it outperforms previous state-of-the-art methods by a large margin across several image-recognition benchmarks.
Problem
Semi-supervised image recognition asks how to use unlabeled images U to improve classifiers learned from labeled images S.
Method
Deep Co-Training models multiple deep networks as views, minimizes prediction divergence on unlabeled data, and uses adversarial examples to enforce view differences.
Results
Deep Co-Training outperforms previous state-of-the-art methods by a large margin on SVHN, CIFAR-10/100, and ImageNet.
Takeaways & Limitations
The co-trained networks provide different and complementary information, supporting Co-Training for semi-supervised image recognition.
Takeaways & Limitations
The approach relies on Co-Training assumptions requiring consistent predictions across views and a view-difference constraint.
Abstract
from arXiv · showhide
In this paper, we study the problem of semi-supervised image recognition, which is to learn classifiers using both labeled and unlabeled images. We present Deep Co-Training, a deep learning based method inspired by the Co-Training framework. The original Co-Training learns two classifiers on two views which are data from different sources that describe the same instances. To extend this concept to deep learning, Deep Co-Training trains multiple deep neural networks to be the different views and exploits adversarial examples to encourage view difference, in order to prevent the networks from collapsing into each other. As a result, the co-trained networks provide different and complementary information about the data, which is necessary for the Co-Training framework to achieve good results. We test our method on SVHN, CIFAR-10/100 and ImageNet datasets, and our method outperforms the previous state-of-the-art methods by a large margin.
1 Introduction
Semi-supervised image recognition seeks to use unlabeled images alongside labeled data, but extending Co-Training to deep networks requires preserving distinct, complementary views. Deep Co-Training addresses this with adversarially enforced view differences and achieves strong benchmark performance.
- Problem: Semi-supervised image recognition uses unlabeled images in addition to labeled images to build better classifiers.The dataset contains labeled S and unlabeled U, with the core problem being how to use U to improve learning on S.
- Co-Training: Co-Training assumes two sufficient views of each instance whose classifiers make consistent predictions on the data distribution.The views may come from different data sources or representations, and the classifiers can iteratively add predictions on U to the labeled set.
- Challenge: Naively training two deep networks with consistent predictions can make them collapse into identical models, eliminating the different and complementary information Co-Training requires.Co-Training is beneficial only when the views differ, ideally conditionally independently given the category.
- Method: The view difference constraint requires examples on which the networks make different predictions, preventing collapse and encouraging complementary information.The constraint is expressed as the existence of a distribution X′ where the two network predictions differ.
- Method: Deep Co-Training minimizes Jensen-Shannon divergence on unlabeled data while training each network to resist adversarial examples generated from the other network.This jointly models prediction consistency and the view difference constraint in an end-to-end loss over labeled and unlabeled data.
- Results: Deep Co-Training outperforms previous state-of-the-art methods by a large margin on SVHN, CIFAR-10/100, and ImageNet.The method is extended from dual-view to scalable multi-view training for these datasets.
2 Deep Co-Training
Deep Co-Training combines supervised learning, agreement on unlabeled data, and adversarially induced view differences to train complementary deep networks. It extends this dual-view procedure to scalable multi-view training and evaluates it across several image datasets.
- Dual-View DCT: DCT trains two deep networks whose convolutional representations serve as views, using labeled examples with standard cross-entropy supervision.Each representation precedes its network’s final fully connected classifier.
- Co-Training Assumption: The co-training loss minimizes Jensen-Shannon divergence between the networks’ predictions on unlabeled data, while agreement on labeled data is already encouraged by supervision.The co-training loss is applied to U rather than S.
- View Difference Constraint: Because agreement alone can make networks collapse, DCT generates adversarial examples that separate predictions and trains each network to resist the other network’s attacks.The view-difference loss uses cross-entropy between one network’s original prediction and the other network’s adversarial-example prediction.
- Training DCT: The objective linearly combines supervised, co-training, and view-difference losses, weighted by λcot and λdif, and optimizes them with repeated gradient-descent iterations.The training loop samples labeled batches for each network, a shared unlabeled batch, and adversarial examples.
- Multi-View DCT: Multi-view DCT scales the dual-view design by pairing views through data-stream bundles and randomly reshuffling view pairs across iterations.This construction reuses dual-view hyperparameters for different numbers of views and supports direct comparison of training dynamics.
- Experiments: DCT is evaluated using a prior training and evaluation framework on SVHN, CIFAR-10/100, and ImageNet, with SVHN using 1,000 labeled images and the remaining training images as unlabeled data.The reported SVHN experiments use only random translations of at most 2 pixels for augmentation.
3 Results
Deep Co-Training is evaluated on SVHN, CIFAR-10, CIFAR-100, and ImageNet, including comparisons with prior methods and ablations of view count and loss components. It improves reported performance across these settings, with especially clear gains on harder datasets and ImageNet comparisons.
- SVHN and CIFAR-10: Deep Co-Training achieves significant improvements over prior state-of-the-art methods on SVHN and CIFAR-10 with 2, 4, or 8 views.Results report averages of single-model error rates without ensembling, and dual-view training does not require more total networks than competing methods.
- SVHN and CIFAR-10: Increasing views from 2 to 4 improves performance, whereas increasing from 4 to 8 yields smaller gains but faster convergence.The authors attribute the stronger 4-view gain partly to majority voting when encouraging agreement on unlabeled data.
- CIFAR-100 and ImageNet: 38.65 to 34.63: adding data augmentation substantially lowers CIFAR-100 error for dual-view Deep Co-Training.Without augmentation, dual-view Deep Co-Training performs similarly to prior state-of-the-art methods that use augmentation; with augmentation, its error drops significantly.
- Ablation studies: The ablation study compares training dynamics using Lcot alone, Ldif alone, or their weighted sum, with Lsup included in every case.The plots cover dual-view training on SVHN and CIFAR-10.
- CIFAR-100 and ImageNet: ∼2.6% and ∼0.9%: DCT outperforms Mean Teacher with the same architecture on ImageNet Top-1 and Top-5 error rates, respectively.DCT also reports lower error rates than methods using more parameters and larger 256 × 256 inputs.
On Lcot and Ldif
The ablations show that modeling Co-Training alone can cause networks to collapse, while minimizing Ldif preserves view difference and supports accuracy. More views speed convergence, but gains diminish beyond four views.
- Ldif ablation: Around the 400th epoch on SVHN, a sudden increase in Ldif accompanies model collapse and a significant validation-accuracy drop.These dynamics link loss abnormalities with the observed collapse.
- Ldif ablation: Without minimizing Ldif, the two models make similar errors and become vulnerable to each other’s adversarial examples.The collapse is especially apparent in the Lcot-only setting.
- Number of views: Increasing views from 2 to 4 improves each individual model’s performance, while increasing them to 8 yields smaller gains.The experiments compare 2-, 4-, and 8-view training on SVHN and CIFAR-10.
- Number of views: More views produce faster convergence, but the improvement from 4 to 8 views is smaller than from 2 to 4.The clearest differences appear between epochs 100 and 200.
- Number of views: The experiments suggest that 4-view Deep Co-Training balances performance and computational efficiency.This conclusion reflects the diminishing performance and convergence benefits of adding views.
4 Discussions
The discussion relates Deep Co-Training to semi-supervised learning, ensembling, multi-agent learning, and knowledge distillation. It emphasizes that cooperative prediction agreement alone is insufficient without maintaining different views.
- Co-Training and semi-supervised learning: Deep Co-Training extends Co-Training’s complementary-view idea to deep networks, where independently identifiable views are not naturally guaranteed.Co-Training benefits only when views provide different and complementary information.
- Co-Training and semi-supervised learning: Training only for prediction agreement is insufficient because the networks can collapse into each other, violating the view-difference requirement.Deep Co-Training addresses this with a force that pushes the models apart.
- Model Ensemble: Unlike conventional ensembling, multi-view Deep Co-Training trains models jointly and evaluates one model rather than averaging predicted probabilities.The discussion characterizes DCT as seeking an initialization-free and data-order-free solution.
- Multi-Agent Learning: Deep Co-Training is framed as cooperative multi-agent learning in which multiple agents interact through their predictions and errors.This contrasts with adversarial interactions in GAN-based methods and cooperative interactions in noise-resistance methods.
- Knowledge Distillation: All Deep Co-Training models learn from other models’ predictions as well as from supervised dataset labels, resembling knowledge distillation.Unlike standard teacher–student framing, all models act as students and also learn from peers’ errors.
5 Conclusion
Deep Co-Training extends Co-Training to deep semi-supervised image recognition by using adversarial examples to preserve different views. Experiments report significant accuracy gains, scalable multi-view training, and strong results across several benchmarks.
- Conclusion: Deep Co-Training uses adversarial examples to generate data on which views make different predictions, preventing networks from collapsing.This adds the view-difference force required by the Co-Training framework.
- Conclusion: The experiments show that adding this force improves accuracies significantly over Co-Training-only modeling.The conclusion attributes the improvement to preventing model collapse.
- Conclusion: Deep Co-Training scales from dual-view to multi-view training, with the two-view hyperparameters remaining suitable as the number of views increases.The reported configurations include 2, 4, and 8 views.
- Conclusion: On SVHN and CIFAR-10, the method outperforms previous state-of-the-art methods by a large margin with 2, 4, and 8 views.Additional results are provided on CIFAR-100 and ImageNet, where most previous methods had not reported performance.