PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Unlabeled Data in CNNs by Self-supervised Learning to Rank

Xialei Liu, Joost van de Weijer, Andrew D. Bagdanov

arXiv:1902.06285v1cs.CV

TL;DR

Scarce and costly labels limit CNN regression for image quality assessment and crowd counting. The paper adds self-supervised ranking from automatically generated unlabeled-image relations, with efficient Siamese backpropagation and ranking-based active learning. Results improve significantly on both tasks, while proxy-task errors can reduce labeling effort; the approach is bounded by patch-context requirements and an ambiguously defined ranking task.

  • Problem

    CNN regression for IQA and crowd counting is constrained by scarce, expensive labeled data, including costly image-quality annotations and person-level crowd labels.

  • Method

    The paper jointly trains regression and ranking outputs using automatically ranked unlabeled data, shared parameters, and efficient backpropagation for Siamese ranking networks.

  • Results

    Significant improvements are reported for both IQA and crowd counting when unlabeled ranking data are added, with the efficient multi-task backpropagation method performing best.

  • Takeaways & Limitations

    Errors on the self-supervised ranking task provide an informativeness measure that can drive active learning and substantially reduce labeling effort.

  • Takeaways & Limitations

    IQA patch sampling requires sub-images at least 1/3 the size of the original image to capture context, limiting the use of very small patches.

Abstract

from arXiv · show

For many applications the collection of labeled data is expensive laborious. Exploitation of unlabeled data during training is thus a long pursued objective of machine learning. Self-supervised learning addresses this by positing an auxiliary task (different, but related to the supervised task) for which data is abundantly available. In this paper, we show how ranking can be used as a proxy task for some regression problems. As another contribution, we propose an efficient backpropagation technique for Siamese networks which prevents the redundant computation introduced by the multi-branch network architecture. We apply our framework to two regression problems: Image Quality Assessment (IQA) and Crowd Counting. For both we show how to automatically generate ranked image sets from unlabeled data. Our results show that networks trained to regress to the ground truth targets for labeled data and to simultaneously learn to rank unlabeled data obtain significantly better, state-of-the-art results for both IQA and crowd counting. In addition, we show that measuring network uncertainty on the self-supervised proxy task is a good measure of informativeness of unlabeled data. This can be used to drive an algorithm for active learning and we show that this reduces labeling effort by up to 50%.

1 INTRODUCTION

The paper uses self-supervised ranking on automatically ranked unlabeled images to improve CNN regression for IQA and crowd counting, while also supporting active learning.

  • Applications: IQA is challenging because image-quality datasets require multiple annotations per image, making collection labor-intensive and costly.The paper generates rankings automatically to add unlabeled images during CNN training.
  • Applications: Crowd counting also suffers from scarce labels because each person must be meticulously annotated, alongside substantial visual variability.Relevant challenges include perspective distortion, clutter, occlusion, illumination, and scale variation.
  • Motivation and framework: The framework combines a shared CNN backbone with regression and ranking outputs, using automatically generated image pairs with known relative ranks.The architecture includes problem-specific layers for the primary regression and self-supervised ranking tasks.
  • Motivation and framework: Ranking tasks provide self-supervised proxy tasks that leverage unlabeled data for regression problems with scarce labels.The approach converts unlabeled data into ranked subsets and adds a ranking loss during multi-task training.
  • Contributions: The paper proposes fast Siamese backpropagation that removes redundant multi-branch computation during ranking training.A single branch computes representations before combining possible minibatch pairs.
  • Results: Experiments report significantly improved performance over the state of the art on both IQA and crowd counting while leveraging unlabeled data.The framework also uses self-supervised ranking to support active learning and reduce labeling effort.

2 RELATED WORK

Related work covers ranking, self-supervised learning, active learning, IQA, and crowd counting, positioning this paper as a ranking-based approach to scarce-label regression.

  • Learning from rankings: Prior learning-to-rank methods learn ranking functions from ground-truth rankings, unlike this paper’s self-supervised use of automatically ranked unlabeled data.The paper’s IQA method uses many automatically generated image-quality rankings to train deeper and wider networks.
  • Self-supervised learning: Earlier self-supervised work used scaling and tiling proxy tasks to guide networks toward counting visual primitives.Those tasks exploit invariances under scaling and additive relationships across image tiles.
  • Active learning: Active learning reduces annotation cost by selecting informative unlabeled samples, but many established strategies focus on classification rather than regression.Reviewed strategies include uncertainty, margin, entropy, expected model change, and similarity-based selection.
  • Active learning: This paper instead measures informativeness for IQA and crowd counting through mistakes on a self-supervised ranking proxy task.The proposed strategy selects unlabeled instances with more proxy-task mistakes for labeling.
  • Image quality assessment: Deep NR-IQA methods are limited by the lack of large labeled datasets, while patch-based approaches augment training examples using small image regions.The paper addresses this shortage with automatically ranked image-quality data.
  • Crowd counting: Crowd-counting CNNs remain difficult to train without overfitting because labeled datasets are limited and auxiliary information can be expensive or unavailable.Existing approaches include density-map learning, side-information use, feature extractors, dilated CNNs, and attention modules.

3 LEARNING FROM RANKINGS

The paper trains regression CNNs with a self-supervised ranking task generated from unlabeled images, combining ranked and labeled data in a multi-task objective. It also reduces Siamese computation and uses ranking uncertainty to prioritize informative samples for active learning.

  • Framework: Ordered transformations generate ranked data without annotations, including distortions for IQA and nested crops for crowd counting.Increasing distortion lowers image quality, while contained image regions contain no more people than larger regions.
  • Framework: Ranking serves as a self-supervised proxy task for regression problems where ordered image data can be generated automatically.The approach converts unlabeled images into ranked subsets and trains the principal regression task alongside ranking.
  • Multi-task learning: The combined objective uses regression loss plus λ times ranking loss, with the regression and Siamese branches sharing parameters.The ranking loss is a hinge loss with margin ε, and the same network function is used for both tasks.
  • Multi-task learning: Multi-task training is preferred to ranking pretraining followed by supervised fine-tuning because the authors found the latter inferior.The paper therefore uses multi-task formulations rather than sequential pretraining and fine-tuning.
  • Efficient backpropagation: Efficient Siamese backpropagation computes each image representation once and evaluates all pairs in the loss layer, avoiding redundant branch computation.For n ranked images, the traditional implementation passes n^2 − n images through the network, while the proposed method passes all images once; expected convergence speedup is lower because pairs are correlated.
  • Active learning: Ranking-based certainty estimates support active learning by selecting uncertain unlabeled images expected to improve performance more than random additions.The certainty function orders the unlabeled pool, and the least certain samples are prioritized for labeling.

4 IMAGE QUALITY ASSESSMENT BY LEARNING TO RANK

For no-reference IQA, the paper uses ranked distortions from arbitrary images to supplement scarce human-labeled quality data. It evaluates this strategy with IQA datasets, a VGG-16-based network, and an active-learning loop.

  • Active learning: The active-learning loop trains on labeled data, evaluates certainty on the unlabeled pool, labels the least confident samples, and updates both sets.The supplied algorithm description defines iterative training and pool reduction.
  • Task and motivation: No-reference IQA predicts perceptual image quality without access to the undistorted reference image.IQA annotations commonly use Mean Opinion Scores from many human evaluators.
  • Task and motivation: IQA datasets are costly to annotate because each image requires multiple human assessments, motivating unlabeled-data augmentation.CNNs improved IQA over handcrafted approaches, but further gains are constrained by limited labeled data.
  • Datasets: LIVE contains 808 distorted images from 29 originals, whereas TID2013 contains 3000 distorted images from 25 reference images.LIVE uses five distortion types and MOS values in [0, 100]; TID2013 uses 24 distortion types and MOS values in [0, 9].
  • Ranking data: Rankings can be generated from arbitrary images by applying distortions at increasing intensities, since stronger distortions imply lower quality.This produces large ranked datasets without knowing absolute IQA scores.
  • Network and training: The IQA network uses VGG-16 with its final layer replaced to output a single quality score, while labeled examples use Euclidean regression loss.The regression loss is optimized jointly with the ranking loss.
  • Network and training: Patch sampling is constrained by context requirements: sampled inputs should be at least one-third of the original image.The experiments use 224 × 224 pixel samples from 300–700 pixel images, making very small patches a serious limitation.

5 CROWD COUNTING BY LEARNING TO RANK

The crowd-counting framework learns from labeled density maps while exploiting unlabeled crowd images by generating ranked crops and applying a ranking-specific network output.

  • Motivation: Crowd counting is highly data-driven, but collecting labels requires meticulously annotating each person, leaving benchmark datasets relatively small.The task is further complicated by perspective distortion, clutter, occlusion, illumination, and scale variation.
  • Generating ranked image sets: Ranked image sets are generated by cropping nested patches from unlabeled crowd images, because an inner patch contains no more people than its enclosing patch.Only verifying that an image contains a crowd is required before extracting ranked patches according to the dataset-generation procedure.
  • Generating ranked image sets: Unlabeled crowd images are collected through keyword queries and query-by-example retrieval, producing datasets substantially larger than the annotated benchmarks.Keyword retrieval yields 1,180 images; query-by-example retrieval yields 256 UCF CC 50, 2,229 ShanghaiTech Part A, and 3,819 Part B images.
  • Crowd density estimation network: The baseline uses an ImageNet-pretrained VGG-16 truncated before the final pooling stage, with a convolutional layer that predicts a crowd-density map.The counting loss is the Euclidean distance between predicted and ground-truth density maps, with N denoting the batch size.
  • Crowd density estimation network: Multi-scale sampling trains on randomly sized square patches, while a summation layer converts each predicted density map into a scalar for the ranking loss.The sampled patch sizes range from 56 to 448 pixels, and the summation output estimates the number of persons in the image.

6 EXPERIMENTAL RESULTS

The experiments evaluate learning from rankings for IQA and crowd counting using the Caffe framework, with an ablation study reported for IQA.

  • Experimental setup: The experiments assess the effectiveness of learning from rankings for Image Quality Assessment and crowd counting.All experiments use the Caffe deep learning framework.
  • Experimental setup: The reported ablation study covers the entire TID2013 database.The table is identified as Table 1.

6.1 Image Quality Assessment (IQA)

The IQA experiments evaluate ranking-based self-supervision, multi-task optimization, generalization to unseen distortions, state-of-the-art performance, and active learning on TID2013 and LIVE.

  • Experimental setup: The experiments compare multi-task ranking with direct fine-tuning, RankIQA, and RankIQA followed by fine-tuning on IQA benchmarks.TID2013 is split into 80% training and 20% testing images, with average SROCC reported across ten runs.
  • Ablation study: 11% improvement over the baseline came from random-sampling fine-tuning, while efficient sampling added 2.4% in LCC.The improvements were reported after fine-tuning on TID2013; similar conclusions held for SROCC.
  • Ablation study: 2.8% higher LCC and 2.6% higher SROCC than RankIQA+FT were obtained with efficient Siamese backpropagation in multi-task training.The comparison covered random pair sampling, hard negative mining, and efficient Siamese backpropagation.
  • Generalization to unseen distortions: Adding more synthetic distortions improved overall performance and also improved accuracy on unseen distortions.The experiment combined auxiliary synthetic distortions with all labeled TID2013 data.
  • Comparison with the state-of-the-art: About 5% higher SROCC than HOSA was achieved on all TID2013 distortions, with roughly 3% additional gain from multi-task training.The method also performed well on distortions for which rankings were not generated.
  • Active learning: 75% labeling-cost reduction was achieved when active learning reached similar accuracy with 20% labeled data to random selection with 50%.The active-learning experiment started with 10% labeled data and added selected distortion levels in 10% cycles.

6.2 Crowd counting

The crowd-counting experiments assess ranking-based multi-task learning, efficient backpropagation, benchmark performance, and active learning across several datasets.

  • Experimental setup: The crowd-counting network predicts a density image whose summed values estimate the number of people in a scene.MAE and MSE are used to evaluate predicted counts against ground-truth counts.
  • Ablation study: Pre-training reduced UCF CC 50 MAE by around 28%, while multi-scale augmentation added another improvement of around 6%.The ablation compared a model trained from scratch with an ImageNet-pre-trained model and augmentation.
  • Ablation study: The efficient backpropagation multi-task method reduced average MAE to 279.6, outperforming random pair sampling and hard mining.The reported reductions were about 15 points versus the baseline multi-task sampling comparison and about 5 points for hard mining versus random sampling.
  • Comparison with the state-of-the-art: MAE fell from 291.0 to 279.6 on UCF CC 50 using unlabeled Keyword Dataset images, while performance remained slightly below CSRNet and ic-CNN.The method had fewer parameters than CSRNet and a simpler inference procedure than ic-CNN, but worse MSE than state-of-the-art methods.
  • Comparison with the state-of-the-art: The approach surpassed the state-of-the-art method on ShanghaiTech Part A while remaining slightly below CSRNet and ic-CNN overall.Using the Query-by-example Dataset improved ShanghaiTech performance by about 2%.
  • Comparison with the state-of-the-art: The method outperformed the baseline in all five WorldExpo’10 cases and achieved comparable MAE to other methods.The comparison used multi-task training with ranked images generated from other scenes in the test set.
  • Comparison with the state-of-the-art: UCF-QNRF MAE decreased from 137 to 124 and MSE reached 196, with better MAE but slightly worse MSE than CompositionLoss.The improvement occurred despite UCF-QNRF having substantially more labeled training samples.

7 CONCLUSION

The paper finds that ranked unlabeled data improves regression training for IQA and crowd counting, while proxy-task errors can guide active learning to reduce labeling effort.

  • Active learning results on Shanghai A plot MAE against the percentage of labeled training data.
  • Ranking serves as a self-supervised proxy task for regression problems where ranked image sets can be obtained without annotation.
  • Efficient Siamese backpropagation removes redundant multi-branch computation by processing each image once and combining all minibatch pairs afterward.
  • Adding unlabeled ranked data significantly improves results for Image Quality Assessment and crowd counting.
  • The best results use efficient backpropagation in a multi-task setting.
  • Proxy-task errors measure unlabeled-image informativeness and can select images whose labeling most improves the network.

APPENDIX DISTORTIONS GENERATED FOR TID2013

The appendix specifies the distortion procedures used to generate TID2013 training data, while documenting several distortions that could not be reproduced exactly.

  • The TID2013 dataset provides 25 reference images, 3000 distorted images, 24 distortion types, and 5 degradation levels.
  • Training uses 17 generated distortion types, with fine-tuning from the remaining generated network for distortions that could not be reproduced.
  • Gaussian noise, color-component noise, spatially correlated noise, masked noise, high-frequency noise, impulse noise, quantization noise, and Gaussian blur use explicitly specified parameter ranges or kernels.
  • JPEG and JPEG2000 compression use specified quality factors or compression ratios, while other distortions use defined patch, shift, contrast, saturation, or quantization settings.
  • Several distortion types could not be reproduced because source details were insufficient or proprietary encoders were unavailable.
Loading 1902.06285v1…