PaperlayerView exact revision on arXiv

Source-linked AI summary

BoxCars: Improving Fine-Grained Recognition of Vehicles using 3-D Bounding Boxes in Traffic Surveillance

Jakub Sochor, Jakub Špaňhel, Adam Herout

arXiv:1703.00686v3cs.CV

TL;DR

Vehicle fine-grained recognition methods are often limited to frontal or rear views or require 3D vehicle models, while surveillance cameras can capture arbitrary viewpoints. The method unpacks vehicle images using 3D bounding boxes, estimates boxes when needed, and augments CNN inputs with color changes and random-noise rectangles. The method improves accuracy by up to +12 percentage points and reduces error by up to 50% versus base CNNs, and outperforms state-of-the-art methods – by 9 percentage points in single image accuracy and 7 percentage points in whole track accuracy.

  • Problem

    Vehicle fine-grained recognition methods are often limited to frontal or rear views or require 3D vehicle models, while surveillance cameras can capture arbitrary viewpoints.

    Source: §I · p. 1

  • Method

    The method unpacks vehicle images using 3D bounding boxes, estimates boxes when needed, and augments CNN inputs with color changes and random-noise rectangles.

    Source: §I · p. 2, A. Image Normalization by Unpacking… · p. 3

  • Results

    The method improves accuracy by up to +12 percentage points and reduces error by up to 50% versus base CNNs, and outperforms state-of-the-art methods – by 9 percentage points in single image accuracy and 7 percentage points in whole track accuracy.

    Source: §VI · p. 10

  • Takeaways & Limitations

    BoxCars116k is a publicly available dataset for fine-grained vehicle recognition in the surveillance domain, featuring greatly varying viewpoints that correspond to surveillance scenarios.

    Source: §VI · p. 10

Abstract

from arXiv · show

In this paper, we focus on fine-grained recognition of vehicles mainly in traffic surveillance applications. We propose an approach that is orthogonal to recent advancements in fine-grained recognition (automatic part discovery and bilinear pooling). In addition, in contrast to other methods focused on fine-grained recognition of vehicles, we do not limit ourselves to a frontal/rear viewpoint, but allow the vehicles to be seen from any viewpoint. Our approach is based on 3-D bounding boxes built around the vehicles. The bounding box can be automatically constructed from traffic surveillance data. For scenarios where it is not possible to use precise construction, we propose a method for an estimation of the 3-D bounding box. The 3-D bounding box is used to normalize the image viewpoint by "unpacking" the image into a plane. We also propose to randomly alter the color of the image and add a rectangle with random noise to a random position in the image during the training of convolutional neural networks (CNNs). We have collected a large fine-grained vehicle data set BoxCars116k, with 116k images of vehicles from various viewpoints taken by numerous surveillance cameras. We performed a number of experiments, which show that our proposed method significantly improves CNN classification accuracy (the accuracy is increased by up to 12% points and the error is reduced by up to 50% compared with CNNs without the proposed modifications). We also show that our method outperforms the state-of-the-art methods for fine-grained recognition.

I. INTRODUCTION

The paper targets fine-grained vehicle recognition in traffic surveillance, where camera viewpoints are not known in advance and images can be small and taken from high elevations. It combines 3D-box-based image normalization with input modifications and estimates boxes when they are not already available.

I. INTRODUCTION

  • The method uses 3D bounding boxes to normalize vehicle images and is presented as orthogonal to recent fine-grained methods, so it can be combined with them.Source: §I · p. 1
  • Input modifications increase classification accuracy by up to 12 percentage points and reduce classification error by up to 50%.Source: §I · p. 2
  • The authors add novel data augmentation techniques and expand the BoxCars dataset to almost twice its previous size.Source: §I · p. 2
  • The authors estimate 3D bounding boxes at training and test time, removing the previous method’s requirement to know the box in advance.Source: §I · p. 2

II. RELATED WORK

The related work surveys general fine-grained recognition approaches and vehicle-specific methods. It frames the latter as constrained by viewpoint or reliance on vehicle 3D models.

II. RELATED WORK

1) Automatic Part Discovery:

  • Automatic part-discovery methods address the annotation burden by finding and localizing discriminative parts during training and testing, with features commonly classified by SVMs.Source: 1) Automatic Part Discovery · p. 2

2) Methods using Bilinear Pooling:

  • Bilinear-pooling approaches classify features extracted from convolutional layers; compact bilinear pooling is reported to match full bilinear pooling with fewer features.Source: 2) Methods using Bilinear Pooling · p. 2

3) Other Methods:

  • Other general approaches exploit label structure, joint localization and alignment, image responses, pose normalization, segmentation, or object proposals.Source: 3) Other Methods · p. 2

B. Fine-Grained Recognition of Vehicles

1) Methods Limited to Frontal/Rear Images of Vehicles:

Vehicle-recognition methods include systems built for frontal or rear views and methods that use 3D CAD models. The paper summarizes these constraints alongside other approaches and its arbitrary-viewpoint alternative.

1) Methods Limited to Frontal/Rear Images of Vehicles:

2) Methods based on 3D CAD Models:

  • CAD-based approaches fit or align 3D models, train geometry and viewpoint classifiers, or match image features against models viewed from discretized viewpoints.Source: 2) Methods based on 3D CAD Models · p. 2

3) Other Methods:

  • Other approaches use viewpoint-specific shape alignment, HOG/DPM recognition, re-identification or night features, and unsupervised parts or spatial pooling.Source: 3) Other Methods · pp. 2–3

4) Summary of Existing Methods:

  • The proposed method accepts arbitrary viewpoints and requires 3D bounding boxes, which can come from surveillance video or be estimated from single images at training and test time.Source: 4) Summary of Existing Methods · p. 3

C. Datasets for Fine-Grained Recognition of Vehicles

Existing fine-grained vehicle datasets are limited in sample or class count, while the proposed approach uses 3D bounding boxes to normalize vehicle images for CNN recognition.

C. Datasets for Fine-Grained Recognition of Vehicles

  • Fine-grained vehicle datasets with make-and-model labels are limited in samples or classes, making them impractical for CNN training and real-world traffic-surveillance deployment.Source: C. Datasets for Fine-Grained… · p. 3

D. Vehicle Detection

  • Vehicle detection can use CNN, AdaBoost, deformable-part, or Hough-transform detectors, with other work addressing geometry, night conditions, and occlusion.Source: D. Vehicle Detection · p. 3

III. PROPOSED METHODOLOGY FOR FINE-GRAINED RECOGNITION OF VEHICLES

  • The method uses CNNs for classification and verification, modifying inputs with 3D-box information and estimating boxes when they are unavailable.Source: §III · p. 3

A. Image Normalization by Unpacking the 3D Bounding Box

  • The 3D box identifies vehicle surfaces, which are perspective-warped into adjacent rectified front/rear, side, and roof regions for CNN input.This Unpack normalization localizes vehicle parts without a separate part-localization algorithm.Source: A. Image Normalization by Unpacking… · p. 3, Figure 3 · p. 4

B. Extended Input to the Neural Nets

The CNN receives 3D-box information as viewpoint vectors and a rasterized box, providing auxiliary inputs for vehicle recognition.

B. Extended Input to the Neural Nets

  • Rast adds a four-channel raster of visible box faces; the auxiliary data slightly improve classification and verification, while alternative vector normalization did not improve results.Source: B. Extended Input to the Neural Nets · p. 4
  • View encodes box orientation as three unit vectors from the box center to the front/rear, side, and roof faces, using vehicle direction to distinguish front from rear.Source: B. Extended Input to the Neural Nets · p. 4

C. Additional Training Data Augmentation

Training-time augmentation varies color and simulates missing image regions, while an estimated 3D box permits unpacking when precise boxes are unavailable.

C. Additional Training Data Augmentation

  • Color alters HSV channels and ImageDrop fills a random rectangle with noise to simulate missing parts; both are applied probabilistically during training, and both improve accuracy alone or in combination.Source: C. Additional Training Data Augmentation · p. 4

D. Estimation of 3D Bounding Box from a Single Image

IV. BOXCARS116K DATASET

BoxCars116k combines cleaned-up data from BoxCars21k with newly collected, tracked vehicle imagery from surveillance cameras around Brno. Annotators assigned fine-grained vehicle types and rejected invalid tracks, with multiple annotations used to check reliability.

A. Dataset Acquisition

  • The new dataset combines cleaned BoxCars21k data with tracked vehicle images collected from surveillance videos at varied street-camera viewpoints.Videos were downsampled to about 12.5 FPS to avoid collecting multiple nearly identical images of the same vehicle.Source: §IV · p. 5, A. Dataset Acquisition · p. 5
  • Annotators labeled each valid track with make, model, submodel, and year, using multiple annotations to confirm vehicle-type labels.Labels were accepted when at least three of about four annotations agreed; uncertain cases were resolved by the authors.Source: A. Dataset Acquisition · pp. 5–6
  • Tracks with imprecisely constructed 3D bounding boxes were invalidated, and the new dataset part generally contains more than three images per track.Source: A. Dataset Acquisition · p. 6

B. Dataset Statistics

BoxCars116k contains fine-grained vehicle labels and data from many surveillance cameras, with high viewpoint variation. Its camera-separated splits test recognition on unseen camera viewpoints, while the experiments evaluate proposed modifications across CNNs and define grouped variants.

B. Dataset Statistics

  • BoxCars116k contains 116,286 images of 27,496 vehicles across 693 fine-grained classes and 137 cameras, with varied viewpoints and 3D-box and foreground-mask information.The classes combine make, model, submodel, and model year; the dataset was made publicly available.Source: B. Dataset Statistics · p. 6

C. Training & Test Splits

  • Training and test data were split by camera, so testing uses vehicles from previously unseen cameras and viewpoints.The hard split distinguishes model years and includes 107 classes, with 11,653 training tracks and 11,125 test tracks.Source: C. Training & Test Splits · p. 6

V. EXPERIMENTS

  • The experiments assess proposed modifications across several CNNs and compare grouped configurations, including all five modifications, image-level modifications, and the earlier CVPR16 variant.The hard split targets precise vehicle types including model year; the medium split merges classes differing only by model year.Source: §V · pp. 6–7, C. Training & Test Splits · p. 6

A. Improvements for Different CNNs

The proposed modifications improve CNN classification accuracy and reduce error across the evaluated results. The full and image-only modification sets perform nearly identically, and results are almost identical between the hard and medium splits.

A. Improvements for Different CNNs

  • The modifications improve classification accuracy by up to +12 percentage points and reduce classification error by up to 50%; accuracy is further improved over the original method.Results are reported as single-sample accuracy/whole-track accuracy, with whole-track accuracy generally better.Source: A. Improvements for Different CNNs · p. 7
  • The difference between ALL and IMAGE modifications is negligible, so image-level modifications are sufficient; only Unpack is applied at test time.Color and ImageDrop are used during fine-tuning, not testing.Source: A. Improvements for Different CNNs · p. 7
  • Hard- and medium-split results are almost identical; the authors attribute the small accuracy difference mainly to the larger new dataset.The hard split is the main target because it distinguishes model years.Source: A. Improvements for Different CNNs · p. 7

B. Comparison with the State of the Art

The authors report that their method significantly outperforms standard CNNs and other evaluated fine-grained recognition methods. Input modification reduces processing speed, but the authors describe the method as still usable for real-time processing.

B. Comparison with the State of the Art

  • Vehicle-focused methods could not be evaluated because they are often limited to frontal or rear views or require 3D models for every vehicle type.Source: B. Comparison with the State of the Art · p. 8
  • 84% accuracy is reported for the authors’ best method, which significantly outperforms the compared standard CNNs and fine-grained methods.Fine-grained methods should be compared against the same base network because results vary across networks.Source: B. Comparison with the State of the Art · pp. 7–8
  • Input modification decreases processing speed, but the speed penalty is described as small and the method remains usable for real-time processing.The speed measurements used a GTX1080 and CUDNN whenever possible.Source: B. Comparison with the State of the Art · p. 8

C. Influence of Using Estimated 3D Bounding Boxes instead of the Surveillance Ones

Estimated 3D bounding boxes preserve significant accuracy gains and extend the method beyond surveillance settings. Viewpoint mismatch particularly benefits from Unpack, while Color, Unpack, and ImageDrop contribute most in individual and combined modification tests.

C. Influence of Using Estimated 3D Bounding Boxes instead of the Surveillance Ones

  • With estimated 3D bounding boxes, the modifications still significantly improve accuracy, and the approach can transfer to non-surveillance scenarios.The passage states that a reliable training set of vehicle directions toward vanishing points—or viewpoints and focal length—is additionally required.Source: C. Influence of Using Estimated 3D… · p. 8

D. Impact of Training/Testing Viewpoint Difference

  • Unpack alone improves accuracy by more than 20 percentage points in the largest viewpoint-difference bin; other modifications add gains independently of viewpoint difference.Improvement from Unpack grows with train-test viewpoint difference; Color and ImageDrop add improvement across viewpoint differences.Source: D. Impact of Training/Testing… · p. 8, Figure 10 · p. 9

E. Impact of Individual Modifications

F. Vehicle Type Verification

Vehicle-track verification results show improved average precision, while the conclusion reports gains over base CNNs and state-of-the-art methods. The paper also presents estimated-box construction and BoxCars116k as contributions.

VI. CONCLUSION

  • The paper estimates 3D bounding boxes from individual input images without surveillance video; performance with estimated boxes is only somewhat lower than with proper boxes.Source: §VI · pp. 9–10

F. Vehicle Type Verification

  • For vehicle-type verification, modifications improve average precision for each CNN and outperform human performance reported in the previous paper.The evaluation uses cosine-distance features and 5 million random vehicle-track pairs.Source: F. Vehicle Type Verification · p. 9, Figure 12 · p. 10

VI. CONCLUSION

  • Accuracy improves by up to +12 percentage points and error falls by up to 50%; the method beats state-of-the-art methods – by 9 points in single-image accuracy and 7 in whole-track accuracy.Source: §VI · p. 10
  • BoxCars116k provides annotated surveillance-domain vehicle images with greatly varying viewpoints and is publicly available for research and evaluation.Source: §VI · p. 10
Loading 1703.00686v3…