PaperlayerView exact revision on arXiv

Source-linked AI summary

GiT: Graph Interactive Transformer for Vehicle Re-identification

Fei Shen, Yi Xie, Jianqing Zhu, Xiaobin Zhu, Huanqiang Zeng

arXiv:2107.05475v3cs.CV

TL;DR

Vehicle re-identification needs both global context and local details because vehicles with the same model and color can be difficult to distinguish. GiT couples transformer layers with local correlation graphs in an interactive architecture, and experiments report superiority over many state-of-the-art approaches on three large-scale datasets.

  • Problem

    Vehicle re-identification must capture subtle local differences in addition to global appearance to distinguish highly similar vehicles.

  • Method

    GiT stacks blocks containing transformers for global relationships across patches and local correlation graphs for discriminative features within patches, with interactive two-way coupling.

  • Results

    GiT is reported superior to many state-of-the-art vehicle re-identification approaches on VeRi776, VehicleID, and VeRi-Wild.

  • Takeaways & Limitations

    The paper concludes that GiT effectively explores interaction between local and global features through graph-transformer coupling.

Abstract

from arXiv · show

Transformers are more and more popular in computer vision, which treat an image as a sequence of patches and learn robust global features from the sequence. However, pure transformers are not entirely suitable for vehicle re-identification because vehicle re-identification requires both robust global features and discriminative local features. For that, a graph interactive transformer (GiT) is proposed in this paper. In the macro view, a list of GiT blocks are stacked to build a vehicle re-identification model, in where graphs are to extract discriminative local features within patches and transformers are to extract robust global features among patches. In the micro view, graphs and transformers are in an interactive status, bringing effective cooperation between local and global features. Specifically, one current graph is embedded after the former level's graph and transformer, while the current transform is embedded after the current graph and the former level's transformer. In addition to the interaction between graphs and transforms, the graph is a newly-designed local correction graph, which learns discriminative local features within a patch by exploring nodes' relationships. Extensive experiments on three large-scale vehicle re-identification datasets demonstrate that our GiT method is superior to state-of-the-art vehicle re-identification approaches.

I. INTRODUCTION

Vehicle re-identification must distinguish highly similar vehicles by combining global appearance with discriminative local details. GiT addresses this need by interactively coupling transformer layers and local correlation graphs, without requiring extra fine-grained part annotations or divisions, and reports superiority on three large-scale datasets.

  • Motivation: Vehicle re-identification is challenged by poor illumination, varying viewpoints, and subtle differences among vehicles with the same model and color.Relevant discriminative details include wheels, lights, and front windows.
  • Prior approaches: Early pure CNN methods learn global appearance features but do not adequately handle local features.This limitation restricts vehicle re-identification performance.
  • Prior approaches: Partition-based CNN methods combine global and local features, but uniform division risks misalignment while part detection requires costly annotations and computation.Subsequent learning is also performed separately on each part, ignoring relationships among part regions.
  • Prior approaches: CNN–graph-network methods model relationships among local regions, but CNN down-sampling and large-stride convolutions reduce feature-map resolution.This restricts recognition of vehicles with similar appearances.
  • Proposed method: GiT couples transformer layers for global relationships across patches with local correlation graphs for local features within patches.The method is designed to let both components cooperate interactively, and its LCG module requires no extra fine-grained part annotations or part divisions.
  • Results: Experiments on VeRi776, VehicleID, and VeRi-Wild report that GiT is superior to many state-of-the-art vehicle re-identification approaches.The paper presents this as an experimental result across three large-scale vehicle datasets.

II. RELATED WORK

Prior vehicle re-identification work progresses from pure CNN feature learning to local-part modeling and CNN–graph-network combinations. These approaches improve local representation but face partition alignment, annotation, computation, and resolution limitations.

  • CNN-based methods: Pure CNN-based methods typically use robust CNNs and feature aggregation structures to learn global or local vehicle features.The local-feature category includes global feature learning, uniform spatial division, and part detection approaches.
  • CNN-based methods: Uniform spatial division methods divide feature maps into regions but may produce inaccurate or misaligned parts.They avoid part annotations but can suffer from partition misalignment.
  • CNN-based methods: Part detection methods improve division accuracy by locating vehicle parts or discriminative regions.Their main drawback is the high cost of manual annotations, training, and inference computation.
  • CNN with graph networks: CNN–graph-network methods use graph networks to extract local regional features and propagate information among parts.These methods combine CNN-derived regional features with graph-based local reasoning.

B. Transformer in Vision

Vision transformers model global relationships among image patches, while GiT adds graph-based local modeling to address within-patch feature learning. Its graph and transformer components are coupled so local and global representations can interact.

  • Transformer in vision: Vision transformers use self-attention-based layers to model global context and long-distance dependencies among image patches.They also retain spatial information by avoiding CNN down-sampling operations.
  • Graph-transformer coupling: GiT is designed to couple graph networks and transformer layers so local and global features interact rather than being learned in separate branches.The paper contrasts its dense, two-way coupling with sparse or single-way coupling in HRNet and Multi-scale DenseNet.
  • GiT framework: Each GiT block contains a local correlation graph for within-patch features and a transformer layer for across-patch relationships.The interaction is intended to complement the local and global modeling strengths of the two components.
  • Local correlation graph: The LCG module samples each patch into local features, projects them, and adds positional information before graph construction.The transformed local sequence and positional embedding preserve local feature structure and position within each patch.
  • Local correlation graph: LCG constructs patch-level spatial graphs whose edges reflect cosine similarity between local-feature nodes.Graph propagation then aggregates information from each node and its neighbors using an adjacency matrix and learnable parameters.
  • Local correlation graph: The graph aggregation output is enhanced with GELU and layer normalization to produce new nodes with discriminative local features.These functions are chosen for consistency with the original transformer layer.

C. Transformer Layer

The transformer layer models global relationships across image patches using multi-head self-attention and a multi-layer perceptron. Because this global modeling does not capture patch-local detail, the LCG module remains necessary for local feature learning.

  • The transformer layer models global relationships among patches with multi-head self-attention and a multi-layer perceptron.Patch features are projected into queries, keys, and values before scaled dot-product self-attention.
  • Multi-head self-attention splits queries, keys, and values across h parallel heads, then concatenates and linearly projects the outputs.
  • Because the transformer models global features across all patches, the LCG module is essential for learning local features.

D. Graph Interactive Transformer

GiT stacks blocks containing an LCG module and a transformer layer so local and global features interact throughout feature learning. The two-directional coupling lets each component use outputs from the other component and from the preceding block.

  • Graph Interactive Transformer: Each GiT block contains an LCG module and a transformer layer whose local and global features interact throughout feature learning.The architecture uses multiple GiT blocks, with three stages of four blocks each.
  • Graph Interactive Transformer: Each patch’s local graph features are reshaped into patch features so the LCG module can couple directly with the transformer when d′ = d.This direct coupling avoids extra linear projection or convolution modules, saving parameters and computational cost.
  • Graph Interactive Transformer: The LCG module learns local features from the preceding block’s local and global outputs, while the transformer uses preceding global features and the current block’s local outputs.
  • Graph Interactive Transformer: GiT uses 16 × 16 image patches, while node counts and sampling sizes vary across stages.
  • Graph Interactive Transformer: The two-directional design couples global and local features more completely than the single-directional Global → Local and Global ← Local variants.

E. Loss Function Design

GiT trains with cross-entropy and triplet losses applied to different normalized and unnormalized feature representations. The default configuration balances the two losses equally and retains the initial triplet loss.

  • Loss Function Design: GiT’s total loss combines cross-entropy loss and soft-margin triplet loss with manually set balancing constants α and β.The default setting uses α = β = 1.
  • Loss Function Design: The class-token features are applied directly to triplet loss, while batch-normalized features are fed to cross-entropy loss without label smoothing.
  • Loss Function Design: Cross-entropy loss uses a training set, class labels, sample count, class count, and learnable class-weight matrix.
  • Loss Function Design: The triplet loss uses anchor, positive, and negative training triples, with Euclidean distance between feature pairs.Hard sample mining selects the most difficult positive and negative pairs in each mini-batch.
  • Loss Function Design: Despite considering weighted regularization triplet loss, the default configuration applies the initial triplet loss because many state-of-the-art methods do not use WRT.

IV. EXPERIMENT AND ANALYSIS

The experiments compare GiT with multiple state-of-the-art vehicle re-identification methods on three large-scale datasets using rank-1 identification rate and mean average precision.

  • IV. EXPERIMENT AND ANALYSIS: GiT is evaluated against multiple state-of-the-art vehicle re-identification approaches on VeRi776, VehicleID, and VeRi-Wild.
  • IV. EXPERIMENT AND ANALYSIS: The evaluation measures rank-1 identification rate and mean average precision.

A. Datasets

The evaluation uses three large-scale vehicle re-identification datasets covering unconstrained traffic, varied data scales, and both daytime and nighttime imagery.

  • VeRi776: VeRi776 contains cross-camera vehicle images captured by 20 cameras in unconstrained traffic scenarios.Each vehicle is captured by 2–18 cameras; evaluation uses 37,746 training images and 1,678 probe images with an 11,579-image gallery.
  • VehicleID: VehicleID contains 221,763 images of 26,267 subjects, with front or rear viewpoints and Test800, Test1600, and Test2400 subsets.The testing subsets evaluate performance at different gallery and subject scales.
  • VeRi-Wild: VeRi-Wild contains 416,314 images of 40,671 subjects captured during both day and night.Its testing data are organized into Test3000, Test5000, and Test10000 subsets.

B. Implementation Details

The implementation uses FP16 PyTorch training and evaluates 768-dimensional batch-normalized features, while comparisons organize methods by their use of global and local representations.

  • Training setup: The model is implemented in PyTorch with FP16 training.Training configurations are summarized alongside the implementation description.
  • Training setup: Training uses ImageNet-pretrained ViT weights, random erasing, z-score normalization, and mini-batch SGD with specified decay and momentum settings.Horizontal flipping and random erasing each have probability 0.5; weight decay is 1×10−4 and momentum is 0.9.
  • Testing: Testing uses 768-dimensional features from the batch-normalization layer and cosine distance for vehicle-image similarity.These batch-normalized features serve as the final representations for re-identification.
  • VeRi776 comparison: On VeRi776, GiT achieves 96.86% R1 and 80.34% mAP, exceeding the best global-only method by 13.37% and 26.89%, respectively.It also exceeds PCRNet by 1.46% R1 and 1.74% mAP without extra semantic annotation information.
  • Comparison tables: The comparison tables report percentage performance and distinguish first-, second-, and third-place methods by row color.The table descriptions identify red, green, and blue rows as the first, second, and third places.

2) Comparisons on VehicleID:

GiT consistently outperforms competing vehicle re-identification methods and ViT across large-scale datasets. Ablations indicate that interactive local-global feature coupling and progressive sampling contribute to performance, while complexity remains comparable to ViT.

  • 2) Comparisons on VehicleID:: GiT consistently outperforms state-of-the-art methods on VehicleID’s Test800, Test1600, and Test2400 subsets.On Test800, GiT exceeds HPGN by 0.52% in R1 and ViT by 3.07% in R1.
  • 2) Comparisons on VehicleID:: GiT achieves first place on VeRi-Wild’s Test3000, Test5000, and Test1000 subsets.On Test1000, GiT exceeds PCRNet by 0.41% R1 and 0.45% mAP, and HPGN by 2.51% R1 and 2.73% mAP.
  • 2) Comparisons on VehicleID:: On VehicleID Test2400, GiT exceeds ViT by 1.02% R1 and 4.16% mAP, demonstrating consistent gains over pure transformers across three datasets.The authors attribute these improvements to coupling graph modules with ViT.
  • D. Ablation Studies and Analysis: Interactive coupling improves performance beyond one-way or absent interaction between LCG and transformer layers.On VeRi776, GiT Interactive exceeds Global → Local by 0.52% R1 and Global ← Local by 0.44% R1; on VehicleID Test2400, it improves mAP by 1.65% and 1.29%, respectively.
  • D. Ablation Studies and Analysis: Progressive LCG sampling sizes outperform fixed sampling sizes across three datasets.On VeRi-Wild Test10000, progressive sampling improves mAP over GiT1, GiT2, and GiT3 by 1.24%, 1.47%, and 2.29%, respectively.
  • D. Ablation Studies and Analysis: GiT has complexity comparable to ViT despite incorporating graph neural networks, although it is less efficient than ResNet.GiT-Tiny defeats ViT-Base in complexity and accuracy, while both GiT and ViT are inferior to ResNet in FLOPs and average inference speed.

E. Generalization to Person Re-identification

GiT is evaluated for generalization beyond vehicle re-identification on Market-1501 and MSMT17 person re-identification datasets. It achieves the highest mAP on Market-1501 and the best R1 and mAP on MSMT17.

  • E. Generalization to Person Re-identification: GiT achieves the highest mAP of 88.9% on the Market-1501 person re-identification dataset.Its R1 is slightly lower than those of RGA-SC and SAN.
  • E. Generalization to Person Re-identification: On the larger MSMT17 dataset, GiT achieves the best R1 and mAP among the compared methods.The passage reports that RGA-SC and SAN lose their dominance on MSMT17.

V. CONCLUSION

GiT combines graphs and transformers to improve cooperation between local and global features for vehicle re-identification. It introduces a local correlation graph and evaluates the method through extensive analyses.

  • GiT couples graphs and transformers to explore interactions between local and global features for vehicle re-identification.The method aims to produce effective cooperation between the two feature types.
  • The local correlation graph learns local features within patches.
  • Experiments analyze module interaction, visualization, sampling strategy, backbones, complexity, and sensitivity.
Loading 2107.05475v3…