Source-linked AI summary
Local Relation Networks for Image Recognition
Han Hu, Zheng Zhang, Zhenda Xie, Stephen Lin
TL;DR
Convolution uses fixed filters that are inefficient for visual elements with varying spatial distributions. The paper introduces local relation layers, which adapt aggregation weights from local pixel-pair composability, and shows that LR-Net improves ImageNet recognition over comparable regular convolution networks.
Problem
Fixed-filter convolution is inefficient at modeling visual elements whose spatial distributions vary.
Method
The local relation layer adaptively weights local pixel pairs using composability measured in a learned embedding space, and replaces convolution in deep networks.
Results
LR-Net surpasses a same-computation-budget 26-layer ResNet by 3% absolute top-1 accuracy on ImageNet and also benefits from larger kernels.
Takeaways & Limitations
Local relation networks expand the practicality of bottom-up compositional aggregation as a basic image feature extractor for large-scale recognition.
Takeaways & Limitations
The study remains at an early stage, with future work needed for faster memory scheduling, stronger comparisons against deformable convolution, and other vision tasks.
Abstract
from arXiv · showhide
The convolution layer has been the dominant feature extractor in computer vision for years. However, the spatial aggregation in convolution is basically a pattern matching process that applies fixed filters which are inefficient at modeling visual elements with varying spatial distributions. This paper presents a new image feature extractor, called the local relation layer, that adaptively determines aggregation weights based on the compositional relationship of local pixel pairs. With this relational approach, it can composite visual elements into higher-level entities in a more efficient manner that benefits semantic inference. A network built with local relation layers, called the Local Relation Network (LR-Net), is found to provide greater modeling capacity than its counterpart built with regular convolution on large-scale recognition tasks such as ImageNet classification.
1. Introduction
The paper argues that convolution composes visual elements inefficiently because fixed filters struggle with spatial variability. It introduces local relation layers that adapt aggregation to local pixel-pair composability, yielding stronger recognition results and additional favorable properties.
- Motivation: Convolutional hierarchies combine edges into parts and objects, but their fixed filter templates inefficiently model meaningful composition.Filters aggregate spatial features using learned weights rather than recognizing how elements can be meaningfully joined.
- Motivation: Spatial variability can create a combinatorial explosion of valid filters needed to compose deformable visual concepts.The paper illustrates this issue with varying relationships between a bird’s eye and beak.
- Contribution: The local relation layer adaptively infers local compositional structure by weighting pixel pairs according to similarity in a learned embedding space.The embedding can additionally incorporate geometric priors.
- Results: 3% absolute top-1 accuracy: a 26-layer LR-Net surpasses a same-budget regular 26-layer ResNet on ImageNet classification.Improved accuracy is also reported with basic residual blocks and 50- and 101-layer networks.
- Results: LR-Net benefits from 7 × 7 or larger kernels and is reported as more robust to adversarial attacks than regular convolution networks.The paper attributes the robustness likely to compositional power in the spatial domain.
- Novelty: Unlike conventional top-down weight determination, the proposed network computes aggregation weights bottom-up while preserving locality and geometric relationships.The paper presents this approach as practical and effective for large-scale recognition.
2. Related Works
Related work spans convolution extensions, relation models, and bottom-up aggregation methods. The paper positions local relation networks as a practical local, geometry-aware replacement for convolution that brings compositional modeling to basic vision feature extraction.
- Convolution Extensions: Convolution extensions improve efficiency, receptive-field size, or deformation modeling through grouped, depthwise, dilated, active, and deformable variants.These methods modify channel scope or spatial aggregation scope while retaining convolutional processing.
- Convolution Extensions: Locally connected and position-adaptive layers relax weight sharing, but locally connected layers lose translation invariance and cross-position knowledge transfer.Other variants generate position-adaptive weights or aggregation scopes.
- Bottom-Up Aggregation: Regular convolution and its extensions determine aggregation top-down, whereas the proposed layer uses local pixel-pair composability to determine weights bottom-up.The paper links this design to more efficient spatial-composition encoding.
- Bottom-Up Aggregation: Capsule Networks also use bottom-up composability, but iterative routing is less aligned with back-propagation and multilayer learning infrastructure.The local relation layer instead computes composability through learned embedding-space similarity.
- Bottom-Up Aggregation: Local relation networks differ through local rather than global aggregation and through explicit geometric priors, making them more practical than existing bottom-up methods.The paper states that these differences support applicability to large-scale recognition.
- Relation Modeling: The work bridges compositional representation methods from sequential and non-grid data with basic feature extraction for computer vision.The paper describes adapting both sides as non-trivial.
3. A General Formulation
The paper formulates basic image feature extractors as weighted aggregation operators over input channels and spatial positions. It distinguishes these operators by how weights are parameterized, which inputs are aggregated, and how aggregation weights are obtained.
- General Formulation: A feature-extraction layer maps input x ∈ R^C×H×W to output y ∈ R^C′×H′×W′ through weighted aggregation.C and C′ denote input and output channels, while H, W, H′, and W′ denote spatial resolutions.
- Aggregation Scope: The aggregation scope specifies which input channels and spatial positions contribute to each output feature value.Channel scope may include all, one, or groups of input channels; spatial scope commonly restricts computation to a local area.
- General Formulation: Basic image feature extractors are primarily differentiated by parameterization method, aggregation scope, and aggregation weights.These three axes organize the formulation and the comparison of existing extractors.
- Parameterization: Parameterization methods define learned model weights, generated adaptive weights, adaptive spatial scopes, or fixed spatial priors.Sharing parameterization across positions supports translation invariance.
- Aggregation Weights: Aggregation weights may be fixed or top-down input-dependent, while some non-convolution methods determine them bottom-up from pixel-pair composability.This distinction motivates the local relation layer’s alternative weighting mechanism.
4. Local Relation Layer
The local relation layer adaptively aggregates nearby features using appearance composability and geometric priors, replacing fixed convolutional aggregation with bottom-up local relations.
- Appearance and geometry: Aggregation weights combine appearance-based composability of projected pixel pairs with a translationally invariant geometric prior.Appearance composability compares query and key projections, while the geometric term uses relative position as input.
- Aggregation strategy: The layer computes composability from local pixel-pair properties, making it a bottom-up alternative to top-down convolutional aggregation.It belongs to the class of bottom-up methods and is designed to encode spatial composition more efficiently.
- Locality: Local aggregation over a 7 × 7 neighborhood is crucial for feature learning, while larger kernels improve accuracy more steadily than in convolution variants.The local relation layer uses a 7 × 7 kernel by default and can model visual patterns of increasing size and complexity.
- Geometric priors: A small network on relative positions computes geometric priors more effectively than directly learned values, especially for large neighborhoods.The authors attribute this advantage possibly to treating relative positions as vectors in metric space rather than independent identities.
- Implementation: SoftMax normalization balances appearance composability and geometric-prior terms, while channel sharing reduces model size without observed accuracy loss up to 8 channels.Channel sharing also achieved more than 3× actual speed-up over one channel per aggregation in the CUDA implementation.
- Implementation: A naive CUDA implementation is several times slower than regular convolution at equal FLOPs because convolution has highly optimized memory scheduling.Improving memory scheduling for the local relation layer is identified as future work.
5. Local Relation Networks
Local relation layers replace spatial convolutions within ResNet residual blocks to form LR-Net, using comparable computation while modestly reducing model size.
- Architecture: Local relation layers directly replace spatial convolution layers in deep networks while preserving the residual-block topology.The replacement covers 3 × 3 convolutions in residual blocks and the first 7 × 7 convolution in ResNet.
- Architecture: LR-Net-50 uses similar FLOPs and a slightly smaller model than ResNet-50 because channel sharing is used during aggregation.The first 7 × 7 convolution replacement has comparable ImageNet accuracy and similar FLOPs.
6. Experiments
Experiments show that LR-Net benefits from local aggregation, geometric priors, scalar composability, channel sharing, and larger kernels, while improving recognition and adversarial robustness over convolutional baselines.
- Impact of spatial scope: 71.9 versus 50.7 top-1 accuracy shows that restricting geometric-free aggregation from the full image to a 7×7 neighborhood substantially improves ImageNet recognition.With geometric priors, accuracy further increases from 68.4 to 75.7.
- LR-Net benefits from large kernels: 70.8→71.5→71.9 and 73.6→74.9→75.7 show steadily improving accuracy as LR-Net kernel size grows from 3×3 to 7×7.Regular ResNet-26 saturates or loses accuracy with larger kernels, while depthwise-convolution accuracy remains nearly unchanged from 3×3 to 9×9.
- Effect of geometric prior: 3.1 points is the gap at 9×9 between learned geometric-prior embeddings and directly learned values, demonstrating the advantage of modeling relative positions.The corresponding gap is 0.4 points at 3×3, and both embedding methods outperform omitting geometric priors.
- Appearance composability: Scalar query/key representations outperform larger dimensions, indicating a better speed-accuracy trade-off than vector representations.The ablation reports decreased accuracy as key/query dimension increases.
- Channel sharing and normalization: m = 8 channel sharing maintains accuracy while achieving 3× the speed of m = 1, reducing implementation cost without an observed accuracy drop.The paper also reports that softmax normalization improves accuracy by 0.9 by balancing appearance and geometric-prior terms.
- Comparison and robustness: LR-Net surpasses regular ResNet by 2.9 points at similar computation, remains effective across deeper architectures, and improves white-box attack performance over ResNet-26 and ResNet-50.The reported robustness gains are +6.3 and +12.4 versus ResNet-26, and +0.8 and +4.3 versus ResNet-50 for targeted and untargeted attacks, respectively.
7. Conclusion and Future Works
The paper introduces the local relation layer as a compositional image feature extractor and reports strong ImageNet results. It identifies faster implementation, stronger comparisons with advanced convolution, and broader task evaluation as future directions.
- 7. Conclusion and Future Works: The local relation layer introduces compositionality into image representation as a basic feature extractor.It is presented as a new layer for building deep networks.
- 7. Conclusion and Future Works: Deep networks built with the new layer demonstrate strong results on ImageNet classification.
- 7. Conclusion and Future Works: The authors identify faster GPU memory scheduling as a future direction for implementation.
- 7. Conclusion and Future Works: Future work includes outperforming advanced convolution methods such as deformable convolution.
- 7. Conclusion and Future Works: The study is still at an early stage, motivating exploration of other properties and applicability to other vision tasks.
A1. Implementation Details
The implementation uses standardized residual-network inputs and training procedures, with architectural adjustments for fair FLOPs comparisons. Training uses large-batch SGD with augmentation, warm-up, scheduled decay, and 110 total epochs.
- A1. Implementation Details: All architectures take a 3×224×224 image as input and use skip connections across residual blocks.Across stages, channel transformation followed by batch normalization is used for the shortcut branch.
- A1. Implementation Details: Downsampling is applied in the first residual blocks of res3, res4, and res5 on the 3 × 3 convolution or local relation layer.
- A1. Implementation Details: The bottleneck ratio α is adapted to ensure the same FLOPs across architectures in ablation experiments.
- A1. Implementation Details: Training uses random crops, scale and aspect-ratio augmentation, and SGD with a mini-batch of 1024 on 16 GPUs.Adversarial-training experiments use 32 GPUs.