Source-linked AI summary
Adaptive Fractional Dilated Convolution Network for Image Aesthetics Assessment
Qiuyu Chen, Wei Zhang, Ning Zhou, Peng Lei, Yi Xu, Yu Zheng, Jianping Fan
TL;DR
Image aesthetics assessment lacks a seamless way to incorporate image aspect ratios, even though cropping and warping can alter composition and introduce label noise. The paper proposes parameter-free adaptive fractional dilated convolution to preserve composition while enabling mini-batch training, and reports state-of-the-art performance on AVA.
Problem
Cropping and warping alter image composition and aspect ratios, while original-size inputs with different ratios cannot be batched, complicating image aesthetics assessment.
Source: §1 · p. 1Method
AFDC adapts convolution-kernel dilation to image aspect ratios on composition-preserving warped images, using interpolated fractional kernels without extra parameters.
Source: §1 · p. 2Results
AFDC achieves state-of-the-art performance for image aesthetics assessment on the AVA dataset.
Source: §1 · p. 2Takeaways & Limitations
The authors state that adaptive dilated convolution can also apply where cropping or warping may introduce label noise.
Source: §5 · p. 8The authors acknowledge that dataset constraints leave the learned aspect-ratio-related perception unsatisfactory, despite training on varied aspect ratios.
Source: §4.3 · p. 8
Abstract
from arXiv · showhide
To leverage deep learning for image aesthetics assessment, one critical but unsolved issue is how to seamlessly incorporate the information of image aspect ratios to learn more robust models. In this paper, an adaptive fractional dilated convolution (AFDC), which is aspect-ratio-embedded, composition-preserving and parameter-free, is developed to tackle this issue natively in convolutional kernel level. Specifically, the fractional dilated kernel is adaptively constructed according to the image aspect ratios, where the interpolation of nearest two integers dilated kernels is used to cope with the misalignment of fractional sampling. Moreover, we provide a concise formulation for mini-batch training and utilize a grouping strategy to reduce computational overhead. As a result, it can be easily implemented by common deep learning libraries and plugged into popular CNN architectures in a computation-efficient manner. Our experimental results demonstrate that our proposed method achieves state-of-the-art performance on image aesthetics assessment over the AVA dataset.
1. Introduction
Image aesthetics assessment must combine global visual cues with localized composition information. The paper proposes aspect-ratio-adaptive convolution to preserve composition while supporting mini-batch training.
1. Introduction
- Image aesthetics assessment predicts an aesthetic score while requiring both global cues and localization information.The task has applications including album photo recommendation, auxiliary photo editing, and multi-shot photo selection.Source: §1 · p. 1
- Cropping and warping can alter composition and object aspect ratios, so assigning the original image's score to the altered image may introduce label noise.This can harm aesthetics assessment.Source: §1 · p. 1, Figure 1 · p. 1
- Feeding original-size images preserves aspect ratios, but images with different ratios cannot be concatenated into batches, slowing training and inference.This constraint applies to the MNA-CNN approach described in the introduction.Source: §1 · pp. 1–2
- AFDC adapts convolution-kernel dilation to image aspect ratios on composition-preserving warped images, interpolating the two nearest integer-dilation kernels.The effective receptive field matches that of regular convolution, and the interpolation uses the same kernel parameters.Source: §1 · p. 2
- The method is parameter-free, supports mini-batch training, reduces computational overhead through grouping, and achieves state-of-the-art performance on AVA.The authors also describe implementation with common deep-learning libraries and compatibility with popular network architectures.Source: §1 · p. 2
2. Related Work
Prior aesthetics-assessment work uses several prediction formulations and strategies to preserve image composition. AFDC differs from conventional dilated and deformable convolution by adapting receptive fields to original aspect ratios through parameter-free fractional dilation.
2. Related Work
- Aesthetics-assessment methods use binary or mean labels, ranking scores, and score distributions, with different losses and multi-task or multi-patch architectures.Examples include Earth Mover’s Distance for score distributions [27] and Huber Loss for subjective ratings [17].Source: §2 · p. 2
- Multi-patch methods preserve aspect ratios but may alter aesthetic factors, while MNA-CNN feeds original images individually and cannot batch different aspect ratios.AFDC instead restores original receptive fields from composition-preserving warped images and supports mini-batch training.Source: §2 · p. 2
- Unlike dilated convolution, which expands receptive fields without down-sampling, AFDC restores the original-image receptive field using dynamically constructed fractional dilation.The overview pairs AFDC on wrapped input with vanilla convolution on the original image at the same receptive field.Source: §2 · p. 2, Figure 2 · p. 3
- Deformable convolution learns sampling locations with parameterized layers, whereas AFDC adapts receptive fields to aspect ratios without extra parameters.The paper also contrasts its mini-batch formulation with deformable convolution’s CUDA rewriting and indexing overhead.Source: §2 · pp. 2–3
3. Adaptive Fractional Dilated Convolution
AFDC adapts convolutional receptive fields to each image’s aspect ratio, using fractional dilation to preserve original receptive fields after composition-preserving warping. The section also describes mini-batch implementation, computational overhead reduction, and composition-aware prediction and loss.
3.1. Adaptive Kernel Interpolation
- AFDC adapts convolutional receptive fields to an image’s original aspect ratio, interpolating the nearest integer dilation kernels when fractional rates would misalign feature sampling.The integer-rate kernels share parameters, so interpolation introduces no extra learning parameters.Source: §3.1 · pp. 3–4
3.2. Mini-Batch Computation and Implementation
- Mini-batch computation combines integer-dilated convolution outputs using per-image interpolation weights, making fractional dilation implementable in common deep-learning frameworks.A grouping strategy lets images with compatible aspect ratios share integer-dilated convolutions; AVA’s training and testing images mostly fall in the stated [1/2, 2] range.Source: §3.2 · pp. 4–5
- AFDC adds computation through multiple integer dilations: the paper estimates roughly 1.5 times the original model’s cost for ResNet50 and approximately 2 times for VGG16.The estimates concern the stated dilation kernels, with additional overhead from interpolation; the paper also reports that measured training and testing speeds track #Muti-Adds approximately linearly.Source: §3.2 · p. 5
3.3. Composition-Aware Structure and Loss
- A composition-aware structure uses multiple adaptive-pooling modules to retain spatial localization information before concatenating their fully connected outputs.The paper uses an Earth Mover Distance loss for predicted score distributions.Source: §3.3 · p. 5
4. Experimental Results
On AVA, AFDC outperforms the compared methods on the reported classification-accuracy and MSE metrics. The experiments examine composition-preserving augmentation, aspect-ratio-aware kernels, and composition cues, while the authors note that learned aspect-ratio perception remains constrained by the training dataset.
4. Experimental Results
- Experiments use AVA’s roughly 250,000 images, with 235,528 for training and 20,000 for testing; ResNet-50 is the backbone.Training uses a grouping strategy and changes from three dilation kernels for 12 epochs to seven for the remaining six.Source: §4 · p. 6, §4.1 · p. 6
4.2. Ablation Study
- Random-size warping into square sizes from 224 to 320 replaces random cropping to preserve image composition; crop-based pretraining favors distortion invariance.The authors describe distortion invariance as contrary to AFDC’s aim of preserving the original aspect ratio.Source: §4.2 · pp. 6–7
- AFDC with spatial pyramid pooling (SPP) performs better than AFDC with global pooling, and the example crops show composition-aware localization.The SPP model uses three scales; the authors report that five grids overfit due to extra parameters.Source: §4.2 · p. 7, Figure 6 · p. 7
4.3. Effectiveness of AFDC
- AFDC improves training and validation results, and test comparisons favor it over other dilated-convolution variants.The authors attribute the learning gains to preserving aspect-ratio information and describe the comparison as possible because AFDC adds no learning parameters.Source: §4.3 · p. 7
- AFDC responds discriminatively to aspect-ratio changes, while vanilla Conv shows small fluctuations attributed to resizing-related sampling changes.The AFDC score distribution is multimodal, which the authors say reflects a complex relation between aspect ratio and aesthetic perception.Source: §4.3 · p. 7, Figure 8 · p. 8
- The learned perception of aspect ratios remains unsatisfactory under the training-dataset constraint, though the authors say a more targeted dataset could support this learning ability.Source: §4.3 · p. 8
4.4. Comparison With the State-of-the-Art Results
- AFDC outperforms the compared methods on cls.acc and MSE in the AVA evaluation.Table 4 compares the state-of-the-art methods; the supplied caption identifies the comparison but gives no cell values.Source: §4 · p. 6, §4.4 · p. 8, Table 4 · p. 8
- Compared with multi-patch methods, the unified end-to-end model uses complete images and avoids sampling multiple patches and aggregating their predictions.The authors describe this approach as more efficient and as better preserving original aesthetic information.Source: §4.4 · p. 8
5. Conclusion
The conclusion presents AFDC as a parameter-free way to model image aspect ratios and reduce computational overhead, with possible use beyond aesthetics assessment when cropping or warping introduces label noise.
5. Conclusion
- AFDC models aspect ratios without extra parameters, uses grouping to reduce computational overhead, and may apply where cropping or warping introduces label noise.Source: §5 · p. 8