Source-linked AI summary
Spatial-Semantic Collaborative Cropping for User Generated Content
Yukun Su, Yiwen Cao, Jingliang Deng, Fengyun Rao, Qingyao Wu
TL;DR
Existing cropping methods often focus on simple landscape or landmark images and overlook object relations, content integrity, and fixed-aspect-ratio requirements in complex UGC. S2CNet mines potential-object regions and uses an adaptive attention graph to propagate spatial-semantic information among visual nodes toward the crop candidate. Replacing the adaptive attention graph with GAT lowers ACC5 to 59.4 and 58.6 on UGCrop5k and GAICv1, respectively.
Problem
Existing cropping methods often focus on simple landscape or landmark images and overlook object relations, content integrity, and fixed-aspect-ratio requirements in complex UGC.
Method
S2CNet mines potential-object regions and uses an adaptive attention graph to propagate spatial-semantic information among visual nodes toward the crop candidate.
Results
Replacing the adaptive attention graph with GAT lowers ACC5 to 59.4 and 58.6 on UGCrop5k and GAICv1, respectively.
Takeaways & Limitations
UGCrop5K provides 450,000 annotated candidate crops for evaluating UGC cropping, and the proposed spatial-semantic design supports aesthetic composition with content integrity.
Takeaways & Limitations
The spatial relation model assumes that nodes farther apart contribute less or nothing to aesthetic composition beyond a distance threshold.
Abstract
from arXiv · showhide
A large amount of User Generated Content (UGC) is uploaded to the Internet daily and displayed to people world-widely through the client side (e.g., mobile and PC). This requires the cropping algorithms to produce the aesthetic thumbnail within a specific aspect ratio on different devices. However, existing image cropping works mainly focus on landmark or landscape images, which fail to model the relations among the multi-objects with the complex background in UGC. Besides, previous methods merely consider the aesthetics of the cropped images while ignoring the content integrity, which is crucial for UGC cropping. In this paper, we propose a Spatial-Semantic Collaborative cropping network (S2CNet) for arbitrary user generated content accompanied by a new cropping benchmark. Specifically, we first mine the visual genes of the potential objects. Then, the suggested adaptive attention graph recasts this task as a procedure of information association over visual nodes. The underlying spatial and semantic relations are ultimately centralized to the crop candidate through differentiable message passing, which helps our network efficiently to preserve both the aesthetics and the content integrity. Extensive experiments on the proposed UGCrop5K and other public datasets demonstrate the superiority of our approach over state-of-the-art counterparts. Our project is available at https://github.com/suyukun666/S2CNet.
Introduction
UGC cropping must produce aesthetically appealing fixed-aspect-ratio thumbnails despite multi-object scenes, confounding backgrounds, and content-integrity requirements. S2CNet addresses these challenges by modeling object relations with an adaptive attention graph and introduces the UGCrop5K benchmark.
- UGC cropping targets fixed-aspect-ratio cover images for diverse user-uploaded media, supporting aesthetic composition and format unity.
- Unlike prior work centered on simpler landscape or human-centric images, UGC contains multiple foreground objects and chaotic backgrounds requiring relational modeling.Existing approaches often use images collected from Flickr or similarly clean settings.
- UGC crops must preserve content such as news headlines and complete lyrics while avoiding incomplete faces in multi-person images.
- S2CNet mines potential-object regions and uses an adaptive attention graph to represent object correlations for arbitrary UGC cropping.The proposed graph uses object regions together with the crop candidate as inputs.
- UGCrop5K is introduced as a densely labeled benchmark with 450,000 high-quality annotated candidate crops, alongside experiments comparing the approach with state-of-the-art methods.
Related Work
Prior cropping research uses handcrafted or deep-learning methods, while region-relation modeling remains limited in cropping. S2CNet's framework combines object-region features with spatial-semantic graph reasoning to predict crop aesthetics.
- Aesthetic cropping research progressed from handcrafted composition and detection features to data-driven deep neural-network methods.
- Region-based relation mining is common in other vision tasks but comparatively rare for image cropping.
- The proposed framework extracts features for potential objects and crop candidates, then applies adaptive attention graph propagation before aesthetic score prediction.
- Earlier graph-based cropping strategies neglect spatial location information, while deeper conventional graph convolutions can suffer over-smoothing.
Methodology
S2CNet models crop candidates and object proposals as a spatial-semantic graph, then uses gated attention and message passing to predict crop quality while preserving relevant content.
- Adaptive Attention Graph: S2CNet explicitly models compositional relations among crop candidates and object proposals so regions can be removed, partially reserved, or retained for aesthetic crops with content integrity.The method uses an adaptive attention graph rather than treating all visual patches equally.
- Visual Feature Mining: Top-N object proposals and convolutional feature maps are aligned through RoIAlign and RoDAlign to produce node features for graph reasoning.Faster RCNN pretrained on Visual Genome supplies the potential visual objects.
- Adaptive Attention Graph: The graph is fully connected, with nodes representing extracted features and edges representing correlations among the crop candidate and all regional objects.Global relations among visual objects are included because they can contribute to aesthetic composition.
- Spatial-Semantic Relations: Semantic edge weights use feature appearance similarity, while spatial edges encode node-center relations through either distance-thresholded DisDrop or embedding-based DisEmb representations.DisDrop removes relations beyond a distance threshold, whereas DisEmb can preserve meaningful long-range relations.
- Graph-Aware Attention Module: The correlation adjacency combines semantic and spatial representations, and the feature aggregation gate uses it to generate dynamically weighted tokens before spatial-semantic self-attention.The resulting attention is semantic-aware and topology-aware rather than treating nodes equally.
- Network Optimization: Graph-aware attention is stacked with LayerNorm and feed-forward updates, after which multilayer perceptrons aggregate node messages into the crop candidate for aesthetic-score prediction.Training combines weighted smooth ℓ1 score regression with a ranking loss over candidate crops in an end-to-end objective.
Experiment
The paper introduces UGCrop5K to address real-life UGC cropping and evaluates candidate-view quality with human-derived scores and ranking metrics.
- Datasets: UGCrop5K contains 5,000 varied images collected from social video sources and self-made content across scenes and aspect ratios.The dataset targets real-life cropping applications and includes diverse capture devices and orientations.
- Annotation: Each image receives 90 predefined crop candidates, with every candidate rated by at least five annotators on a 1–5 composition scale.Mean opinion scores provide the ground-truth quality labels.
- Datasets: The benchmark contains 450,000 high-quality annotated candidate crops derived from the 5,000 images.
- Metrics: Evaluation uses averaged Spearman’s rank-order correlation and top-k accuracy for k = 5 and k = 10 instead of IoU.
G Ma Mp UGCrop5K GAICv1 ACC5 ACC10 ACC5 ACC10
Experiments show that S2CNet performs competitively across UGCrop5K, GAICv1, and GAICv2 while its components improve graph-based crop scoring and preserve content under varied display constraints.
- Ablation Analysis: Combining all proposed components achieves the best ablation performance, with the spatial-message component improving more than the semantic-message component.The authors attribute the stronger spatial contribution to unique and useful topological cues in self-attention.
- Ablation Analysis: Replacing the adaptive attention graph reduces ACC5 from 60.8 to 56.2 on UGCrop5K and from 61.0 to 57.3 on GAICv1 with conventional GCNs.Using GAT instead yields ACC5 values of 59.4 and 58.6 on the two benchmarks, respectively.
- Ablation Analysis: The learned graph produces more discriminative crop, reserved, partially reserved, and removed-region features than the pre-graph representation.Crop features become closer to reserved features and farther from removed features, supporting explicit discrimination of good and bad views.
- Quantitative Results: S2CNet outperforms state-of-the-art methods on UGCrop5K and achieves satisfactory results on GAICv1 and GAICv2.The quantitative comparison covers all three datasets; SFRC is reported under the inductive setting for fairness.
- Quantitative Results: 162.8 FPS is achieved while maintaining competitive results, supporting efficient and practical cropping.Runtime and complexity were measured on a single NVIDIA RTX 2080Ti GPU.
- Qualitative Analysis: Qualitative results show that S2CNet preserves foreground subjects and useful background or textual information while producing aesthetically pleasing crops.It also finds suitable views for multiple aspect ratios and circular cropping without retraining.
Conclusion
The paper presents a spatial-semantic collaborative cropping network and a densely labeled UGCrop5K dataset for user-generated content cropping. By modeling semantic appearance and spatial topology, the method outperforms existing cropping approaches qualitatively and quantitatively.
- S2CNet combines semantic appearance and spatial topology information to address user-generated content cropping comprehensively.The approach is designed for user-generated content and targets both the visual and relational aspects of cropping.