PaperlayerView exact revision on arXiv

Source-linked AI summary

ProCrop: Learning Aesthetic Image Cropping from Professional Compositions

Ke Zhang, Tianyu Ding, Jiachen Jiang, Tianyi Chen, Ilya Zharkov, Vishal M. Patel, Luming Liang

arXiv:2505.22490v1cs.CV

TL;DR

Aesthetic cropping must capture complex composition while overcoming limited diversity in rule-based methods and scarce annotated training data. ProCrop retrieves similar professional compositions, fuses their features with query-image features, and supplies a 242K-image weakly annotated dataset. It achieves state-of-the-art performance across supervised and weakly supervised settings, including results comparable to fully supervised approaches when trained on the new dataset.

  • Problem

    Aesthetic cropping is difficult because composition involves complex interactions, while existing methods lack diversity or require scarce, labor-intensive annotated datasets.

  • Method

    ProCrop retrieves compositionally similar professional images, fuses their features with query-image features, and generates a 242K-image weakly supervised dataset through outpainting and iterative refinement.

  • Results

    ProCrop achieves state-of-the-art performance across supervised and weakly supervised cropping, surpassing prior weakly supervised methods and matching fully supervised approaches on the new dataset.

  • Takeaways & Limitations

    Professional compositions and composition-aware weak supervision provide a practical basis for improving aesthetic cropping across image-retrieval and cropping benchmarks.

  • Takeaways & Limitations

    The evaluation metrics may not fully reflect true aesthetic quality, and user-specific control over cropping preferences remains unexplored.

Abstract

from arXiv · show

Image cropping is crucial for enhancing the visual appeal and narrative impact of photographs, yet existing rule-based and data-driven approaches often lack diversity or require annotated training data. We introduce ProCrop, a retrieval-based method that leverages professional photography to guide cropping decisions. By fusing features from professional photographs with those of the query image, ProCrop learns from professional compositions, significantly boosting performance. Additionally, we present a large-scale dataset of 242K weakly-annotated images, generated by out-painting professional images and iteratively refining diverse crop proposals. This composition-aware dataset generation offers diverse high-quality crop proposals guided by aesthetic principles and becomes the largest publicly available dataset for image cropping. Extensive experiments show that ProCrop significantly outperforms existing methods in both supervised and weakly-supervised settings. Notably, when trained on the new dataset, our ProCrop surpasses previous weakly-supervised methods and even matches fully supervised approaches. Both the code and dataset will be made publicly available to advance research in image aesthetics and composition analysis.

1. Introduction

Aesthetic cropping is difficult because composition involves complex visual interactions, while existing rule-based and data-driven methods face limits in expressiveness or annotated-data scale. ProCrop addresses these gaps by retrieving professional compositions and introducing a 242K-image weakly supervised dataset.

  • Existing methods struggle because rule-based systems cannot capture sophisticated compositions, while data-driven systems depend on labor-intensive annotated datasets.The largest existing dataset is described as containing only about 10K images, limiting compositional and stylistic diversity.
  • ProCrop retrieves professional photographs with similar aesthetic compositions to guide cropping, requiring no annotations for the reference database.The method uses professional photography as external compositional knowledge to improve diversity beyond rules and data-driven training alone.
  • The proposed weakly supervised pipeline generates 242K annotated aesthetic images by outpainting professional photographs and iteratively refining diverse crop proposals.The pipeline uses GPT-4 for layout descriptions, SAM for compositional masks, and ControlNet for image outpainting.
  • Experiments report that ProCrop significantly outperforms existing methods, while training on the new dataset surpasses prior weakly supervised methods and matches fully supervised approaches.The contribution summary presents this comparison across the reported experiments.
  • The work plans to release its code and dataset publicly to support image cropping, image aesthetics, and composition analysis research.

2. Related work

Aesthetic image cropping differs from retargeting by prioritizing composition quality, but existing rule-based and data-driven approaches face limitations in nuanced composition, annotation cost, and dataset scale. ProCrop combines retrieval augmentation with weakly supervised outpainting to address these constraints.

  • 2.1. Aesthetic image cropping: Aesthetic cropping generates and scores candidate crops for visual quality, whereas image retargeting primarily focuses on preserving content.
  • 2.2. Image cropping methods: Rule-based methods use hand-crafted features or aesthetic rules but often struggle with nuanced compositions, while data-driven methods predict candidate regions or crop coordinates.
  • 2.3. Dataset construction: Supervised cropping datasets such as GAICv1, GAICv2, CPC, FCDB, and SACD are labor-intensive and expensive to create.
  • 2.3. Dataset construction: Prior outpainting work was limited to single crop suggestions, faced reliability issues in generated content, and was not publicly accessible.
  • 2.4. Retrieval augmentation: Retrieval augmentation fetches relevant information from an external database, commonly through nearest-neighbor matching in a precomputed embedding space.

3. Method

ProCrop retrieves professional images with similar compositional characteristics and fuses their features with query-image representations to generate aesthetic crop proposals. A composition-aware weakly supervised pipeline further creates diverse training pairs through GPT-4 descriptions, SAM masks, ControlNet outpainting, and iterative model-in-the-loop refinement.

  • 3.2. ProCrop: Retrieval-driven aesthetic cropping: ProCrop retrieves professional photographs by compositional similarity and uses their features to guide aesthetic cropping.SAM embeddings identify top-K matching professional images, while the retrieval pipeline is designed to capture composition rather than category similarity.
  • 3.2. ProCrop: Retrieval-driven aesthetic cropping: Retrieved features are projected and cross-attended with query-image features before entering the decoder, incorporating professional compositional knowledge.The fused representation combines projected retrieved features with a cross-attended feature derived from the query and retrieved images.
  • 3.2. ProCrop: Retrieval-driven aesthetic cropping: The model also fuses BLIP-derived multimodal features that describe salient objects and their spatial arrangements with the image features.These image-text embeddings provide language-based compositional cues alongside retrieved visual features.
  • 3.3. Composition-aware dataset generation: The weakly supervised dataset pipeline uses GPT-4 for within- and beyond-image composition descriptions, SAM masks, and ControlNet outpainting of professional images.Beyond-image descriptions are reported as essential for effective outpainting, while the generated dataset contains diverse image categories, professional crop proposals, and compositional descriptions.
  • 3.3. Composition-aware dataset generation: Instead of retaining one crop, iterative model-in-the-loop refinement generates varied proposals whose top-k outputs become pseudo-labels.Random crops initialize training, after which ProCrop repeatedly proposes crops; the selected top-k crops increase annotation diversity and generalization across cropping scenarios.

4. Experiments

Experiments evaluate ProCrop’s datasets, retrieval behavior, supervised and weakly supervised performance, and component ablations. Across these settings, professional-image retrieval and diverse composition sources consistently improve cropping results.

  • Datasets and metrics: The experiments use professional retrieval data from 55,000 top-scoring AVA images and a 242,000-image CAD generated from 80,000 curated professional photographs.Evaluation reports IoU, boundary displacement, and top-k accuracy, with crop equivalence defined by IoU greater than 0.85.
  • Comparative assessment: ProCrop significantly outperforms previous approaches across supervised datasets and metrics by retrieving professional compositions during training and inference.The supervised evaluation uses GAICv1, GAICv2, and CPC-trained models evaluated with ACC1/5, ACC1/10, and IoU.
  • Comparative assessment: With 90 predicted crops, ProCrop outperforms existing ensemble methods on unseen SACD in both IoU and Disp, while also producing multiple effective candidates.The full retrieval model further outperforms its no-retrieval variant in the weakly supervised setting.
  • Ablation study: Professional AVA retrieval achieves ACC5 0.860 and ACC10 0.945, exceeding retrieval from GAICv1 or CPC sources.CPC retrieval reaches ACC5 0.840 and ACC10 0.940, while GAICv1-only retrieval is comparable to the baseline.
  • Ablation study: Performance improves as more images are retrieved and stabilizes around ten images, where diverse layout information benefits IoU and Disp.Models trained on CAD and evaluated on SACD show similar IoU values with one retrieved image before improving with larger retrieval sets.
  • Ablation study: Image retrieval improves average IoU from 0.7103 to 0.7489 and lowers Disp from 0.0667 to 0.0555, with text embeddings providing further gains.These component results are reported for the weakly supervised setting.

5. Conclusion

The paper presents a composition-aware cropping framework that retrieves professional images with similar aesthetic compositions and introduces a large-scale compositional cropping dataset. Evaluations across retrieval and cropping tasks report state-of-the-art performance and broad benchmark applicability.

  • 5. Conclusion: ProCrop combines professional-image retrieval with query-image features and a large-scale composition-aware dataset to support aesthetic image cropping.The framework is evaluated across image retrieval, supervised cropping, and weakly supervised cropping tasks.

Supplementary Material

The supplementary material adds ablations and implementation analyses for ProCrop’s retrieval components and computational behavior.

  • Supplementary Material: Supplementary experiments analyze the retrieved-image encoder, retrieval time, memory consumption, and additional retrieval-related design choices.The supplementary material also covers the number of generated crops, efficiency, transferability, and retrieved-image influence.

A. Additional ablations

Additional ablations examine retrieval encoders, feature alignment, retrieval efficiency, and changing the retrieval database at inference. Results show benefits from SAM retrieval features, cross-attended fusion, professional retrieval sets, and efficient cached retrieval.

  • Retrieval encoder: SAM retrieval features slightly improve performance over cDETR while requiring less memory, attributed to SAM’s pretraining and boundary-feature extraction.The comparison uses GAICv2 with professional AVA images as the retrieval set.
  • Feature alignment: Cross-attended fusion further improves performance beyond directly concatenating query and retrieved features, demonstrating the benefit of enhanced feature interaction.The ablation compares no retrieved features, direct concatenation, and concatenation with cross-attention on SACD.
  • Retrieval efficiency: Retrieval takes 0.1–1 seconds for 10 images across databases containing 10,000–255,000 images, and cached relationships can reduce cost for larger sets.The retrieval process uses ElasticSearch and remains efficient across small, medium, and large databases.
  • Test-time retrieval flexibility: Using CPC as the retrieval set lowers ACC5 and ACC10, whereas Unsplash-lite performs comparably to AVAP, showing transferability across professional retrieval datasets.The model is trained on GAICv2 with AVAP and evaluated with alternative retrieval sets during inference.
  • Retrieved-image connections: Retrieved images need not match the query exactly; similar line compositions can still guide reasonable crop proposals.The examples use Unsplash-lite as the retrieval set for GAICv2.

B.1. Developed dataset

The developed dataset uses GPT-based descriptions and outpainting to create diverse crop proposals from professional images. Examples show that dual-space descriptions produce more realistic, detailed outpainted content.

  • Text generation and outpainting: Dual-space GPT-4 descriptions produce more realistic and detailed outpainting results than within-image descriptions or BLIP descriptions.Figure 11 compares BLIP, GPT-4 within-image, and GPT-4 dual-space descriptions with their corresponding outpainted images.
  • Dataset examples: The CAD examples pair outpainted images with diverse crop proposals, illustrating the versatility of the dataset-generation approach.The examples are presented as additional visualizations of outpainted images and their crop proposals.
  • Dataset diversity: The retrieval-dataset diversity analysis considers aesthetic scores, layout types, and semantic tags.Figure 13 visualizes the diversity of retrieval datasets.

B.2. Details of retrieved datasets

The retrieval system is designed to use professional photography and compositional layout features, with BLIP selected for practical text-embedding extraction. SAM features provide layout cues that complement image embeddings, while text adds only marginal gains.

  • Retrieval-set selection: The best results come from a professional photography retrieval set, notably AVA’s top 55,000 highest-rated images.The AVA subset’s diversity is analyzed through aesthetic scores, layout types, and semantic tags.
  • Text feature extraction: BLIP supplies text embeddings for multimodal fusion because it is open-source and practical on new test sets, whereas GPT-4 is more precise but costly.BLIP is used consistently between training and testing; GPT-generated text pairs are reserved for CAD dataset outpainting.
  • Multimodal fusion: Text embeddings provide only marginal performance improvements, while retrieved aesthetic rules remain central to generating high-quality crops.The motivation identifies language as a guide to salient image regions but reports limited gains from adding text embeddings.
  • Complex-scene composition: The method is intended to capture layout combinations and line compositions, including in complex scenes where salient-object cues are insufficient.The saliency visualization illustrates complex scenarios, while the layout assumption follows prior work on layout combinations.
  • Layout representation: SAM-derived line and layout features are treated as aesthetic guidelines and fused with image embeddings rather than using geometric masks directly.The extracted features are reported to correlate highly with geometric masks, as illustrated by K-means clustering.

D.2. Comparison to rule-based methods

Rule-based methods based on salient-object detection work well for simple center compositions but struggle in complex scenes without a clear salient object. ProCrop instead evaluates overall layout through line structures and retrieves professionally composed references with similar layouts.

  • Rule-based limitations: Saliency-based rule methods suit simple center compositions with one prominent object but struggle when complex scenes lack a clear salient object.The comparison is illustrated in Figure 15.
  • ProCrop comparison: ProCrop addresses these scenes by analyzing overall line structure and retrieving professional images with similar compositional layouts.The approach uses layout similarity rather than relying only on a single salient object.

E. Discussion

The paper identifies two limitations: aesthetic-quality metrics may not fully capture subjective quality, and user-specific cropping preferences remain unexplored. It also points to retrieved images as references for improving generation control and fine-grained segmentation under data scarcity.

  • Aesthetic-quality metrics may not fully reflect the true aesthetic quality of cropped areas because subjective annotations are imperfect.
  • The cropping process does not yet support user-specific composition preferences for personalized cropping styles.
  • Future work: Retrieved images could guide image generation for finer control and improved aesthetic quality, while also supporting fine-grained segmentation when data are scarce.
Loading 2505.22490v1…