PaperlayerView exact revision on arXiv

Source-linked AI summary

Discrete Annotation, Continuous Preference: Rethinking Supervision for Accurate and Generalizable Aesthetic Image Cropping

Ziqing Zhang, Xiao Liu, Kai Liu, Jianze Li, Weihang Zhang, Linghe Kong, Yulun Zhang

arXiv:2610.00582v1cs.CV

TL;DR

Aesthetic cropping relies on annotations that can be subjective and confined to fixed grids, limiting the supervision available for crop preference and evaluation. The paper reconstructs a continuous preference field from discrete annotations and uses it to train CPIC and recalibrate benchmarks. It reports state-of-the-art accuracy and exceptional out-of-domain generalization for CPIC, alongside a continuous evaluation standard through CPICD.

  • Problem

    Existing cropping datasets rely on small, subjective rater pools and restrict candidate boxes to sparse grids or predefined aspect ratios.

    Source: §1 · p. 2, §3.1 · p. 4

  • Method

    CPF reconstructs a dense, multi-peaked, continuous, and sharp preference landscape from discrete annotations, and its reward guides GRPO training of the VLM-based CPIC model.

    Source: §1 · pp. 2–3

  • Results

    CPIC achieves state-of-the-art accuracy and exceptional out-of-domain generalization, while CPICD recalibrates mainstream benchmark boxes to provide a continuous evaluation standard.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The paper presents discrete annotations as usable starting points for continuous preference modeling and benchmark recalibration.

    Source: §1 · pp. 2–3

    The continuous approximation assumes that discrete, individually noisy boxes collectively sample the core tendencies of human preference.

    Source: §1 · p. 2

Abstract

from arXiv · show

Aesthetic image cropping aims to identify the optimal crop of an image in terms of aesthetics and composition. While supervision based on annotated data is fundamental, the field has been hindered by a long-standing problem: existing datasets suffer from (1) human subjectivity and (2) rigid discreteness confined to fixed sampling grids. These flawed annotations not only limit the accuracy and generalization of trained models but also severely distort fair evaluation. To overcome this, we propose to model human cropping preference as a multi-peaked, continuous, and sharp field over the crop space. We introduce the Continuous Preference Field (CPF), which recovers a dense preference landscape from discrete annotations through (1) peak clustering, (2) off-lattice refinement, (3) negative shaping, and (4) field assembly. Based on this, we train CPIC, a VLM-based cropping model optimized via GRPO with the CPF reward, which overcomes template collapse, achieving state-of-the-art performance and exceptional out-of-domain generalization. Finally, to resolve the long-standing benchmark evaluation crisis, we introduce CPICD, a comprehensive recalibration of existing ground-truth boxes. By leveraging the CPF to correct grid-bound artifacts across mainstream benchmarks, CPICD establishes a rigorous and reliable foundation for future cropping research. Extensive experiments and user studies demonstrate the superiority of our CPF, CPIC, and CPICD. Code, model, and data are available at https://github.com/zzqingz/CPIC.

1 INTRODUCTION

Aesthetic cropping datasets are subjective and grid-bound, while human crop preference is multi-peaked, continuous, and sharp. The paper introduces CPF, trains CPIC with its reward, and uses CPF to recalibrate benchmark annotations as CPICD.

1 INTRODUCTION

  • Existing datasets reflect small pools of raters and restrict crop candidates to sparse grids or predefined aspect ratios, creating subjective and discrete supervision.The introduction gives 6 to 19 raters as examples of the small pools.Source: §1 · p. 2
  • Human crop preference can support multiple disjoint framings, tolerate small pixel shifts, and sharply penalize crossing critical visual boundaries such as clipping a face.The paper argues that fixed-grid annotations do not represent these geometric properties of preference.Source: §1 · p. 2
  • CPIC uses the CPF reward with GRPO to optimize VLM cropping in continuous space, overcoming template collapse and achieving state-of-the-art accuracy and exceptional out-of-domain generalization.The paper also reports that experiments and user studies validate the superiority of its framework.Source: §1 · p. 3
  • CPICD applies CPF to recalibrate mainstream benchmark ground-truth boxes, addressing grid artifacts and establishing a continuous evaluation standard.The paper characterizes existing evaluation ground truths as flawed anchors.Source: §1 · p. 3
  • CPF models cropping preference as a multi-peaked, continuous, and sharp field, recovering a dense reward landscape from discrete annotations through peak clustering, off-lattice refinement, negative shaping, and field assembly.The paper frames discrete annotations as noisy individually but useful collectively as starting points for a continuous approximation.Source: §1 · pp. 2–3

2 RELATED WORK

Prior cropping methods progress from proposal scoring to coordinate regression and VLM decoding, while cropping datasets constrain what crops can be annotated or evaluated. This paper instead builds on existing annotations to evaluate beyond candidates and refine ground-truth boxes.

2.1 AESTHETIC IMAGE CROPPING

  • Aesthetic cropping methods evolved from proposal scoring to direct coordinate regression and VLMs that decode bounding boxes as discrete text tokens, whose vocabularies can reproduce annotation-lattice artifacts.The passage describes the current state of the art as autoregressive VLM cropping via prompting, instruction tuning, or pretraining.Source: §2.1 · p. 3

2.2 CROPPING DATASETS

  • Cropping datasets range from professional, crowd-sourced, and ranked crops to fixed aspect ratios or anchor grids; this paper builds on existing annotations to evaluate beyond candidates and refine ground-truth boxes.Examples include GAIC's approximately ninety candidates per image on a 12 × 12 anchor grid [Zeng et al., 2019; 2020].Source: §2.2 · p. 3

3 METHODOLOGY

The method addresses subjective, grid-constrained supervision by constructing CPF as a continuous preference field, then uses it to train CPIC and recalibrate benchmark annotations as CPICD.

3.1 PRELIMINARY: RETHINKING THE SUPERVISION SIGNAL

  • Across 200 images, published croppers emit only 18 to 43 distinct coordinate values, while benchmarks rely on just 6 to 19 annotators and coarse grids.The annotations thus reflect small rater pools and restrict candidate crops to coarse anchor grids or sparse predefined aspect ratios.Source: §3.1 · p. 4

3.2 CPF: CONSTRUCTING THE CONTINUOUS PREFERENCE FIELD

  • CPF clusters high-scoring crops into preference peaks, then refines them off-grid using human preference, composition, and subject-preservation scores.The refinement combines the three scores equally and is intended to reduce subjective annotation bias.Source: §3.2 · pp. 4–5
  • Five structured negative classes model undesirable crops, including clipped subjects, broken compositions, discarded context, templated crops, and low-MOS annotations.These negatives encode boundary conditions where aesthetic quality can drop sharply.Source: §3.2 · p. 5
  • CPF assembly combines peak-distance scores with a soft gate based on proximity to structured negatives, taking the maximum weighted response across preference peaks.Its distance uses overlap, boundary displacement, scale, and shape terms that contribute equally.Source: §3.2 · pp. 5–6

3.3 CPIC: LEVERAGING CPF AS SUPERVISION SIGNAL

  • CPIC cold-starts with refined peaks as supervised targets, then uses GRPO with CPF scores as rewards for sampled crops.The authors state that CPF keeps a healthy reward gap between good and bad boxes, avoiding reward collapse.Source: §3.3 · p. 6

3.4 CPICD: RECALIBRATION OF BENCHMARK GROUND TRUTH WITH CPF

  • CPICD recalibrates benchmark boxes by perturbing each historical annotation and selecting a refined crop with the rule-regularized preference score.The resulting annotations provide an alternative continuous ground truth for evaluation.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments compare CPIC with prior cropping methods, report strong out-of-domain performance and favorable user-study results, and ablate CPF’s reward and design components. The studies also specify the training and evaluation setup.

4.1 EXPERIMENTAL SETTINGS

  • CPIC is trained on CPC and GAIC and evaluated on GAIC, FLMS, and FCDB using dataset-specific ranking, accuracy, overlap, and boundary metrics.Training uses SFT followed by GRPO, with Qwen3-VL-2B-Instruct, LoRA, and 4 RTX 4090 GPUs for the GRPO stage.Source: §4.1 · p. 7

4.2 COMPARISON WITH THE STATE-OF-THE-ART METHODS

  • CPIC ranks first or second on ten of eleven metric columns and performs strongly on unseen FLMS and FCDB benchmarks.On GAIC, GAIC and Venus lead ACC10, while CPIC surpasses both on SRCC and LAION-Aesthetics.Source: §4.2 · p. 8
  • In qualitative examples, CPIC balances two birds and frames a fox with context, while baselines leave distracting space or crop too tightly.Figure 4 presents in-domain GAIC and out-of-domain FCDB examples.Source: §4.2 · p. 8, Figure 4 · p. 8
  • Participants selected CPIC most often for crop quality and significantly preferred CPICD over original benchmark annotations.The first study selected among crops from six methods; the second used two-alternative choices between original and recalibrated annotations.Source: §4.2 · p. 8

4.3 ABLATION STUDIES

  • Only CPF-guided GRPO yields comprehensive gains across reference-based and no-reference metrics; naive discrete-IoU GRPO emits just 92 distinct boxes.CE with refined continuous targets expands diversity but remains limited in accuracy, and GRPO on those targets still misses aesthetic boundaries.Source: §4.3 · pp. 8–9
  • Restricting CPF to one peak discards alternative valid framings and reduces out-of-domain generalization on FLMS and FCDB.The single-peak variant slightly overfits the GAIC training domain; multi-peak supervision provides a more robust balance across domains.Source: §4.3 · p. 9
  • Removing off-lattice refinement returns targets to the annotation grid and reduces crop diversity to 88 distinct boxes; the full model emits 199.The off-grid target supports sub-grid coordinates and significantly boosts overlap metrics.Source: §4.3 · p. 9
  • Removing structured negatives weakens penalties for critical visual violations, while CPF’s sharp valleys improve LAION aesthetic quality and box accuracy.The example violation is slightly cutting into a subject’s face, which produces only a minor IoU shift.Source: §4.3 · p. 9

5 CONCLUSION

The conclusion presents CPF as a continuous preference reward for training CPIC and recalibrating benchmark labels as CPICD. It reports state-of-the-art accuracy and out-of-domain generalization for CPIC, and describes CPICD as a continuous evaluation foundation.

5 CONCLUSION

  • CPF converts coarse labels into a dense reward landscape, supporting CPIC training and CPICD recalibration of benchmark boxes.The field is assembled through peak clustering, off-lattice refinement, structured negative shaping, and field assembly; the conclusion reports CPIC’s state-of-the-art accuracy and exceptional out-of-domain generalization.Source: §5 · p. 9

APPENDIX

The supplementary material gathers implementation and inference details, interactive cropping demonstrations, extended ablations, human-evaluation protocols, and further visualizations.

APPENDIX

  • The appendix covers implementation and inference, interactive cropping, additional ablations, user-study protocols, and qualitative visualizations.Source: APPENDIX · p. 13

A IMPLEMENTATION DETAILS

The appendix specifies CPF construction and CPIC optimization settings, then describes the single-crop inference procedure and its coordinate format.

A IMPLEMENTATION DETAILS

  • CPF retains at most four peaks, samples 40 perturbations per peak, and removes negatives with IoU ≥0.85 to their refined peak.The field uses shared temperature τ = 0.10 and drift penalty λ = 0.15.Source: A IMPLEMENTATION DETAILS · p. 13
  • CPIC uses LoRA rank 64, effective batch size 16, and learning rates of 10^-4 for SFT and 3 × 10^-5 for GRPO.GRPO uses G = 8, β = 0.04, and ϵ = 0.2; rollouts use temperature 1.0, top-p = 0.95, and at most 64 tokens.Source: A IMPLEMENTATION DETAILS · p. 13
  • At inference, CPIC deterministically returns one crop as JSON bbox_2d coordinates in a normalized 0–1000 image coordinate system.The prompt requests the single best aesthetic crop; greedy decoding avoids sampling.Source: A IMPLEMENTATION DETAILS · p. 13

B ZERO-SHOT INTERPRETABILITY AND INTERACTION

The section tests whether box-focused CPIC can explain crop choices and follow natural-language edits without explicit rationale or dialogue training. Adding explanations to its output did not consistently improve results.

B ZERO-SHOT INTERPRETABILITY AND INTERACTION

  • Adding a short crop explanation produced no consistent benefit: it improved GAIC and FCDB but degraded FLMS, with only small overall changes.CPIC therefore outputs boxes only, isolating CPF supervision and avoiding extra generation overhead.Source: B ZERO-SHOT INTERPRETABILITY AND… · p. 13
  • The same checkpoint is probed without demonstrations or parameter updates for crop explanations and natural-language requests to change framing or subjects.The model was not trained with aesthetic-rationale annotations or multi-turn crop-refinement dialogues.Source: B ZERO-SHOT INTERPRETABILITY AND… · p. 14
  • In representative examples, CPIC grounds explanations in subject, arrangement, contrast, and visual effect, and reframes musicians to remove a distracting pedestrian.The figure pairs a rationale example with an interactive crop-refinement example.Source: B ZERO-SHOT INTERPRETABILITY AND… · p. 14, Figure 5 · p. 14

C ADDITIONAL JUSTIFICATION

Additional analyses examine the assumptions and construction choices behind CPF and CPICD. They qualify the continuity claim, compare refinement choices, and assess recalibrated boxes through human preference.

C ADDITIONAL JUSTIFICATION

  • 1, 0.9099, and 0.8273 oracle IoU are reported for GAIC, FLMS, and FCDB, respectively, under GAIC’s 12 × 12 anchor template.The template matches GAIC’s annotations, while integer coordinates in [0, 1000] yield near-unit oracle IoU across the three datasets; the model is not claimed to produce strictly continuous coordinates.Source: C ADDITIONAL JUSTIFICATION · pp. 14–15, Table 8 · p. 14
  • Adding composition and subject-preservation regularization improves all three metrics in successive refinement steps.Direct learned-preference optimization gives modest gains; its GAIC and FLMS improvements are smaller than with the IoU reward, while FCDB IoU is equal.Source: C ADDITIONAL JUSTIFICATION · p. 15, Table 9 · p. 15
  • Structured negatives lower CPF scores for distracting content, excessive margins, and subject truncation even when IoU remains high.They preserve tolerance around good compositions while sharply penalizing visually damaging changes.Source: C ADDITIONAL JUSTIFICATION · p. 15
  • Participants favor CPICD recalibrated crops over original annotations, providing human-preference evidence beyond agreement with the construction score.CPICD retains human annotations as anchors while relaxing discrete sampling constraints and incorporating composition and subject preservation.Source: C ADDITIONAL JUSTIFICATION · p. 15

D USER STUDY DETAILS

The web-based study collected human judgments on cropping quality and benchmark recalibration. Both sub-studies used balanced samples from GAIC, FLMS, and FCDB, with anonymized and randomized choices.

D USER STUDY DETAILS

  • 1,200 human preference evaluations covered cropping quality and ground-truth recalibration, with 20 participants choosing aesthetically pleasing compositions based on first impressions.The authors report no strictly correct answers.Source: D USER STUDY DETAILS · pp. 15, 17
  • Each sub-study used 30 test images—10 each from GAIC, FLMS, and FCDB.For recalibration, users chose between the original annotation and CPICD; for crop quality, they selected their top 1 to 3 among six crops.Source: D USER STUDY DETAILS · p. 17
  • Method names were hidden as A–F, and question order and option layouts were shuffled for every participant.The crop-quality choices included GAIC, CACNet, UNIC, InstructCrop, Venus, and CPIC.Source: D USER STUDY DETAILS · p. 17

E ADDITIONAL VISUALIZATIONS

Additional visualizations compare CPIC with other croppers and show examples of CPICD recalibrations. The examples span GAIC, FCDB, and FLMS.

E ADDITIONAL VISUALIZATIONS

Related papers

Loading 2610.00582v1…