PaperlayerView exact revision on arXiv

Source-linked AI summary

GenScale: A Benchmark for Relative Object Scale in Image Generation and Editing

Lingxiao Li, Max Whitton, Ledell Wu, Boqing Gong

arXiv:2609.00525v1cs.CV

TL;DR

Image generators and editors can preserve visual realism while violating real-world relative object scale. The paper introduces GenScale, a physically grounded benchmark and calibrated ordinal evaluator, alongside Rescale for localized correction; experiments find scale remains unreliable, while Rescale consistently improves plausibility. The results position relative scale as a distinct, measurable, and actionable capability.

  • Problem

    Image generation and editing systems can produce photorealistic, prompt-aligned images while rendering familiar objects at implausible relative sizes.

  • Method

    GenScale evaluates pairwise anchor-target scale relations across three regimes using physical-size metadata and a human-calibrated ordinal judge, while Rescale performs model-agnostic localized geometry-aware correction.

  • Results

    State-of-the-art generators and editors cannot reliably observe relative scale, while Rescale consistently improves scale plausibility across generated and edited images.

  • Takeaways & Limitations

    Relative object scale is a distinct, measurable, and actionable dimension for image generation systems.

  • Takeaways & Limitations

    SD3.5-Large has much lower scored coverage than other systems, especially in S4, so its results require cautious interpretation.

Abstract

from arXiv · show

Modern image generation and editing systems can produce photorealistic, prompt-aligned images, but still often render familiar objects at implausible relative sizes. To measure this failure mode, we introduce GenScale, a benchmark and evaluation protocol for real-world relative object scale in image generation and editing. GenScale contains 900 image-level entries and 1,643 pairwise anchor-target scale relations across common-object generation, human-product generation with metric dimensions, and scale correction from failed generations. We further design a human-calibrated ordinal judge for scalable pairwise scale evaluation. Last but not the least, we introduce Rescale, a model-agnostic post-processing agent for localized scale correction without modifying the source generator. Experiments reveal that state-of-the-art image generators and editors cannot reliably observe relative scale yet, while Rescale consistently improves scale plausibility across generated and edited images. Together, GenScale establishes relative object scale as a distinct, measurable, and actionable capability for image generation systems.

1 Introduction

GenScale frames relative object scale as a distinct physical-realism failure in image generation and editing, then introduces a benchmark, calibrated evaluator, and correction agent to measure and address it.

  • Motivation: Photorealistic, prompt-aligned images can still place familiar objects at mutually implausible relative sizes, undermining visual credibility.The issue matters for advertising, product visualization, virtual character creation, and professional content production.
  • Motivation: Across text-only generation, human-product generation, and scale correction, models often render small objects too large and large objects too small.The paper calls this systematic pattern mean regression in generated object scale.
  • Benchmark and evaluation: GenScale evaluates relative scale as pairwise anchor-target relations grounded in external object-size metadata across three regimes.The regimes are common-object size priors, human-product metric scale, and post-generation scale correction; the benchmark contains 900 image-level entries and 1,643 relations.
  • Benchmark and evaluation: Because scale depends on identity, physical size, perspective, depth, occlusion, and placement, GenScale uses a human-calibrated ordinal judge rather than raw pixel ratios.The Gemini-based judge is calibrated against annotations from nine human raters.
  • Correction: Rescale is a model-agnostic agent for localized, geometry-aware scale correction that does not modify the source generator.It can post-correct generated or edited images and provide corrected examples for model training.
  • Results: Experiments reveal persistent relative-scale failures in state-of-the-art generators and editors, while Rescale consistently improves scale plausibility across generated and edited images.The benchmark’s coverage spans composition, physical plausibility, and spatial-scale evaluation rather than treating scale as only a broader capability.

2 Related Work

Related benchmarks have improved evaluation of image quality, prompt following, composition, reasoning, and spatial relations, but relative object scale remains embedded within broader capabilities.

  • Generation progress: Text-to-image systems have progressed from diffusion and autoregressive models to transformer-based diffusion and rectified-flow systems.Recent open and closed models extend high-fidelity generation, reference conditioning, and image editing.
  • Benchmark landscape: Existing evaluations cover distributional quality, image-text similarity, prompt suites, compositional generalization, object attributes, reasoning, factuality, and spatial control.These benchmarks assess many capabilities but do not isolate relative object scale as the central relation.
  • Benchmark landscape: Relative object scale is related to spatial reasoning and physical commonsense, yet prior benchmarks usually subsume it under broader capabilities.GenScale instead targets physically grounded object-object scale relations directly.

3 GenScale: A Benchmark for Physically Grounded Relative Scale

GenScale constructs a physically grounded, pairwise benchmark spanning implicit object priors, explicit human-product metric scale, and scale correction, with human-calibrated ordinal evaluation.

  • Benchmark overview: The benchmark grounds scale judgments in object identities, physical reference lengths, expected 3D ratios, scenario labels, and product reference images when applicable.This structure avoids treating raw object-space ratios as sufficient evidence of plausibility.
  • Benchmark overview: GenScale contains 900 image-level entries and 1,643 pairwise scale relations across common-object generation, human-product scale, and post-generation correction.Each anchor-target pair is the atomic evaluation unit, while image-level grouping supports model comparison.
  • Task 1: Implicit Common-Object Scale: Task 1 tests implicit common-object size priors from text-only prompts, including natural-depth and same-plane scenarios.The same-plane setting reduces perspective ambiguity so image-space proportions more directly reflect real-world ratios.
  • Task 2: Explicit Metric Scale: Task 2 combines product reference images with metric dimensions and size-conditioned human anchors to test plausible human-product proportions.Products are paired with hands, heads or faces, feet or legs, or full bodies according to their dimensions.
  • Task 3: Scale Correction: Task 3 converts failed Task 1 and Task 2 generations into editing cases for automatic scale diagnosis and precise numeric resizing.S4 withholds the editable object, resize direction, and scale factor, requiring the model to diagnose and choose a local correction.
  • Human-Calibrated Scale Evaluation: The five-point ordinal rubric maps scores 1/2 to undersizing, 3 to plausible scale, and 4/5 to oversizing, with score 3 corresponding to approximately ±20% error.Pairs are invalid only when reliable judgment is impossible because objects are missing, merged, duplicated, or too degraded to identify.
  • Human-Calibrated Scale Evaluation: Human consensus is derived from nine raters over 285 calibration images and 527 anchor-target pairs, then used to calibrate a Gemini-based judge.The judge receives the image, object names, physical lengths, expected 3D ratio, and rubric; its disagreements are concentrated near the diagonal, with 2.85% differing by two or more levels.

4 Rescale: Agentic Relative-Scale Correction

Rescale is a model-agnostic post-processing pipeline that uses structured scale information to identify implausible object sizes and apply localized, geometry-aware corrections while preserving surrounding image content.

  • Rescale changes only implausibly scaled objects while preserving identity, layout, lighting, and background.
  • A multimodal agent grounds objects and converts pairwise scale evidence into an edit plan for target selection, resizing, and contact-preserving placement.For common-object scenes it aggregates inconsistent relations into a conservative object-level plan and may edit over multiple rounds; human-product scenes keep the body-part anchor fixed.
  • Rescale verifies the edited result and performs additional correction rounds when necessary.
  • The correction pipeline extracts the target, completes the background, adjusts masks and depth, and reinserts the resized object locally.It uses segmentation, background completion, enlarged edit masks, monocular depth, and a modular insertion backend.

5 Evaluation Results

GenScale evaluations show that relative scale remains unreliable across generation and editing, with depth sometimes masking errors and Rescale consistently improving plausibility.

  • Relative scale remains far from solved in common-object generation, human-anchored product generation, and general-purpose correction.The benchmark identifies mean regression, product magnification, and weak autonomous diagnosis as distinct failure modes.
  • Products are more often enlarged than shrunk in human-anchored generation despite explicit metric cues and human anchors.These cues improve scale realization relative to Task 1, but do not solve it.
  • Explicit localized resize instructions are easier for general-purpose editors to follow than autonomous physical-scale diagnosis.
  • Rescale reduces scale errors across Task 1 and Task 2 source models and produces its largest gain on Task 3 correction inputs.The results suggest that structured diagnosis and local editing outperform generic image editing for scale correction.
  • SD3.5-Large has much lower scored coverage than other models, especially in S4, so its results require cautious interpretation.

6 Conclusion

GenScale measures relative object scale across generation and editing, while Rescale improves scale plausibility; the benchmark also defines a bounded scope for physical-size reasoning.

  • GenScale covers common-object priors, human-product metric scale, and scale correction from failed generations.
  • A human-calibrated ordinal evaluator grounds pairwise judgments in object-size metadata rather than raw pixel ratios.
  • Relative scale remains unreliable in contemporary image generators, while Rescale consistently improves scale plausibility.
  • GenScale targets visually identifiable categories with stable physical dimensions, excluding deformable, fine-grained, or context-dependent objects.Broader validation is needed across viewpoints, occlusions, domains, and future models.

Overview

The appendix documents GenScale’s dataset construction, calibrated evaluation, Rescale implementation, quantitative and qualitative analyses, benchmark comparison, and physical-size knowledge base.

  • The appendix details GenScale construction, including physical-size metadata, filtering, prompts, task sampling, failure-case selection, and released metadata.
  • It specifies the human-calibrated protocol, including ordinal annotation, judge prompts, validity criteria, calibration diagnostics, and bootstrap confidence intervals.
  • Rescale’s appendix covers agentic diagnosis, edit planning, localization, segmentation, background completion, depth conditioning, insertion, training, and ablations.
  • The appendix reports complete results for GenScale Tasks 1–3 and Rescale correction, including score distributions, directional errors, coverage, confidence intervals, and quality preservation.
  • Additional appendix sections provide qualitative examples and a direct comparison between GenScale and GenSpace.
  • GenScale’s physical-size knowledge base contains 100 entries, including 97 common objects and three human body-part anchors.The categories derive from COCO and LVIS, while product dimensions and reference images come from ABO.
  • Task 1 evaluates implicit common-object priors using prompts with two to four objects and no object-specific metric sizes or reference images.It contains 400 prompts and 1,143 evaluated relations split nearly evenly between S1 Natural Depth and S2 Same Plane.
  • Depth analysis estimates within-image depth ranks and relates them to real-world size ranks using average ranks and Spearman correlation.Across seven generators, S1 places smaller objects closer and larger objects farther, whereas S2 largely removes this pattern.

A.3 Task 2: ABO Product Filtering and Human-Anchor Assignment

Task 2 uses ABO product metadata and size-conditioned human anchors to evaluate whether generated human-product images realize metric scale plausibly. Products are filtered by data validity and characteristic length, then paired deterministically with increasingly large human anchors.

  • Task 2 setup: Task 2 combines an ABO product, product reference image, metric dimensions, and a size-conditioned human anchor for evaluating product-to-human proportions.The prompt also requests natural interaction, catalog-style visibility, and mild perspective so scale remains interpretable.
  • Product filtering: Products must have an English title, valid dimensions, a retrievable main image, and characteristic length between 5 and 250 cm.Characteristic length is the maximum of the converted length, width, and height dimensions.
  • Human-anchor assignment: Human anchors are assigned by product length: hand below 30 cm, face/head from 30–60 cm, foot/leg from 60–100 cm, and full body above 100 cm.Canonical anchor lengths come from anthropometric references, enabling interpretable product-to-human scale comparisons.
  • Pairwise evaluation: Each Task 2 image contains one product and one human anchor, producing one evaluated product-to-anchor scale relation.The ground-truth ratio uses product_scale.typical_len_cm and the corresponding canonical human-anchor length.

B Human and VLM Evaluation Details

GenScale evaluates relative object scale through independently judged anchor–target pairs, using human-calibrated ordinal criteria that account for scene geometry and visibility. Its calibration supports a Gemini judge, while Rescale uses depth-aware local editing for scale correction.

  • Annotation interface and benchmark records: Each image is evaluated through anchor–target pairs, with benchmark metadata supplying object identities and reference physical lengths.Reference lengths are semantic scale references rather than pixel measurements, and anchor roles must not be reversed.
  • Ordinal scale rubric: The five-point rubric labels targets from severely undersized to severely oversized, with score 3 denoting physically plausible relative scale.Approximate error guidance assigns ±20% to score 3, 20–60% to scores 2 and 4, and over 60% to scores 1 and 5.
  • Scene-aware judgment: Evaluators correct for perspective, depth, foreshortening, occlusion, and partial visibility before penalizing apparent size discrepancies.The protocol uses the target’s intended full extent and treats visible but incorrectly scaled objects as scorable rather than invalid.
  • Human calibration: Human calibration achieves 65.15% ± 6.98% exact agreement and 94.03% ± 3.16% within-one agreement with aggregate consensus.The leave-one-rater-out comparison remains 58.00% ± 6.56% exact and 93.35% ± 3.03% within-one, with QWK 0.7058 ± 0.0695.
  • VLM judge calibration: Gemini reaches 63.95% exact agreement, 97.15% within-one agreement, MAE 0.3890, Pearson correlation 0.8328, and QWK 0.8234 against human consensus.These results support using the calibrated judge for large-scale model comparison on the combined Task 1+2 calibration split.
  • Scale-correction backend: Rescale’s depth-aware backend retains a reference-side appearance cue while injecting depth residuals only into target-half tokens for localized correction.Under generic edited-object crop metrics, InsertAnything remains strongest overall, whereas the specialized backend targets mask–extent decoupling and depth-controlled resizing.

C.5 Correction-Model Training Ablation

The correction-backend ablation finds that guidance and depth conditioning are the main effective components, while detail-preservation additions are not strongly reflected in current crop-level metrics.

  • C.5 Correction-Model Training Ablation: Guidance conditioning substantially improves representation-level consistency, especially DINO, and also improves both no-reference quality metrics.The baseline uses the same diptych insertion formulation without additional guidance, depth, detail-aware loss, or high-frequency processing.
  • C.5 Correction-Model Training Ablation: Adding depth conditioning further improves CLIP-I, DINO, SSIM, LAION-Aes, and Q-Align-IQ, supporting explicit geometry for scale correction.The ablation identifies depth-aware conditioning as an effective design choice for the correction backend.
  • C.5 Correction-Model Training Ablation: Guidance and depth conditioning are the main effective components, whereas the detail-preservation branch is not strongly reflected by current crop-level metrics.The combined ablation and backend substitution results characterize the depth-aware model as a functional attempt at more appropriate local insertion.
  • C.5 Correction-Model Training Ablation: Future work should target numeric resize fidelity, contact preservation, and decoupling mask extent from object extent.The generic InsertAnything checkpoint remains a very strong synthesis backend, motivating a specialized model and directly targeted losses and metrics.

D.5 Rescale Correction on Task 1

Rescale is evaluated on matched before–after pairs across Task 1, Task 2, and Task 3, using calibrated scale metrics and visual-quality measures to assess correction effects.

  • D.5 Rescale Correction on Task 1: Rescale Task 1 results use matched scorable pairs scored on both the original and corrected images.The correction table reports Error before, Error after, Gain, Plausible, and Severe under this matched-pair protocol.
  • D.5 Rescale Correction on Task 1: Rescale correction is also reported for human–product scale and the Task 3 correction benchmark under matched-pair evaluation.Task 2 has one evaluated pair per image, and Task 3 includes both S4 and S5 source images.
  • D.5 Rescale Correction on Task 1: Visual evaluation measures reference consistency, image preservation, high-frequency preservation, and no-reference image quality before and after correction.The metrics include CLIP-I, DINO, SSIM, SSIM-HF, LAION-Aes, and Q-Align-IQ.
  • D.5 Rescale Correction on Task 1: Qualitative examples visualize inputs, outputs, and correction behavior, while quantitative conclusions rely on calibrated metrics from the main paper and appendix tables.Additional examples cover benchmark tasks and Rescale corrections, including natural-depth and same-plane Task 1 settings.

E.4 Task 3: General-Purpose Editor Correction Examples

General-purpose editors often preserve visual realism during Task 3 correction but may fail to produce a sufficiently localized and accurate scale edit.

  • E.4 Task 3: General-Purpose Editor Correction Examples: Task 3 examples show a diagnosis–execution gap in which editors may under-correct, over-correct, or regenerate content beyond the intended localized edit.The first column is the erroneous source image, followed by editor outputs under corresponding correction prompts.
  • E.4 Task 3: General-Purpose Editor Correction Examples: Task 3 correction outcomes are evaluated with matched-pair scale metrics and identity-preservation and visual-quality measures.The relevant tables report Task 3 before–after correction results and visual consistency and quality after Rescale correction.
  • E.4 Task 3: General-Purpose Editor Correction Examples: Rescale examples emphasize resizing and reinserting the target while keeping the surrounding scene, reference object, lighting, and background fixed.The intended correction is localized rather than a full-scene regeneration.
  • E.4 Task 3: General-Purpose Editor Correction Examples: In S4, failures often originate in diagnosis; in S5, failures more often reflect insertion-backend limitations such as blending, distortion, or unintended local changes.S4 requires autonomous discovery, whereas S5 specifies the edit plan and therefore isolates more of the insertion problem.

F Comparison to GenSpace

GenScale evaluates physical relative scale more finely than GenSpace’s binary larger-versus-smaller criterion, revealing errors that the coarser benchmark can miss.

  • F Comparison to GenSpace: GenSpace labels an object pair correct when the prompt-specified larger object has at least 1.2 times the predicted volume of the smaller object.Its Relative Size criterion is part of a broader spatial-awareness suite.
  • F Comparison to GenSpace: GenScale instead uses object-pair-specific physical size ratios and tolerance intervals to assess real-world proportions.This produces a finer-grained evaluation of rendered relative scale.
  • F Comparison to GenSpace: Many pairs that pass GenSpace are judged incorrect by GenScale, indicating that GenScale detects errors invisible to a coarse larger-versus-smaller test.The comparison excludes 14.5% of Task 1 pairs that GenSpace failed to detect and omits Task 2 as out of distribution.
  • F Comparison to GenSpace: The benchmark spans natural-depth and same-plane common-object generation, human–product scale, and correction scenarios with either auto-discovery or precise instructions.These scenarios provide distinct input conditions for evaluating relative-scale capability and correction.
  • F Comparison to GenSpace: Rescale correction examples isolate localized resizing and reinsertion, while failure cases expose diagnosis and insertion limitations across S4 and S5.The correction scenarios separate autonomous scale-error discovery from explicitly specified resizing.
Loading 2609.00525v1…