PaperlayerView exact revision on arXiv

Source-linked AI summary

PaintBench: Deterministic Evaluation of Precise Visual Editing

Kai Xu, Ellis Brown, Shrikar Madhu, Rob Fergus, He He, Saining Xie

arXiv:2606.00188v1cs.GRcs.CVcs.LG

TL;DR

Precise single-answer visual editing remains difficult to evaluate because many existing methods rely on subjective or learned judgment. PAINTBENCH addresses this gap with procedurally generated tasks and deterministic pixel-level scoring, finding low performance across models and strong correlation with applied visualization editing.

  • Problem

    Precise single-answer edits remain difficult to evaluate reliably because existing benchmarks often use human judgment, learned judges, or perceptual proxies.

  • Method

    PAINTBENCH procedurally generates 20 fundamental editing operations with configurable scene parameters and evaluates outputs by deterministic pixel-level comparison without judge models.

  • Results

    Across 11 models, the best-performing model reaches only 17.1% mIoU, while geometric transformation, formula-based color change, and most structural manipulation tasks remain consistently difficult.

  • Takeaways & Limitations

    PAINTBENCH provides a configurable diagnostic for precise editing, and its scores strongly correlate with performance on TinyGrafixBench data-visualization editing.

  • Takeaways & Limitations

    The framework may grade small or thin edit-regions more strictly than their error magnitude warrants, and non-unique edits may require more complex deterministic metrics.

Abstract

from arXiv · show

While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introduce PaintBench, a dynamically scalable benchmark targeting 20 fundamental precise visual editing operations across four categories: geometric transformation, structural manipulation, color change, and symbolic reasoning. Procedural generation with configurable complexity enables an effectively infinite, contamination-resistant evaluation suite, and deterministic pixel-level evaluation eliminates reliance on bias-prone judge models. Across 11 image editing models, we find overall low performance, with the current highest-performing industry leader scoring only 17.1% (mIoU). Task decomposition reveals especially challenging operation types (geometric transformation, most structural manipulation, formula-based color change) and model-specific specializations. Fine-grained benchmark diagnostics further show performance degradations induced by scene variations in object count, background complexity, color scheme, and edit-region size. To test generalization of PaintBench scores to applied task performance, we create a procedural, deterministic evaluation for data visualization editing (TinyGrafixBench) and find strong linear correlation with PaintBench scores ($R^2 = 0.91$, $p < 0.001$). Altogether, PaintBench provides a rigorous foundation for measuring and driving progress in precise multimodal visual editing.

1. Introduction

PAINTBENCH addresses the difficulty of evaluating precise edits with unique correct outputs by combining procedurally generated tasks with deterministic pixel-level verification. Across 11 models, performance remains low and varies substantially by operation and scene condition.

  • PAINTBENCH targets editing tasks with unique correct outputs, where deterministic evaluation can replace human or model-based judgment.Examples include moving a shape, flood-filling a region, and removing the smallest instance of a shape.
  • Procedural generation creates input images, instructions, and answer images from seeds and configurable scene parameters.This supports fresh task sets and targeted evaluation of operations or visual conditions.
  • 17.1% mIoU is the highest score among 11 evaluated image editing models.Geometric transformation, formula-based color change, and most structural manipulation tasks are consistently difficult, while removal and single-color operations are more tractable.
  • Striped backgrounds, high object counts, nonstandard palettes, and small edit-regions substantially degrade performance.Models also show task-specific specializations that do not align perfectly with their overall rankings.
  • 20 fundamental operations span geometric transformation, structural manipulation, color change, and symbolic reasoning in a 1,920-problem test set.The benchmark also uses a pixel-level protocol that evaluates edit and preservation quality without bias-prone judge models.

2. Related Work

PAINTBENCH extends deterministic synthetic evaluation to precise visual editing, contrasting with benchmarks that commonly use subjective judgments, learned judges, or perceptual proxies. Its procedural design also supports dynamic, contamination-resistant testing without relying on a static benchmark snapshot.

  • Prior image-editing benchmarks commonly evaluate real-image edit triplets using human judgment, vision-language model scoring, or perceptual metrics.These approaches suit open-ended edits with many valid outputs but are less direct for correctness verification.
  • Dynamic generation addresses benchmark drift by producing fresh problems at will, helping prevent saturation and contamination.This positions PAINTBENCH as a scalable alternative to static evaluation sets.
  • PAINTBENCH follows a synthetic visual reasoning tradition established by programmatic benchmarks with deterministic answers.Its pixel-level verification is philosophically aligned with HumanEval's execution-based correctness testing.
  • PAINTBENCH evaluates native pixel-space output and does not cover approaches that write code or call external tools for editing.The benchmark tests whether models that understand scenes can also edit them exactly.
  • Figure 2 depicts a pipeline from seeded scene generation to pixel-wise comparison against the answer and input images.The comparison avoids reliance on bias-prone judge models.

3. Benchmarking Precise Visual Editing

PAINTBENCH is a procedurally generated, deterministic benchmark for precise single-answer visual editing. It decomposes evaluation into atomic operations and controlled scene conditions, enabling systematic testing of model sensitivity.

  • Design and scale: Each problem is an input image, instruction, and answer image generated from a seed.The test set contains 20 tasks, 8 visual conditions, and 12 problems per task-condition combination, totaling 1,920 problems.
  • Design principles: Determinism defines a unique correct output through A = f(I, t) and evaluates model output by pixel-level comparison against the answer and input.The design excludes judge models, perceptual proxies, and ambiguity.
  • Design principles: Dynamic generation uses random seeds to produce fresh problem sets, while explicit scene parameters enable controlled difficulty and precise ablations.The parameters include canvas dimensions, object count, background texture, and color palette.
  • Task categories: The 20 task types are organized into geometric transformation, structural manipulation, color change, and symbolic reasoning categories.These categories cover transformations, scene-composition changes, color operations, and reasoning-based edits.
  • Visual conditions: Each visual condition changes exactly one scene parameter relative to a baseline of 1024 × 1024, n = 3 shapes, a standard palette, and a single-color background.This isolates sensitivity to aspect ratio, palette, background texture, and object count.

4. Pixel-Level Evaluation

PAINTBENCH evaluates precise edits by separating changed edit regions from unchanged preservation regions and comparing model outputs with answer images at pixel level. Its mIoU score aggregates correctness across color tolerances while penalizing both failed edits and unintended changes.

  • Edit-region E contains pixels differing between input and answer images, while preservation-region P contains identical pixels.
  • IoU@t increases with looser color tolerance t, and mIoU averages IoU@t across t ∈ {0, . . . , 10}.The sweep spans exact pixel matching through a tolerance of 10.
  • The evaluation classifies pixels as correctly or incorrectly edited and preserved according to their color-distance tolerance.The four sets are correctly edited, incorrectly edited, correctly preserved, and incorrectly preserved.
  • IoU@t jointly penalizes failed edits and corruption of the preservation region, remaining robust when the preservation region is much larger.
  • Table 1 reports mIoU (%) per task and model, with category averages over five tasks and a benchmark average over 20 tasks.The benchmark average includes 95% bootstrap confidence intervals.

5. Leading Models Fail to Execute Precise Edits

Across 11 models, precise editing remains difficult: geometric transformation, formula-based color change, and most structural manipulation tasks are consistently hard, while performance varies substantially by operation and scene condition.

  • Overall performance: 17.1% mIoU is the best overall score across the 1,920-problem benchmark, with GPT-IMAGE-2 at 16.3% and open-weights models ranging from 6.7% to below 1%.Aggregate rankings also conceal a clear difficulty gradient and pronounced per-task specialization.
  • Operation difficulty: No model exceeds 17.5% mIoU on any geometric task, while shearing and scaling reach at most 7.8% across models.Most structural manipulation tasks and formula-based color changes are likewise consistently challenging.
  • Operation difficulty: 50.6% is achieved on removal, compared with 27.1% on flood fill and 30.4% on recolor, making removal and single-color operations more tractable.These tasks require filling a connected region with one color or removing content, rather than per-pixel computation.
  • Model specialization: GPT-IMAGE-2 leads geometric transformation and structural manipulation, while NANO-BANANA-2 leads color change and symbolic reasoning.The leading models therefore specialize in complementary categories despite similar overall performance.
  • Failure modes: Color imprecision improves only at lenient tolerances, whereas execution omission stays near zero across tolerances and structural catastrophe collapses all metrics.Structural imprecision instead plateaus at moderate edit-region accuracy, providing distinct per-problem diagnostic signatures.
  • Scene sensitivity: Striped backgrounds and high object counts cause the largest mIoU drops, while nonstandard palettes sharply reduce exact edit- and preservation-region accuracy.For example, BAGEL’s preservation accuracy falls from 19.2% to 0.3% under the nonstandard palette.
  • Scene sensitivity: 0.9% mIoU at edit-regions below 322 pixels rises to 28.7% at regions of at least 2,562 pixels for NANO-BANANA-2.All models over-edit, by roughly 50–1,400× for the smallest regions, driving sharp declines on small edits.

6. TINYGRAFIXBENCH: Generalization Beyond Synthetic Shapes

TinyGrafixBench transfers PaintBench’s procedural, deterministic evaluation to chart editing across five chart types and four operations. Scores are slightly lower but strongly correlated with PaintBench performance.

  • Benchmark design: TinyGrafixBench contains 600 deterministic problems across bar, scatter, line, heatmap, and network charts, with construction, transformation, removal, and recoloring tasks.Charts are rendered at 1024 × 768 pixels and evaluated with the same mIoU protocol as PaintBench.
  • Results: 15.9% is NANO-BANANA-2’s TinyGrafixBench score, followed by GPT-IMAGE-2 at 15.6%, while the other models score 5.3% or below.Overall TinyGrafixBench performance is slightly lower than PaintBench performance.
  • Generalization: R^2 = 0.91, p < 0.001 for the correlation between model scores on TinyGrafixBench and PaintBench.The result supports the benchmark’s relevance to applied data-visualization editing.
  • Model-specific patterns: TinyGrafixBench exposes model-specific chart-editing strengths, including BAGEL on filter series and FLUX.1-KONTEXT-DEV on swap axes.It also reveals large capability gaps between NANO-BANANA-2 and NANO-BANANA-1 on several chart tasks.

7. Discussion

PAINTBENCH addresses deterministic evaluation for visual edits with unique correct outputs, using procedural generation and configurable scene variation. Results indicate broad model brittleness, while the benchmark’s scores strongly correlate with performance on applied chart-editing tasks.

  • Benchmark scope: PAINTBENCH targets unique-correct-output edits underserved by subjective evaluation frameworks, using procedurally generated problems and pixel-level comparison without judge models or perceptual proxies.Its configurable generation supports fresh, contamination-resistant problem sets and controllable difficulty.
  • Findings: Geometric transformation, formula-based color change, and most structural manipulation tasks remain consistently difficult, whereas removal and single-color operations are more tractable.The benchmark also exposes task-specific model specializations despite generally low scores.
  • Implications: R2 = 0.91, p < 0.001 for the linear association between PAINTBENCH and TINYGRAFIXBENCH mIoU scores across models.The OLS fit closely tracks y = x, supporting comparison with the applied chart-editing benchmark.
  • Implications: The procedurally generated, deterministically evaluated framework can extend beyond 2D raster editing to scientific visualization, engineering drawing, simulation, and 3D scene manipulation.The paper presents this broader applicability as a future direction for tasks with unique correct edits.
  • Benchmark scope: 20 task types span geometric transformation, structural manipulation, color change, and symbolic reasoning.The taxonomy includes 35 task-modes across four categories.
  • Diagnostics: PAINTBENCH varies shape vocabulary, orientation, aspect ratio, control points, palettes, and striped backgrounds to diagnose sensitivity to scene conditions.Scene parameters are generated deterministically, with background and holdout colors excluded from the object pool.

A.7. TINYGRAFIXBENCH: Benchmark Design

TINYGRAFIXBENCH transfers the deterministic-edit framework from synthetic shape scenes to Matplotlib-rendered analytical charts. Its seeded state-then-render design constructs unique answers while controlling visual style and numerical consistency.

  • Task design: TINYGRAFIXBENCH covers five chart types with four editing tasks each, yielding 20 task-modes.The chart types are bar, heatmap, line, network, and scatter plots.
  • Generation pipeline: A seeded state-builder creates a base chart description, task functions mutate a copy, and deterministic renderers produce the input and answer figures.Each task returns the two states and a natural-language instruction.
  • Visual controls: Chart styling samples contrasting background/text pairs and perceptually distinct object colors, while gibberish labels prevent reliance on real-world label semantics.Object colors are sampled in CIE L*a*b* space with perceptual-distance constraints.
  • Consistency: Instruction values are rounded to three significant figures before being used in both instruction strings and answer-state render parameters.This prevents displayed and instructed numerical values from drifting and creating persistent edit-region errors.
  • Unambiguity: Unique correct answers are enforced by construction through separated bar values, nondegenerate line gaps, available heatmap targets, and a fixed maximum-residual scatter point.These constraints make the relevant chart-edit targets visually unambiguous.

A.8. Evaluation Pipeline

The shared evaluation pipeline compares model outputs with deterministic answer images after normalization, separating changed and preserved pixels. It uses CIE76 color distances and mean IoU across tolerances to measure both edit and preservation quality.

  • Preprocessing: Outputs are rescaled with aspect ratio preserved and center-cropped to the answer resolution using nearest-neighbor interpolation.Nearest-neighbor resizing avoids synthesizing intermediate boundary colors.
  • Pixel partition: The change mask identifies pixels differing between input and answer, while all remaining pixels form the preservation region.This separates the pixels requiring edits from backgrounds and unchanged shapes.
  • Color comparison: Per-pixel color error is measured as Euclidean distance in CIE L*a*b* space using the CIE76 metric.Both output and answer images are converted from sRGB before distance computation.
  • Tolerance metrics: IoU penalizes both missed edits and erroneous changes to preservation regions.The metric is analogous to intersection over union for segmentation masks.
  • Tolerance metrics: Mean-tolerance IoU averages IoU across all 11 tolerances and serves as the primary reported metric.This avoids committing evaluation to a single color tolerance.
  • Models: The experiments evaluate 11 image-editing models, including proprietary native multimodal generators and open-weights flow-matching and diffusion editors.The model set includes NANO-BANANA-2, NANO-BANANA-1, GPT-IMAGE-2, and eight locally run models.

B.3. Benchmark Configuration

The benchmark configuration fixes a 1,920-problem PAINTBENCH test set while varying one visual parameter at a time across independently seeded conditions. Automatically generated instructions and metadata preserve the exact transformation parameters for downstream analysis.

  • Experimental configuration: The experiments use configuration values recorded separately from construction-side details, with proprietary API models using provider defaults and locally run models using documented settings.The locally run models use fixed random seeds, and inference runs on NVIDIA H200 GPUs via Slurm.
  • Problem counts: PAINTBENCH contains 1,920 problems from 20 tasks, eight visual conditions, and 12 problems per task-condition cell.Images use square or horizontal/vertical aspect-ratio resolutions depending on condition.
  • Visual conditions: Each visual condition changes exactly one parameter relative to baseline and uses an independent seed set.This isolates scene-variation effects across conditions.
  • Object counts: Most tasks use three shapes at baseline, with ablations increasing counts to 10, 25, or 60 shapes.Some task groups use adjusted count ranges suited to their structure.
  • Instructions: Instructions are generated from transformation parameters and specify the target shapes, operation, and all parameters needed for the unique answer.Each problem stores input and answer PNGs plus an instruction JSON sidecar containing the seed and task metadata.

C. Additional Experiments

The case study tests whether reasoning traces and structured solutions from a multimodal language model improve precise image editing. Augmentation helps mainly with target identification and transformation planning, but can hurt rote operations; the analysis uses bootstrap confidence intervals for aggregate results.

  • Prompt augmentation: Prompt augmentation inserts a multimodal-LLM reasoning stage before the image editor receives the original inputs and generated solution.GEMINI 3.1 THINKING produces the trace; NANO-BANANA-2 then performs the edit.
  • Scope: The case study uses NANO-BANANA-2 and an earlier PAINTBENCH version, leaving comprehensive re-evaluation across the full model lineup for future work.
  • Results: Flood Fill (−5.1%), Gradient (−3.9%), and Scaling (−3.2%) regress under augmentation.The authors suggest additional context may distract models on operations already executed through rote patterns.
  • Uncertainty analysis: Aggregate confidence intervals resample per-problem IoU values 10,000 times and preserve the benchmark’s hierarchical macro-averaging structure.The task lists are treated as fixed, so intervals reflect variation from resampled instances rather than alternative task selections.

D.2. Full Results Tables

The full-results materials provide per-task, per-mode, visual-condition, and benchmark-level breakdowns for PAINTBENCH and TINYGRAFIXBENCH. Galleries complement the tables with representative inputs, instructions, ground truths, and model outputs across benchmark categories and chart families.

  • PAINTBENCH task results: Tables 16–18 report PAINTBENCH mean IoU, edit-region accuracy, and preservation-region accuracy for all 35 task-modes.Aggregate-level bootstrap confidence intervals are reported separately.
  • Visual conditions: Tables 19–21 report PAINTBENCH mean IoU, edit-region accuracy, and preservation-region accuracy across eight visual conditions and 20 tasks.Object-count conditions use task-group-specific levels.
  • TINYGRAFIXBENCH results: Tables 22–24 provide TINYGRAFIXBENCH mean IoU, edit-region accuracy, and preservation-region accuracy for all 20 task-modes.
  • Confidence intervals: Tables 13–15 contain aggregate bootstrap confidence intervals for PAINTBENCH categories, TINYGRAFIXBENCH plot types, benchmark averages, and visual conditions.
  • PAINTBENCH galleries: Figures 12–15 show representative PAINTBENCH problems and outputs from all eleven models across geometric, structural, color, and symbolic categories.
Loading 2606.00188v1…