PaperlayerView exact revision on arXiv

Source-linked AI summary

Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing

Xiaochuan Zhong, Yifan Hou, Chenxi Pang, Shaobo Cui

arXiv:2609.08657v1cs.CVcs.LG

TL;DR

Existing chart benchmarks primarily assess final answers or rendered fidelity rather than layer-wise chart behavior. LayerWiseBench builds executable-program charts into understanding and editing evaluations grounded in layer structure. Across both tracks, visibility relations between overlapping components are a recurring challenge.

  • Problem

    Existing chart benchmarks assess task-level answers, code execution, or rendered fidelity, while chart composition motivates evaluating understanding and editing at the layer level.

    Source: §1 · p. 1

  • Method

    LayerWiseBench derives layer-wise understanding questions and targeted editing tasks from executable chart programs, construction records, and spatially aligned layer renderings.

    Source: §1 · p. 2

  • Results

    Visibility ordering is the lowest-scoring understanding dimension for all seven VLMs, and visibility-constrained edits have the lowest mIoU for all four editors.

    Source: §7 · p. 9

  • Takeaways & Limitations

    Tasks involving front-to-back relations between overlapping chart components are a recurring challenge across understanding and editing.

    Source: §7 · p. 9

    The study covers program-generated charts from the 14 paradigms supported by its construction library, and editing metrics measure agreement with reference outputs rather than internal layer representations.

    Source: §7 · p. 9

Abstract

from arXiv · show

Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This structural view motivates evaluating whether models can understand and manipulate charts at the layer level. Existing chart benchmarks, however, primarily assess the correctness or fidelity of final outputs and do not directly evaluate these layer-wise behaviors. We present LayerWiseBench, a benchmark organized around three core concepts, layer attribution, layer binding, and visibility ordering, that structure its chart-understanding and chart-editing evaluations. Generated from executable chart programs, LayerWiseBench pairs each rendered chart with spatially aligned per-layer RGBA assets and construction-derived labels for functional roles, semantic bindings, and visibility relations. From this layer-wise representation, we derive controlled understanding questions, editing targets, reference images, and evaluation regions. It contains 2,800 source charts across 14 chart paradigms, from which we derive 7,329 layer-wise understanding questions and 53,791 instruction-guided editing variants. Among the evaluated VLMs, Qwen3.5-27B, which achieves the highest QA macro-average, obtains 93.04% accuracy on layer attribution and 97.46% on layer binding, but only 61.46% on visibility ordering. Across the four evaluated image editors, overall mIoU ranges from 1.49% to 4.93%, and visibility-constrained edits have the lowest mIoU for every editor, ranging from 0.37% to 2.00%. Taken together, these results identify tasks involving front-to-back relations between overlapping components as a recurring challenge across understanding and editing, motivating more explicit modeling of component identity and visibility relations.

1 INTRODUCTION

LayerWiseBench evaluates chart understanding and editing through functional layers, semantic bindings, and visibility relations. Across both tracks, front-to-back relations between overlapping components emerge as a recurring challenge.

1 INTRODUCTION

  • Charts can be described compositionally through graphical components, mappings, guides, and view structures, motivating evaluation at the layer level.Existing chart benchmarks often assess task-level answers, code execution, or rendered-chart fidelity, while newer work also uses visual grounding and scene-graph comparison.Source: §1 · p. 1
  • The representation separates reference, data, text, and dependent components, and records within-layer properties, cross-layer correspondences, and front-to-back order.Visibility ordering is included because properties and correspondences alone do not fully determine overlapping components' appearance.Source: §1 · p. 2, Figure 1 · p. 2
  • LayerWiseBench evaluates chart understanding through questions and chart editing through targeted editing tasks.Executable chart programs supply construction records and spatially aligned layer renderings for annotations of functional roles, semantic correspondences, and visibility relations.Source: §1 · p. 2
  • The benchmark contains 2,800 source charts across 14 paradigms, yielding 7,329 understanding questions and 53,791 editing variants.These instances are organized around layer attribution, layer binding, and visibility ordering.Source: §1 · p. 2
  • Visibility ordering is the lowest-scoring understanding dimension for all seven VLMs, and visibility-constrained edits have the lowest mIoU for all four editors.Qwen3.5-27B scores 93.04% on layer attribution and 97.46% on layer binding, but 61.46% on visibility ordering; visibility-constrained editing mIoU ranges from 0.37% to 2.00%.Source: §1 · pp. 2–3

2 RELATED WORK

Related work covers chart understanding and chart generation or editing, with benchmarks expanding task coverage and evaluation signals. LayerWiseBench organizes these evaluations around functional layers and their relations.

2.1 CHART UNDERSTANDING PATTERNS AND BENCHMARKS

  • Chart understanding methods either map rendered charts directly to answers or text, or recover an intermediate representation before downstream reasoning.Examples include chart components, data tables, rendering code, and structured triples.Source: §2.1 · p. 3
  • Chart-understanding benchmarks broadened coverage to diverse questions, chart types, reasoning, real-world settings, visual grounding, localization, and infographic charts.Their formulations generally do not make functional organization into layers an explicit organizing principle.Source: §2.1 · p. 3

2.2 CHART GENERATION PATTERNS AND EDITING BENCHMARKS

  • Chart generation and editing use either structured chart specifications followed by rendering or direct transformation of rendered images from natural-language instructions.Structured specifications expose executability and symbolic structure; direct image editing is typically assessed through the visual transformation.Source: §2.2 · p. 3
  • Existing benchmarks assess reconstruction, code-oriented editing, end-to-end image editing, multi-turn interaction, dependency propagation, and final chart quality.LayerWiseBench uses an end-to-end editing interface and construction-derived annotations to report Local target, Binding-consistent, and Visibility-constrained families separately.Source: §2.2 · pp. 3–4

3 LAYER-WISE CHART REPRESENTATION AND FORMULATION

The paper represents charts in its controlled construction domain through components and the structures organizing and relating them. This representation captures functional layers, semantic bindings, and visibility order.

3 LAYER-WISE CHART REPRESENTATION AND FORMULATION

  • A chart is represented as S = (C, Λ), where C is its instantiated components and Λ records how they are organized and related.The executable source program produces both the rendered image and construction records used to instantiate S.Source: §3 · p. 4
  • Components include visible chart elements and chart-level layout structures, together with resolved data, content, appearance, and spatial properties.This component-level description applies within the paper's controlled construction domain.Source: §3 · p. 4
  • Within Λ, λ assigns functional layers, B records cross-layer semantic bindings, and V records resolved front-to-back order for overlapping component pairs.The three structures define layer attribution, layer binding, and visibility ordering; compositing order can change the visible result.Source: §3 · p. 4

4 LAYERWISEBENCH CONSTRUCTION

LayerWiseBench uses layer-aware organization and executable chart programs to construct understanding questions and editing variants. Its templates produce charts, aligned layer renders, and construction-derived structures that ground both evaluation tracks.

4.1 SOURCE CHART GENERATION

  • The source library contains 14 paradigms spanning varied mark, layout, and overlap structures, with its composition fixed before model evaluation.It contains 200 source charts per paradigm, totaling 2,800; splits are assigned by source chart and inherited by derived instances.Source: §4.1 · p. 4
  • Parameterized templates produce source charts and reference edits, registering task-relevant elements into functional layer groups and rendering spatially aligned transparent RGBA layers.Instantiated elements and construction records provide C and λ, B, and V in Λ, while aligned renders ground layers in the composite image.Source: §4.1 · p. 4, Figure 2 · p. 5

4.2 LAYER-WISE UNDERSTANDING INSTANCE GENERATION

  • Understanding queries are derived from component properties and relations, and are instantiated only when visually identifiable and uniquely answerable from construction records.Visibility queries additionally require verified local overlap; structured specifications fix the target and gold answer before question wording, and generated questions undergo consistency and leakage checks.Source: §4.2 · p. 5
  • The understanding branch contains 7,329 questions across 2,600 source charts and 13 paradigms, covering attribution, binding, and visibility ordering.The question-family counts are 3,172, 3,176, and 981, respectively.Source: §4.2 · p. 5

4.3 LAYER-WISE EDITING VARIANT GENERATION

  • Editing variants come from controlled parameter interventions and rerendering, with instructions and updates constrained by template schemas and checked against rebuilt renderer profiles.Families distinguish local target changes, coordinated visible changes to construction-linked components, and changes within verified overlaps that retain front-to-back order; the process generates 53,791 candidates.Source: §4.3 · pp. 5–6

5 EVALUATION SETTING

The evaluation tests layer-wise chart understanding with multiple-choice questions and chart editing with instruction-guided image transformations. Both tracks use construction-derived layer structure, with editing evaluated against reference changes and scored by pixel-level metrics.

5.1 LAYER-WISE UNDERSTANDING SETTING

  • The understanding test contains 727 questions: 316 on layer attribution, 315 on layer binding, and 96 on visibility ordering.Models receive a rendered chart, a multiple-choice question, answer choices, and four functional layer definitions; unmappable responses count as incorrect.Source: §5.1 · p. 6
  • The understanding results report family accuracy, unweighted macro-average accuracy, and invalid-response rate across all 727 test items.Invalid responses count as incorrect in family accuracies and are also reported separately.Source: §5.1 · p. 6

5.2 LAYER-WISE EDITING SETTING

  • Four image editors receive only a source chart and instruction; reference edits and construction annotations are reserved for evaluation.The common test set has 4,614 items: 3,447 Local target, 444 Binding-consistent, and 723 Visibility-constrained edits.Source: §5.2 · p. 6
  • Editing is scored primarily with mIoU, which rewards reference-matching pixels in the changed region while penalizing incorrect target pixels and deviations elsewhere.mIoU averages over tolerances and items; Edit Accuracy and Preservation Accuracy are complementary diagnostics, while SSIM and OCR-NES cover additional checks.Source: §5.2 · pp. 6–7

6 EXPERIMENTS AND ANALYSIS

The experiments compare three layer-wise understanding dimensions across vision-language models and assess image editors on construction-grounded edits. Across both tracks, visibility-related cases involving front-to-back relations between overlapping components emerge as the recurring weakness.

6.1 LAYER-WISE UNDERSTANDING EXPERIMENTS

  • 64.58% is the best visibility-ordering accuracy among evaluated VLMs, versus 50% under uniform random guessing on 96 binary questions.Visibility ordering is the lowest-scoring task family for every VLM in the main comparison.Source: §6.1 · p. 7
  • Text-only layer-attribution accuracy remains 78.48–92.41%, indicating that many questions in this family can be answered without the chart image.Source: §6.1 · p. 8
  • Qwen3.5 visibility-ordering accuracy is 64.58% for 2B, 59.38% for 9B, and 61.46% for 27B, despite rising macro-average accuracy with model size.The reported chart compares Qwen3.5 understanding accuracy, with different axis ranges.Source: Figure 4 · p. 7, §6.1 · p. 8

6.2 LAYER-WISE EDITING RESULTS

  • Overall editing mIoU ranges from 1.49% to 4.93% across 4,614 items, indicating difficulty combining pixel-accurate target changes with preservation of the surrounding chart.Table 2 reports overall and family-level editing mIoU.Source: §6.2 · p. 8, Table 2 · p. 8
  • Visibility-constrained edits score 0.37%–2.00% mIoU, below local target edits at 1.77%–6.22% and binding-consistent edits at 1.14%–3.22%.Visibility-constrained edits are the lowest-scoring editing family for every evaluated editor.Source: §6.2 · p. 8, Table 2 · p. 8
  • Figure 5 contrasts Qwen Image Edit 2509 outputs showing broad unintended changes versus closer agreement with reference edits.Source: §6.2 · p. 8

6 EXPERIMENTS AND ANALYSIS

  • Visibility ordering is the lowest-scoring understanding dimension for all seven VLMs, and visibility-constrained edits have the lowest mIoU for all four editors.The shared gap centers on tasks involving front-to-back relations between overlapping components.Source: §6.1 · p. 7, §6.2 · p. 8

7 CONCLUSION

LayerWiseBench evaluates chart understanding and editing through construction-derived layers, semantic bindings, and visibility relations. Across both tracks, visibility ordering and edits involving overlaps are the recurring weak points, within a study limited to program-generated charts from 14 supported paradigms.

7 CONCLUSION

  • Visibility ordering is the lowest-scoring understanding dimension for all seven VLMs, while visibility-constrained edits score 0.37%–2.00% for every editor.Qwen3.5-27B obtains 61.46% on visibility ordering; overall editing mIoU ranges from 1.49% to 4.93%.Source: §7 · p. 9
  • The study covers program-generated charts from 14 supported paradigms; future work proposes natural charts, broader paradigms, and matched editing sets.The editing metrics measure agreement with reference outputs, not whether models internally represent chart layers.Source: §7 · p. 9

A TERMINOLOGY

  • Layer Attribution, Layer Binding, and Visibility Ordering name the three core evaluation dimensions; editing uses separate family labels.The editing families are Local target, Binding-consistent, and Visibility-constrained.Source: A TERMINOLOGY · p. 13

B.1 FUNCTIONAL LAYER ONTOLOGY AND ELEMENT MAPPING

  • The ontology assigns chart elements by function—not rendering primitive—to Reference, Data, Text, or Dependent, with each canonical type assigned exactly one layer.For example, legends and mark-tied labels are Dependent, while axes and positional guides are Reference.Source: §B.1 · pp. 14–15

C BENCHMARK CONSTRUCTION DETAILS

  • The benchmark has two tracks: understanding queries functional roles and structural relations, while editing applies controlled natural-language modifications.Both are built from rendered charts that retain a layer-wise representation.Source: C BENCHMARK CONSTRUCTION DETAILS · p. 15

C.1 UNDERSTANDING DATA CONSTRUCTION

  • Understanding items derive from program-based canonical facts and ask layer membership, semantic partners, or which locally overlapping element is in front.Questions are verbalized only after the fact and answer space are fixed.Source: §C.1 · pp. 15–16

C.2 EDITING VARIANT CONSTRUCTION

  • Editing variants pair instructions with aligned input and reference renderings; controlled parameter updates are rendered from source programs, then routed by construction-derived relations.The family verifier checks visible change in eligible regions; for visibility-constrained edits, it does not verify foreground direction or that stacking order changed.Source: §C.2 · pp. 16–18

C.3 CONSTRUCTION VALIDATION AND DATASET ACCOUNTING

Construction checks validate understanding items and editing variants before scoring, producing a common 4,614-item editing test set. Evaluation uses fixed inference settings and deterministic parsing, with unparseable responses counted as incorrect.

C.3 CONSTRUCTION VALIDATION AND DATASET ACCOUNTING

  • In the 100-item family-stratified understanding audit, 99 passed before a wording correction; all 20 Bar3D Visibility Ordering test questions passed their separate check.The audit assessed gold-answer correctness, visual answerability, and answer uniqueness.Source: §C.3 · p. 19
  • 5,148 of 5,392 held-out editing candidates pass both construction checks; excluding 514 no-op edits and 20 without layer assets yields the 4,614-item scored set.The instruction-alignment check allows rounded numerical forms, so passing it does not guarantee exact agreement with stored reference values.Source: §C.3 · p. 19
  • For 45 items using one multiplier instruction template, excluding displayed-value discrepancies changes overall mIoU by at most 0.017 percentage points and leaves editor rankings unchanged.This check does not exhaustively assess all numerical instructions; the Visibility-constrained family remains lowest for each editor.Source: §C.3 · p. 19

D INFERENCE SETTINGS

  • Understanding models run without sampling with a fixed seed, while each image editor produces one output per item using a fixed seed of 20260530.Understanding generation is capped at 32 tokens; editors use model-specific settings and output sizes, with evaluation normalization where needed.Source: D INFERENCE SETTINGS · p. 19

E.1 UNDERSTANDING EVALUATION

  • The 727-item understanding test split has 316 Attribution, 315 Binding, and 96 Visibility Ordering items; failed or unparseable responses remain in the denominator as incorrect.Macro Average weights the three families equally, unlike accuracy over all items, which weights each item equally.Source: §E.1 · p. 20

E.2 EDITING EVALUATION

The editing evaluation scores predictions on shared item sets after image normalization, using pixel-based PaintBench measures, SSIM, and OCR string agreement. Each metric captures a different aspect of the edit, and OCR assesses text preservation rather than requested font or style correctness.

E.2 EDITING EVALUATION

  • 4,614 items form the common PaintBench and SSIM set, while OCR is evaluated separately on 568 eligible items among 581 applicable Text/Font edits.Missing or invalid prediction images invalidate evaluation rather than reducing its denominator.Source: §E.2 · p. 20
  • Predictions are EXIF-corrected and converted to RGB; mismatched aspect ratios use crop-based resizing when ϵ ≤ 0.03 and fit-with-padding otherwise.The resulting prediction matches the input and reference dimensions, with no further resizing or alignment during scoring.Source: §E.2 · p. 21
  • PaintBench compares aligned prediction and reference pixels in CIE Lab, reporting Edit Accuracy, Preservation Accuracy, and mIoU as percentages, with higher values better.The scored set has changed pixels; Preservation Accuracy is defined as one when there are no unchanged pixels.Source: §E.2 · p. 21
  • SSIM is averaged over valid positions and RGB channels, and its change from the input baseline is interpreted alongside PaintBench because SSIM measures global similarity.Higher SSIM is better, while ∆Si > 0 indicates improvement over the input baseline.Source: §E.2 · p. 21
  • OCR Exact and OCR-NES measure exact string match and normalized character similarity, respectively, on the shared eligible subset.The OCR rule selects a common crop and rotation for all models; OCR-NES measures readable-text preservation, not correctness of the requested font or style change.Source: §E.2 · pp. 21–22

F.1 COMPLEMENTARY UNDERSTANDING RESULTS

The complementary understanding results compare image-plus-text evaluation with text-only controls and report additional runs that exceed the main invalid-response threshold. Image removal lowers macro-average accuracy overall, but visibility-ordering performance does not improve uniformly with images.

F.1 COMPLEMENTARY UNDERSTANDING RESULTS

  • All seven VLMs have 10.16–25.77 percentage-point lower macro-average accuracy without images than in the image-plus-text comparison.The text-only control retains the questions and choices across all 727 test items.Source: §F.1 · p. 22, Table 6 · p. 22
  • Visibility Ordering ranges from 36.46% to 67.71% without images and from 50.00% to 64.58% with images, so image availability does not uniformly improve this family.InternVL3-8B scores 67.71% without images and 56.25% with the image.Source: §F.1 · p. 22
  • Qwen3.5-0.8B and Qwen3.5-4B produce 96 and 58 invalid responses with images, versus 13 and 1 without images, respectively.For Qwen3.5-4B, 51 of 96 image-run Visibility Ordering responses are invalid; its 28.13% accuracy reflects incorrect choices and frequent invalid responses.Source: §F.1 · p. 23, Table 7 · p. 23

F.2 COMPLEMENTARY EDITING RESULTS

Complementary editing results report pixel agreement, image similarity, and OCR string agreement on shared evaluation sets. The highest-performing editor differs across metrics, and OCR exact matching and partial-match similarity also have different leaders.

F.2 COMPLEMENTARY EDITING RESULTS

  • FLUX Kontext Dev has the highest Edit Accuracy, while Qwen Image Edit 2509 has the highest Preservation Accuracy.These complementary metrics are reported on the common set of 4,614 items.Source: §F.2 · p. 23, Table 8 · p. 23
  • InstructPix2Pix has the highest SSIM and smallest decrease from baseline despite the lowest Edit Accuracy, and all four editors remain below the no-edit SSIM baseline.The Table 8 baseline is 95.80.Source: §F.2 · p. 23, Table 8 · p. 23
  • OmniGen2 has the highest OCR Exact score, whereas Qwen Image Edit 2509 has the highest OCR-NES on the 568-item eligible text-editing subset.OCR Exact rewards complete reference-string matches; OCR-NES gives credit for partial string similarity.Source: Table 9 · p. 23, §F.2 · p. 24

G SENSITIVITY TO VISIBILITY THRESHOLDS

The sensitivity analysis varies visibility overlap thresholds for understanding and the active-pixel cutoff for editing while holding model outputs fixed. The reported family-level patterns remain stable across the tested threshold ranges.

Understanding

  • Visibility Ordering accuracy for Qwen3.5-27B remains between 60.64% and 61.46% across the tested overlap thresholds, while the three family rankings remain unchanged.Attribution and Binding stay at 93.04% and 97.46% because their item sets do not depend on those thresholds.Source: G SENSITIVITY TO VISIBILITY THRESHOLDS · p. 24

Editing

  • The Visibility-constrained family has the lowest mIoU for every editor at every tested active-pixel cutoff, with a maximum displayed-curve change of 0.15 percentage points.The analysis pool remains fixed at 4,614 items; from k = 8 to k = 32, nine cases move from Visibility-constrained to Local target.Source: G SENSITIVITY TO VISIBILITY THRESHOLDS · p. 24, Figure 9 · p. 25
  • The Visibility-constrained family falls from 723 to 714 edits as k rises from 8 to 32, while Binding-consistent remains at 444 edits, preserving the main-evaluation pattern within the tested range.Only family assignment changes; existing item scores are reused.Source: G SENSITIVITY TO VISIBILITY THRESHOLDS · pp. 24–25

Related papers

Loading 2609.08657v1…