PaperlayerView exact revision on arXiv

Source-linked AI summary

PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides

Xiaoqiu Wang, Yizhe Chi, Wenyi Li, Deyao Hong, Zhihan Shan, Mingju Gao, Kaisen Yang, Youjie Zheng, Calvin Xiao, Qinhuai Na

arXiv:2609.29718v1cs.CLcs.AI

TL;DR

Visual coding requires agents to infer a target’s structure and express it programmatically, but existing benchmarks offer limited tests of end-to-end editable reconstruction. PPTBench evaluates reconstruction of scientific flow diagrams as editable slides; across 31 configurations, the best reaches 67.80 of 100, while agents still struggle with semantic and visual correctness.

  • Problem

    Visual coding requires agents to infer a target’s visual structure and express it programmatically, while existing benchmarks offer limited evaluation of end-to-end visual reconstruction as editable output.

    Source: §1 · p. 2

  • Method

    PPTBench uses 500 scientific flow-diagram tasks requiring single-page PPTX reconstructions from native editable objects, evaluated across four stages by an Agentic Judge.

    Source: §1 · p. 2, §5 · p. 21

  • Results

    67.80 of 100 for the best configuration, Kimi K3 at high effort, while agents generally produce valid decks but still struggle with semantic and visual correctness.

    Source: §1 · p. 3, §5 · p. 21

  • Takeaways & Limitations

    PPTBench makes the ability to understand and reconstruct visual material as structured, editable artifacts measurable.

    Source: §5 · p. 21

    The voting experiment covers the two hard gates, not Stage 3 detail findings.

    Source: §3.4 · p. 17

Abstract

from arXiv · show

Coding agents are beginning to act in the visual world. They now build webpages, GUIs, games, 3D scenes, diagrams, and documents. Success in such visual coding requires bridging two spaces: inferring visual structure and expressing it programmatically. Slides are a core medium of knowledge work, widely used to communicate ideas and collaborate in a form that people can directly inspect and edit. Therefore, they provide an ideal testbed for visual coding, as they require agents to recover visual structure and realize it as editable objects. However, existing benchmarks either rely on subjective open-ended evaluation, produce non-editable code outputs, or focus only on local editing rather than end-to-end visual reconstruction. We introduce PPTBench, which benchmarks visual coding through editable slide reconstruction. It contains 500 tasks, each based on a scientific flow diagram from a real arXiv paper and requiring agents to reconstruct it as a single PPTX page composed of native, editable objects. A four-stage Agentic Judge evaluates artifact validity, semantic correctness, rendering quality, and fine-grained visual quality. Across 31 configurations spanning model families, effort levels, and harnesses, the best configuration, Kimi K3, reaches only 67.80, while the median scores 19.47. We find that agents can reliably produce valid PPTX files but still struggle with semantic and visual correctness, especially text details. More reasoning mainly helps agents pass hard gates, while stronger verification is more consistently associated with higher quality. PPTBench advances the vision of coding agents that can understand and reconstruct the visual world through structured, editable code.

1 Introduction

PPTBench tests whether agents can infer visual structure and express it as editable slide objects, addressing the limits of benchmarks that do not measure end-to-end reconstruction. Its staged evaluation separates artifact validity, semantic correctness, and visual quality, while results show that valid files do not guarantee faithful reconstructions.

1 Introduction

  • Editable slides expose an agent’s inferred object structure, requiring it to choose objects and connections rather than paste the reference image.The artifact contract mechanically forbids embedding a rasterized copy of the reference.Source: §1 · pp. 2–3
  • Existing benchmarks lack end-to-end visual reconstruction: some use subjective targets, others return source files, and office-editing tasks specify the object changes in advance.PPTBench instead fixes a reference image and requires a native, editable one-page reconstruction, scored by a staged judge.Source: §1 · p. 2, Figure 2 · p. 3
  • PPTBench contains 500 screened tasks, each requiring reconstruction of a scientific flow diagram from an arXiv paper as one PPTX page of native editable objects.The diagrams span multi-panel layouts, hierarchical blocks, repeated grids, complex connector routing, text-dense layouts, and vector icons.Source: §1 · pp. 2–3
  • The four-stage Agentic Judge checks artifact validity, semantic correctness, rendering quality, and fine-grained visual differences, with failed gates scoring zero.The judge gates and enumerates findings; fixed program logic converts those findings into scores.Source: §1 · pp. 2–3
  • Kimi K3 at high effort scores 67.80 of 100, while the median configuration scores 19.47; agents often produce valid PPTX files but struggle with semantics and visual correctness.Only 2.08% of model–task pairs fail to produce a usable artifact, while 70.43% lose process semantics after producing a readable deck.Source: §1 · p. 3
  • Text accounts for 51.6% of deductions, while additional reasoning mainly helps pass hard gates and render inspection is more consistently associated with quality.The association between render-inspection frequency and score is r = 0.881; edit frequency carries little signal (r = 0.255).Source: §1 · p. 3

2 Introducing the PPTBench Benchmark

PPTBench formalizes visual reconstruction as rebuilding a scientific flow diagram from an image into a semantically equivalent, visually faithful, natively editable slide. It defines the task, constructs and screens its benchmark, specifies a four-stage judge, and illustrates scoring with a reconstruction case study.

2.1 Task formulation

  • A single-task reconstruction contract requires a semantically equivalent, visually faithful, natively editable PPTX, while allowing different object-level implementations when they preserve discernible structure.The agent sees a reference image and any approved irreducible raster assets; the artifact must be one PPTX page, and the contract rules out copying the reference as a raster.Source: §2.1 · p. 4

2.2 Benchmark construction and integrity

  • Figure 3 traces sourcing and preprocessing through human screening of 1,000 candidates to 500 frozen tasks, then gives agents a reference image and approved resources for PPTX reconstruction.Candidates come from real arXiv figures; screening removes diagrams that are too simple or not reliably judgeable, among other exclusions.Source: Figure 3 · p. 5, §2.2 · p. 5

2.3 Benchmark composition and coverage

  • The 500 tasks span 50 arXiv primary categories, with systems, architecture and software engineering, AI and machine learning, and computer vision and imaging making up just under half.Computing and electrical engineering fields account for 373 tasks (74.6%); the remaining 127 (25.4%) come from other scientific areas. Domain indicates provenance, not difficulty.Source: §2.3 · p. 6

2.4 Evaluation protocol

  • The four-stage judge checks artifact validity, semantic correctness, rendering quality, and fine-grained visual quality; failed gates terminate evaluation, while later stages enumerate localized defects.Three independent judged runs are aggregated by a fixed rule: findings are merged using the largest reported affected proportion, and at least two gate votes produce a zero score.Source: §2.4 · pp. 6–7

2.5 Case study

  • In the case study, all three rounds pass the semantic and render/text gates despite severe corruption confined to one node, leaving localized defects for fine-grained scoring.The seven merged findings cost 20.01 points, yielding 79.99; text deductions account for 16.51 points, compared with 1.25 for layout.Source: §2.5 · p. 8
  • A single round’s finding can carry full deduction weight, while four symptom-level text findings for one mis-sized frame jointly cost 10.51 points.The authors note that a root-cause taxonomy would score this case higher; the case’s judging cost of $0.401 should not be generalized beyond its task and generator configuration.Source: §2.5 · pp. 8–9

3 Experiments

The experiments evaluate 31 configurations on 500 tasks, examining performance, reasoning, rendered-output inspection, and score reliability. Kimi K3 emerges as the winner, while semantic failures and text defects remain obstacles and neighboring leaderboard positions and task-level detail scores remain uncertain.

3.1 Setup and Metrics

  • The primary metric is mean S across the frozen 500 tasks, with invalid artifacts scored zero and retained in the denominator, reported alongside artifact validity and stage gate-pass rates.All 31 configurations saw the same tasks, enabling paired task-level comparisons; each candidate was judged three times, for 46,500 total judgments.Source: §3.1 · p. 9

3.2 Overall Performance

  • Kimi K3 at high effort scored 67.80, ahead of GPT-5.6 Sol Max at 49.28 and Qwen 3.8 Max at 47.69; the median configuration scored 19.47.Accepted-run cost ranged from $20.67 for GPT-5.6 Luna at low effort to $1,157.91 for Kimi K3; GLM-5.3-Flash at max scored 32.42 for $52.53. The 31 configurations were compared on all 500 tasks.Source: Table 1 · p. 10, §3.2 · pp. 10–11

3.3 Analysis of Agent Behavior

  • 97.92% of runs passed the artifact gate, but 70.43% failed the semantic gate versus 2.28% failing render/text; only 25.21% cleared both judged gates.Overall, 74.79% were rejected before detail scoring, and only 120 of 15,500 model–task pairs had no recorded finding.Source: §3.3 · p. 11, Figure 7 · p. 12
  • Among gate-passing records, text accounted for 51.6% of penalty points versus 13.6% for layout; unexpected_wrap alone cost 5,002.0 points across 1,591 cases.Text-fitting errors dominate because their rendered overflow is not apparent from the OOXML geometry alone; unexpected_wrap was also the most reproducible finding, with only 6.7% of cases seen by one judge round.Source: §3.3 · pp. 12–13, Table 2 · p. 13
  • GPT-5.6 Sol gained 44.29 mean-score points from none to max effort, as gate pass rate rose 46.8 points while conditional detail quality rose only 7.68 points.Across the effort sweep, additional compute mainly improved process recovery, not the drawing quality of successful outputs. The two Claude families were worse or no better at their most expensive setting: Opus 5 declined at max, while Sonnet 5 plateaued and lost artifact validity.Source: §3.3 · pp. 14–15, Figure 10 · p. 15
  • Across 27 trajectory-parsed configurations, mean score correlated with review turns (r = 0.881) far more than revision turns (r = 0.255), but effort confounds this association.Review means inspecting the candidate render, where text-fitting defects become visible; the correlation is descriptive, not evidence that forcing more inspection at fixed effort raises scores.Source: §3.3 · pp. 15–16, Figure 11 · p. 16

3.4 Is the Measurement Trustworthy?

  • On 200 validation cases, the judge’s majority agreed with the human majority on 86.5% (κ = 0.696), while human annotators were unanimous on 81.5% (κ = 0.728).These headline agreement rates are not directly comparable; the comparable chance-corrected statistics place the judge just below human reliability. The judge was slightly stricter, recording 15 extra gates against 12 missed gates.Source: §3.4 · pp. 16–17, Table 3 · p. 17
  • Two independent three-round Luna High majorities agreed on 96.0% of cases, while annotators’ issue sets matched exactly on only 17.3% of 52 cases.Single-run Luna High self-consistency was 87.5%; detail findings were less reproducible than the gates, with mean pairwise Jaccard similarity of 0.392 among annotators.Source: §3.4 · p. 17, Table 4 · p. 18
  • Aggregation preserves per-finding round support, but per-task detail scores carry more uncertainty than configuration-level aggregates.A deduction supported by only one of three rounds remains recoverable from the released records.Source: §3.4 · p. 18
  • At 500 tasks, resampled rankings matched the full-set order at Kendall’s τ = 0.946 (5–95% range: 0.918–0.970).The mean 95% confidence interval on a configuration’s score was 5.70 points wide, with a maximum width of 8.29; tasks were resampled in paired fashion across configurations.Source: §3.4 · p. 18, Figure 13 · p. 19
  • 100% of 500-task resamples ranked Kimi K3 first, 18.52 points ahead of GPT-5.6 Sol Max (paired 95% interval [13.43, 23.58]).The exact top-3 set also survived every full-size resample.Source: §3.4 · pp. 18–19, Table 5 · p. 19, Figure 13 · p. 19
  • Only 4 of 31 adjacent leaderboard pairs separated significantly; Sol Max’s 1.59-point lead over Qwen 3.8 Max remained unresolved.Their paired 95% interval was [−3.50, 6.64]; the exact top-2 set recurred in 72.6% of resamples, and the top-5 set in 60.2%.Source: §3.4 · pp. 18–19, Table 5 · p. 19

4 Related Work

PPTBench combines a fixed visual target, native editability, and topology gating, dimensions that related benchmarks typically address separately. Its design also distinguishes structural reconstruction from chart-code generation, diagram understanding, document authoring, and model-judge evaluation.

4 Related Work

  • Chart-code and screenshot-to-markup work renders predicted code for comparison, but chart data can anchor correctness and textual source editability differs from editable slide objects.PPTBench instead requires direct manipulation of the native object graph.Source: §4 · p. 19
  • Only PPTBench requires a fixed visual target, scored native editability, and gated node-and-edge semantics together.Other slide-generation work has underdetermined targets, while document-editing work starts from an existing decomposition.Source: Table 6 · p. 20, §4 · p. 20
  • PPTBench fixes both content and appearance at whole-slide granularity, separating visual-semantic understanding from geometric control while leaving narrative and design quality unmeasured.Related document generation evaluates what to say as well as how to draw, while document editing starts from an already decomposed artifact.Source: §4 · p. 20
  • Prior diagram-reading work finds text recognition more reliable than structural recovery, while generated scientific diagrams are evaluated structurally against the same visually plausible but structurally wrong failure.PPTBench supplies one image target and requires no explicit recovered graph.Source: §4 · p. 20
  • PPTBench’s judge returns gates and localized findings rather than scores, which fixed program logic converts; reliability is checked against blinded human judgments and independent repetitions.Its agent evaluation also sweeps effort, motivated by chain-of-thought prompting [60] and test-time compute scaling [51].Source: §4 · p. 20

5 Conclusion

Across 31 configurations, agents often produce valid decks but struggle with semantic and rendering gates, with text wrapping a prominent residual error. PPTBench measures editable slide reconstruction through a staged judge and deterministic scoring.

5 Conclusion

  • PPTBench evaluates 500 hand-screened scientific-flow-diagram tasks requiring one editable PPTX page, with four stages covering validity, semantic correctness, rendering, and fine-grained visual quality.The judge gates and enumerates findings, while a deterministic ranker assigns the score.Source: §5 · p. 21
  • 67.80 of 100 for Kimi K3 at high effort and 19.47 for the median summarize 31 configurations; nearly every run returns a valid deck, but most fail a semantic or rendering gate.Unintended wrapping dominates residual error; additional reasoning mainly lifts gate-pass rate, while render inspection tracks quality and deck editing does not.Source: §5 · p. 21
  • The measurement cannot separate harness effects from model effects and, because targets are fixed, does not measure authoring.The paper concludes that PPTBench advances the vision of agents reconstructing the visual world through structured, editable code.Source: §5 · p. 21

A Task Catalog

The task catalog releases specifications for 500 benchmark tasks, with materialized source documents, reference images, and approved resources reconstructed locally. Its domain distribution is summarized in Table 7.

A Task Catalog

  • 500 tasks are released as downloader-only specifications, and the materializer verifies source data before locally reconstructing each source document, rendered reference image, and approved resources.The full per-task catalog is released with the benchmark.Source: A Task Catalog · p. 27
  • The benchmark catalog is provided separately rather than reproduced in the paper, alongside the downloader-only task specifications.The released materializer locally reconstructs the source document and rendered reference from verified source data.Source: A Task Catalog · p. 27
  • Table 7 reports the distribution of the frozen benchmark across ten domains and gives one representative source paper for each domain.Its final column records a source paper, and each listed arXiv class includes its assigned task count.Source: Table 7 · p. 27

B Additional Cases

The worked reconstructions all clear hard gates, spanning scores from 95.50 to 76.75. Their detail scores emphasize textual fidelity more than layout, while examples show that substantial geometric or icon differences can preserve diagram meaning.

B Additional Cases

  • A 92.75 reconstruction substitutes icons and squares 15 rounded nodes yet remains the same diagram, whereas a 76.75 case has text misaligned and broken mid-word.The latter is described as where a reader starts to struggle; local imprecision is treated differently from a diagram that reads wrongly.Source: B Additional Cases · p. 27
  • The bottom two worked cases lose 12.50 and 16.51 points on text, while layout costs no more than 2.25 points.All four worked reconstructions clear every hard gate, and the detail-score range is deliberately compressed.Source: Table 8 · p. 27
  • 95.50 to 76.75 is the score span across three additional composites, and every reconstruction clears both hard gates.Figure 14 shows the full composites; Table 8 gives per-dimension scores.Source: Figure 14 · p. 28

C Detail Scoring Taxonomy

Table 9 organizes fine-grained visual issues by dimension, category, severity base, and subtype; after hard gates, affected prevalence scales each subtype’s fixed severity base.

C Detail Scoring Taxonomy

  • After both hard gates, each issue subtype’s fixed severity base is multiplied by affected prevalence; Table 9 groups subtypes that share a base.Repeated dimensions and categories are merged.Source: C Detail Scoring Taxonomy · p. 29, Table 9 · p. 29
  • Typography issues are divided between medium-severity alignment, wrapping, and text-rendering errors and low-severity size, style, and spacing errors.The taxonomy assigns medium base 1.5 and low base 0.5 to these categories.Source: C Detail Scoring Taxonomy · p. 30
  • Missing or extra shapes and connectors receive the highest severity base, 3.0; many geometry and attachment errors are medium, while several connector styling errors are low.Node corner radius and line shadow are listed as low-severity and diagnostic, respectively.Source: C Detail Scoring Taxonomy · pp. 30–31
  • Node styling errors are low severity, while node and connector shadows are diagnostic issues with zero severity base.The taxonomy also labels this area “Local graphics, nodes and connectors.”Source: C Detail Scoring Taxonomy · p. 30
  • Vector and icon confusion includes missing, extra, substituted, or malformed icons and grid or mini-diagram errors, while many position and rendering issues are low severity.These groups carry medium base 1.5 and low base 0.5, respectively.Source: C Detail Scoring Taxonomy · p. 30
Loading 2609.29718v1…