PaperlayerView exact revision on arXiv

Source-linked AI summary

Crafter: A Multi-Agent Harness for Editable Scientific Figure Generation from Diverse Inputs

Haozhe Zhao, Shuzheng Si, Zhenhailong Wang, Zheng Wang, Liang Chen, Xiaotong Li, Zhixiang Liang, Maosong Sun, Minjia Zhang

arXiv:2605.30611v1cs.CVcs.AIcs.CL

TL;DR

Scientific figure generation remains limited across diverse figure types and input conditions, while raster outputs are difficult to revise locally. Crafter introduces a multi-agent harness for generation and editable raster-to-SVG conversion, outperforming baselines across both benchmarks, including by 16.61 points on PaperBanana-Bench against the strongest agentic baseline.

  • Problem

    Existing systems mainly target text-to-image generation for narrow figure types and do not address diverse input conditions or locally editable outputs.

  • Method

    Crafter uses a multi-agent harness with shared structured specifications, planning, verification, and typed revision to generate figures, while CraftEditor converts raster outputs into editable SVGs.

  • Results

    CRAFTER achieves the highest overall score across both benchmarks and all evaluated tasks and dimensions, leading the strongest agentic baseline by 16.61 points on PaperBanana-Bench.

  • Takeaways & Limitations

    The paper provides a unified generation-to-editing pipeline with cross-type, cross-condition evaluation and performance exceeding the reported baselines.

  • Takeaways & Limitations

    Existing evaluations and systems have limited scope because they predominantly address text-to-image generation and do not assess generalization across figure types or input conditions.

Abstract

from arXiv · show

Scientific figures are among the most effective means of communicating complex research ideas, yet producing publication-quality illustrations remains one of the most labor-intensive parts of paper preparation. Existing automated systems each target a single figure type under text-only input, leaving the diversity of types and conditions researchers actually use unaddressed; their raster outputs further cannot be locally revised. Because scientific figures are structured compositions of discrete semantic components, the localized errors generators produce on such layouts demand not a stronger backbone but a harness. We instantiate this harness in two complementary systems: Crafter, a multi-agent harness for figure generation that generalizes across figure types and input conditions without architectural changes, and CraftEditor, which applies the same pattern to convert raster outputs into editable SVGs. Moreover, we introduce CraftBench, a benchmark spanning three figure types and four input conditions with human quality annotation. Experiments show that Crafter substantially outperforms both standalone generators and the agentic baseline on PaperBanana-Bench and CraftBench, with ablations confirming each component's independent contribution; CraftEditor faithfully converts outputs into editable SVGs that surpass all baselines. Our code and benchmark are available at https://github.com/HaozheZhao/Crafter.

1 Introduction

Existing scientific-figure generators remain narrow in figure types and input conditions, while raster outputs are difficult to revise locally. The paper introduces a multi-agent generation-and-editing harness, CRAFTBENCH, and evaluation protocol to address these limitations.

  • Limitations: Existing systems predominantly target text-to-image methodology figures, leaving broader figure types and non-text conditioning inputs unsupported.Researchers commonly work from rough sketches, partial layouts, or reference visual elements and icons.
  • Limitations: Raster generators produce static images that cannot be locally revised, while code-generation methods provide editability but lack visual richness.The limitation affects tasks such as changing labels, color schemes, or component arrangements.
  • Motivation: Structured scientific figures require a harness because generators produce localized errors, including garbled labels and misaligned connectors, that rephrasing alone cannot resolve.Figures comprise semantic components such as labeled boxes, arrows, icons, and annotations with precise spatial relationships.
  • CRAFTER: CRAFTER uses cooperating agents that share an evolving figure specification, combining plan exploration, critique, specification refinement, and convergence control around an image-generation backend.The agents include an intent reasoner, plan generator, critic, specification refiner, and convergence judge.
  • CRAFTEDITOR: CRAFTEDITOR extends the harness to raster-to-SVG conversion through extraction, asset processing, and iterative composition into an editable SVG skeleton.The extraction phase removes text overlays and clutter; processing captions assets and classifies them as vector or raster.
  • Evaluation: CRAFTBENCH contains 279 samples spanning three figure types and four input conditions, with human quality annotation and VLM-based evaluation against real images.The samples were curated across eighteen research areas, award-tier conference posters, and research blogs.

2 Related Work

Related work includes code-generation and agentic approaches to scientific figure creation, while existing benchmarks evaluate narrower generation settings. Code-based methods provide editable diagrams but lack visual richness, whereas agentic pipelines target high-quality raster figures for methodology plots.

  • Scientific figure creation: Code-generation methods synthesize editable diagrams, such as TikZ, from text descriptions but remain restricted to schematic diagrams and lack visually rich icons and stylized layouts.These methods are associated with Belouadi et al. (2024), Zala et al. (2024), and Greisinger and Eger (2026).
  • Scientific figure creation: Agentic pipelines combine LLM agents with image generators to produce high-quality raster figures for methodology plots.The passage identifies Zhu et al. (2026a,b) as examples of this approach.
  • Benchmarks and evaluation: Existing benchmarks primarily evaluate text-to-image generation, with PaperBanana-Bench and Paper2SysArch focusing on academic diagrams and SridBench omitting conditional inputs.IGenBench instead targets text-to-infographic reliability using a decomposed verification framework.

3 Method

Crafter uses a four-role harness that plans, renders, verifies, and revises through typed edits to a shared specification, addressing variance, prompt degradation, and unstructured feedback. CraftEditor reuses this harness to convert raster figures into editable, coordinate-faithful SVGs through extraction, processing, and critic-driven composition.

  • 3.1 Harness Formulation: The harness wraps a pluggable executor with designer, executor, verifier, and reviser roles operating on a shared evolving specification.The verifier emits directive diagnostics, while the reviser applies structured operations rather than appending free-text prompt corrections.
  • 3.2 Crafter: Crafter uses five cooperating agents to infer intent, propose visual plans, render raster candidates, critique them, and write typed edits into the specification.Prompt-level adaptation allows the same pipeline to generalize across diverse figure types and input conditions.
  • 3.2.1 Plan-Level Search: Crafter treats inter-sample variance as a search problem by branching into K intent-conditioned candidate plans before rendering.Plan-level branching can avoid unsuitable compositional choices without spending rendering budget on their refinement.
  • 3.2.2 Verification and Refinement: A directive critic scores six dimensions, identifies defects, and suggests corrections, while a verify-then-refine loop runs for up to T=3 rounds with best-so-far rollback.An early-exit gate skips refinement when first-round outputs meet critical acceptance thresholds.
  • 3.3 CraftEditor: CraftEditor applies the same harness to raster-to-vector conversion through extraction, element processing, and critic-driven SVG composition.Extraction isolates assets, processing captions and classifies them, and composition assembles and refines a coordinate-faithful editable SVG.

4 CRAFTBENCH

CRAFTBENCH evaluates scientific figure generation across three figure types and four input conditions using 279 curated samples. It is constructed through multi-stage filtering and human curation, then evaluated with a redesigned VLM-as-judge protocol for cross-type, cross-condition comparisons.

  • Benchmark scope: CRAFTBENCH covers three figure types and four conditions, including text-to-image, mask-completion, key-element composition, and sketch-conditioned generation, with 279 curated samples.The benchmark includes one text-to-image task and three reference-conditioned tasks.
  • Data construction: The benchmark is assembled from academic figures, conference posters, and research-blog infographics through classification, complexity, claim-alignment filtering, and human curation.Filtering leaves 553 candidates before curation produces the final benchmark balanced across tasks and styles.
  • Benchmark composition: Text-to-image contributes 179 of 279 samples, while mask-completion, sketch-conditioned, and key-element composition contribute 30, 40, and 30 samples, respectively.Academic figures are the largest style share with 140 samples; sketch-conditioned and key-element composition use only academic figures.
  • Evaluation protocol: Evaluation scores each candidate and human-drawn target independently with a Gemini 3.5 Flash judge, using lenient win-rate reporting and avoiding pairwise position bias.The protocol follows a referenced VLM-as-judge philosophy but is redesigned for cross-type and cross-condition evaluation.

5 Experiments

Experiments show that CRAFTER generalizes across benchmarks, backbones, quality dimensions, and tasks, outperforming standalone generators and agentic baselines. CRAFTEDITOR also surpasses prior editable-figure baselines, while ablations identify planning, correction, and iterative composition as important mechanisms.

  • Main results: CRAFTER achieves the highest overall score on both benchmarks, leading the strongest agentic baseline by 16.61 point on PaperBanana-Bench and 22.20 point on CRAFTBENCH.The evaluation covers PaperBanana-Bench and CRAFTBENCH, scored with referenced VLM-as-judge protocols against human-drawn targets.
  • Main results: Across both backbones, CRAFTER scores best in every PaperBanana quality dimension and every CRAFTBENCH task, indicating broad rather than narrow gains.It improves over its standalone generator on every quality dimension and task.
  • Crafter ablations: Every mechanism removal lowers PaperBanana-Bench’s overall score, with drops ranging from 5.04 to 8.90 point.Using a single candidate plan causes an 8.56 point drop, while replacing typed edits with free-text revision instructions costs 8.90 point overall.
  • CraftEditor results: CRAFTEDITOR leads on all seven evaluation axes, scoring 8.04 overall versus 6.91 for AutoFigure-Edit and 3.69 for Edit-Banana.The comparison uses 80 held-out CRAFTER outputs and an ensemble of three VLM judges.
  • CraftEditor ablations: Removing iterative composition causes a uniform overall drop of −2.15 across seven axes, while removing agentic cleaning causes a consistent −0.33 effect.The largest per-axis drop from removing agentic cleaning occurs on icons.

6 Conclusion … C.2 Quality Gates

The paper concludes that CRAFTER and CRAFTEDITOR address limited cross-type, cross-condition generalization and non-editable outputs, while CRAFTBENCH enables corresponding evaluation. The appendices document experimental setup, benchmark construction, quality gates, evaluation protocols, human validation, limitations, case studies, and failure analysis.

  • 6 Conclusion: CRAFTER, CRAFTEDITOR, and CRAFTBENCH address cross-type, cross-condition figure generation, editable raster-to-SVG conversion, and benchmark evaluation.The conclusion identifies these systems as responses to limited generalization and non-editable outputs.
  • A Appendix: The appendices cover harness experiments, CRAFTBENCH construction, CRAFTER implementation, evaluation protocols, judge prompts, human validation, limitations, case studies, and failure cases.The appendix roadmap assigns these topics to Appendices B–L.
  • B Experimental Setup: The experimental setup evaluates CRAFTER on PaperBanana-Bench and CRAFTBENCH, and CRAFTEDITOR on raster-to-vector conversion.The setup section states that these experiments are summarized in Section 5.
  • B.1 CRAFTER on PaperBanana-Bench and CRAFTBENCH: PaperBanana-Bench contains 292 methodology figures, while CRAFTBENCH contains 279 samples and uses benchmark-specific VLM judging protocols.PaperBanana-Bench uses Gemini 3.1 Pro; CRAFTBENCH uses Gemini 3.5 Flash and reports lenient win-rate.
  • B.1 CRAFTER on PaperBanana-Bench and CRAFTBENCH: CRAFTER is compared with five vanilla generators and two agentic frameworks under controlled backbone and vision-language-model assignments.The controlled comparison uses Nano Banana 2 and gemini-3.1-pro-preview, with Nano Banana Pro additionally testing executor pluggability.
  • B.2 CRAFTEDITOR on Raster-to-Vector Conversion: CRAFTEDITOR is evaluated on a balanced held-out subset of 80 rasters against Edit-Banana and AutoFigure-Edit.Outputs are scored by a three-VLM ensemble across seven axes on a 0–10 scale, and two ablations use the same subset and ensemble.
  • C.1 Source Pool Composition and Crawl Windows: CRAFTBENCH contains 279 samples drawn from five source pools, including broad-domain and method/architecture arXiv crawls targeting method-figure coverage.The method/architecture crawl targets figures whose captions contain overview, pipeline, architecture, method, approach, or framework.
  • C.2 Quality Gates: Every CRAFTBENCH candidate passes a seven-stage quality pipeline beginning with caption keyword filtering, strict content classification, and complexity rescoring.Accepted fine-grained types include diagram, method illustration, architecture, and teaser; photographs, charts, screenshots, equation-only renders, and tables are rejected.

C.3 Reference-Conditioned Task Construction

CRAFTBENCH constructs three reference-conditioned tasks from source figures through semi-automatic preparation and manual quality assurance. Inputs are reviewed, edited or confirmed, and admitted only after unanimous human agreement.

  • Quality-controlled construction: All reference-conditioned inputs undergo semi-automatic construction, manual editing or confirmation, and quality filtering before entering the benchmark.The reported counts include only samples surviving the full quality pipeline with unanimous human agreement.
  • Mask-completion: Mask-completion (n=30) removes a hand-labeled semantic region while preserving unmasked ground-truth pixels, with mask areas averaging about 40%.Masks range from roughly 20% to over 90% and leave the remaining figure interpretable.
  • Key-element composition: Key-element composition (n=30) extracts icon-level spatial placeholders without text or arrows, applies random displacement, and receives human correction or confirmation.Annotators deleted incorrectly extracted elements and equalized difficulty; 29 samples were hand-edited and 1 hand-confirmed.
  • Sketch-conditioned generation: Sketch-conditioned generation (n=40) uses hand-drawn or pen sketches (n=15), AI-drafted rough sketches (n=14), or rasterized SVG wireframes (n=11).AI-drafted sketches use captions and surrounding context rather than source pixels, while every sketch passes human quality assurance.
  • Manual quality assurance: Three graduate-level annotators independently review every input in task-specific browser interfaces, revising disagreements until all three agree on acceptance.Revisions may include hand-recropping, hand-painting, regeneration, or switching sketch families before re-review.

D CRAFTER: Implementation Details · D.1 Scaling Behavior of K and T · D.2 Computational Cost

Crafter coordinates five agents around an evolving figure specification, using structured edits, iterative judging, and configurable plan diversity and refinement. Increasing candidate plans and rounds improves performance, while the added inference cost remains modest relative to manual figure production.

  • D CRAFTER: Implementation Details: Crafter coordinates five agents around an evolving specification S, with all agents reading and writing S rather than passing free-text addenda.The image-generation backend E supplies raster outputs but is not itself an agent.
  • D CRAFTER: Implementation Details: Crafter supports text-to-image and reference-conditioned generation through distinct routing branches, with new reference tasks requiring only instruction-builder and role-hint additions.Text-to-image samples call E once per plan, while reference-conditioned samples pass the reference image through E’s multimodal interface.
  • D CRAFTER: Implementation Details: Typed corrective edits update S through structured operations, while the critic reports six quality-axis scores, issues, suggestions, and a revised description.The repair agent reads only the issues and suggestions when selecting edits, keeping the next prompt coherent.
  • D CRAFTER: Implementation Details: The convergence judge combines hard stopping rules with vision-language acceptance and selects the highest-scoring artifact across rounds.After termination, OCR-based typo repair is guarded by a revert when correction introduces an artifact.
  • D.1 Scaling Behavior of K and T: +7.19 points is the largest single gain from increasing K from 1 to 3, showing that plan-level diversity is critical for avoiding unsuitable framings.Adaptive K adds +1.37 points over fixed K=3 and improves faithfulness by +3.93 points on the hardest content-correctness samples.
  • D.1 Scaling Behavior of K and T: +5 points results from increasing refinement rounds T from 1 to 3 on PaperBanana-Bench.The supplied passage reports this gain but does not include the remainder of the scaling comparison.
  • D.2 Computational Cost: Crafter costs approximately 2–3× more per figure than PaperBanana, while generating all 279 CraftBench samples costs under $90.CraftEditor adds $0.85 per raster-to-SVG conversion, with most of that cost coming from LLM tokens during iterative SVG refinement.

E CRAFTEDITOR: Implementation Details and Ablations · F CRAFTEDITOR: Judge Ensemble Protocol

CraftEditor combines agentic extraction with iterative SVG composition, and its ablations show that extraction generally improves category-level performance. Its editable-output quality is evaluated using a three-VLM ensemble with explicit scoring and retry rules.

  • E CRAFTEDITOR: Implementation Details and Ablations: The extraction phase uses a vision-language designer, an image-editing executor, and a verifier that iterates for at most T=3 rounds.The verifier accepts candidates or returns directive diagnostics for refinement.
  • E CRAFTEDITOR: Implementation Details and Ablations: The composition phase generates SVG skeletons at temperatures 0.20 and 0.45, selects one with a convergence judge, and performs T=4 hybrid-critic refinement rounds.The critic scores text presence, arrow endpoints, layout consistency, and color drift; best-so-far reversion provides protection during refinement.
  • E CRAFTEDITOR: Implementation Details and Ablations: Four external services are isolated behind interface adapters, making backend swaps, such as changing the segmentation model, a single configuration change.The wrapped services cover LLM inference, image editing, segmentation, and background removal.
  • E CRAFTEDITOR: Implementation Details and Ablations: The two ablations disable agentic cleaning or iterative composition and are evaluated on the same 80-sample subset with the same three-VLM judge ensemble.The former skips Stage 1 and uses segmentation plus background removal, while the latter sets the composition refinement loop to T=0.
  • E CRAFTEDITOR: Implementation Details and Ablations: CRAFTEDITOR wins or ties the “w/o agentic cleaning” ablation in 11 of 12 source categories under the headline judge ensemble.The only exception is a 3-sample text-to-image infographic subset, where differences fall within the per-sample noise band and are statistically inconclusive.
  • F CRAFTEDITOR: Judge Ensemble Protocol: The judge ensemble consists of Gemini 3.1 Flash-Lite, GPT-5.4, and Doubao-Seed-2.0-Pro, each scoring seven editable-output axes from 0–10.Judges receive the original raster and rendered SVG, use temperature 0.15, and return per-axis scores plus structured issues in JSON.
  • F CRAFTEDITOR: Judge Ensemble Protocol: The headline score is the mean of the three judges’ overall scores, with one retry replacing any original overall score below 3.0 when the retry is higher.This rule addresses VLM-judge volatility on visually unfamiliar inputs.
  • F CRAFTEDITOR: Judge Ensemble Protocol: The seven-axis editable-output rubric measures raster-to-SVG reproduction fidelity, whereas PaperBanana-Bench and CRAFTBENCH rubrics measure generation quality against human-drawn targets.These results appear in separate tables and should not be compared directly.

G Evaluation Protocol Details · H Judge Prompts

CRAFTBENCH evaluates generated figures against human-drawn ground truths with a fixed, per-image judging protocol and a lenient win-rate aggregation. Its four task-specific judge jobs share a skeptical review scaffold designed to detect textual, semantic, and input-fidelity errors, especially in edit tasks.

  • G Evaluation Protocol Details: Each generated figure and human-drawn ground truth is independently scored one image at a time against the same generator inputs.A Gemini 3.5 Flash judge uses temperature 0 and a fixed seed; inputs include the caption, paper context, and reference image when applicable.
  • G Evaluation Protocol Details: The protocol rates each image from 0 to 10 on a selected set of figure-quality aspects.The supplied passage specifies that each image is rated on a small aspect set, but does not include the full list.
  • G Evaluation Protocol Details: Aspect scores are combined by a weighted mean, emphasizing content faithfulness and input fidelity at 3.0 while weighting readability and format at 1.0 to 1.5.The weighting scheme produces one total score per image.
  • G Evaluation Protocol Details: A candidate is labeled Model, Human, or Tie according to whether its total exceeds, trails, or falls within 0.30 of the ground truth.A missing generation counts as Human, and the benchmark score is the sample mean of the {100, 50, 0} outcome mapping.
  • H Judge Prompts: The CRAFTBENCH judge runs as four separate jobs covering text-to-image, key-element composition, sketch-conditioned generation, and mask-completion.All jobs share the system scaffold and scoring anchors, while aspects and user messages are filled from task-specific metadata and attached JPEG inputs.
  • H Judge Prompts: The shared prompt presents a skeptical reviewer who compares one candidate figure at a time with the original illustration request and, when applicable, an input image.The prompt identifies one human-made and one AI-generated candidate across the reviewed candidates.
  • H Judge Prompts: The judge is instructed to detect garbled or fake text, incorrect components, generic placeholders, and silent alteration or neglect of input images in edit tasks.Edit jobs additionally foreground input fidelity in the shared scoring anchors.

I Human Evaluation

A blind pairwise human study evaluated whether the CRAFTBENCH VLM judge reflects human preferences across four tasks and three figure types. The automatic score matched the majority human verdict on 72% of cases, with Cohen’s κ = 0.58.

  • Study protocol: Three graduate-level annotators each rated a random sample of 60 cases spanning four tasks and three figure types.Annotators compared model outputs against original human-drawn figures in a custom web interface and were compensated at $25 per hour.
  • Study protocol: The blind pairwise study mapped each comparison to a MODEL, TIE, or HUMAN verdict and compared it with the automatic judge.The study used three annotators and measured agreement against the majority human verdict.
  • Agreement with human judgment: 72% of cases matched the majority human verdict at Cohen’s κ = 0.58, supporting CRAFTBENCH as a reliable proxy for human judgment.This agreement held across all four tasks and three figure types.

J Limitations · K Case studies · L Failure cases

The paper identifies proprietary-model dependence, nontrivial deployment cost, and limited benchmark scale as key limitations. Case studies show CRAFTER’s advantages on input-honoring edits and multi-component figures, while failure cases expose specific weaknesses in panel counting, infill continuity, and structural critique.

  • J Limitations: Headline results depend on closed-source image-generation backbones and evaluation judges, making the harness conditional on proprietary access and judge biases.The backbones include Gemini 3.1 Flash Image, Gemini 3.0 Pro Image, and openai/gpt-image-2; judges include Gemini 3.1 Pro and Gemini 3.5 Flash.
  • J Limitations: A single CRAFTER run can execute four parallel generations and three refinement rounds, while CRAFTEDITOR adds roughly four VLM rounds and SVG composition, increasing latency and API cost.Scaling deployment therefore requires corresponding API budget and wall-clock time.
  • K Case studies: On input-honoring editing tasks, baselines regenerate from captions and ignore conditioning inputs, whereas CRAFTER preserves their spatial structure and scores much higher on input fidelity.The comparison concerns CRAFTBENCH editing samples shown in Figure 5.
  • K Case studies: For multi-component academic figures, baselines omit paper-named components and substitute generic visuals, while CRAFTER’s verify-then-refine loop detects omissions and adds corrective layout guidance.The corrective layer pins the missing component to the correct sub-region for the next refinement round.
  • L Failure cases: CRAFTER does not win every sample: three cases are judged better for the human-drawn target, each isolating a failure mode and responsible harness stage.These cases are collected in Figure A8.
  • L Failure cases: In text-to-image generation, intent reasoning can collapse a multi-panel caption into one panel, preventing later verification and refinement from recovering the missing panels.The failure is characterized as dropped panels in multi-panel a/b/c captions.
  • L Failure cases: Mask completion can produce clashing boxed infill that breaks continuity with the preserved diagram at the mask boundary.The stated cause is attributed to the backbone and to a critic that evaluates structure without checking input-faithful infill.
  • L Failure cases: Sketch-conditioned generation can yield a literal skeleton, motivating concrete fixes including panel-count and mask-boundary continuity checks in the critic.The supplied passage identifies the literal-skeleton mode but truncates its further description.
Loading 2605.30611v1…