PaperlayerView exact revision on arXiv

Source-linked AI summary

FigMirror: Ground It, Code It, Plot It

Xiaohan Zhao, Jiacheng Liu, Yaxin Luo, Zhiqiang Shen

arXiv:2608.28814v1cs.CVcs.AI

TL;DR

Existing chart-to-code systems mainly reproduce reference figures instead of transferring their visual style to new data. FigMirror addresses this gap with Grounded Measurement and a Drawer–Reviewer framework, evaluated on PlotTwin-Bench, where it consistently outperforms prior methods. The approach treats figure style as measurable and reusable while leaving judgments about data-appropriate presentation to the user.

  • Problem

    Existing methods focus on pixel-level reproduction, while reference-conditioned style transfer requires preserving a figure’s visual style when plotting new data.

  • Method

    FigMirror uses coordinate grounding to locate style elements, executable code to measure their values, and a Drawer–Reviewer loop to apply and revise those attributes.

  • Results

    FigMirror consistently outperforms prior methods on reference-conditioned style transfer, evaluated by PlotTwin-Bench through code- and image-level style criteria.

  • Takeaways & Limitations

    The framework makes scientific figure style a measurable and reusable design choice for transferring presentation style after a user selects a reference.

  • Takeaways & Limitations

    FigMirror does not decide whether a presentation fits the data; users remain responsible for scales, labels, units, legends, captions, and checking the visual encoding.

Abstract

from arXiv · show

Converting scientific figures into executable code has gained increasing attention, yet existing methods primarily focus on reproducing the reference figure itself. A more practical setting is to plot new data while preserving the visual style of a reference figure (e.g., color scheme and typography). Prior approaches mimic the reference through pixel-level optimization and struggle to carry its style to new data. We show that the key to this task lies in the coordinate grounding and coding capabilities present in modern computer-use models. We propose FigMirror, an agentic framework that unlocks these capabilities through Grounded Measurement, which locates visual elements by coordinates and measures their properties through executable code. We further introduce PlotTwin-Bench, an expert-curated benchmark with fine-grained code and image-level style metrics. Experiments show that FigMirror consistently outperforms existing methods on reference-conditioned style transfer. All plots in this paper are generated by FigMirror, except those produced by other methods for comparison. Our code and data are available at: https://github.com/VILA-Lab/FigMirror.

1 INTRODUCTION

Scientific figure style transfer targets a practical gap left by chart reproduction: preserving a reference’s visual style while plotting new data. FigMirror addresses this through coordinate-grounded measurement, an agentic Drawer–Reviewer process, and a benchmark that evaluates style in code and rendered images.

  • Motivation: Producing publication-quality figures requires repeated manual adjustment of spacing, panel sizes, and fonts.Researchers have traditionally adopted figures from top venues as references, manually copying spacing, colors, and layouts.
  • Motivation: Existing chart-to-code methods primarily reproduce reference figures at the pixel level rather than transfer visual style and layout to new data.The paper identifies reference-conditioned style transfer as a more practical setting than reproducing the original figure.
  • Problem formulation: Precise style measurement is the core challenge because transferable attributes must be identified and their exact values computed from pixels.Examples include measuring a color series or subplot spacing as a hex code or width in points.
  • Approach: Grounded Measurement repurposes coordinate grounding to locate each style-bearing element and read its value with executable code.FigMirror organizes these measurements in an agentic Drawer–Reviewer framework that renders user data into a style template.
  • Evaluation: PlotTwin-Bench evaluates figure style transfer with expert-curated figure–code pairs and per-reference criteria measured through both code and rendered images.Its vision channel detects rendering-only issues such as overlapping text or clipped labels, while FigMirror consistently outperforms prior methods.

2 RELATED WORK

Prior work developed chart-to-code reproduction and computer-use grounding separately, but neither line directly studies transferring a reference figure’s style to new data. The paper positions reference-conditioned style transfer as a distinct task requiring attribute-level style reading.

  • Chart-to-code generation: Chart-to-code methods generate executable plotting code from chart images, with later systems scaling data, code-oriented backbones, and training methods.The cited line began in chart understanding and became generative when multimodal models could emit executable programs.
  • Computer-use agents: Computer-use agents rely on GUI grounding to locate interface elements before operating software across web and desktop environments.This grounding capability supplies the coordinate interface that FigMirror reuses for plot elements.
  • Research gap: Neither research line reaches reference-conditioned style transfer: chart-to-code targets reproduction, while computer-use agents apply grounding to on-screen actions.The paper states that no method reads reference style attributes one by one or benchmarks how faithfully style transfers to new data.

3 METHOD

FigMirror converts reference style into measurable, reusable attributes and iteratively applies them to new data. Grounded Measurement localizes each visual element, computes its value with code, and supports a Drawer–Reviewer loop that preserves matches and repairs localized mismatches.

  • 3 METHOD: FigMirror uses a Style Checklist to identify reference attributes, then resolves them through Grounded Measurement before rendering user data in a template.The Drawer keeps open and resolved attributes, moving each measured value into the resolved set before writing plotting code.
  • 3 METHOD: Scientific figures and GUIs share discrete, code-rendered elements with sharp boundaries and flat fills, enabling coordinate grounding for ticks, legends, and marks.The reused grounding interface returns coordinates for visual elements in the reference figure.
  • 3.1 GROUNDED MEASUREMENT: Grounded Measurement first localizes an attribute-bearing region and then computes its exact value from a local image crop with a short program.The method supports continuous attributes such as colors, widths, and aspect ratios, as well as discrete attributes such as grid-line presence.
  • 3.2 THE DRAWER–REVIEWER LOOP: The Reviewer compares the draft with the reference using global and high-resolution views, marking localized mismatches for repair.Marked regions make feedback targets precise, allowing the Drawer to revise the relevant attribute while leaving matched attributes untouched.
  • 3.2 THE DRAWER–REVIEWER LOOP: A stateful Drawer and stateless Reviewer alternate until the Revision List is empty, while the Preserve List carries accepted style choices across iterations.FigMirror is packaged as a portable skill containing the checklist, measurement procedure, and iterative loop.

4 BENCHMARK AND EVALUATION

PlotTwin-Bench evaluates reference-conditioned style transfer through complementary code and vision channels, focusing on reference-specific departures from default plotting. The combined score uses both channels with fixed weights and is intended to align with human judgments.

  • Evaluation design: PlotTwin-Bench scores reference-specific style choices through complementary code and rendered-image channels.The code channel evaluates departures from Matplotlib defaults, while the vision channel identifies visible defects such as layout and spacing mismatches.
  • Evaluation design: The code channel gives proportional credit as candidate attributes move from defaults toward exact reference values.Categorical attributes use exact agreement, while continuous attributes use normalized distance.
  • Evaluation design: The vision channel starts at 100 and deducts 5, 10, or 25 points for minor, major, or critical visual defects.Each defect must cite visible evidence through bounding boxes and, when applicable, relevant code lines.
  • Combined score: The combined score gives the vision channel the larger fixed weight because readers judge rendered figures.The same weights are used across experiments, with both channel scores reported separately.

5 EXPERIMENTS

The experiments evaluate FigMirror against four external chart-to-code baselines on a curated subset of PlotTwin-Bench. All methods produce self-contained plotting scripts under a shared GPT-5.5 evaluation setup.

  • Experimental setup: The evaluation uses 150 of 400 references, including all 50 hand-curated figures and 100 randomly sampled augmented references.The exact evaluation subset is included in the benchmark release.
  • Experimental setup: Every method returns a self-contained plotting script, and the evaluator and underlying models use GPT-5.5 at x-high reasoning effort.FigMirror runs as a skill in the Codex harness.
  • Baselines: FigMirror is compared with Plot2Code, METAL, ChartGalaxy, and ChartIR.The baselines span one-shot generation, iterative feedback, prompt-based generation, and successive code repair.

5.3 ABLATIONS

Ablations show that Reviewer feedback and Grounded Measurement both improve style transfer, while additional iterations raise the combined score. The case study illustrates how localized review boxes drive successive repairs without disturbing accepted attributes.

  • 5.3 ABLATIONS: Removing the Reviewer costs 2.1 points, removing Grounded Measurement costs 3.7, and removing the skill costs 11.7.Naked Codex falls to the level of external baselines, while Grounded Measurement removal causes a 7.2-point vision-channel loss despite a slight code-channel increase.
  • 5.4 ITERATION BUDGET: The combined score rises from 71.8 at one iteration to 72.7 at three and 74.3 at five.Three iterations improve vision by 1.4 points while leaving the code score essentially unchanged; five improve both channels.
  • 5.5 MECHANISM-LEVEL CASE STUDY: Review boxes convert composition-wide searches into local edits, such as separating overlapping insets in the next iteration.Unboxed attributes are preserved while the Drawer edits only localized mismatches.
  • 5.5 MECHANISM-LEVEL CASE STUDY: Each review re-audits the full draft, allowing subtler mismatches to surface after dominant defects are cleared.In the illustrated run, every boxed defect is resolved in the following iteration and accepted attributes remain undisturbed.
  • 5.5 MECHANISM-LEVEL CASE STUDY: The measurement probes return exact attribute values that feed the Drawer’s subsequent repairs.The example includes recovered colors, panel positions, gaps, and line properties.

6 CONCLUSION

FigMirror treats scientific figure style transfer as a measurable problem: style elements are localized, measured, and reapplied to new data. Its Grounded Measurement and Drawer–Reviewer framework, evaluated by PlotTwin-Bench, makes style explicit and revisable.

  • Conclusion: Reference-conditioned style transfer requires localizing, measuring, and reapplying style elements because target data differ from the reference.The paper links this process to the discrete, code-rendered structure shared by scientific figures and graphical interfaces.
  • Conclusion: Grounded Measurement reads each style element’s value with code, while the Drawer–Reviewer loop makes attributes explicit, measurable, and revisable.PlotTwin-Bench evaluates candidates through both code and rendered-image criteria.

ETHICS STATEMENT

FigMirror transfers a chosen reference’s presentation style but leaves responsibility for whether that style fits the data with the user. Reference figures used outside the benchmark also require respect for licensing and attribution terms.

  • Users remain responsible for axis scales, labels, units, legends, captions, and checking that visual encodings support the underlying claim.
  • Applying FigMirror outside the benchmark requires respecting the reference figure’s license and attribution terms.

A LIMITATIONS

FigMirror’s limitations follow the capabilities of its underlying code-capable multimodal model. Errors in localization, measurement, code generation, or visual review can pass through the Drawer–Reviewer loop.

  • FigMirror depends on the underlying model to localize plot elements, measure style values, write executable code, and detect residual visual mismatches.
  • An error in either the Drawer’s or Reviewer’s abilities can pass through the loop.
  • If future models perform these tasks in one shot, the grounding mechanism and Drawer–Reviewer loop become less necessary.

B.1 BENCHMARK COMPOSITION

The benchmark combines hand-curated scientific figures with augmented references generated by rewriting existing plotting code toward greater structural complexity and visual polish. Figure 8 reports the complete chart-type distribution across both sources.

  • Figure 8 reports counts for all 12 chart types in the augmented and hand-curated sources.
  • The augmented source is built by rewriting ChartMimic plotting scripts toward higher structural complexity and visual polish, retaining references that satisfy both criteria.

B.3 TRANSFER-DATA CONSTRUCTION

Transfer data are constructed by reading each reference’s visual structure, inventing a different-domain scientific story, and generating compatible but varied CSV data. The process preserves the reference’s macro visual organization while varying selected data, scale, semantic, or categorical dimensions and checking output validity.

  • For each reference, the generator reads chart types, panel structure, major regions, and hierarchy before defining a different-domain scientific story and writing one CSV.
  • The reference remains a viable template by preserving its main chart types, macro layout, reading order, and panel hierarchy.
  • Variation changes one to three dimensions, including data shape, axis and scale, or semantic and categorical properties.
  • Regular grids remain complete grids, while heterogeneous figures preserve macro regions and may change local panel allocation, grouping, or schema.
  • The generated CSV is checked for consistent row width, complete role coverage, finite values, plausible numerical relationships, and valid file structure.

B.4 EVALUATION PROTOCOL DETAILS

The evaluation separates code-visible style fidelity from rendered visual mismatches, then combines both channels after generation. The protocol transfers reference style—not reference data or fixed panel counts—through an iterative Drawer–Reviewer workflow and explicit style-transfer constraints.

  • Scoring protocol: Each reference receives separately auditable code and vision scores after generation, without rerendering or exposing scoring criteria to the method.The evaluator reads source code, images, and render records only after the plotting script has run.
  • Code channel: The code channel scores deviations on reference-specific style attributes, comparing reference and candidate values against Matplotlib defaults.Only attributes differing from the defaults enter the reference-specific set, and per-attribute credit measures closeness to the reference value.
  • Vision channel: The vision channel evaluates layout, spacing, alignment, and hierarchy through evidence-grounded visual defects, with minor, major, and critical penalties of 5, 10, and 25.Each defect must cite visible bounding-box evidence and, when applicable, relevant code lines.
  • Combined score: The combined score is S = 0.35Scode + 0.65Svision, preserving complementary evidence from code-visible choices and rendered effects.The vision channel receives the larger weight because code cannot settle every distinctive visual choice.
  • Human validation: Across five methods, the combined score and human win rate have Spearman correlation ρs = 0.90, but the small sample makes this evidence suggestive rather than conclusive.The validation used 50 references, over 300 pairwise judgments, and 76% annotator agreement on doubly labeled comparisons.
  • Transfer constraints: The transfer workflow plots target data while preserving the reference’s visual style, adapting panel count to the data and retaining the reference chart type and mark family.The instructions explicitly distinguish style transfer from reproduction and prohibit changing chart family merely because data values, series, or panel counts differ.
Loading 2608.28814v1…