PaperlayerView exact revision on arXiv

Source-linked AI summary

VisAudit: Evaluating Multimodal Agents for Visual Diagnosis and Repair

Shicheng Liu, Adam Kahirov, Qi Zhang, Zhimin Hu, Song Wang, Junhong Lin, Julian Shun, Yada Zhu

arXiv:2610.02399v1cs.LG

TL;DR

Autonomous visualization review requires deciding whether a chart is wrong, what to change, and whether the repair worked, while existing benchmarks generally specify the task in advance. VisAudit evaluates that review loop across three tracks using controlled chart perturbations and correct examples. Current multimodal agents remain far from reliable autonomous review, struggling with diagnosis, complex repair, and avoiding unnecessary changes.

  • Problem

    Existing visualization benchmarks generally evaluate predefined generation, editing, or defect-detection tasks rather than autonomous decisions about whether and how to intervene.

    Source: §1 · p. 1

  • Method

    VisAudit evaluates iterative diagnosis, code repair, execution, and verification across diagnosed-repair, autonomous-repair, and open-world-verification tracks.

    Source: §1 · p. 2

  • Results

    Current agents struggle to diagnose unspecified defects, resolve multiple simultaneous flaws, and avoid unnecessary modifications to correct charts.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    VisAudit provides a benchmark for studying when multimodal agents should act, what to change, and whether an intervention succeeds.

    Source: §6 · p. 10

Abstract

from arXiv · show

Multimodal agents are increasingly used for data visualization tasks but remain limited in autonomous review. Unlike humans, they may fail to recognize when a visualization is incorrect, determine what to change, repair it without disrupting correct content, and verify whether the intervention succeeded. Existing benchmarks largely evaluate predefined individual capabilities such as chart generation, instruction-guided editing, or defect detection, and therefore do not capture this gap in autonomous review. We introduce VisAudit, a benchmark for evaluating visualization diagnosis, repair, and verification. Given a rendered chart and configurable auxiliary evidence, including its source data table, intended text summary, and visualization code, an agent iteratively diagnoses potential defects, modifies and executes visualization code, inspects execution and visual feedback, and determines when no further intervention is needed. VisAudit defines three tracks spanning diagnosed repair, autonomous repair, and open-world verification, and contains 1,900 flawed instances across 21 chart types and 10 flaw categories, together with 300 initially correct charts. We construct the benchmark through controlled perturbations of validated source visualizations, with systematic verification and human-aligned quality control to ensure that injected defects are well-defined and recoverable from the available evidence. Experiments with leading multimodal models reveal a substantial gap from reliable autonomous review: the strongest evaluated model fully recovers only $47.4\%$ of flawed charts in the autonomous-repair setting.

1 INTRODUCTION

Autonomous visualization review requires agents to discover defects, decide whether and how to intervene, preserve correct content, and verify the outcome. VisAudit evaluates this full review process through three tracks and a controlled benchmark.

1 INTRODUCTION

  • Existing benchmarks mostly specify the task in advance, leaving autonomous decisions about whether, how, and when to intervene unmeasured.Prior work targets chart generation, instructed editing, or detection of predefined properties.Source: §1 · pp. 1–2
  • VisAudit evaluates agents that use a chart and configurable auxiliary evidence to edit, execute, inspect, and revise visualization code.The agent decides when to continue or stop and whether the resulting visualization is satisfactory.Source: §1 · p. 2
  • Its three tracks progress from repairing a disclosed defect to autonomous diagnosis and then deciding whether any intervention is needed.Open-world verification includes initially correct visualizations.Source: §1 · p. 2
  • The benchmark contains 1,900 flawed instances across 21 chart types and 10 flaw categories, plus 300 initially correct charts.Instances are produced by controlled perturbations of correct source visualizations.Source: §1 · p. 2
  • Evaluation measures executability, recovery, preservation, and correct-chart recognition to assess repair and avoidance of unnecessary intervention.Source: §1 · p. 2
  • Agents struggle to diagnose unspecified defects, resolve simultaneous flaws, and avoid unnecessary changes to already-correct charts.The introduction identifies these challenges as limitations of current multimodal models.Source: §1 · pp. 2–3

2 RELATED WORK

Prior visualization benchmarks evaluate generation, instructed editing, or misleading-chart detection, generally with a predefined task. VisAudit instead targets the complete autonomous review loop.

Chart generation and instruction-driven editing

  • Generation and editing benchmarks specify the desired output or modification, even when they support iterative refinement or preservation of unaffected content.Examples include ChartMimic and Chart2Code for chart-to-code generation or reconstruction, and ChartEditBench and VisEditBench for iterative editing.Source: §2 · p. 3

Misleading visualization detection and correction

  • Misleading-visualization benchmarks assess recognition or correction of deceptive designs, rather than the full autonomous review process.Misleading ChartQA and Misviz focus on detection, while MisVisFix also provides interactive correction.Source: §2 · p. 3

Comparison with VisAudit

  • VisAudit combines diagnosis, selective repair, verification of repair consequences, and no-op decisions when charts are already correct.Table 1 contrasts these capabilities with representative benchmark settings.Source: §2 · p. 3, Table 1 · p. 3

3 VISAUDIT BENCHMARK

VisAudit evaluates agents reviewing charts against configurable reference evidence in an iterative code-execution environment. Its tracks progressively increase the need for diagnosis and verification.

Task Formulation

  • Each instance supplies an initial chart and may include the source table, intended summary, and original visualization code as auxiliary modalities.The chart is always provided; auxiliary evidence varies by experimental configuration.Source: §3 · p. 4
  • A successful output must execute, represent the data and intended message faithfully, repair defects, and preserve already-correct aspects.When original code is unavailable, the agent generates code from the chart and available reference modalities.Source: §3 · p. 4

Interaction

  • Agents create or modify code, execute it, inspect rendered and execution feedback, and may revise their output before deciding to stop.Evaluator outcomes are not exposed during interaction.Source: §3 · p. 4

Benchmark Tracks

  • Diagnosed Repair provides the flaw description, Autonomous Repair withholds it, and Open-World Verification also includes charts that may already be correct.The tracks respectively test repair, diagnosis plus repair, and whether intervention is needed at all.Source: §3 · p. 4

Benchmark composition

  • The benchmark derives 1,900 flawed instances from 854 source charts across 21 chart types and 10 flaw categories using controlled code perturbations.The source table and intended summary remain reference evidence while each perturbation introduces a targeted discrepancy.Source: §3 · p. 5

Evaluation-set composition

  • The evaluation set includes 400 single-flaw, 900 double-flaw, and 600 triple-flaw instances; Track C adds 300 initially correct charts for 2,200 total instances.Tracks A and B use the same 1,900 flawed tasks and differ in whether instructions disclose the flaws.Source: §3 · p. 5, Table 3 · p. 5

4 EVALUATION PIPELINE

The pipeline evaluates repair by execution success, recovery of properties altered by injected flaws, and preservation of unaffected properties. It also measures recognition of initially correct charts and reports efficiency and coverage separately.

4 EVALUATION PIPELINE

  • Recovery and preservation jointly assess whether code runs, repairs altered chart properties, and avoids unintended changes to unaffected properties.Evaluation uses structured properties such as titles, axis labels, series, plotted values, legends, and axis configurations.Source: §4 · pp. 5–6
  • Recovery compares submitted properties with ground truth over the perturbation footprint, the properties changed between the ground-truth and initial charts.For perturbed axis limits, the comparator checks coverage of the required data extent or summary-specified range rather than exact autoscaling padding.Source: §4 · p. 6
  • Preservation rewards retaining unaffected properties and penalizes unsupported additions, excluding automatically changed properties from scope when plotted data are modified.For example, automatically scaled axis limits are excluded from preservation scope after data modifications.Source: §4 · p. 6
  • Correct-Chart Recognition Rate measures whether an agent leaves initially correct Track C charts unchanged and terminates without modifying their code.It is assessed on 300 initially correct instances.Source: §4 · p. 6
  • The pipeline reports tool steps, model-response counts, available token use, and execution outcomes separately; technical failures and unstarted tasks count as coverage gaps.Completed unsuccessful repairs remain in outcome statistics, and each metric uses its own denominator.Source: §4 · p. 6

5 EXPERIMENTS

Across the three tracks, agents remain limited in autonomous review, with performance varying by defect type and count and by whether charts may already be correct. The experiments also examine how input modalities and interaction budgets affect review.

Experimental setup

  • The evaluation tests seven models across three tracks, using chart, code, source table, and summary inputs, and measures executability, recovery, preservation, interaction, and cost.Tracks A and B use the same 1,900 flawed instances; Track C also includes 300 initially correct charts.Source: §5 · pp. 6–7

Overall performance

  • Across tracks, recovery on flawed Track C charts is generally comparable to Track B, while preservation decreases for several models and turn counts change relatively little.The similar turn counts suggest that performance differences primarily reflect diagnosis and decision making rather than longer interaction.Source: §5 · p. 8

Behavior on initially correct charts

  • Correct-chart recognition ranges from 32.7% to 72.0%, and binary preservation is below 90% for every model on initially correct charts.Agents frequently modify already-correct charts rather than terminating without intervention.Source: §5 · p. 8, Table 6 · p. 8

Performance by flaw category

  • Wrong titles are among the easiest flaws to repair, while wrong chart types remain challenging even when the defect is provided.Providing the flaw yields particularly large gains for missing legends, dropped series, and color contradictions, while gains are generally smaller for readily identifiable flaws.Source: §5 · p. 8, Table 7 · p. 8

Performance by number of flaws

  • Binary recovery generally declines as flaws accumulate; Qwen's binary recovery falls from 53.0% to 17.8% in Track B, 49.0% to 17.5% in Track C, and 74.2% to 46.3% in Track A.Graded recovery is more stable because it measures the fraction of affected properties recovered, not whether every flaw was fixed.Source: §5 · p. 9, Table 8 · p. 9

Interaction budget

  • Beyond a sufficient interaction budget, additional turns provide little recovery benefit and can reduce preservation by introducing unnecessary changes.The results indicate that interaction is needed to complete repairs, but more turns do not necessarily improve them.Source: Table 10 · p. 9, §5 · p. 10

6 CONCLUSIONS

VisAudit evaluates autonomous visualization diagnosis, repair, and verification across diagnosed-repair, autonomous-repair, and open-world-verification tracks. Experiments show that current multimodal agents remain far from reliable autonomous review, with performance varying by flaw type and count.

Conclusion

  • VisAudit is a benchmark for evaluating autonomous visualization review through diagnosis, repair, and verification.Source: §6 · p. 10
  • Across three tracks, experiments show that current multimodal agents remain far from reliable autonomous review.Source: §6 · p. 10
  • Performance varies substantially across flaw types and counts, and VisAudit provides a foundation for developing agents that determine when to act, what to change, and when an intervention succeeds.Source: §6 · p. 10

A DATASET CONSTRUCTION

VisAudit is built by perturbing aligned, validated visualizations while preserving source evidence, then checking that injected flaws are present and repairable. Its evaluation includes 1,900 flawed charts and 300 initially correct charts.

Source visualizations

  • Each source visualization supplies aligned plotting code, a rendered chart, a CSV table, and a natural-language summary; perturbations change the code and chart while leaving the table and summary unchanged.The original code-rendered chart and structured figure properties serve as evaluation references.Source: A DATASET CONSTRUCTION · p. 11

Perturbation taxonomy

  • Ten flaw categories introduce specific chart discrepancies while keeping source evidence unchanged, including altered plotted values and missing legends.For example, a value perturbation conflicts with the correct table value; a missing legend tests whether a series mapping is needed even if the summary does not request one.Source: A DATASET CONSTRUCTION · p. 11

Generating single and combined flaws

  • Category-specific prompts generate single flaws, while two or three perturbations are applied sequentially for multi-flaw tasks; axis truncation instead uses a deterministic transformation.The combinations retain 36 pair scenarios and 60 triple scenarios, excluding conflicting or insufficiently attributable effects.Source: A DATASET CONSTRUCTION · p. 11

Quality control and human alignment

  • Automatic checks test executability and figure-property changes, while category-specific checks and a cross-family LLM panel assess whether flaws are instantiated and repairable from the source evidence.Instances require positive judgments from a majority of at least three panel models; 8,446 of 10,945 single-flaw candidates pass all checks.Source: A DATASET CONSTRUCTION · p. 12
  • Human assessment of 150 randomly sampled instances agrees with LLM-judge decisions at over 96.64%, supporting the panel’s use for validation at scale.The assessments check whether the intended defect is present and whether source evidence supports its repair.Source: A DATASET CONSTRUCTION · p. 12

Evaluation set

  • The evaluation set contains 400 single-, 900 double-, and 600 triple-flaw instances across 106 scenarios; Tracks A and B use the same 1,900 instances but differ in whether instructions reveal flaw categories.Flaw count, track, available modalities, and interaction budget provide separate dimensions for studying task difficulty.Source: A DATASET CONSTRUCTION · p. 12

Initially correct charts and release

  • Track C adds 300 validated, unperturbed charts to the 1,900 flawed instances, with correct charts making up 13.6% of that track.The correct charts are mixed with flawed ones, and the instruction does not reveal whether a chart needs repair.Source: A DATASET CONSTRUCTION · p. 12
  • Released instance identifiers and versioned manifests identify task files and evaluation code, while reference code and perturbation metadata are reserved for evaluators.Source: A DATASET CONSTRUCTION · p. 12

A.1 DATASET STATISTICS

The dataset-statistics passage describes chart-type labels, length measures, and table-size measures reported in Table 11.

A.1 DATASET STATISTICS

  • Table 11 reports source chart-type labels per task, with counts including repeated uses of a source chart across distinct perturbations.Source: §A.1 · p. 12
  • Code length counts physical lines, while summary length counts whitespace-separated words.Source: §A.1 · p. 12
  • CSV length counts data records after the header, and header width is reported separately because source tables need not be rectangular.Source: §A.1 · p. 12

A.2 FLAW-GENERATION PROMPTS

The flaw-generation prompts specify constrained code edits that create visible, verifiable visualization defects while preserving truthful source evidence and avoiding unrelated changes. They cover several defect types and require checks that the intended repair is recoverable from the available evidence, with exact evidence and target metadata.

A.2 FLAW-GENERATION PROMPTS

  • Flawed candidates come from Mistral-Medium-3.5-128B at temperature 0 and seed 42, while axis truncation uses a deterministic transformation rather than a prompt.In multi-flaw chains, each later request receives the previously perturbed script.Source: §A.2 · p. 12
  • A flaw must change rendered pixels, remain plausible and repairable, and alter only the targeted code while preserving the complete script’s other content and execution conditions.The table and summary remain untouched and truthful; the evidence must quote the original state that the flawed chart contradicts. If a category does not fit or the edit would be invisible or break the script, the prompt requires a decline.Source: §A.2 · pp. 13–15
  • V2 requires category-specific target metadata and physical evidence, declining when competing, conditional, negated, or former-state claims undermine the proposed repair truth.For example, title truth must have one direct affirmative summary clause, and axis labels must be supported by a qualifying summary relation or a uniquely bound CSV column.Source: §A.2 · p. 16
  • Wrong-title flaws change only the title literal and require a direct, exact, unambiguous summary statement of the original title.Wrong-axis-label flaws change one label or swap the x/y labels; the original label must be physically supported and no competing label claim may exist in scope.Source: §A.2 · pp. 17–19
  • Missing-legend flaws must remove the legend without changing plotted data, and every removed series label must remain recoverable through a complete ordered code or CSV structure.The prompt warns that deleting an explicit legend call can be a no-op when another code path redraws it.Source: §A.2 · p. 20
  • Swapped-legend flaws exchange two series names while leaving data, colors, markers, and plot-call order unchanged, and require visually distinguishable series whose names appear in the summary or table.If the script’s palette or other naming changes would create a second defect or restore truthful labeling, the prompt requires declining the instance.Source: §A.2 · p. 21

Value perturbation

  • A value perturbation changes exactly one measured data value—not years, indices, x positions, or axis ticks—by multiplying it by 0.45, 1.6, or 2.25, preserving the original numeric precision.Prefer a value in the middle of the range, not the largest or zero; the change should move the mark by roughly at least 10% of the plotted value range.Source: §A.2 · p. 22

Swapped series

  • Series swaps exchange two equal-length numeric arrays between plotted series while preserving labels, colors, markers, and drawing order.The arrays must have at least five elements, distinct value profiles, and table-verifiable values; the edit must be a real transposition rather than copying one series over the other.Source: §A.2 · p. 22

Dropped series

  • Series deletion removes exactly one identifiable series through the smallest fitting code change, while preserving the remaining series and avoiding a blank chart.The prompt covers multiple plotting mechanisms and rejects edits that require unrelated changes, fail to remove the series from all its marks, or cannot uniquely bind its values to the source table and recover its mark kind.Source: §A.2 · pp. 23–24

Wrong chart type

  • Chart-type errors replace one plotting function or trace class with an allowed alternative at every call site, keeping the data and visual attributes unchanged.Only specified incompatible keywords may be removed or renamed; the edit is rejected if it needs unsupported surgery, loses a described data channel, or produces no visible marks.Source: §A.2 · pp. 24, 26

Color contradiction

  • Color contradictions repaint a visible chart element with a clearly different color that conflicts with an attributable summary statement.The prompt requires tracing the edited literal to the rendered artist, preserving color notation, and declining when the summary relation is ambiguous or the code’s color literal is overridden or unused.Source: §A.2 · pp. 26–27

A.3 QUALITY-CONTROL JUDGE PROMPT

The quality gate checks whether injected defects are detectable from truthful summaries and tables, distinguishing explicit from semantic contradictions. It rejects instances when the evidence does not contradict the flaw, regardless of visual obviousness.

A.3 QUALITY-CONTROL JUDGE PROMPT

  • Four judge models receive the same prompt; a judgment is positive when the detectable field is true, with unparseable responses retried up to three times.For multi-flaw tasks, each component flaw is judged separately.Source: §A.3 · p. 27
  • The judge evaluates detectability from the untouched summary and table, ignoring whether the flaw is visually obvious.An instance is fair only if those text and numeric references expose the defect.Source: §A.3 · p. 27
  • Explicit detectability names the changed element, while semantic detectability follows from a contradiction in meaning even without naming it.The prompt contrasts a quoted title with an incompatible y-axis label as examples.Source: §A.3 · p. 27
  • Generic evidence that remains consistent with the flawed chart is insufficient, and the judge must quote the fragment that exposes a detectable flaw.A removed legend is rejected when the summary does not identify the individual series.Source: §A.3 · pp. 27–28

B INTERACTIVE ENVIRONMENT

The interactive environment gives agents a writable workspace and provider-neutral tools for iterative chart review. It returns execution and visual feedback while leaving termination to the agent under the main protocol.

B INTERACTIVE ENVIRONMENT

  • Each episode starts in a fresh writable workspace with only condition-available modalities, and agents act sequentially using six provider-neutral tools.The tools execute Python, read and write files, list files, view images, and finish; each action and observation is auditable.Source: B INTERACTIVE ENVIRONMENT · p. 28
  • A persistent Python worker preserves variables, imports, and open figures across turns, while timeouts can trigger a restart that loses in-memory state.Rendering uses Matplotlib Agg, and file paths are restricted to the episode workspace.Source: B INTERACTIVE ENVIRONMENT · p. 28
  • Execution returns outputs, errors, warnings, tracebacks, and changed files, but agents must explicitly view images and choose when to finish.Under the main uncapped protocol, verifier outcomes are hidden and do not trigger early stopping; infrastructure safeguards still apply.Source: B INTERACTIVE ENVIRONMENT · p. 28

B.1 AGENT PROMPTS AND TOOL INTERFACE

The benchmark keeps a fixed system prompt and varies task instructions across tracks. Agents use a sandboxed tool workspace and are instructed to inspect rendered output before finishing.

B.1 AGENT PROMPTS AND TOOL INTERFACE

  • Agents are instructed to view their rendered chart and check it against the task before finishing, since clean script execution alone does not ensure a correct chart.The workspace is sandboxed, contains only local files, and has no network access or package installation.Source: §B.1 · p. 29
  • Track B tells agents to find and repair flaws while changing only what is wrong; Track C uses the same instruction for flawed and correct charts without revealing their status.For a faithful Track-C chart, the agent should leave the code unchanged, run it once, and finish.Source: §B.1 · pp. 29–30
  • Track A appends diagnosis text to Track B’s instruction, naming either one flaw or k flaws and directing repair of exactly those.The bracketed diagnosis fields use category descriptions from Table 12.Source: §B.1 · p. 29
  • The workspace provides Python execution and file operations, with image viewing and a finish tool; the persistent kernel retains variables, imports, and open figures.The protocol specifies fixed tool schemas and task prompts, identified by checksums.Source: §B.1 · pp. 29–30

C USAGE OF LLMS

LLMs polish writing and assist dataset generation; a Mistral model generates flawed data.

C USAGE OF LLMS

  • LLMs polish writing and assist dataset generation, with a Mistral model used to generate flawed data.Further details are in Appendix A.Source: C USAGE OF LLMS · p. 30
Loading 2610.02399v1…