PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving VLMs on Scientific Chart Analysis via Autonomous Evolution

Abhinav Sudhakar Dubey, Vidushi Goel, Kiran Rana, Shikhar Mathur, Vaggos Chatziafratis

arXiv:2610.35077v1cs.CVcs.CLcs.LG

TL;DR

Open VLMs often struggle with scientific-chart questions, raising the question of how much accuracy can be recovered without changing the model. The paper searches over inference programs around frozen models, and mean descriptive exact-match accuracy rises from 16.6% to 41.3% across 12 model-benchmark pairs.

  • Problem

    Open VLMs often struggle with scientific-chart questions, and the paper asks how much accuracy can be recovered without changing the model.

    Source: §1 · pp. 1–2

  • Method

    The paper evolves inference programs around frozen VLMs by using an LLM to propose edits from concrete failures and selecting the final program on a disjoint holdout.

    Source: §1.1 · p. 2, §3.1 · p. 5

  • Results

    16.6% to 41.3%: mean descriptive exact-match accuracy rose across all 12 model-benchmark pairs, with a largest gain of 55.2 percentage points.

    Source: §4.1 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    Inference-program evolution can improve chart-question accuracy for frozen VLMs without retraining them.

    Source: §1.1 · p. 3, §5 · p. 9

    The model and decoding mode remain fixed, so measured accuracy changes are attributable to inference-code evolution within that setting.

    Source: §1.1 · p. 3

Abstract

from arXiv · show

Understanding and operating with scientific charts is a standard approach in data analysis, and while humans are good at interpreting such charts, open vision-language models (VLMs) are known to perform poorly when it comes to question answering based on such data types. Motivated by an initial observation that many of their errors are due to inexact answering, we raise the following natural question: how much accuracy can we recover without changing the model at all? In this paper, we study this question by searching over inference programs and the code around a "frozen" version of the model. Using ideas from autonomous evolution (famously used in AlphaEvolve), we first start from a baseline inference program, and in each round we show an LLM judge the code together with a few failures on training questions, and ask for an improving evolutionary move. As usual in autonomous evolution approaches, the candidate programs are kept in a small pool ranked by training accuracy, and the next edit is performed on top-scorers from this pool. We run this approach on three VLMs and four popular benchmarks on scientific charts, and we measure the fraction of descriptive questions whose answer exactly matches the ground truth. The accuracy improves in all experimental settings. On average, it rises from 16.6% with the starting program to 41.3% with the evolved one, and the largest single increase is 55.2 percentage points. As a side benefit, our evolved program is 3.8x faster than the starting program, because short answers are easier to grade and faster to generate. Our results demonstrate that a large part of the observed errors arise due to the inference pipeline, and that optimizing it can improve the "frozen" VLM without the need to retrain it.

1 INTRODUCTION

Open VLMs often struggle with scientific-chart questions, and exact-match grading can mark verbose or poorly formatted answers wrong. The paper searches over inference programs around frozen models and reports improved accuracy across all tested model-benchmark pairs without retraining.

1 INTRODUCTION

  • VLMs often struggle with scientific plots, and benchmark evidence reports a human-accuracy gap that is relatively largest for small open models.Descriptive chart questions include reading axis ticks, identifying series gaps, and extracting bar values; the cited evidence says the gap widens on reasoning questions.Source: §1 · pp. 1–2

1.1 OUR CONTRIBUTION

  • The paper optimizes the inference program around a frozen model, including prompts, response budgets, and answer post-processing, rather than changing model weights.The authors contrast this broader search space with prompt-only optimization, which cannot shrink verbose answers or remove trailing percent signs.Source: §1 · p. 2, §1.1 · p. 2
  • An LLM proposes code edits using concrete failures; candidates are ranked by training accuracy, while the final program is selected on a disjoint holdout.A static validator rejects malformed candidates before GPU evaluation, and the model and decoding mode stay fixed during the search.Source: §1.1 · pp. 2–3, Figure 1 · p. 3
  • Across three differently sized open VLMs and four chart benchmarks, the fastest evolved Qwen3-VL-2B/ChartQA variant was 3.8 times faster than the unevolved model; it matched hand-tuned accuracy and beat its combined accuracy-throughput score.Source: §1.1 · pp. 3–4
  • 16.6% to 41.3%: mean descriptive exact-match accuracy rose across the 12 model-benchmark pairs, and every pair improved, with a largest gain of 55.2 percentage points.The largest gain was Qwen3-VL-2B on ChartQA, from 0.5% to 55.7%.Source: §1.1 · p. 3, Table 1 · p. 4, Figure 2 · p. 4

2 RELATED WORK

Prior chart-QA work changes model weights, prompts, or inference programs, while prompt-only and single-trajectory approaches constrain the search space. This work searches executable inference code, complementing serving-system optimizations.

Chart understanding

  • Chart-QA methods have modified model weights or converted charts to intermediate representations, whereas this approach leaves weights untouched and edits inference code.The code includes the prompt, decoding budget, and output parser; this also makes the approach applicable to frozen models.Source: §2 · pp. 4–5

Inference-time and prompt optimization

  • Prompt optimizers search text strings, while this work searches executable programs that can also change decoding parameters, token budgets, and output post-processing.Other application-layer approaches optimize declarative modules, text-valued parameters, or execution.Source: §2 · p. 5

Evolutionary and self-improving code search

  • LLM-driven code evolution builds on population-based search and concrete-error feedback, enabling structural changes that prompt-only optimization cannot express.The paper describes this as the first application of LLM-driven code evolution to VLM inference pipelines.Source: §2 · p. 5

LLM serving systems

  • Inference-program optimization complements serving-system work by changing model instructions, generation limits, and output parsing rather than serving infrastructure.The resulting programs can be deployed on existing serving systems.Source: §2 · p. 5

3 AUTONOMOUS EVOLUTION OF VLMS

The paper treats the executable code around a fixed VLM as the search target for chart question answering. An LLM proposes edits to candidate programs, which are validated and evaluated before selection favors training accuracy for parent choice and holdout accuracy for the saved result.

3.1 PROBLEM SETTING AND KEY IDEA

  • An inference program maps a chart, question, and question type to a graded string, and the search optimizes its exact-match accuracy on unseen questions.The VLM itself is fixed; the program may call it zero or more times.Source: §3.1 · p. 5

3.2 EVOLUTION SEARCH OVER CANDIDATE PROGRAMS

  • Only a marked code block in one inference function is editable; prompt, per-question hints, token budgets, and output normalization may change, while model identity and greedy decoding stay fixed.Fixed decoding makes output changes reproducible and attributable to code edits rather than sampling noise.Source: §3.2 · p. 6

3.3 THE EVOLUTION LOOP

  • Each generation uses an LLM to make a targeted edit informed by six sampled failures from different question types; every five generations, crossover combines strong parents.The loop also loads train and holdout windows, validates candidates before GPU evaluation, and writes the best-by-holdout candidate to disk.Source: §3.3 · p. 6, Figure 3 · p. 6, §3.1 · p. 6
  • A validator performs five static checks before GPU evaluation, while the evaluator uses descriptive exact match as the sole selection signal and tracks reasoning exact match, latency, and errors as non-fitness quantities.Reasoning exact match is treated only as a proxy because the official reasoning metric is open-ended.Source: §3.3 · p. 7, Table 2 · p. 7
  • The population holds at most ten programs ranked by training accuracy, but the saved candidate is the one with best holdout accuracy to reduce overfitting.Parent selection balances exploitation and exploration through tournaments and occasional draws from the weaker half.Source: §3.3 · p. 7

4 EXPERIMENTS

The experiments evaluate evolved inference programs for three open VLMs across four chart benchmarks, using descriptive exact-match accuracy as the primary metric. The search improves all twelve model-benchmark runs, with a detailed Qwen3-VL-2B/ChartQA comparison showing gains in both accuracy and throughput.

4 EXPERIMENTS

  • The evaluation uses Qwen3-VL-2B-Instruct, InternVL3, and SmolVLM2 across CharXiv, ChartQA, EvoChartQA, and PlotQA, with descriptive training and disjoint holdout windows.Reasoning questions are evaluated but not used for selection; final accuracy is computed over each benchmark's full descriptive portion.Source: §4 · pp. 7–8
  • The primary comparison is against a seed program without hints or post-processing; Qwen3-VL-2B/ChartQA also includes a hand-tuned baseline and three deployment variants.Prompt-only optimizers were not run as separate baselines.Source: §4 · p. 8

4.1 RESULTS

  • 16.6% to 41.3% mean descriptive exact-match accuracy across twelve runs, with every configuration improving and a mean gain of 24.7 points.The largest gain was 55.2 points; gains were larger when correct answers from the seed were ungradable and smaller when outputs were already cleanly formatted.Source: §4.1 · p. 8
  • Baseline examples show that gains involve answer formatting and extraction as well as visual understanding: evolved programs return standardized strings where seed outputs are often long and ungradable.Table 3 examples span Qwen3-VL-2B, InternVL3, and SmolVLM2; wrong seed values are underlined.Source: §4.1 · p. 8, Table 3 · p. 8
  • On Qwen3-VL-2B/ChartQA, best_speed reaches 58.6% accuracy at 1.24 queries per second, versus the seed's 38.3% at 0.33 queries per second.The comparison covers 128 descriptive and 64 reasoning samples; best_speed is fastest and highest on the aggregate score, beating the hand-tuned baseline.Source: §4.1 · p. 8, Table 4 · p. 9
  • Evolution moves the Qwen3-VL-2B/ChartQA program up and to the right of the seed in accuracy and throughput, with best_speed dominating on both axes.The aggregate score ranks deployment variants; the search itself optimizes descriptive exact match alone.Source: §4.1 · p. 8, Figure 4 · p. 9

5 CONCLUSION

The paper concludes that evolving inference programs can improve scientific chart question answering without retraining VLMs. Across three models and four benchmarks, all twelve model-benchmark pairs improve, while the method is less effective for perceptual failures and absent responses.

5 CONCLUSION

  • Across all 12 model-benchmark pairs, mean descriptive exact-match accuracy rises from 16.6% to 41.3%, with a largest gain of 55.2 points.The fastest evolved Qwen3-VL-2B/ChartQA program is 3.8 times faster than the seed and more accurate than a hand-tuned baseline.Source: §5 · p. 9

AI USE STATEMENT

Generative AI tools helped implement and edit the work, with the mutation model producing candidate inference programs that were validated and evaluated. The authors state that they reviewed AI-assisted work and take responsibility for the final content.

AI USE STATEMENT

  • Gemini 2.5 Flash served as the mutation operator, writing candidate inference programs that were automatically validated and empirically evaluated.The authors say these programs are research artifacts of the work.Source: AI USE STATEMENT · p. 9
  • AI tools also assisted with readability edits, paper structure, reference formatting, and compliance checks; the authors reviewed the work and take responsibility for the final content.The statement excludes AI use for theoretical models, hypotheses, methodology design, and results interpretation.Source: AI USE STATEMENT · pp. 9–10

REPRODUCIBILITY STATEMENT

The reproducibility statement specifies deterministic decoding and points to the code, programs, and logs for all twelve runs.

REPRODUCIBILITY STATEMENT

  • Greedy decoding and a validator-fixed model id make saved-program reruns reproduce outputs exactly.The source code, seed and evolved programs, and evolution logs for all twelve runs are available in the project repository.Source: REPRODUCIBILITY STATEMENT · p. 10

A AGGREGATE SCORE FOR THE DEEP DIVE

Deployment variants are ranked with a harmonic-mean score combining descriptive accuracy and capped throughput, with a 30% accuracy floor; evolutionary search itself optimizes descriptive exact match.

A AGGREGATE SCORE FOR THE DEEP DIVE

B CODE STRUCTURE AND LOGGING

The implementation reuses an evolution engine across runs, with separate scripts for seeds, benchmark-specific evolution, evaluation, and final deployment variants.

B CODE STRUCTURE AND LOGGING

  • The engine is reused unchanged across runs and handles population management, parent selection, mutation and crossover, validation, and the generation loop.Source: B CODE STRUCTURE AND LOGGING · p. 12
  • Evaluation scripts load images and answers, run each candidate, and compute accuracy and timing, while evolved winners are selected on questions unseen during evolution.Source: B CODE STRUCTURE AND LOGGING · p. 12
  • Each generation logs parent, child train and holdout accuracy, and the current best, with JSON history recording event types, accuracies, and timing.Source: B CODE STRUCTURE AND LOGGING · p. 12
  • Figure 5 sorts all 12 runs by absolute descriptive exact-match gain; gains cluster by whether seed failures involved formatting or perception.Formatting failures show large recoverable gains, while perception failures leave smaller headroom.Source: Figure 5 · p. 13

C PER-RUN GAINS, SORTED

Figure 5 ranks the twelve runs by their absolute gain in descriptive exact match.

C PER-RUN GAINS, SORTED

D ENGINEERING FRICTIONS

Practical access, loading, and efficiency issues constrained which models and strategies could be evaluated in the unified inference setup.

D ENGINEERING FRICTIONS

  • Some gated checkpoints were delayed by access approvals, while Molmo, Pixtral, and Phi-3.5-Vision failed to load under the unified inference template.Source: D ENGINEERING FRICTIONS · p. 13
  • Qwen3-VL-2B request batching encountered a right-padding issue in Qwen2VLProcessor and capped accuracy at 29.7%.Source: D ENGINEERING FRICTIONS · p. 13
  • INT4 quantization achieved only 0.04 queries per second on a T4 GPU, roughly 75 times slower than float16, so it was not pursued.Source: D ENGINEERING FRICTIONS · p. 13
  • These issues were practical difficulties in building a uniform evaluation harness across heterogeneous VLMs, not algorithmic failures.Source: D ENGINEERING FRICTIONS · p. 13

Related papers

Loading 2610.35077v1…