PaperlayerView exact revision on arXiv

Source-linked AI summary

PlotChain: Deterministic Checkpointed Evaluation of Multimodal LLMs on Engineering Plot Reading

Mayank Ravishankara

arXiv:2602.13232v1cs.AIcs.SE

TL;DR

Engineering plot reading requires quantitative extraction and light reasoning that generic chart and captioning benchmarks do not specifically measure. PlotChain addresses this with a deterministic generator-based benchmark, exact ground truth, checkpoint diagnostics, and standardized tolerance-based evaluation. Across four MLLMs, frontier models cluster near 78–80% field-level pass rates while strict end-to-end completion exposes divergences and frequency-domain bottlenecks.

  • Problem

    Engineering plot reading requires extracting values from axes and curves and performing derived calculations, while existing evaluations often target generic chart QA, OCR, or captioning rather than engineering plot families.

  • Method

    PlotChain generates 15 engineering plot families with exact ground truth, adds cp_ checkpoint fields for intermediate reads, and evaluates strict JSON numeric outputs with deterministic decoding and per-field tolerances.

  • Results

    Frontier models cluster near 78–80% field-level pass rates, but diverge under strict end-to-end completion; derived frequency-response and spectrum tasks remain persistent bottlenecks.

  • Takeaways & Limitations

    Checkpoint metrics enable fine-grained profiling by distinguishing upstream perception or axis-calibration failures from downstream derivation and error-compounding failures.

Abstract

from arXiv · show

We present PlotChain, a deterministic, generator-based benchmark for evaluating multimodal large language models (MLLMs) on engineering plot reading-recovering quantitative values from classic plots (e.g., Bode/FFT, step response, stress-strain, pump curves) rather than OCR-only extraction or free-form captioning. PlotChain contains 15 plot families with 450 rendered plots (30 per family), where every item is produced from known parameters and paired with exact ground truth computed directly from the generating process. A central contribution is checkpoint-based diagnostic evaluation: in addition to final targets, each item includes intermediate 'cp_' fields that isolate sub-skills (e.g., reading cutoff frequency or peak magnitude) and enable failure localization within a plot family. We evaluate four state-of-the-art MLLMs under a standardized, deterministic protocol (temperature = 0 and a strict JSON-only numeric output schema) and score predictions using per-field tolerances designed to reflect human plot-reading precision. Under the 'plotread' tolerance policy, the top models achieve 80.42% (Gemini 2.5 Pro), 79.84% (GPT-4.1), and 78.21% (Claude Sonnet 4.5) overall field-level pass rates, while GPT-4o trails at 61.59%. Despite strong performance on many families, frequency-domain tasks remain brittle: bandpass response stays low (<= 23%), and FFT spectrum remains challenging. We release the generator, dataset, raw model outputs, scoring code, and manifests with checksums to support fully reproducible runs and retrospective rescoring under alternative tolerance policies.

I. INTRODUCTION

PlotChain addresses the need for engineering-plot benchmarks with exact ground truth and diagnostics that separate visual reading from downstream reasoning. It combines deterministic generation, checkpoint fields, standardized scoring, and reproducible artifacts.

  • Motivation: Existing chart benchmarks commonly emphasize generic chart QA, captioning, or heterogeneous modalities rather than canonical engineering plot families.The motivation is to target engineering-specific reading tasks and exact quantitative recovery.
  • Benchmark design: PlotChain renders 15 engineering plot families with 450 items from known parameters and computes exact ground truth deterministically.Items use controlled clean, moderate, and edge difficulty settings and require strict JSON numeric responses.
  • Diagnostic evaluation: Checkpoint fields isolate intermediate plot reads such as cutoff frequency, peak magnitude, intercept, slope region, or axis scale for failure localization.They distinguish successful intermediate reading from failure on derived quantities, which final-answer scores cannot reveal.
  • Reproducibility: Released generators, datasets, raw outputs, evaluation scripts, and checksumed manifests support exact reruns and retrospective rescoring.Alternative tolerance or parsing policies can be applied without re-querying model APIs.
  • Evaluation protocol: The protocol fixes prompting and decoding at temperature = 0, requires strict JSON numeric outputs, and scores predictions with human-aligned tolerances.This standardization supports automated comparison under realistic plot-reading precision.

A. Chart and Plot Question Answering Benchmarks

Prior chart benchmarks span synthetic QA, scientific plots, real-world charts, derendering, summarization, and general multimodal reasoning. Together, they motivate evaluations that separate visual extraction from reasoning and extend beyond final-answer scores.

  • Synthetic chart QA: FigureQA and DVQA use programmatically generated charts for controlled evaluation but emphasize restricted chart styles, labels, or answer forms.DVQA also highlights brittleness to appearance variation and chart-specific answers.
  • Plot and chart QA: PlotQA targets scientific plots with real-valued answers, while ChartQA combines visual and logical reasoning over real-world charts.These benchmarks demonstrate that chart understanding is a compositional perception–reasoning problem rather than pure OCR.
  • Structured chart representations: ChartOCR, ChartReader, Pix2Struct, UniChart, and DePlot investigate chart derendering, unified comprehension, structured extraction, or table-based reasoning.These approaches treat conversion to structured representations as a foundation for downstream reasoning and accessibility.
  • Chart summarization: Chart-to-text benchmarks prioritize narrative faithfulness and content selection, whereas plot-reading evaluation prioritizes numeric extraction under realistic tolerances.The distinction separates accessibility-oriented summarization from quantitative engineering reading.
  • Evaluation gap: Broader multimodal benchmarks include charts but are not designed specifically for quantitative engineering plots or checkpoint-based diagnostic analysis.Recent chart evaluations therefore reinforce the need to separate visual extraction from reasoning and characterize failures beyond one final score.

E. Benchmarking Methodology and Reproducibility

PlotChain evaluates numeric plot reading through fixed schemas, deterministic decoding, and tolerance-based scoring across canonical engineering plot families. Its checkpoint fields expose intermediate reads, while artifact release supports reproducible analysis.

  • Reproducibility: Deterministic generation, fixed decoding, checksumed artifacts, and stored raw outputs enable reruns and rescoring under revised tolerance policies.The release follows reproducibility practices for multimodal evaluation.
  • Task definition: Models receive canonical engineering plot images and questions, then return one JSON object containing numeric values or null.The task requires interpreting axes, including log scales, reading curve values, and computing derived quantities such as cutoff frequency or settling time.
  • Plot families: The benchmark covers canonical families including Bode magnitude and phase, bandpass response, FFT spectrum, spectrograms, step response, IV curves, stress–strain, pump curves, and torque–speed plots.Table I pairs each family with representative target variables; checkpoint fields are omitted from the table for brevity.
  • Target fields: Final fields are headline numeric answers, while cp_ fields are intermediate reads used to localize failures.Examples include cutoff frequency, peak magnitude, and other plot-reading sub-skills.

B. Plot Families

PlotChain organizes 15 engineering plot families across controls, signals, circuits, mechanical systems, fluids, fatigue, and materials, with representative numeric outputs for each family.

  • Coverage: PlotChain contains 15 plot families with 30 items per family, totaling 450 items.The families reflect common analysis patterns across controls, signals, circuits, mechanical systems, and materials.

C. Deterministic Generation and Ground Truth

PlotChain uses a frozen generator to deterministically produce plot parameters, images, and exact labels, with controlled rounding and human-readable difficulty variation. The release also includes item metadata and validation artifacts for reproduction.

  • Deterministic generation: A frozen generator maps (master_seed, family, index) to plot parameters, rendered PNG images, and ground-truth dictionaries computed from analytic or numeric baselines.The corresponding parameterization and baseline are released for every numeric target.
  • Ground truth: Ground-truth values are quantized with family- and field-specific decimal precision to reflect realistic plot-reading precision.Examples include integer Hz cutoffs and 0.1 dB gains.
  • Difficulty design: Each family uses an approximately 40% clean, 30% moderate, and 30% edge difficulty mixture while preserving manual human readability.Edge cases weaken visual aids such as gridlines or axis cues without becoming pathological.
  • Release artifacts: The distribution includes JSONL items, per-family PNG directories, and generator-produced validation CSVs.These artifacts support row-level and summary-level sanity checks.
  • Item structure: Each JSONL item stores identifiers, questions, final and checkpoint ground truth, generating parameters, and deterministic generation metadata.Metadata includes the seed, difficulty tag, and explicit final-versus-checkpoint field lists.

IV. EXPERIMENTAL SETUP AND EVALUATION

The evaluation compares four multimodal LLMs using a shared, deterministic configuration and fixed run metadata. Models receive standardized item field schemas and return numeric JSON outputs.

  • Model suite: Four multimodal LLMs are evaluated: GPT-4.1, GPT-4o, Claude Sonnet 4.5, and Gemini 2.5 Pro.The models span three providers.
  • Deterministic decoding: All evaluations use temperature = 0, a shared prompt template, and a fixed maximum output token budget.Run timestamps, model identifiers, and artifact checksums are recorded in the release manifest.
  • Output interface: Each item defines ordered final and cp_ checkpoint fields, and the evaluator requires one strict JSON object containing numeric values or null for those keys.Checkpoint fields are intended to diagnose failures between intermediate reads and derived quantities.

C. Robust Output Parsing

PlotChain combines tolerant response parsing with numeric scoring that separates field, item, model, and checkpoint outcomes. Its tolerances are designed around realistic human plot-reading precision.

  • Robust parsing: The evaluator accepts direct JSON, fenced JSON, or the first braced object, and sanitizes simple fractions and trailing commas before reparsing.If parsing still fails, all fields are scored as missing.
  • Numeric scoring: A field passes when either absolute error or relative error falls within its permitted tolerance.The errors are eabs = |p − g| and erel = |p − g| / max(|g|, 10^-12).
  • Tolerance policy: Tolerances are defined per family and field under the plotread policy to reflect axis resolution and visual affordances.This aligns scoring with realistic human plot reading.
  • Metrics: The benchmark reports field pass rate, final-field item pass, checkpoint item pass, model final pass rate, and model checkpoint pass rate.It also reports mean absolute and relative errors and preserves raw outputs for rescoring.
  • Reproducibility: Raw model outputs and per-call latency are retained so metrics can be recomputed under alternative tolerance or parsing rules without new API calls.This supports retrospective evaluation changes.

F. Paired Significance Testing

Results are evaluated with paired item-level comparisons and multiple metrics, revealing strong overall performance for the leading models but substantial family-specific bottlenecks. Frequency-domain and derived-quantity tasks remain especially difficult.

  • Paired testing: Because every model sees the same 450 items, headline strict all-pass comparisons are paired and evaluated with McNemar’s exact test plus Holm correction.Paired bootstrap confidence intervals are also reported for strict all-pass differences.
  • Overall ranking: Gemini 2.5 Pro leads strict all-pass at 72.0%, followed by GPT-4.1 at 68.4%, Claude Sonnet 4.5 at 61.3%, and GPT-4o at 32.4%.Field-level pass rates are higher, with the top three clustered near 78–80%.
  • Family-level analysis: Family-level heatmaps aggregate final-field pass rates across all items in each plot family, complementing strict item-level completion.This reveals end-target recovery even when not every field in an item passes.
  • Family variation: Resistor I–V is near-solved at 100% for all models, whereas bandpass response reaches only 7–23% and FFT spectrum 25–50%.These results identify frequency-domain and derived-quantity families as persistent bottlenecks.
  • Checkpoint interpretation: Checkpoint all-pass is computed only for items containing checkpoint fields, separating checkpoint-based diagnostic outcomes from final-field results.Final and checkpoint metrics are therefore not identical populations in all cases.

C. Checkpoint diagnostics vs. final fields

Checkpoint fields separate intermediate plot reads from final derived quantities, enabling more localized diagnosis than final-field scores alone. Paired comparisons assess whether observed strict all-pass differences are statistically meaningful.

  • Checkpoint diagnostics: Checkpoint fields expose whether models can read intermediate cues but fail to complete multi-step derivations.The diagnostic design separates localized reads such as crossing frequencies and peak locations from final derived quantities.
  • Checkpoint diagnostics: GPT-4o and Claude Sonnet 4.5 tend to score relatively better on checkpoint fields than final fields.This pattern is consistent with extracting intermediate cues but failing to complete multi-step derivations reliably.
  • Paired comparisons: Gemini 2.5 Pro and GPT-4.1 are not statistically distinguishable on strict all-pass after Holm correction at α = 0.05.The paired analysis uses McNemar’s exact test over discordant item outcomes and corrects all pairwise comparisons.
  • Paired comparisons: GPT-4.1 and Gemini 2.5 Pro significantly outperform Claude Sonnet 4.5, while all three significantly outperform GPT-4o.These comparisons use the same fixed item set and Holm-corrected pairwise tests.

A. Frontier models cluster on field-level accuracy, but diverge on strict end-to-end completion

Frontier models often recover individual numeric targets, but performance diverges when every dependent final field must be correct simultaneously. Derived-quantity families expose compounding errors, while latency varies substantially across models.

  • Field-level accuracy: Gemini 2.5 Pro, GPT-4.1, and Claude Sonnet 4.5 cluster around ∼78–80% field-level pass rates.The field-level results indicate that these models often recover individual numeric targets under human-realistic tolerances.
  • Strict end-to-end completion: 72.0% and 68.4% are Gemini 2.5 Pro’s and GPT-4.1’s strict all-pass rates, respectively, when all final fields must be correct.The item-level metric exposes compounding errors that field-level scores can conceal.
  • Family-dependent bottlenecks: 0% strict all-pass across models occurs on BANDPASS RESPONSE, despite some intermediate values falling within tolerance.Dependent reads such as f1 and f2 can compound into invalid bandwidth or Q estimates.
  • Accuracy–latency tradeoff: 1.97 s, 20.66 s, 6.13 s, and 2.22 s are the reported latencies for GPT-4.1, Gemini 2.5 Pro, Claude Sonnet 4.5, and GPT-4o, respectively.The results present an accuracy–latency tradeoff relevant to throughput and cost constraints.

B. Tolerance dependence

PlotChain’s results depend on evaluation choices and deployment conditions, including tolerance policies, strict output compliance, model versions, and the synthetic scope of the benchmark. The released artifacts support auditing and rescoring, but do not remove these boundaries.

  • Tolerance dependence: Rankings can shift under different per-family/per-field tolerance policies, especially for borderline reads near resolution limits.The paper publishes the tolerance policy and raw outputs to enable rescoring without additional API calls.
  • Interface constraints: Strict numeric-JSON compliance can penalize otherwise-correct reasoning when models produce invalid JSON.Strict all-pass therefore includes adherence to the evaluation contract in addition to numeric correctness.
  • Model and platform variability: Results reflect specific model versions and run configurations, so provider-side updates may affect outcomes.Run dates, settings, and checksums are recorded to support reproduction under the same conditions when possible.
  • Task scope: PlotChain emphasizes numeric extraction and derived-quantity computation from canonical engineering plots rather than open-ended explanation or external-knowledge tasks.Those capabilities are described as complementary and suitable for evaluation with other benchmarks.
  • Benchmark scope: 15 canonical plot families and 450 items provide controlled difficulty and edge cases for reproducible capability profiling.The benchmark is generator-based and uses exact ground truth derived from known parameters.
  • Reproducibility and extensions: Released artifacts support transparent, repeatable benchmarking and rescoring under alternative tolerance policies without additional model queries.The paper positions the benchmark as a foundation for broader plot styles, real-world noise sources, and new diagnostic families.
Loading 2602.13232v1…