PaperlayerView exact revision on arXiv

Source-linked AI summary

QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

Shuxiang Cao, Zijian Zhang, Abhishek Agarwal, Grace Bratrud, Niyaz R. Beysengulov, Daniel C. Cole, Alejandro Gómez Frieiro, Elena O. Glen, Hao Hsu, Gang Huang, Raymond Jow, Greshma Shaji, Tom Lubowe, Ligeng Zhu, Luis Mantilla Calderón, Nicola Pancotti, Joel Pendleton, Brandon Severin, Charles Etienne Staub, Sara Sussman, Antti Vepsäläinen, Neel Rajeshbhai Vora, Yilun Xu, Varinia Bernales, Daniel Bowring, Elica Kyoseva, Ivan Rungger, Giulia Semeghini, Sam Stanwyck, Timothy Costa, Alán Aspuru-Guzik, Krysta Svore

arXiv:2604.25884v1quant-phcs.CV

TL;DR

Quantum calibration requires interpreting experimental plots, but systematic evidence about VLM performance in this domain has been lacking. QCalEval benchmarks 18 VLMs across zero-shot and in-context settings, finding that frontier models improve with demonstrations while many open-weight models degrade, and that current models still struggle to connect visual features with operational outcomes.

  • Problem

    No systematic evaluation existed for how well VLMs interpret quantum calibration plots, despite their importance for calibration workflows.

  • Method

    QCalEval evaluates 18 VLMs on 243 samples spanning 87 scenario types, 22 experiment families, six question types, and both zero-shot and in-context learning settings.

  • Results

    72.3 mean score is achieved by the best general-purpose zero-shot model, while frontier closed models and Gemma improve substantially with labeled examples and many open-weight models degrade.

  • Takeaways & Limitations

    The benchmark exposes a capability gap between identifying visual features and mapping them to operational calibration outcomes, with in-context learning helping frontier models but not reliably helping open-weight models.

  • Takeaways & Limitations

    Per-family sample sizes range from 2–21 samples, limiting the statistical power of fine-grained conclusions about individual experiments.

Abstract

from arXiv · show

Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-language models (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score.

1 Introduction

Quantum calibration produces complex, drifting data that must be interpreted visually for increasingly automated workflows, yet VLM performance on these plots had not been systematically evaluated. QCalEval establishes this evaluation and shows substantial differences across models, prompting conditions, and fine-tuning strategies.

  • Motivation: Calibration targets drift with environmental fluctuations and hardware instabilities, while scaling creates combinatorial experiment dependencies.Each qubit requires dozens of characterization experiments, and one calibration result can invalidate others.
  • Motivation: Calibration plots serve as a platform-independent visual interface for interpreting experimental results and guiding calibration agents.VLMs complement parametric fitting because fitting can fail silently when its assumed model is violated.
  • Benchmark and findings: QCalEval evaluates 243 samples across 87 scenario types and 22 experiment families spanning superconducting qubits and neutral atoms.The benchmark uses six question types under zero-shot and in-context learning conditions.
  • Benchmark and findings: 72.3 was the best general-purpose zero-shot mean score, establishing a challenging baseline for quantum calibration plot understanding.The benchmark evaluates 18 VLMs.
  • Benchmark and findings: Frontier closed models and Gemma improve under multimodal in-context learning, whereas many open-weight models degrade with multi-image prompts.Improvements reach up to +29 scores on calibration diagnosis.
  • Benchmark and findings: SFT improves zero-shot performance but does not close the multimodal in-context learning gap.The ablation evaluates five 9-billion-parameter Qwen3.5 recipes; its strongest sequential curriculum is ICL→zero-shot.
  • Benchmark and findings: NVIDIA Ising Calibration 1 is released as an open-weight 35B mixture-of-experts reference model trained with the strongest sequential curriculum.The curriculum uses two-phase SFT: ICL followed by zero-shot training.

2 Related Work

Prior work covers multimodal adaptation, chart reasoning, quantum-calibration automation, and scientific agents, but these areas do not benchmark fine-grained operational diagnosis of quantum calibration plots. QCalEval targets that missing evaluation setting.

  • Vision-language models: VLM research combines pretrained vision encoders and language models, while multimodal few-shot systems enable image-text adaptation without guaranteeing robust in-context learning.The related work discusses CLIP, BLIP-2, Flamingo, and OpenFlamingo as representative approaches.
  • Scientific figures: Chart benchmarks expose needs for OCR, numerical grounding, and structural relations, but standard chart systems target extraction and question answering rather than quantum diagnostic reasoning.The cited chart resources focus mainly on bar, line, and pie charts.
  • Quantum calibration: Quantum-calibration automation uses agents to select experiments, interpret results, and adapt workflows, making plot interpretation an operational bottleneck.Calibration plots must indicate success, failure causes, and possible next actions.
  • Scientific agents: Scientific instrument agents orchestrate tools and workflows across domains, but they do not benchmark fine-grained visual diagnosis of quantum calibration plots.This gap motivates evaluating VLMs on actionable calibration artifacts.

3 QCalEval Benchmark

QCalEval decomposes quantum calibration-plot understanding into six question types and evaluates them across heterogeneous experiment families using zero-shot and demonstration-based settings. The design separates visual perception, scientific interpretation, fit assessment, parameter extraction, and actionable diagnosis.

  • Evaluation design: QCalEval evaluates six question types under zero-shot and in-context learning conditions.Zero-shot uses a query plot and textual background; in-context learning adds labeled examples from the same experiment family.
  • Benchmark scope: The benchmark spans 22 experiment families, mainly superconducting qubits and neutral atoms, with 1D traces, 2D maps, histograms, and image-like measurements.It includes both simulated and real-hardware data.
  • Benchmark scope: Each experiment family contains multiple scenario types and samples with plot images, scenario labels, and ground-truth answers for all six question types.Scenario types typically represent distinct success or failure modes.
  • Task taxonomy: The taxonomy isolates failures across plot description, coarse outcome classification, scientific significance, fit reliability, parameter extraction, and calibration diagnosis.Q1 tests visual grounding, while Q2–Q6 progressively assess domain reasoning and operational decisions.
  • Task taxonomy: Q6 assigns a family-specific status code and may require a corrective range or suggested action.Unlike Q2’s shared coarse label, Q6 is an actionable diagnosis tailored to each experiment family.
  • Evaluation design: In-context learning is restricted to Q3, Q5, and Q6, where demonstrations can transfer reasoning patterns, output schemas, or family-specific label vocabularies.Scenario types with only one sample are excluded because they cannot provide a separate support example.

4 Results and Analysis

QCalEval evaluates 18 VLMs across six question types under zero-shot and in-context settings, revealing strong visual-feature detection but persistent domain-interpretation and fit-judgment gaps. Demonstrations help frontier models and Gemma, while multi-image prompts harm several open-weight models.

  • Evaluation setup: QCalEval evaluates 18 VLMs across six question types, reporting zero-shot and multimodal in-context learning results.Q1, Q2, and Q4 are evaluated zero-shot; Q3, Q5, and Q6 are evaluated under both settings.
  • Zero-shot performance: 72.3 is Gemini-3.1-Pro’s best-base-model mean score, while Ising-Cal-1 reaches 74.7 and leads outcome classification, fit assessment, and calibration diagnosis.Gemini-3.1-Pro leads parameter extraction with Q5: 71.5; Ising-Cal-1 leads Q2: 67.1, Q4: 90.5, and Q6: 75.3.
  • Zero-shot performance: Q4 fit assessment exposes false confidence and no-fit blindness, with unreliable fits called “Reliable” from 6% to 74% and no-fit cases mislabeled “Unreliable” from 0% to 91%.Models that learn to output “No fit” at the 31.7% ground-truth rate perform better than models that rarely predict it.
  • In-context learning: +28.9 points is Claude Opus 4.6’s Q6 gain with demonstrations, compared with +18.6 for Gemini-3.1-Pro and +23.9 for Gemma-4-31B-IT.All seven closed-source models improve on every axis, while Gemma-4-31B-IT is the only benchmarked open model with comparable gains.
  • In-context learning: Qwen3.5-9B and Qwen3.5-35B-A3B lose 18.4 and 16.7 Q6 points, respectively, under multi-image demonstrations, whereas one-shot Q5 extraction improves for nearly all models.N-way scaling shows frontier models improve with more demonstrations, while Qwen3.5, MiniCPM-o, and Kimi-VL peak at one shot and then degrade.
  • Failure patterns: Optimistic bias accounts for 85% of Q6 errors, and “No fit” blindness affects 38% of samples in the qualitative failure analysis.Visual-similarity confusion is another recurring failure mode when scenarios differ in subtle features such as oscillation frequency.

5 SFT Ablation and Case Study

The 9-billion-parameter ablation shows that supervised fine-tuning substantially improves zero-shot calibration-plot understanding, but sequential training and multimodal in-context learning remain difficult. Guided by the strongest curriculum, the paper releases Ising-Cal-1 as a stronger single-plot reference model rather than a solution to the MM-ICL gap.

  • Ablation study: The ablation trains Qwen3.5 9B models on synthetic QCalEval data using a train/test split by experiment family and evaluates five SFT recipes.The datasets contain 25.8K zero-shot QA pairs and 12.9K ICL QA pairs.
  • Ablation study: Zero-shot SFT raises Q6 from 61.1 to 70.6, while sequential curricula raise Q4 from 28.6 to about 60 on unseen experiment families.The reported gains are +9.5 points for Q6 and roughly 31 points for Q4.
  • Ablation study: The best ICL result for Q3 is 24.1, below the base score of 27.1, and no SFT recipe improves free-text scientific reasoning under in-context learning.Different recipes improve different classification axes, but Q5 and Q6 gains remain modest and inconsistent.
  • Ablation study: ICL→zero-shot is the strongest overall sequential curriculum in the train/test-family ablation.The authors hypothesize that zero-shot-first training teaches reliance on the query image and makes later ICL training less effective.
  • Case study: Ising-Cal-1 is an open-weight Qwen3.5-35B-A3B MoE model trained with the ICL→zero-shot curriculum identified in the ablation.The model is released as a reference case study based on the ablation findings.
  • Case study: Ising-Cal-1 improves zero-shot plot understanding over its base model but retains the base model’s multi-demonstration failure pattern, so it is not a solution to the MM-ICL gap.Its clearest gains are in diagnosis, scientific interpretation, and fit-related judgments.

6 Limitations

QCalEval’s coverage is broad but uneven: its 243 samples span 22 experiment families, with relatively few samples per family limiting fine-grained conclusions. The authors identify expansion to additional experiments, platforms, and modalities as future work.

  • Dataset size and coverage: QCalEval contains 243 samples across 22 experiment families, mainly covering superconducting qubits and neutral atoms, with 186 simulated and 57 hardware samples.Future work can expand coverage to more experiment types, hardware platforms, and modalities.
  • Dataset size and coverage: Per-family sample sizes of 2–21 limit the statistical power for fine-grained conclusions about individual experiments.This limitation concerns the benchmark’s ability to support detailed per-experiment comparisons.

7 Conclusion

QCalEval establishes a benchmark for VLM understanding of quantum calibration plots across zero-shot and in-context settings, finding strong visual detection but weaker domain interpretation, fit judgment, and multi-image reasoning. The paper combines broad evaluation with controlled safeguards, human-AI ground-truth construction, and an SFT ablation, while identifying persistent MM-ICL limitations.

  • 7 Conclusion: QCalEval evaluates 18 models across six question types under zero-shot and in-context learning settings.The benchmark covers 22 experiment families, primarily superconducting qubits and neutral atoms.
  • 7 Conclusion: Models detect visual features at Q1 scores of 65–91% but perform worse when mapping them to outcomes at Q2 scores of 32–67% and Q6 scores of 37–75%.The conclusion identifies systematic optimistic bias toward predicting success.
  • 7 Conclusion: Sequential curricula raise Q4 from 28.6 to about 60 and Q6 from 61.1 to 70.6 in zero-shot evaluation, but no configuration closes the free-text ICL gap on Q3.Zero-shot SFT degrades in-context learning performance overall.
  • Benchmark scope: The benchmark contains 243 samples spanning 87 scenario types, with 309 unique images and six question-answer pairs per sample.This yields 1,458 total QA pairs for single-turn evaluation.
  • Evaluation design: In-context evaluation excludes seven singleton scenario types and covers Q3, Q5, and Q6, where demonstrations can transfer reasoning patterns, schemas, or label vocabularies.This reduces the in-context evaluation set to 236 samples.
  • Ground-truth construction: Ground truth combines expert scenario annotations, independent GPT-5.4 and Gemini-3.1-Pro expansions, cross-validation, rubric generation, and human verification.Experts review all flagged cases and a random sample of agreeing cases.
  • Inference protocol: Inference uses independent single-turn requests with greedy decoding at temperature 0 and a maximum output budget of 16,384 tokens.Zero-shot requests contain the query plot and textual background; ICL requests prepend demonstration images and labels.

B.2 Scoring Methods

QCalEval combines deterministic and judge-based scoring across six calibration-plot question types, with separate zero-shot and multimodal in-context prompt protocols. Evaluation uses fixed decoding settings and released prompt templates.

  • Scoring axes: Q1 combines exact programmatic checks with LLM matching of human-verified visual key points.The programmatic component checks plot type and axis scales; judges assess 3–5 visual elements.
  • Scoring axes: Q2 uses 4-way exact-match outcome classification, while Q4 uses 3-way exact-match fit assessment.Q2 labels outcomes as Expected behavior, Suboptimal parameters, Anomalous behavior, or Apparatus issue; Q4 labels fits Reliable, Unreliable, or No fit.
  • Scoring axes: Q3 averages two LLM judges’ scores on three scientific key points, producing a 0–100 significance-analysis score.Each key point receives 0, 0.5, or 1 for missing, partial, or full coverage.
  • Scoring axes: Q5 scores structured parameter extraction field by field using type-specific tolerances, while failed JSON parses score 0.The final Q5 score averages across fields.
  • Scoring axes: Q6 performs multi-way exact-match status classification after normalizing whitespace, capitalization, and common synonyms.Each experiment family defines 2–7 possible status labels.
  • Evaluation protocol: All models use deterministic decoding at temperature 0 with a 16,384-token output limit, while prompt strings and evaluation settings are released.Images are transported as base64 PNG data URLs, and multi-image prompts place support examples before the query.
  • Prompt protocols: Zero-shot and in-context evaluation use independent user turns, with ICL support images and labels interleaved before the query image.Q3 uses one example per scenario type, Q5 uses one sibling example, and Q6 uses one labeled example per scenario type.

D.2 Confusion Analysis

Confusion analysis reveals a systematic optimistic bias: models often collapse calibration failures into benign labels, especially SUCCESS or Expected behavior. The hardest cases are failure modes with subtle visual signatures.

  • Q2 outcome classification: Models frequently misclassify non-Expected outcomes as Expected behavior or Suboptimal parameters rather than severe failure labels.The asymmetry is stronger for failures than for successes: clear successes are rarely flipped into severe categories.
  • Q6 diagnosis classification: Models overpredict SUCCESS on ambiguous Q6 plots, particularly in readout, Ramsey, and spectroscopy families.The recurring error reflects difficulty distinguishing subtle failure signatures from successful calibrations.
  • Universally misclassified scenarios: 24 of 87 scenario types score below 10% for every base model on Q2, and all are failure modes misclassified as Expected behavior.The universally difficult scenarios are grouped by experiment family in Table 9.
  • Interpretation: The confusion pattern is a domain-wide limitation rather than a weakness confined to one model family.No current VLM reliably separates the listed calibration failures from expected behavior under the coarse four-way taxonomy.

E Qualitative Analysis: Universally Misclassified Failure Modes

Case studies show that models can describe visible structure yet fail at domain-specific calibration judgments. Recurring errors include treating noise, failed excitation, beating, or bad fits as successful or quantitatively usable results.

  • Cross-case limitation: Non-deterministic inference backends can change outputs across runs, although programmatic scores remain stable for most repeated evaluations.Individual response wording may vary despite stable Q2, Q4, Q5, and Q6 scores.
  • DRAG sweep: On the DRAG sweep, GPT-5.4 scores 32.2/100 while weak fits and scattered data expose failures across classification, reasoning, extraction, and diagnosis.The model correctly identifies the scatter plot but treats a physically meaningless zero-crossing as potentially informative.
  • Single-shot readout: The model extracts plausible cluster centers and recognizes overlap, yet declares SUCCESS when identical population ratios indicate that excitation failed.This case scores 61.1/100 and passes Q1, Q3, Q4, and Q5 but fails Q2 and Q6.
  • Ramsey frequency calibration: The model correctly labels the Ramsey fit Unreliable but does not identify beating as the physical cause of the mismatch.This separates fit-quality assessment from domain-specific diagnosis.
  • Ramsey frequency calibration: For Ramsey beating, the model reads fitted values literally and predicts SUCCESS despite amplitude modulation making the single-frequency fit unreliable.Gemini-3.1-Pro scores 40.3/100 on this case.
  • Coupler flux spectroscopy: In coupler spectroscopy, clean heatmaps and visible branches are mistaken for successful calibration even though the overlaid fit systematically misses the measured dispersion.GPT-5.4 scores 33.3/100 and fails Q2, Q3, Q4, and Q6.

G.3 Zero-Shot SFT Results

The 9B SFT ablation shows substantial gains in zero-shot transfer, with sequential training making recipe order important. These gains do not resolve the difficulty of multimodal in-context learning, especially free-text scientific reasoning.

  • Zero-shot results: Zero-shot-only SFT raises Q6 from 61.1 to 70.6 and reaches a 61.6 average on unseen experiment families.The result is reported as the strongest single-format recipe in the ablation.
  • Zero-shot results: Sequential curricula raise Q4 from 28.6 to about 60, with ICL→zero-shot reaching a 62.2 average and +8.6 over base.ICL→zero-shot is the strongest sequential curriculum in the train/test-family ablation.
  • In-context results: After SFT, ICL gains remain modest and inconsistent: the best Q5 score is 84.3, the best Q6 score is 42.1, and no recipe improves Q3.The best Q3 result is 24.1 with ICL→ZS, below the base score of 27.1.
  • Reference model: The full NVIDIA Ising Calibration 1 recipe uses Qwen3.5-35B-A3B and expands training data across all 22 experiment families.The generated data contains 48.7K zero-shot QA pairs and 23.8K ICL-formatted QA pairs.

H.2 Training Recipe

The training recipe uses two sequential supervised fine-tuning phases: first on ICL-formatted demonstrations, then on augmented zero-shot examples. Optimization uses full-parameter SFT with a frozen vision tower, and the resulting open weights are available from NVIDIA.

  • Training phases: Phase 1 trains on 23.8K ICL-formatted QA pairs to process multi-image demonstrations and relate labeled examples to query plots.Training uses a learning rate of 10^-5 for one epoch.
  • Training phases: Phase 2 continues training on 48.7K LLM-augmented zero-shot QA pairs to strengthen single-plot understanding while preserving ICL capabilities.This phase uses a learning rate of 5 × 10^-6 for one epoch.
  • Optimization: Both phases use AdamW with cosine learning-rate decay, 3% linear warmup, BF16 precision, and an effective batch size of 128.The setup performs full-parameter SFT with a frozen vision tower and zero weight decay.
  • Availability: The open-weight model is identified as Ising-Calibration-1-35B-A3B and is available through NVIDIA's Hugging Face repository.The stated availability link is https://huggingface.co/nvidia/.
Loading 2604.25884v1…