PaperlayerView exact revision on arXiv

Source-linked AI summary

QuPAINT: Physics-Aware Multimodal Reasoning for Quantum Material Characterization

Sankalp Pandey, Xuan-Bac Nguyen, Hoang-Quan Nguyen, Tim Faltermeier, Nicholas Borys, Hugh Churchill, Khoa Luu

arXiv:2609.12202v1cs.CVcs.AIcs.LG

TL;DR

Determining flake thickness from optical microscopy is difficult because layer count must be inferred from subtle optical contrast, and workflows vary across settings. QuPAINT combines synthetic data, image-grounded reasoning, and physics-informed attention; QuPAINT-8B achieves the strongest reported detection performance.

  • Problem

    Optical microscopy requires inferring flake thickness from subtle contrast, while workflows vary across materials, substrates, laboratories, and imaging conditions and transfer comparisons lack standardized protocols.

    Source: §1 · p. 1, §1.3 · p. 3

  • Method

    QuPAINT combines Synthia-generated microscopy images, verified image-specific QMat-Instruct reasoning, multimodal instruction tuning, and a Physics-Informed Attention prior.

    Source: §8 · pp. 14–15

  • Results

    QuPAINT-8B achieves the strongest performance across reported detection metrics, with AP 45.6 for general flake detection and 37.3 for mono-layer detection.

    Source: §7.2 · p. 12

  • Takeaways & Limitations

    Image-grounded reasoning improves confidence ranking across mono-layer and general detection without substantially changing which flakes are detected.

    Source: §7.3 · p. 14

    Synthetic images cannot capture every variation across microscopes, substrates, defects, and imaging conditions, so a domain gap may remain in new settings.

    Source: §8 · p. 15

Abstract

from arXiv · show

Characterizing two-dimensional (2D) quantum materials by optical microscopy requires localizing exfoliated flakes and determining their layer thickness from subtle optical contrast and interference color to select suitable flakes for device fabrication. However, models face synthetic-to-real domain shifts and variation across materials, substrates, laboratories, and imaging conditions. We present QuPAINT, a physics-aware multimodal framework for transferable quantum flake characterization. The Synthetic Materials Framework (Synthia) generates diverse synthetic microscopy images while preserving layer-dependent optical behavior. Using these images, we construct QMat-Instruct, a multimodal instruction dataset with image-specific reasoning traces generated from verified annotations and constrained to observable optical cues. QuPAINT integrates these signals through Physics-Informed Attention (PIA), which injects substrate-relative optical priors into the visual representation to support grounded multimodal reasoning. For evaluation, we introduce QF-Bench, to our knowledge, the largest real-world benchmark for this problem, spanning diverse microscopy and substrate conditions. Using its verified annotations, we study counting, visual grounding, reasoning quality, confidence calibration, and transfer to an unseen material. QuPAINT-8B substantially outperforms prior methods and establishes state-of-the-art performance for both general and monolayer flake detection. Additional experiments show that image-grounded supervision improves strict spatial grounding and confidence calibration while preserving robust general flake detection on the unseen material.

1 INTRODUCTION

Optical microscopy offers a scalable route to characterizing 2D flakes, but layer count is inferred from subtle, acquisition-sensitive optical cues, and AFM confirmation adds manual effort. QuPAINT addresses these challenges with synthetic data, image-grounded supervision, substrate-relative visual reasoning, and a benchmark spanning multiple materials and settings.

1 INTRODUCTION

  • Layer count is not directly visible: adjacent categories can look alike, while material and imaging conditions alter the weak contrast used to distinguish them.Finding flakes is only the first step; accurate thickness estimates matter for selecting flakes for device fabrication.Source: §1 · p. 1

1.1 Extension of the Conference Version

  • The journal extension replaces template-based explanations with image-specific reasoning traces grounded in verified labels and observable optical cues, and broadens evaluation beyond detection.The evaluation adds instruction following, grounding, reasoning quality, and transfer to a material excluded from training.Source: §1.1 · pp. 1–2

1.2 Why Conventional Vision Models Struggle?

  • Conventional visual cues often locate flakes without identifying their layers, because adjacent categories share geometry and differ mainly in weak optical changes.The paper motivates relative optical reasoning because acquisition changes can outweigh differences between neighboring layer categories, while verified mono- and few-layer examples are rare.Source: §1.2 · p. 2

1.3 Limitations of Prior Work

  • Prior systems cover limited materials and acquisition settings, models trained on one pipeline may not transfer reliably, and the field lacks standardized cross-domain evaluation.Different datasets and experimental configurations make direct comparisons difficult and leave transferability uncertain.Source: §1.3 · pp. 2–3

1.4 Physics-Aware Reasoning

  • QuPAINT frames characterization as multimodal reasoning that compares each flake with its surrounding substrate and returns locations, layer categories, and supporting rationale.Relative brightness, darkness, or color differences are used instead of relying only on absolute appearance.Source: §1.4 · p. 3

1.5 Contributions of this Work

  • Synthia renders realistic flakes using thin-film interference, CIE color representations, and substrate-specific calibration, then composites them onto real microscopy images to augment rare thin-layer examples.The framework avoids overlap with existing flakes.Source: §1.5 · p. 4
  • QMat-Instruct replaces fixed templates with image-specific reasoning traces from verified annotations, constrained to visible optical evidence and relative substrate differences.The supervision does not directly provide material identity or oxide thickness.Source: §1.5 · p. 4
  • PIA uses relative LAB contrast between image regions and substrate to emphasize faint mono- and few-layer flakes over regions with the strongest absolute contrast.This physics-informed attention is part of the QuPAINT multimodal framework.Source: §1.5 · p. 4

2 RELATED WORK

Prior 2D-material characterization spans dense prediction, classical methods, instrumentation, and transferable representations, but cross-material and synthetic-to-real transfer remain difficult. Multimodal models also struggle with subtle optical discrimination, motivating QuPAINT’s physics-informed approach.

2.1 Automated 2D Material Characterization

  • 2D-material characterization has progressed from dense prediction and classical methods to automated microscopy and transferable representations, but accuracy often fails to transfer across materials or from synthetic to measured flakes.Continual-learning methods address cross-material transfer, while adaptation anchored in optics can help synthetic-to-real transfer [23], [44].Source: §2.1 · pp. 4–5

2.2 Multimodal Large Language Models (MLLMs)

  • Multimodal models handle routine factual recall and extraction better than subtle perceptual discrimination or inference chained through physical constraints,.Layer counting is challenging because its evidence may be only a few percent of contrast, and vision-centric models struggle to generalize across domains.Source: §2.2 · p. 5
  • QuPAINT extends the conference framework with a physics-informed multimodal design linking observed appearance to the optical properties that produce it.The related work positions this design for layer counting, where optical evidence is subtle and domain transfer is difficult.Source: §2.2 · p. 5

3 CHALLENGES AND LIMITATIONS OF EXISTING QUANTUM MATERIALS DATASETS

Quantum-flake datasets are difficult to collect because exfoliation produces uncontrolled flakes, optical images do not directly reveal thickness, and verification requires laborious AFM matching. Existing synthetic data also has optical-calibration and placement limitations that can widen the synthetic-to-real gap.

3.1 Challenges in Data Collection

  • Mechanical exfoliation produces flakes with uncontrolled shapes, sizes, and thicknesses, so collecting examples of specific layer counts requires searching among many unsuitable regions.A single exfoliation can generate thousands of flakes, but thickness is not directly controlled.Source: §3.1 · p. 5
  • Optical microscopy does not directly measure layer thickness, and matching candidates to nanometer-scale AFM measurements requires substantial manual effort.Appearance varies with illumination, camera response, substrate configuration, and other imaging conditions, limiting data-collection throughput.Source: §3.1 · p. 5

3.2 Limitations of Existing Synthetic Datasets

  • MaskTerial lacks image-specific color calibration and explicit CIE color conversion, limiting fidelity across microscope settings.Its flake placement also ignores flakes in the background, risking unrealistic overlap or physically inconsistent stacking; Table 1 compares synthesis techniques.Source: §3.2 · p. 5, Table 1 · p. 5

4 SYNTHETIC MATERIALS FRAMEWORK (SYNTHIA)

Synthia generates controlled synthetic microscopy images using optical modeling and image-aware corrections, and those images support QMat-Instruct. The dataset combines task supervision with image-specific reasoning grounded in visible optical cues.

4 SYNTHETIC MATERIALS FRAMEWORK (SYNTHIA)

  • Synthia generates microscopy images with controlled material types, layer numbers, locations, and optical properties, then supplies images and verified annotations for QMat-Instruct.The framework combines a multilayer optical model with color calibration and substrate-aware placement to reduce the synthetic-to-real difference.Source: §4 · pp. 5–6, §4.3 · p. 7

4.1 Multilayer Optical Model

  • The multilayer optical model uses transfer-matrix calculations to estimate wavelength-dependent reflectance and flake contrast from material and layer properties.It represents a stack of thin layers by complex refractive indices and physical thicknesses, then converts simulated reflectance to RGB using CIE color-matching functions and an illuminant.Source: §4.1 · p. 6

4.2 Details of Synthia

  • Synthia adjusts generated colors to the target image’s color characteristics and identifies valid substrate regions for physically plausible flake placement.Its placement avoids existing flakes, and the qualitative comparison reports subtler flake–substrate contrast than MaskTerial, closer to real monolayers.Source: §4.2 · p. 6

4.3 QMat-Instruct Dataset

  • QMat-Instruct pairs microscopy images and instructions with structured responses for counting, localization, layer identification, and binary verification.Its supervision supports identifying and localizing flakes while connecting visible optical appearance to predicted layer category through vision and language.Source: §4.3 · p. 7
  • QMat-Instruct replaces fixed reasoning templates with image-specific traces generated using a vision–language model and verified annotations.The traces are restricted to visible relative contrast, color, boundaries, and flake appearance, excluding external measurements such as AFM or Raman spectroscopy.Source: §4.3 · p. 7

5 THE PROPOSED METHOD

QuPAINT combines image features, substrate-relative optical contrast priors, and instructions to generate flake characterizations. Its learnable correction adjusts the prior’s influence during training while retaining optical contrast as a physical signal.

5 THE PROPOSED METHOD

  • QuPAINT combines a vision encoder, Physics-Informed Attention (PIA) optical contrast prior, and an MLLM to generate responses to image-and-text inputs.Outputs can include detected flake counts, spatial locations, and predicted layer categories.Source: §5 · p. 7

5.1 Vision Encoder with Physics-Informed Attention (PIA)

  • PIA assigns each image patch a contrast value measuring its difference from the surrounding substrate in CIELAB space.The prior can be calibrated during training to accommodate illumination, white balance, camera response, and substrate appearance.Source: §5.1 · p. 7

5.2 Learnable PIA Correction

  • A learnable sigmoid correction adjusts each PIA score’s scale and bias without requiring an additional supervision signal.The corrected weight is applied to its corresponding visual token, allowing training to adjust the prior while retaining optical contrast.Source: §5.2 · p. 7
  • For monolayer identification, template reasoning follows a fixed structure, whereas LLM reasoning describes each candidate’s image-specific optical appearance before predicting.The comparison includes contrast, color shift, transparency, and interference cues.Source: Figure 6 · p. 8
  • Synthia’s synthetic monolayers show subtler flake–substrate contrast, closer to real examples, than MaskTerial’s more visually separated flakes.This is a qualitative comparison of synthetic monolayers.Source: Figure 7 · p. 8

5.3 Text Encoder

The text encoder maps instruction tokens to textual representations, and QMat-Instruct covers flake-counting, localization, layer-identification, and verification tasks. The encoded instruction and physics-modulated visual sequence condition the MLLM, which generates a response tailored to the instruction.

5.3 Text Encoder

  • Instruction tokens are mapped to textual representations zt by the text encoder.Source: §5.3 · p. 8
  • QMat-Instruct instructions cover counting, localization, layer identification, and verification, with responses supervising predicted flakes and their distinguishing optical properties.Source: §5.3 · p. 8

5.4 Multimodal Fusion and Decoding

  • The physics-modulated visual sequence ˜zv and encoded instruction zt condition MLLM generation of the output sequence Y.Source: §5.4 · p. 8
  • Depending on the instruction, outputs can report flake counts, locations, predicted layer categories, and supporting optical evidence.Source: §5.4 · p. 8

5.5 Training Objective

QuPAINT trains on QMat-Instruct responses with an autoregressive language-modeling loss and uses Physics-Informed Attention (PIA) to modulate visual features with image-derived, substrate-relative optical priors. PIA estimates optical contrast from the image because acquisition-specific parameters needed by a transfer-matrix model may be unavailable across microscopy settings.

5.5 Training Objective

  • QuPAINT is trained with the standard autoregressive language-modeling loss over QMat-Instruct responses.Source: §5.5 · p. 8
  • No separate supervision is applied to the PIA correction; its parameters are optimized with the model through the language-modeling loss.Source: §5.5 · pp. 8–9
  • PIA extracts optical contrast priors from microscopy images to modulate visual features, which are combined with text instructions for physics-aware reasoning and flake localization.Source: Figure 8 · p. 9

5.6 Physics-Informed Attention (PIA)

  • PIA uses image-derived differences from the surrounding substrate as a prior, reflecting layer-dependent color and intensity changes from thin-film interference.Source: §5.6 · p. 9
  • Acquisition-specific optical constants, illumination, substrate parameters, and camera response needed for direct transfer-matrix modeling are not always available across settings.Source: §5.6 · p. 9
  • PIA compares image regions with the surrounding substrate in CIELAB space and uses color difference as an image-space proxy for optical variations associated with material presence and thickness.Source: §5.6 · p. 9

5.7 PIA Approximation of Optical Contrast

The PIA approximation estimates each region’s optical difference from the bare substrate by modeling image intensity, converting color measurements to CIELAB, and computing a perceptual distance. It constructs the map from patch-level contrast scores using an estimated background appearance.

5.7 PIA Approximation of Optical Contrast

  • For each camera channel, measured intensity depends on spectral reflectance, illumination spectrum, and camera sensitivity; the method distinguishes region reflectance from bare-substrate reflectance.Source: §5.7 · p. 9
  • Measurements across the three color channels form RGB values for image regions and substrate, which are converted to CIELAB space.Source: §5.7 · p. 9
  • PIA quantifies optical difference as the Euclidean distance between a region’s and substrate’s CIELAB representations.Source: §5.7 · p. 9
  • A larger ∆E indicates greater deviation from substrate appearance; the method estimates substrate color from dominant background appearance and evaluates contrast over non-overlapping patches.Source: §5.7 · p. 9

5.8 Implementation

  • In implementation, the substrate is approximated by the image-wide median RGB value, and each patch is represented by its median RGB value.Source: §5.8 · p. 9

Algorithm 2 Physics-Informed Attention Module

Algorithm 2 builds a perceptual attention map from patch-level differences relative to an estimated substrate appearance. The resulting PIA map emphasizes regions that differ more strongly from the substrate without requiring material-specific optical parameters.

Algorithm 2 Physics-Informed Attention Module

6 IMAGE-GROUNDED REASONING SUPERVISION

QuPAINT replaces shared reasoning templates with image-specific explanations grounded in verified flake annotations and visible optical cues. The explanations are constrained to information available in the microscopy image and describe flakes relative to their substrate.

6.1 Image-Specific Reasoning

  • Image-specific traces describe each sample’s visible contrast, color, boundaries, and appearance relative to the substrate, unlike the shared explanations produced by fixed templates.Different images with the same layer category can therefore receive different explanations based on observed optical properties.Source: §6.1 · p. 10

6.2 Reasoning Generation from Verified Ground Truth

  • Verified Synthia locations and layer labels are fixed before explanation generation, keeping annotator localization or classification errors from changing the supervision targets.The annotator explains visual evidence for the supplied annotations rather than independently detecting flakes or assigning layer categories.Source: §6.2 · p. 10

6.4 Annotator Model

  • The final instruction-tuning response retains its established structure: verified annotations determine the enumeration and conclusion, with generated reasoning replacing the template explanation.Reasoning traces are generated with Qwen3-VL-30B-A3B-Thinking; the pipeline retains only the reasoning span and allows another annotator VLM to be substituted.Source: §6.2 · p. 10, §6.4 · p. 11

6.3 Constraints on Generated Reasoning

  • Reasoning uses visible optical cues, including relative contrast, interference color, transparency, edge visibility, and interior uniformity, while excluding external characterization unavailable at inference.Flakes are described relative to the local substrate rather than by fixed absolute color or intensity, which vary with material and acquisition conditions.Source: §6.3 · pp. 10–11

7 EXPERIMENTAL RESULTS

Experiments evaluate QuPAINT’s detection, instruction following, grounding, reasoning supervision, physics-aware components, and transfer across materials. QuPAINT-8B leads the reported detection metrics, while image-grounded supervision improves several stricter localization and confidence measures and preserves performance on held-out WTe2.

7.1 Implementation Details and Benchmarking

  • QF-Bench combines public and in-house microscopy data across eight materials and contains 280,526 annotated flakes; training also uses physics-based synthetic samples.The extension corpus contains 40,000 composite images, and each compared variant uses the same 10,000-image subset; no QF-Bench image is used for training.Source: §7.1 · pp. 11–12

7.2 Main Flake Detection Results

  • 45.6 AP for general flake detection and 37.3 AP for mono-layer detection: QuPAINT-8B achieves the strongest performance across all reported detection metrics.Mono-layer detection remains harder across methods because these flakes generally have weaker contrast against the surrounding substrate.Source: §7.2 · p. 12, Table 3 · p. 13

7.3 Instruction-Following and the Effect of Reasoning Supervision

  • Grounding accuracy at IoU 0.5 is 29.3 for both supervision strategies, while IoU 0.75 grounding rises from 13.8 to 15.5 and localization precision at IoU 0.5 from 18.3 to 19.6.Mean IoU changes only slightly for matched mono-layer boxes (0.6875 to 0.6931) and general-flake boxes (0.7178 to 0.7122).Source: §7.3 · pp. 12–13, Table 4 · p. 13
  • Mono-layer confidence AUC rises from 0.820 to 0.846 with image-grounded supervision, while general-flake AUC rises from 0.821 to 0.826.The counting error increases from 2.57 to 2.77, and both models primarily over-count.Source: §7.3 · pp. 13–14, Table 4 · p. 13
  • Image-grounded descriptions are distinct on 81.5% of images, and their optical vocabulary tracks measured contrast: median flake ∆E is 20.1 for low-contrast terms and 21.6 for higher-contrast terms.The contrast difference is significant (Mann–Whitney p = 0.03, n = 1,186); mono-layer flakes have median ∆E 8.1 versus 20.8 for thick flakes.Source: §7.3 · p. 14

7.4 Ablation: Physics-Aware Description and PIA

  • 34.1 AP with both PAD and PIA is the strongest mono-layer ablation result, compared with 28.1 AP without PIA and 30.6 AP with PIA alone.PAD alone reaches 31.7 AP; the combined +6.0 AP gain is close to the sum of the reported individual gains (+2.5 and +3.6), suggesting largely complementary information.Source: §7.4 · p. 14, Table 5 · p. 14

7.5 Cross-Material and Unseen-Material Generalization

  • Image-grounded supervision raises macro-F1 from 27.1 to 28.6 across materials, while held-out WTe2 retains the same F1 score of 23.0 under both strategies.WTe2 is excluded entirely from the extension training corpus; these reduced-budget extension results are not directly comparable in absolute value with the released models.Source: §7.5 · p. 14

8 CONCLUSION AND LIMITATIONS

The paper presents Synthia, QMat-Instruct, and QuPAINT as a framework for characterizing 2D quantum-material flakes from optical microscopy. It also identifies remaining domain, ambiguity, and data-balance limitations.

Conclusion

  • The framework combines Synthia’s physics-based synthetic microscopy generation, QMat-Instruct’s multimodal instruction dataset, and QuPAINT’s multimodal tuning with a Physics-Informed Attention prior.The conclusion describes the framework as identifying and characterizing flakes from optical microscopy images.Source: §8 · pp. 14–15
  • Image-grounded supervision improves macro-F1 across QF-Bench while preserving performance on held-out WTe2, the material excluded from extension training.The cited cross-material result is reported at IoU 0.5.Source: Table 6 · p. 15

Limitations

  • Synthetic images cannot capture every microscope, substrate, defect, or imaging-condition variation, so a domain gap may remain in new settings.Layer appearance also depends on material and acquisition factors, making identification potentially ambiguous from a single optical image; QF-Bench is imbalanced toward thick flakes, with verified mono- and few-layer samples rarer.Source: §8 · p. 15
Loading 2609.12202v1…