PaperlayerView exact revision on arXiv

Source-linked AI summary

VERDICT: Agreement Beats Pixel-Space Verification in Real-Document OCSR

Yani Guan, Dengpan Dong, Shuang Luo, Zi Wei, Joah Han, Dan Hannah, Yumin Zhang, Qichao Hu, Kang Xu

arXiv:2608.22183v1cs.CVcs.IRcs.LG

TL;DR

OCSR needs a reliable way to decide which predictions to retain when arbitrary document figures lack ground truth. VERDICT compares molecular-identity agreement among distinct recognizers, filters generic structures, and uses abstention tiers; agreement outperformed pixel-space verification on real-document evaluations. The framework supports high-precision label release, but its validation and deployment scope remain limited.

  • Problem

    OCSR can convert molecular structure figures into machine-readable SMILES, but arbitrary paper figures lack ground truth for deciding which predictions to retain.

  • Method

    VERDICT converts four recognizer outputs to stereochemistry-preserving InChIKeys, votes on molecular identity, assigns agreement tiers, and filters invalid or generic structures.

  • Results

    Agreement among four recognizers reached 0.916 AUROC on real ACS depictions, while pixel-space round-trip verification reached 0.547 AUROC.

  • Takeaways & Limitations

    VERDICT turns agreement into a selective labeling gate that can retain high-confidence structures while abstaining on uncertain images.

  • Takeaways & Limitations

    Validation remains limited by small or non-random samples, VLM-derived segmentation references, incomplete chemical adjudication, and untested patent-domain scope.

Abstract

from arXiv · show

Optical Chemical Structure Recognition (OCSR) converts 2D molecular depictions in the published literature into SMILES, and is increasingly important for constructing large-scale chemical training datasets. Automation at that scale requires identifying unreliable predictions in the absence of ground truth. Three families of label-free signals were compared on $263$ ACS journal depictions with verified ground truth: model confidence, re-rendering similarity, and agreement among recognizers. Pixel-space re-rendering performed little better than chance (AUROC $0.547$, $95\%$ CI $[0.465,0.629]$), and an oracle-tuned threshold on it reduced correct labels per image from $0.745$ to $0.205$. Agreement among four architecturally distinct recognizers instead reached an AUROC of $0.916$ ($[0.880,0.952]$). The two-of-four rule accepted $81.7\%$ of images at $88.8\%$ precision, the three-of-four rule $52.1\%$ at $98.5\%$. The same pattern held on CLEF-IP, UOB, and USPTO. This distinction is obscured on synthetic benchmarks, where re-rendered predictions naturally resemble their inputs. A substance filter removed $2{,}193$ false agreements on wildcards and R-group fragments, after which the three-of-four rule rejected all $68$ generic depictions. VERDICT was then applied to PMC Open Access, producing $6{,}146$ structure labels for $4{,}833$ molecules; chemist adjudication of $400$ released labels in two independent samples yielded precisions of $0.995$ for the three-of-four tier and $0.958$ for the two-of-four tier. VERDICT therefore enables validated labels for multimodal molecular databases linking structure images, machine-readable representations, and source-publication information. In SES AI's Molecular Universe platform, VERDICT further serves as an image-based interface for searching and retrieving molecular records.

1 Introduction

OCSR can convert molecular depictions into machine-readable structures, but real documents make recognizer outputs unreliable and ground truth unavailable. VERDICT addresses this gap by using agreement among recognizers to identify predictions suitable for retention.

  • Large portions of chemical multimodal knowledge remain trapped in structure figures, motivating OCSR systems that can select reliable predictions without ground truth.
  • 0.940 average exact-match accuracy across four rendered conditions fell to 0.231 across four real-document sets for a fine-tuned VLM recognizer.The gap reflects unpredictable depiction style and size plus dense page context requiring crop isolation.
  • VERDICT compares four architecturally distinct recognizers on molecular identity, rather than relying on pixel-space similarity or exact SMILES-string matching.Its substance filter removes agreements involving wildcard- or R-group-containing fragments.
  • 0.916 AUROC for recognizer agreement on 263 ACS depictions exceeded 0.547 for pixel-space round-trip verification.The same ordering was reproduced on CLEF-IP, UOB, and USPTO.
  • 81.7% of images were accepted at 0.888 precision, while 52.1% were accepted at 0.985 precision.VERDICT establishes which crops were read correctly and abstains on the rest.

2 VERDICT System Framework

VERDICT builds a consensus gate by converting recognizer outputs to stereochemistry-preserving molecular identities, filtering invalid or generic structures, and assigning agreement tiers. Its design also emphasizes diverse error patterns and records training overlap for released labels.

  • Four recognizers produce predictions that are canonicalized and converted to stereochemistry-preserving InChIKeys for molecular-identity comparison.The recognizers are DECIMER, MolScribe, MolNexTR, and a supervised fine-tuned VLM.
  • Gold labels require agreement count a≥3, consensus labels require a=2, and a≤1 is routed to review without emitting a label.The tiers are based on agreement count after identity-key comparison.
  • Recognizer diversity matters because the four engines differ in architecture, tokenization, training data, and error patterns across conditions.The framework treats error independence as more important than apparent complementarity in aggregate performance.
  • 15.0% of PMC molecules overlapped with the in-house voter’s training pool, while none of the 257 distinct ACS molecules did.Released records include a training-overlap flag, and using only the three public recognizers removes dependence on the in-house voter.
  • The gate evaluation uses identity-key matching on 263 ACS images with verified ground truth, so its scores are not directly comparable to Table 1’s canonical-string scores on 331 images.
  • Voting on molecular identity accounts for representational variability among equivalent SMILES strings.The winning InChIKey is the mode of the remaining valid keys; predictions that fail parsing or InChI generation are discarded.
  • The substance filter requires a valid InChIKey, no dummy atom, and at least six heavy atoms, excluding incomplete or generic structures.The deployed heavy-atom threshold is ≥6 among strict-InChIKey labels.

3 VERDICT Performance and Discussions

Agreement among recognizers is substantially more informative than pixel-space re-rendering for identifying reliable OCSR predictions. With abstention, agreement sharply reduces accepted-label errors and transfers across depiction domains, but its operating threshold must be specified relative to roster size.

  • 3.1 Comparison of four ground-truth-free signals: AUROC 0.547 (95% CI [0.465, 0.629]) for pixel-space verification contrasts with AUROC 0.916 (95% CI [0.880, 0.952]) for multi-engine agreement.Pixel-space thresholding reduced correct labels per image from 0.745 to 0.205, whereas agreement separated correct and incorrect predictions.
  • 3.1 Comparison of four ground-truth-free signals: Agreement is computed on molecular identity, while pixel similarity is dominated by differences in bond style, spacing, font, and layout.Predictions are reduced to InChIKeys, removing representational differences among equivalent SMILES strings.
  • 3.2 Abstention plays a role in agreements among engines: 0.745 exact match for the four-engine quorum exceeds MolScribe’s 0.707 on ACS by less than four points when abstention is disabled.Across four real-document sets, the difference between quorum and best-engine accuracy ranges from −0.6 to +3.4 percentage points.
  • 3.2 Abstention plays a role in agreements among engines: Accepted-prediction error rates fall by factors of 1.8 to 11.8 across CLEF-IP, UOB, USPTO, and ACS when at least three engines agree.Raw accuracy moves by at most 3.4 points while abstention retains between 52.5% and 95.0% of images.
  • 3.3 From ACS to other depiction domains: 0.992 precision on UOB and 0.989 on USPTO show that ACS-calibrated thresholds transfer to hand-drawn and patent-derived depictions.The non-ACS runs use the first 500 archived rows of each benchmark rather than random samples, so these values are not full-benchmark sampling estimates.
  • 3.4 Agreement fraction is the operating specification: The operating specification is an agreement fraction together with roster size, because the same vote count represents different acceptance requirements for different numbers of engines.Adding a fifth voter makes three-of-five weaker than three-of-four, so thresholds must be recalibrated when the roster changes.
  • 3.5 Operating points: 81.7% coverage at 0.888 precision is used for corpus construction, whereas 52.1% coverage at 0.985 precision is used when incorrect structures cost more than missing ones.Unanimity covers only 22.4% without increasing observed precision, and the reported point estimates are optimistic because thresholds were selected on the reporting images.

4 Frontier-Model Backstopping

The frontier model is evaluated as a backstop for quorum abstentions rather than as a replacement. It rescues some rejected images, but agreement-based routing retains better overall performance and lower cost.

  • Backstop evaluation: The gpt-4o and gpt-5.5 backstop runs use different subsets, so their rescue rates are not a head-to-head comparison.The archived gpt-4o run resolved one of 15 abstentions, whereas gpt-5.5 resolved 20 of 48.
  • Backstop evaluation: 20 of 48 abstentions were resolved by gpt-5.5, raising coverage from 81.7% to 92.0% while precision fell from 0.888 to 0.872.The rescue rate was 0.417 with interval [.288, .557].
  • Backstop evaluation: The two-of-four quorum exceeds the standalone frontier model in coverage, precision, and yield.The quorum reaches 81.7% coverage, 0.888 precision, and 0.726 yield, versus 76.8%, 0.812, and 0.624 for the frontier model.
  • Backstop evaluation: Adding the backstop reaches 92.0% coverage at 0.872 precision for an estimated $1.31, versus $5.13 for the frontier model alone.The backstop therefore increases coverage without replacing the quorum gate.
  • Routing by agreement: The frontier model is better where no two recognizers agree but worse by 15 to 26 points elsewhere.Agreement count identifies which images should be handled by the quorum or routed to the model with different failure patterns.

5 Journal-Domain Corpus

The journal-domain corpus pipeline combines broad segmentation with consensus filtering and human validation. Its main operational constraints are segmentation recall, substantive engine contributions, and conservative precision outside ACS.

  • Corpus construction: 31,776 candidate crops were produced from 18,021 PMC figure files, with consensus filtering yielding 6,146 released structure labels.The corpus began with 2,600 chemistry-related PMC Open Access articles, 1,934 of which contained figures.
  • Consensus filtering: Parseability ranks engines misleadingly: MolScribe produces parseable output for 93.0% of crops but substantive molecular votes for only 39.1%.The ranking by parseable output is close to the reverse of the ranking by substantive vote.
  • Segmentation: Segmentation recall sets a ceiling on corpus coverage, whereas gate precision sets only a floor on corpus quality.Extra crops can be rejected later, but structures never segmented cannot be recovered.
  • Segmentation: 0.878 estimated segmentation recall on a 20-figure probe made over-segmentation the dominant error, which the gate can tolerate.The estimate used 41 structures counted by a frontier VLM and 36 recovered by DECIMER-Segmentation; its 95% Wilson CI was [0.745, 0.947].
  • Substance filtering: The substance filter removed all 2,193 labels introduced by the broader matching rule.The corrected filtering rule addressed false agreements from non-molecular or repeated disconnected components.
  • Human validation: Pooled chemist review found 0.995 precision for gold labels and 0.958 for consensus labels across two independent samples.Unscoreable labels were nine times more common in consensus, and the thresholds were conservative on PMC relative to ACS.

6 A Grounded Molecular Knowledge Base

The grounded knowledge-base pipeline enriches accepted molecular records with article-derived information and shows that label quality affects both molecule identity and downstream field coverage. Traceability is strongest for numeric fields and weaker for prose evidence.

  • Record construction: Each accepted molecule stores its SMILES, structure crop, source article, PMCID, and figure number before downstream extraction.A paid frontier-model call then extracts names, properties, synthesis information, bioactivity, summaries, and supporting text.
  • Label-quality effects: 2,552 distinct molecules were obtained from 6,146 released labels, versus 1,698 from the older 10,896-label set.Records decreased from 3,931 to 3,112 because invalid identity keys had previously collapsed unrelated crops.
  • Label-quality effects: Summary coverage rose from 14.7% to 23.5%, synthesis coverage from 5.6% to 11.4%, and evidence coverage from 13.3% to 23.6%.The comparison used the same 100 articles, prompt, model, and 50-molecule limit per article at unchanged cost of about $7.80.
  • Traceability: 98.3% of numeric fields were traceable, while only 325 of 653 evidence snippets met the support threshold.Compound names reached 81.4%, and 115 evidence snippets had no verbatim overlap with the source article.
  • Label-quality effects: Filtering input labels increased molecule and field coverage without changing extraction cost.The downstream comparison isolates label quality by holding the articles, prompts, model, and per-article limit constant.

7 Limitations

The evaluation is constrained by model dependence, limited and non-random validation, incomplete Markush evaluation, and molecule-level provenance and licensing boundaries.

  • The operating curve reflects this engine mix and evaluation period, not a permanent ranking of recognizers.Frontier-model behavior changed substantially between backstop runs, and the round-trip result depends on the Appendix B implementation.
  • Validation uses 263 ACS images, 48 backstop abstentions, and 20 segmentation figures with VLM-derived rather than human reference counts.Cross-dataset experiments use the first 500 archived rows rather than random samples, so coverage and precision are not full-benchmark sampling estimates.
  • Markush structures are excluded from identity-key evaluation, and knowledge-base records have not received full chemical adjudication.The authors call for larger randomly sampled audits to characterize rare errors and support patent-domain deployment.
  • Provenance auditing is molecule-level because original image manifests were not preserved, and corpus licenses differ across articles.Commercial users must filter on the recorded license field because the corpus is not uniformly licensed for commercial use.

8 Conclusion

VERDICT treats deployable OCSR as a prediction-selection problem, using recognizer agreement to gate labels and substance filtering to prevent false agreements. It produces validated structure labels while retaining a model-dependent reliability scope.

  • Agreement on molecule identity among architecturally distinct recognizers provides reliable information for deciding which OCSR predictions to keep.The system reports AUROC 0.916 on real labeled journal figures.
  • 81.7% coverage reaches 0.888 precision, while 52.1% coverage reaches 0.985 precision under the VERDICT gate.Distribution-free lower bounds are 0.829 and 0.940, respectively, at 3.5 seconds per image and no marginal API cost.
  • Molecule-identity matching and explicit substance filtering address 2,193 erroneous wildcard or R-group fragment matches caused by a loose matching rule.A frontier-model backstop recovered part of the abstained portion, raising coverage to 92.0% at 0.802 yield.
  • VERDICT produced 6,146 deduplicated structure labels on PMC journals.

A The reconciliation core

The reconciliation core deterministically combines independently deployed recognizer adapters, fixed tie-breaking, and archived outputs to produce reproducible labels and tiers.

  • Adapters expose predict(image) → SMILES | None while running in separate environments, allowing engines with incompatible dependencies to connect to one voting core.Adding an engine does not modify the voting logic.
  • Deterministic reconciliation uses highest agreement followed by fixed engine priority, so identical engine outputs always produce the same label and tier.
  • 3.49 seconds is the median ACS latency, with p90 6.92 seconds, p95 9.27 seconds, and mean 4.68 seconds.These timings use four engines in parallel on one node.
  • Archived prediction files make the reconciliation numbers exactly recomputable, but rerunning the engines remains a separate reproducibility question.The archived runs lack a full dependency lock, and residual variation may arise from library and driver versions.
  • 988 of 993 predictions were character-identical across CPU and GPU reruns of the three open engines.The five differences were largely unparseable alternatives; one image changed from unparseable to parseable but incorrect, remaining scored wrong.

B The round-trip score no longer gated on

The round-trip score deterministically compares an input crop with a rendered candidate image, but its fixed, unregistered implementation contributes little useful signal on real crops.

  • The round-trip score averages global SSIM and Jaccard overlap after grayscale conversion, 256 × 256 resizing, and ink-mask binarization.Candidate SMILES are rendered deterministically with RDKit.
  • The earlier fused gate used c = σ(4a + 4s_rt − 4) and accepted predictions at c ≥ 0.60.On real crops, the round-trip score occupied [0.0005, 0.098], so its logit contribution was at most 0.39 versus 4 per agreement vote.
  • The implementation performs no registration, scale or rotation search, bond-length or line-width matching, or local or masked SSIM.These untested choices could explain why a better implementation might separate correct from incorrect predictions.
  • The oracle-tuned cut point is dominated by the emit-everything baseline, indicating the score is not merely being read at the wrong operating point.

C Corpus datasheet

The PMC Open Access corpus spans thousands of chemistry-focused articles and figure crops, with a substantial subset yielding parseable predictions and validated structure labels.

  • 2,600 articles yielded 18,021 figure files and 31,776 segmented crops from chemistry-dense journals.Of these articles, 1,934 contained figures.
  • 29,764 crops had at least one parseable prediction, while 9,074 reached a quorum and 6,320 survived the substance filter.The filtered labels comprised 3,097 gold labels and 3,223 consensus labels.
  • 6,146 labels survived evaluation-set deduplication across 4,833 distinct molecules.

D Reproducing every number

The paper provides a public archive, source code, analysis scripts, and supporting artifacts that reproduce the reported analyses and corpus counts, with two large-scale checks run against archived files.

  • The corpus, reconciliation core, per-engine adapters, and analysis scripts are deposited under DOI 10.5281/zenodo.21753106 with CC-BY-4.0 licensing.The code is also mirrored on GitHub, and the analysis does not require cluster access.
  • The artifacts copied into the paper source’s data directory provide the underlying files for reproducing the reported analyses.
  • detail_4eng.csv contains per-engine predictions for all 29,764 crops, enabling per-engine, pairwise-agreement, and three-engine-ablation analyses without a GPU.
  • The released artifacts include corpus funnel counts, the n=263 ACS four-engine run, quorum labels, knowledge-base records, backstop results, contamination audits, segmentation recall, and robustness reports.These files are identified as the sources for the corresponding analyses and reproducibility checks.
  • The analysis scripts recompute knowledge-base extraction, grounding numbers, backstop results, exposure analyses, release-corpus files, and reported quantities from raw outputs.The scripts cover the analyses in sections 3–6 and the revised sections 4.1 and 6.
  • The contamination audit and extraction-grounding check require the archived 365,776-row training-pool manifest and approximately 1.4 GB of full text from 100 source articles.Their scripts, invocations, outputs, and provenance records are included in the deposit.
Loading 2608.22183v1…