PaperlayerView exact revision on arXiv

Source-linked AI summary

ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching (ALD/E) Scientific Figures

Fahad Ahmed, Sören Auer, Jennifer D'Souza

arXiv:2607.26848v1cs.CV

TL;DR

Scientific-figure comprehension lacks domain-specific datasets and reasoning capabilities for complex scientific figures. This paper introduces an expert-annotated ALD/E figure benchmark and competition, whose results identify data extraction and visual question-answering as particularly challenging.

  • Problem

    Scientific-figure comprehension lacks domain-specific datasets and reasoning capabilities, while existing scientific-figure datasets and competitions have limited scope and task formulations.

    Source: §1 · p. 2

  • Method

    The paper introduces an expert-annotated, domain-specific ALD/E figure dataset and competition for end-to-end scientific comprehension and reasoning across four task categories.

    Source: §1 · p. 2, §3.3 · p. 5, §5 · pp. 15–16

  • Results

    Competition results identify data extraction and visual question-answering as particularly challenging despite advances across the four task categories.

    Source: §5 · p. 16

  • Takeaways & Limitations

    The findings highlight directions for future research on reliable comprehension of specialized scientific documents.

    Source: §5 · p. 16

    Significant challenges remain in data extraction and visual question-answering, which require robust multimodal reasoning and precise scientific understanding.

    Source: §5 · p. 16

Abstract

from arXiv · show

Scientific figure comprehension and reasoning using multimodal AI requires integrating visual perception with domain-specific reasoning to extract meaningful knowledge, often not presented in the text of a research publication. The Sci-ImageMiner benchmark dataset, accompanied by a community-driven competition, raises the bar over prior scientific competitions by curating a comprehensive, expert-annotated dataset across four end-to-end complementary tasks. The competition attracted 68 active participants and 1,263 public/private submissions from 9th January 2026 to 8th April 2026. Our results show that state-of-the-art multimodal models perform well on classification and summarization tasks but struggle with data extraction and scientific reasoning, particularly in visual question-answering. These findings reveal key limitations and highlight challenges and opportunities for improving domain-aware multimodal AI systems. Overall, the Sci-ImageMiner benchmark and competition establish a rigorous platform for advancing research in scientific figure comprehension and reasoning and demonstrate the potential of state-of-the-art approaches for a challenging and complex research area.

1 Introduction · 2 Sci-ImageMiner Benchmark Dataset

The paper introduces Sci-ImageMiner, an expert-annotated, domain-specific benchmark for end-to-end comprehension and reasoning over ALD/E scientific figures, and describes its collection, organization, and annotation.

1 Introduction

  • Scientific figure reasoning combines visual perception with domain knowledge to interpret trends and relationships and derive structured knowledge from figures [18].The paper situates this task in ALD and ALE, foundational technologies for advanced materials and next-generation nanoelectronics.Source: §1 · pp. 1–2
  • Sci-ImageMiner addresses the lack of large-scale, expert-curated data for end-to-end reasoning over authentic, domain-grounded scientific figures.The paper notes that multimodal models struggle with scientific figures and that prior competitions and datasets have limited coverage or task scope.Source: §1 · p. 2
  • The benchmark is presented as a comprehensive, expert-annotated dataset for investigating multimodal model capabilities and the challenges of domain-specific figure comprehension.The competition examines model performance, approaches that may improve it, and which tasks remain most challenging; the dataset and evaluation scripts are available under a CC BY 4.0 license.Source: §1 · p. 2

2.1 Data Collection & Preparation

  • The dataset draws on 205 ALD/E materials-science publications and uses a 49-figure taxonomy.The taxonomy’s visualizations and descriptions are shared in the repository.Source: §2.1 · p. 2
  • MinerU [16] extracts structured text and high-resolution figures from publications, preserving semantic structure and visual fidelity for downstream analysis.The workflow is illustrated in Figure 1.Source: §2.1 · p. 3, Figure 1 · p. 3

2.2 Dataset Organization

  • Sci-ImageMiner separates ALD from ALE and experimental from simulation-based studies, with standard train/dev/test splits.Figure 2 shows the organization hierarchy, and Table 1 presents dataset statistics by split, process type, and study type.Source: §2.2 · p. 3, Figure 2 · p. 3, Table 1 · p. 4

2.3 Annotations

  • Ten domain experts annotated the dataset using a custom platform; Qwen2.5-VL-7B-Instruct pre-annotated classification, extraction, and summarization, while experts manually annotated VQA.The team used subfigure-level bounding boxes and a unified JSON schema; Fleiss’ Kappa was 0.46 (moderate agreement) on a shared sample of five papers. Annotations underwent bi-weekly automated validation.Source: §2.3 · p. 4
  • For VQA, annotators created four question-answer pairs per figure at the subfigure level, using four question types and four standardized answer formats.Question types cover process-oriented mechanisms or workflows, comparative/trend relationships, structure–property links, and application/performance insights; answers may be yes/no, factoid, list, or paragraph.Source: §2.3 · p. 4

3 Competition

The competition evaluates end-to-end scientific figure understanding and reasoning through four complementary tasks, supported by a phased competition schedule and organized submission process.

3.1 Timeline Schedule

  • The schedule ran from the 9 January 2026 development launch through evaluation ending 8 April, with complete train/dev and blind-test data released on 6 March.The website launched on 28 November 2025; evaluation began on 16 March 2026.Source: §3.1 · p. 5

3.2 Competition Organization

  • Separate CodaBench tracks and automated evaluation scripts organized submissions, while the competition proceeded through development and evaluation phases.A website and Google Group supported participant information, announcements, and Q&A; winning teams were invited to submit system papers after the competition.Source: §3.2 · p. 5

3.3 Tasks

  • The four tasks collectively evaluate end-to-end scientific figure understanding and reasoning.Source: §3.3 · p. 5

Task 1: Classification

  • Figures are classified into 49 predefined taxonomy categories, with F1-score as the primary leaderboard metric.Accuracy, precision, and recall are also evaluated.Source: Task 1: Classification · pp. 6–7

Task 2: Data Extraction

  • Charts are reconstructed as structured Markdown tables and evaluated using RMS and TEDS, whose weighted combination determines the final ranking.Source: Task 2: Data Extraction · p. 7

Task 3: Summarization

Task 4: Visual Question Answering

  • Domain-specific reasoning is evaluated on expert-annotated question-answer pairs, with each answer type scored individually and the final ranking score weighting paragraph, factoid, list, and yes/no results equally.Paragraph answers use ROUGE-1/2/L and BERTScore-F1; factoids use ROUGE-1/2/L and Exact Match; lists use case-insensitive set-based F1; yes/no answers use Accuracy and F1-score.Source: Task 4: Visual Question Answering · p. 7

4 Results and Discussion

The competition drew substantial participation, and baseline LVLMs showed no consistently strong performance across tasks. The section also surveys top-team approaches and presents the Task 1 classification leaderboard.

4.1 Overview

  • 68 active participants contributed 1,263 public and private submissions.Full leaderboards are available on the competition website.Source: §4.1 · p. 7, Table 3 · p. 8

4.2 Baseline Results

  • No single evaluated LVLM baseline performed well across all tasks, despite using prompts 9 for consistency.The evaluated models included Gemma 4 E4B 8b, Qwen3-VL-8B-Instruct [14], GLM-4.6V-Flash [15], and Intern VL 3.5 8b [17].Source: §4.2 · pp. 7–8

4.3 Competition Results

  • The section summarizes methodologies proposed by top-five leaderboard teams with available method reports, with an overview of approaches shown in Table 6.Source: §4.3 · p. 8

Task 1: Classification Results

1. Ricoh_SRCB

Ricoh_SRCB combines hierarchical classifier fine-tuning with global and locally cropped image inputs, fusing model outputs for final predictions.

1. Ricoh_SRCB

  • Ricoh_SRCB trains a coarse 6-class model, refines it with a 49-class model and auxiliary binary and 4-class classifiers, then fuses predictions from global and locally cropped inputs.The input views provide complementary contextual and fine-grained visual features.Source: §1 · p. 8

2. IIT_PATNA_CV_1 · 3. DocMiner

IIT_PATNA_CV_1 reformulates panel classification as taxonomy-aware generation with Qwen2.5-VL-7B-Instruct, while DocMiner enriches context and refines prompts and predictions through iterative, multi-agent processes.

2. IIT_PATNA_CV_1

  • IIT_PATNA_CV_1 generates panel labels with Qwen2.5-VL-7B-Instruct, prompted by the input image and a taxonomy-aware description rather than an explicit classification head.The model is fine-tuned using Low-Rank Adaptation (LoRA).Source: §2 · p. 8
  • IIT_PATNA_CV_1 uses class-aware oversampling and parameter-efficient training to improve robustness.Source: §2 · p. 8
  • At inference, IIT_PATNA_CV_1 applies test-time augmentation and majority voting for stability, then maps predicted labels to the predefined taxonomy.Source: §2 · p. 8

3. DocMiner

  • DocMiner augments inputs with category-specific sample documents, image captions, and source text to improve semantic understanding.Source: §3 · pp. 8–9
  • DocMiner iteratively optimizes prompts using development-set feedback, refining category descriptions especially to resolve ambiguity between visually similar classes.Source: §3 · p. 9
  • DocMiner uses independent model predictions and reconciles them through consensus or re-evaluation to mitigate individual-model bias.Source: §3 · p. 9

4. VLMinators · 1. TeleOCR-VL · 2. VLMinators

The merged sections describe a classification approach and two chart-to-structure extraction methods, using prompt cues, specialized training, synthetic data, and inference refinements.

4. VLMinators

  • QLoRA fine-tuning of Qwen2.5-VL-7B-Instruct and LLaMA-based vision models was the classification team’s most effective configuration; selective label hints and post-processing added gains.Heterogeneous ensembles proved ineffective because of high inter-model disagreement.Source: §4 · p. 9
  • Table 4 is identified as reporting Task 1 classification scores for the best baseline and top-five teams; its caption supplies no score values.The supplied passages do not include the table cells.Source: Table 4 · p. 9

Task 2: Data Extraction Results

  • The Task 2 section introduces the top-five submissions and states that Table 5 presents the task leaderboard.The supplied passage provides no leaderboard values.Source: Task 2: Data Extraction Results · p. 9

1. TeleOCR-VL

  • TeleOCR-VL combines structure-aware training, synthetic chart data, pseudo-label refinement, and heterogeneous ensemble inference for chart-to-structure extraction.Separate training on structure-only and full-content samples decouples layout learning from content recognition; the ensemble balances structural fidelity and numerical accuracy.Source: §1 · p. 9
  • TeleOCR-VL uses synthetic data to mitigate annotation scarcity and improve generalization, with iterative pseudo-label refinement to reduce noise.Its inference ensemble combines heterogeneous models to balance structural fidelity and numerical accuracy.Source: §1 · p. 9

2. VLMinators

  • The VLMinators extraction approach fine-tunes Qwen2.5-VL-7B-Instruct with QLoRA and uses prompt constraints, chart semantics, and injected chart-type metadata to guide table generation.Metadata is prepended at inference to align chart semantics with the table schema and reduce structural ambiguity with minimal architectural modification.Source: §2 · p. 10

3. Ricoh_SRCB · 4. Vassilis Sioros · 5. DocMiner

Ricoh_SRCB, Vassilis Sioros, and DocMiner describe approaches that combine visual and textual context with task-specific processing. The supplied passages also identify tables covering Task 2 data-extraction scores and competition-wide team methods and rankings.

3. Ricoh_SRCB

  • Ricoh_SRCB combines full charts with cropped subgraphs in multi-image prompts and uses category-wise augmentation to address class imbalance.The approach is intended to capture global and local visual patterns, provide more uniform training coverage across chart types, and improve generalization.Source: §3 · p. 10

4. Vassilis Sioros

  • Vassilis Sioros uses Qwen3.5-9B with early vision-language fusion, Gated Delta Networks, sparse Mixture-of-Experts, and cropped subfigures paired with surrounding document text.The model generates dense string representations of tables, which are parsed into structured Markdown.Source: §4 · p. 10

5. DocMiner

  • DocMiner incorporates captions and surrounding document text and routes question types dynamically among specialized agents.The textual inputs are used to enhance semantic grounding, while the agents handle distinct question types.Source: §5 · p. 10
  • Table 5 presents Task 2 data-extraction scores for the best baseline and top-five teams.Source: Table 5 · p. 10
  • Table 6 overviews top teams, methods, LVLMs, and rankings across the competition task leaderboards.The passage notes that the full leaderboards are released on the competition website.Source: Table 6 · p. 11

Task 3: Summarization Results

1. DeepVitminC … 4. Vassilis Sioros

The described teams combine visual inputs with document context and use different alignment, preference-learning, reranking, and architecture choices for summarization or visual question answering. The section also introduces the top-five approaches and points to a task leaderboard.

1. DeepVitminC

  • DeepVitminC retrieves contextual information primarily from figure captions and injects it into model inputs through retrieval-augmented generation.This Context Extraction Module supports summarization and visual question answering.Source: §1 · p. 12
  • DeepVitminC’s Consistent Alignment Module jointly processes visual and contextual signals, with supervised fine-tuning teaching task-specific response formats.Source: §1 · p. 12

2. Ricoh_SRCB

  • Ricoh_SRCB fine-tunes Qwen2.5-VL-7B-Instruct with LoRA and DPO, using stochastically corrupted correct answers as hard negatives for preference learning.Its prompts also incorporate captions, explicit subgraph enumeration, and strict formatting constraints.Source: §2 · p. 12

3. TeleOCR-VL

  • TeleOCR-VL combines chart images, captions, surrounding manuscript text, and chart-type priors, then reranks candidate summaries using semantic, consensus, and cross-modal checks.Conflict resolution filters inconsistent outputs.Source: §3 · p. 12

4. Vassilis Sioros

  • Vassilis Sioros uses Qwen3.5-9B with early fusion, Gated Delta Networks, sparse Mixture-of-Experts, cropped subfigures, and localized context retrieved with a sliding window.Training data is normalized and filtered under token-budget constraints.Source: §4 · p. 12

Tasks 4: Visual Question-Answering Results

1. DeepVitminC · 2. Ricoh_SRCB · 3. DocMiner

The three teams use distinct strategies for multimodal figure understanding: retrieval-enriched supervised fine-tuning, staged SFT and GRPO alignment, and iterative multi-agent VQA refinement. Their approaches emphasize contextual grounding, training-data limitations, and specialized error-driven optimization.

1. DeepVitminC

  • DeepVitminC combines rule-based caption and document-text extraction with retrieval-augmented generation to provide complementary context.It uses caption-based and full-text contexts as inputs to enriched prompts.Source: §1 · pp. 12–13
  • DeepVitminC uses enriched prompts to improve semantic grounding across caption-based and full-text contexts.Both context types contribute to the multimodal inputs.Source: §1 · p. 13
  • DeepVitminC trains a vision-language model with supervised fine-tuning and uses a Consistent Alignment Module to enforce task-specific formats and semantic expectations.Source: §1 · p. 13

4. Vassilis Sioros · 5. VLMinators

The discussion compares participant approaches across all competition tasks and finds that top-performing strategies combine strong vision-language models with integrated context, structure, and task-specific optimization.

4. Vassilis Sioros

  • Vassilis Sioros fine-tunes Qwen3.5-9B in a unified multimodal pipeline, using cropped inputs, localized text, precomputed summaries and tables, and answer-type-specific token budgets.Its hierarchical pipeline treats table extraction and summarization as evidence for downstream VQA, with preprocessing and format-aware post-processing supporting consistent outputs.Source: §4 · p. 13

5. VLMinators

  • VLMinators applies QLoRA fine-tuning to Qwen2.5-VL-7B-Instruct and chains upstream task outputs into VQA prompts to improve numerical precision and contextual reasoning.Its adaptive decoding and answer-type constraints control output structure and length; chained figure classification, table extraction, and summarization context particularly benefit factoid and paragraph queries.Source: §5 · pp. 13–14

4.4 Discussion

  • Ricoh_SRCB achieves the strongest classification performance with hierarchical subtasks and global-plus-cropped image inference.IIT_Patna_CV_1 reframes classification as generation, while DocMiner and VLMinators use prompt engineering and contextual enrichment; decomposition and ambiguity handling emerge as key factors.Source: §4.4 · p. 14
  • TeleOCR-VL leads data extraction with structure-aware training, synthetic augmentation, and ensemble inference, while VLMinators achieves competitive results through fine-tuning and context injection.The discussion identifies data design, contextual input engineering, and ensembling as the most robust combination; Ricoh_SRCB and Vassilis Sioros emphasize input representation and architecture, while DocMiner explores modular reasoning.Source: §4.4 · pp. 14–15
  • Summarization approaches combine retrieval and alignment, preference optimization, or consensus re-ranking, while Vassilis Sioros remains competitive through preprocessing and normalization.The discussion highlights contextual grounding, alignment, and inference-time validation as important strategies.Source: §4.4 · p. 15
  • VQA results favor structured context integration and task-specific optimization, with Vassilis Sioros and VLMinators achieving strong results through pipeline design.DeepVitminC uses structured context extraction and retrieval, DocMiner uses iterative prompt refinement and agent-based reasoning, and Ricoh_SRCB gains little from training-only improvements.Source: §4.4 · p. 15

Comparative Analysis and Key Takeaways

  • Top approaches combine strong Qwen-based vision-language backbones with data engineering, prompt design, model adaptation, and inference-time optimization.Task 1 favors decomposition and ambiguity handling, Task 2 data-centric design and ensembling, Task 3 contextual grounding and validation, and Task 4 structured reasoning and context chaining.Source: Comparative Analysis and Key Takeaways · p. 15

5 Conclusions

The ICDAR Sci-ImageMiner competition establishes an expert-annotated benchmark for scientific comprehension and reasoning over ALD/E figures. Its results show that substantial challenges remain in data extraction and visual question-answering.

5 Conclusions

  • The ICDAR Sci-ImageMiner competition is the first benchmark dataset and competition dedicated to scientific comprehension and reasoning over ALD/E figures, with expert annotations reflecting the domain’s complexity.The competition covers four task categories.Source: §5 · pp. 15–16
  • The competition attracted substantial global participation across all four task categories, and teams proposed novel methodologies and approaches.Source: §5 · p. 16
  • Significant challenges remain in data extraction and visual question-answering, which require robust multimodal reasoning and precise scientific understanding.These challenges underscore the difficulty of reliable comprehension in specialized scientific documents and point to directions for future research.Source: §5 · p. 16

Related papers

Loading 2607.26848v1…