PaperlayerView exact revision on arXiv

Source-linked AI summary

Image2Struct: Benchmarking Structure Extraction for Vision-Language Models

Josselin Somerville Roberts, Tony Lee, Chi Heem Wong, Michihiro Yasunaga, Yifan Mai, Percy Liang

arXiv:2410.22456v1cs.CVcs.AI

TL;DR

VLM benchmarks often struggle to evaluate complex, real-world outputs without costly human judgment or restrictive multiple-choice formats. Image2Struct addresses this gap with automatic round-trip evaluation across fresh data and three structure-extraction domains. Results vary widely across models and domains, while the benchmark remains challenging and its image metrics imperfect.

  • Problem

    Existing VLM benchmarks often require costly human evaluation or use multiple-choice formats that cannot represent many real-world uses such as code generation.

    Source: §1 · p. 1

  • Method

    Image2Struct prompts VLMs to generate structures from images, renders those structures, and automatically compares the rendered images with the inputs across Webpages, LaTeX, and Musical Scores.

    Source: §1 · p. 2

  • Results

    Performance varies considerably across 14 VLMs and domains; GPT-4 Omni has the highest mean win rate, while maximum EMS reaches 0.402 for sheet music and 0.830 for LaTeX equations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Fresh-data round-trip evaluation differentiates model performance on challenging structure-extraction tasks where no model performs well across all domains.

    Source: §1 · p. 2, §6 · p. 11, §7 · p. 11

    The benchmark is limited to structure extraction, also reflects formal-language knowledge, and does not measure robustness to noise or image perturbations.

    Source: §6.1 · p. 11

Abstract

from arXiv · show

We introduce Image2Struct, a benchmark to evaluate vision-language models (VLMs) on extracting structure from images. Our benchmark 1) captures real-world use cases, 2) is fully automatic and does not require human judgment, and 3) is based on a renewable stream of fresh data. In Image2Struct, VLMs are prompted to generate the underlying structure (e.g., LaTeX code or HTML) from an input image (e.g., webpage screenshot). The structure is then rendered to produce an output image (e.g., rendered webpage), which is compared against the input image to produce a similarity score. This round-trip evaluation allows us to quantitatively evaluate VLMs on tasks with multiple valid structures. We create a pipeline that downloads fresh data from active online communities upon execution and evaluates the VLMs without human intervention. We introduce three domains (Webpages, LaTeX, and Musical Scores) and use five image metrics (pixel similarity, cosine similarity between the Inception vectors, learned perceptual image patch similarity, structural similarity index measure, and earth mover similarity) that allow efficient and automatic comparison between pairs of images. We evaluate Image2Struct on 14 prominent VLMs and find that scores vary widely, indicating that Image2Struct can differentiate between the performances of different VLMs. Additionally, the best score varies considerably across domains (e.g., 0.402 on sheet music vs. 0.830 on LaTeX equations), indicating that Image2Struct contains tasks of varying difficulty. For transparency, we release the full results at https://crfm.stanford.edu/helm/image2struct/v1.0.1/.

1 Introduction

Image2Struct addresses the difficulty of evaluating VLM outputs for real-world structure-extraction tasks by rendering predicted structures and comparing them with source images. Its benchmark spans three domains and 14 VLMs, revealing substantial variation across models and task difficulty.

  • Image2Struct evaluates structure extraction from images, including equation-to-LaTeX conversion and webpage-screenshot-to-HTML generation.Source: §1 · pp. 1–2
  • The benchmark uses a round-trip process that converts an image into predicted structure, renders it back into an image, and scores similarity automatically.Source: §1 · p. 2, Figure 1 · p. 2
  • It covers Webpages, LaTeX, and Musical Scores, using fresh data from active online communities and releasing the full results publicly.Source: §1 · p. 2, Figure 2 · p. 3, LaTeX · p. 3
  • 0.402 versus 0.830: maximum EMS varies substantially between sheet music and LaTeX equations, showing that benchmark tasks have different difficulty levels.Source: §1 · p. 2
  • Performance varies considerably across 14 prominent VLMs, with GPT-4 Omni best by mean win rate but ranking third on Musical Scores.Source: §1 · p. 2

2 Image2Struct

Image2Struct collects fresh, filtered examples across webpages, LaTeX, and musical scores, then evaluates VLM-generated structures by rendering them and comparing images. It combines established image metrics with CIS and EMS, including a patch-based approximation for scalable earth mover comparison.

  • The benchmark covers webpages, scientific-document diagrams, and sheet music, with data collected from active sources including GitHub Pages, arXiv, and IMSLP.Source: §2 · p. 3, §2.1 · p. 4, §2.1.1 · p. 4
  • The evaluation pipeline downloads live data, filters and conditions it, queries VLMs, renders their outputs, and computes image similarity scores.Source: §2.1 · pp. 3–4, Figure 3 · p. 4
  • Images are compared after rendering predicted structures, with unsuccessful compilation assigned a zero score and metrics normalized from 0 for dissimilarity to 1 for identical images.Source: §2.2 · p. 5
  • The benchmark uses pixel similarity, LPIPS, SSIM, CIS, and EMS, introducing CIS and EMS for image comparison and efficiency.Source: §2.2 · pp. 5–6
  • EMS augments earth mover distance with pixel coordinates and approximates the costly O(W^2H^2) computation by comparing image patches.Source: §2.2 · p. 6

3 Experiments

The experiments evaluate VLMs under standardized zero-shot prompts, compile their generated structures into images, and score the resulting reconstructions with rendering and similarity measures.

3.1 Prompts

  • Models receive domain-specific prompts that specify the expected output format, with zero-shot prompting used as the common evaluation setting.The prompts are designed to prevent poor performance caused by misunderstanding renderer-accepted formats.Source: §3.1 · pp. 6–7

3 Experiments

  • Generated responses are parsed, supplemented when needed, and compiled with Jekyll, TeX, or LilyPond before screenshots are compared.Rendering failures can result from invalid generated code, and simple syntax or package errors are sometimes fixed during post-processing.Source: §3.2 · p. 7
  • Eight closed-API and six open-weight VLMs are evaluated once with temperature zero to improve reproducibility, using mean win rate from compilation success and EMS scores.The evaluation uses 5.9M input text tokens, 30K input images, and 17.9M output text tokens.Source: §3.3 · p. 7

3.3 Models

  • Table 1 lists the 14 VLMs tested in the initial benchmark.The tested set comprises eight closed-API and six open-weight models.Source: §3.3 · p. 7, Table 2 · p. 8

4 Results

Image2Struct distinguishes model performance and reveals substantial differences across domains, model families, and subtasks, while error analysis shows persistent difficulty with visual nuance and sheet-music interpretation.

4.1 Model performance

  • 0.324 average EMS for LaTeX, 0.370 for Webpages, and 0.069 for Musical Scores shows that the benchmark remains unsaturated.The best model reaches a maximum EMS of 0.724 in Webpages.Source: §4.1 · p. 8
  • GPT-4o achieves RSR 0.807 and EMS 0.660 for LaTeX, RSR 0.980 and EMS 0.710 for Webpages, but RSR 0.491 and EMS 0.340 for Musical Scores.Performance is higher on Webpages and LaTeX than on Musical Scores, with equations easier than plots within LaTeX.Source: §4.1 · p. 8
  • Closed-API models outperform open-weight models, but no model dominates every task; GPT-4o leads overall, ranking first in Webpages and LaTeX and third in Musical Scores.The top-performing models exhibit different task niches.Source: §4.1 · p. 8
  • Error analysis finds that models extract elements such as text and images but miss visual nuances, and none of the tested models can interpret sheet music.Some models generate valid LilyPond code without demonstrating sheet-music interpretation.Source: §4.1 · p. 8
  • The benchmark supports image-based comparison when structures have multiple valid representations, while structure similarity can be computed when ground-truth source code is available.Levenshtein edit distance is given as an example of a structure-level comparison.Source: §4.2 · p. 8

4.3 Test on misspecified data

On misspecified examples from Word documents and magazines, powerful models can still generate valid structures, but reconstructed LaTeX and webpages retain visible fidelity gaps.

4.3 Test on misspecified data

  • GPT-4o and Gemini 1.5 Pro produce valid structures for additional LaTeX and webpage images despite those images originating from Word documents or magazines.The test contains two LaTeX and two webpage examples.Source: §4.3 · p. 9
  • For LaTeX, the best reconstructions preserve text and equations but fail to replicate paragraph alignment.The result indicates that valid structure generation does not guarantee full visual matching.Source: §4.3 · p. 9

5 Related works

Related work spans VLM benchmarks, image-to-code datasets, image-comparison metrics, and round-trip evaluation, with Image2Struct positioned as an automatic evaluation framework for structure generation.

VLM benchmarks

  • Existing VLM benchmarks assess tasks such as captioning and visual question answering but face ambiguity, scaling difficulty, data leakage, and reliance on human or reference-based evaluation.Human evaluation can be suitable for some tasks, while reference outputs provide another evaluation strategy.Source: §5 · p. 10

Other benchmarks for image-to-code

  • Prior image-to-code datasets largely target webpage-to-HTML conversion and commonly use manually curated or synthetic data.The cited examples include synthetic datasets with more than 75,000 code-screenshot pairs.Source: §5 · p. 10

Metrics

  • Image2Struct uses established image metrics including LPIPS, EMD, and SSIM alongside alternatives such as CLIP Score and cross-correlation.The related metric literature also includes Siamese networks and SIFT for visual matching.Source: §5 · p. 10

Round-trip evaluation

  • Its input image → structure → rendered image procedure is related to backtranslation and cycle-consistency, but those works generally train bidirectional mappings whereas Image2Struct evaluates models.The distinction is between training a mapping and measuring model performance through a round trip.Source: §5 · p. 10

6 Discussion

Image2Struct supports fresh, repeatable evaluation, but its conclusions are bounded by prompt sensitivity, metric imperfections, and a focus on structure extraction rather than broader reasoning.

  • Prompting and adaptation choices can affect VLM performance, and evaluating alternatives such as chain-of-thought or auto-prompting remains future work.Source: §6 · p. 10
  • The benchmark can be refreshed quickly with unseen data, but all models must be rerun after each refresh for apples-to-apples comparison.Source: §6 · p. 11
  • Its scores depend on imperfect image metrics, including EMS limitations under affine transformations beyond translation.Source: §6.1 · p. 11
  • Image2Struct evaluates structure extraction but not abstract concept generation, reasoning over extracted structure, or robustness to noise and perturbations.Source: §6.1 · p. 11
  • The framework could extend beyond webpages, scientific documents, and sheet music to domains such as radiology images and electronic health records.Source: §6.2 · p. 11
  • Automatic, repeatable evaluation and fresh test sets can improve transparency and comparison among VLMs, while generated structures may also be misused.Source: §6.2 · p. 11

7 Conclusion

The paper introduces Image2Struct to evaluate VLM extraction of structured content from images using fresh data and automated image comparisons.

  • Image2Struct evaluates VLMs extracting structures such as webpages, LaTeX, and music sheets from images.Source: §7 · p. 11
  • The benchmark continuously downloads current real-world data and compares images rendered from model predictions with original images using automated metrics.Source: §7 · p. 11
  • Its design enables fresh and diverse test examples for evaluation without relying on human judgment.Source: §7 · p. 11

Checklist

The checklist records the paper’s reproducibility materials, experimental reporting, asset provenance, and ethics-related disclosures.

  • The paper discusses work limitations, broader impacts, and the absence of theoretical results requiring assumptions or proofs.Source: Checklist · p. 16
  • The authors provide code, data, and reproduction instructions, while experimental details are reported in Section 3.Source: Checklist · p. 16
  • The evaluation reports token usage but does not compute error bars because repeated runs are prohibitively expensive.Source: Checklist · p. 16
  • The authors state that existing assets are cited and licensed, while no new assets or human-subject compensation were applicable.Source: Checklist · pp. 16–17
  • The dataset uses downloads from GitHub, arXiv, and IMSLP under the respective platform policies, with copyright retained by original owners.Source: Checklist · p. 16

A Classical metrics

This appendix describes classical image-similarity metrics, EMDblock’s multiscale flow formulation, rendering post-processing, and examples illustrating valid or imperfect reproductions.

Classical metrics

  • Pixel Similarity averages binary equality comparisons between corresponding pixels in two images.Source: A Classical metrics · p. 18
  • Classical pixel comparison can penalize shifted images and cause false rankings, so the implementation ignores modal color and permits minor color differences.Source: A Classical metrics · p. 18
  • SSIM compares luminance, contrast, and pixel variance, while LPIPS uses VGG activations and normalized scores where higher means more similar.Source: A Classical metrics · p. 18

Earth Mover Distance

  • EMD represents images as grayscale-value distributions and optimizes flows between values under nonnegative and mass constraints.Source: A Classical metrics · pp. 18–19

EMDblock

  • EMDblock approximates the high-resolution cost computation with image patches, while its illustration shows block movement, within-patch pixel movement, and color-change costs.Source: Figure A1 · p. 19

Rendering and post-processing

  • Post-processing removes headers and standardizes LaTeX or LilyPond outputs to improve compilation, whereas webpages receive no additional post-processing.Source: D Post-processing · p. 21

Multiple correct answers

Examples

H Detailed results

This section reports detailed benchmark statistics through appendix tables covering LaTeX, Webpages, and Music performance.

H Detailed results

I Results of error analysis

The error analysis shows that GPT-4o often extracts textual or basic visual elements but fails on precise formatting, layout, data, and musical structure.

I Results of error analysis

  • GPT-4o extracts text with minor or no error in 42 of 48 analyzed instances across domains.The remaining seven music instances contain no relevant text.Source: I Results of error analysis · p. 29

I.1 LaTeX

  • LaTeX equations and algorithms commonly miss newlines, bold or italic styling, and modifiers such as hats and bars.These errors occur in 3, 3, and 1 of 14 instances, respectively.Source: §I.1 · p. 29
  • None of the seven LaTeX plot renderings is satisfactory because relative geometry, positions, scatterplot points, and non-primary colors are poorly reproduced.Simple shapes and text are created, but color replication succeeds only for primary colors such as red or blue.Source: §I.1 · p. 30, Figure A11 · p. 30, Figure A12 · p. 30

I.2 Musical scores

  • All seven musical-score predictions appear unrelated to the originals, despite GPT-4o achieving a compilation success rate of 0.491 on the full dataset.The predictions have incorrect keys, numbers of measures, or notes, indicating a failure beyond LilyPond syntax compilation.Source: §I.2 · p. 31, Figure A15 · p. 31

I.3 Webpages

  • Across 21 webpage instances, GPT-4o reproduces text and simple elements but frequently misplaces, mis-sizes, or misaligns elements and fails on color gradients.Bad relative size, position, or alignment appears in 12 instances, while all three gradient cases are problematic.Source: §I.3 · p. 31, Figure A16 · p. 32, Figure A17 · p. 32

J Misspecified data

This section presents misspecified LaTeX and HTML data examples alongside dataset documentation describing instance composition, collection, access, and usage boundaries.

J Misspecified data

  • Appendix figures show examples of misspecified data intended for replication in LaTeX and HTML.Source: Figure A18 · p. 33, Figure A19 · p. 33
  • Average performance on two misspecified LaTeX instances and two misspecified webpage instances is reported in separate appendix tables.The cited table descriptions state that scores are conditioned on successful rendering.Source: Table A5 · p. 34, Table A6 · p. 34

Dataset composition

Dataset access

Dataset structure

Data considerations

Data collection

  • Data are collected from public GitHub, arXiv, and IMSLP sources using software APIs or scraping, with search-based selection and manual inspection for quality and diversity.The authors state that the whole datasets are used rather than a sampled subset.Source: §L.3 · p. 37, Q23 What mechanisms or procedures were… · p. 37

Intended use

Related papers

Loading 2410.22456v1…