PaperlayerView exact revision on arXiv

Source-linked AI summary

SO-Bench: A Structural Output Evaluation of Multimodal LLMs

Di Feng, Kaixin Ma, Feng Nan, Haofeng Chen, Bohan Zhai, David Griffiths, Mingfei Gao, Zhe Gan, Eshan Verma, Yinfei Yang, Zhifeng Chen, Afshin Dehghan

arXiv:2511.21750v3cs.CVcs.AIcs.CLcs.RO

TL;DR

Visual structured-output capability remains underexplored despite the need for schema-compliant outputs from image-based tasks. SO-Bench evaluates MLLMs across four visual domains and diverse schemas, finding persistent gaps in fully correct outputs even among frontier models. Training experiments show that targeted supervision improves structured-output accuracy.

  • Problem

    No benchmark systematically quantifies how well MLLMs produce schema-compliant outputs grounded in visual evidence, despite the need to align image information with customized schemas.

    Source: §1 · pp. 1–2

  • Method

    SO-Bench pairs nearly 1.8K images across four visual domains with unique schemas and instructions, and evaluates schema adherence, structural fidelity, and value-level accuracy.

    Source: §1 · p. 2

  • Results

    Frontier models exceed 95% accuracy in schema following but reach only nearly 19% accuracy (fuzzy match) on fully correct structured outputs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SFT and RLVR improve schema validation and field matching accuracy, underscoring the importance of targeted supervision for structured reasoning.

    Source: §1 · p. 2

    Benchmark labels may contain errors, and annotations capture only one ground truth even when multiple outputs could be correct.

    Source: Discussions and Limitations · p. 12

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are increasingly deployed in real-world, agentic settings where outputs must not only be correct, but also conform to predefined data schemas. Despite recent progress in structured generation in textual domain, there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs. In this work, we conduct a comprehensive study of visual structural output capabilities for MLLMs with our carefully designed SO-Bench benchmark. Covering four visual domains, including UI screens, natural images, documents, and charts, SO-Bench is built from over 6.5K diverse JSON schemas and 1.8K curated image-schema pairs with human-verified quality. Benchmarking experiments on open-sourced and frontier proprietary models reveal persistent gaps in predicting accurate, schema compliant outputs, highlighting the need for better multimodal structured reasoning. Beyond benchmarking, we further conduct training experiments to largely improve the model's structured output capability. We make the benchmark and evaluation publicly available at https://github.com/apple/ml-sobench

1 Introduction

Real-world visual tasks require outputs that are both semantically correct and compliant with customized schemas, yet schema-grounded extraction and reasoning over images remain underexplored. SO-Bench evaluates this capability across visual domains and finds that accurate, fully compliant output remains a challenge even for frontier models, while targeted training improves performance.

1 Introduction

  • Visual structured output must align image information with hierarchical schemas and user instructions, a capability not systematically quantified by prior work.Prior visual-language work often emphasizes captioning or semantic parsing in limited domains, while key information extraction commonly uses simple fields.Source: §1 · p. 2
  • SO-Bench pairs nearly 1.8K curated images with unique schemas and instructions across four visual domains, drawn from 112K images, 6.5K schemas, and 60K user profiles.Its construction uses multimodal embedding search, synthetic schema generation, human-in-the-loop refinement, and expert inspection.Source: §1 · p. 2
  • Its evaluation separately measures schema adherence, structural fidelity, and value-level accuracy using exact or fuzzy match.The pipeline assesses schema instruction following and behavior under constrained decoding.Source: §1 · p. 2
  • Frontier models exceed 95% accuracy in schema following but reach only nearly 19% accuracy (fuzzy match); SFT and RLVR raise schema validation and field matching accuracy up to 20% and 13%, respectively.The benchmark also finds substantial structured-generation and schema-compliance gaps between small models and proprietary counterparts.Source: §1 · p. 2
  • The benchmark and evaluation pipeline are publicly available.The project is hosted at https://github.com/apple/ml-sobench.Source: §1 · p. 2

2 Related Works

Prior structured-output benchmarks largely assess text inputs or narrow visual tasks. SO-Bench instead targets multimodal generation grounded in visual inputs and diverse JSON schemas.

2 Related Works

  • Text-only benchmarks test structure generation and format fidelity, while visual benchmarks cover tasks such as screenshot-to-HTML parsing or synthetic 3D reconstruction in limited domains.Examples include StructEval (Yang et al., 2025b), JSONSchemaBench (Geng et al., 2025), StructBench (Tang et al., 2023b), Pix2Struct (Lee et al., 2023), Image2Struct (Roberts et al., 2024), and IR3D-Bench (Liu et al., 2025).Source: §2 · p. 3
  • Agentic tool-use benchmarks assess executable calls or multi-step interactions, whereas SO-Bench grounds multimodal structured generation in visual inputs and diverse JSON schemas.The cited comparisons include BFCL, Tau-Bench (Yao et al., 2024), ToolVQA (Yin et al., 2025), and GTA (Wang et al., 2024).Source: §2 · p. 3
  • OCR and document benchmarks typically use simple, one-layer templates, whereas SO-Bench targets realistic JSON schemas with nested structures.The passage names OCRBenchV2 (Fu et al., 2024), CC-OCR (Yang et al., 2024), and OmniDocBench (Ouyang et al., 2025).Source: §2 · p. 3

3 The SO-Bench Dataset

SO-Bench frames visual structured output as generating JSON that satisfies a supplied schema while reflecting image content and user intent. It builds and evaluates a diverse benchmark through a human-reviewed curation pipeline.

3.1 Overview

  • The task is to generate output Y from image I, schema S, and instruction X, with both syntactic schema conformity and semantic fidelity to the image and instruction.The benchmark addresses diverse images and schemas through a multi-stage auto-labeling pipeline with human review.Source: §3.1 · p. 3

3.2 Data Collection

  • SO-Bench combines images across UI, natural-image, document, and chart domains with real-world and synthetic JSON schemas for visual applications.Its image sources span ten public datasets, while its schema repository retains 6K unique visual-application schemas and adds 500 synthetic schemas.Source: §3.2 · pp. 3–4

3.3 Data Curation

  • The curation workflow links schemas to images, adds varied user instructions, then generates and refines structured responses with model and human review.It samples user profiles from 60K synthesized profiles and has eight domain experts check intermediate outputs.Source: §3.3 · p. 4, Figure 2 · p. 4

3.3.1 Image-Schema Association

  • Image–schema association retrieves top-k schemas using image and caption embeddings, then prompts a multimodal model to select a semantically aligned, structurally appropriate schema.The similarity score combines image-to-schema and caption-to-schema cosine similarities; k is typically 20, and human checks filter mismatches.Source: §3.3.1 · pp. 4–5

3.3.2 Schema Generation with Multi-image Grouping

  • Multi-image grouping clusters similar images and generates a unified nested schema to increase schema diversity and structural depth beyond single-image templates.The experiment uses m = 3 neighboring images per cluster.Source: §3.3.2 · p. 5

3.3.3 Progressive Response Generation and Refinement

  • Response generation uses available OCR, layout, or HTML signals, validates JSON outputs, and iteratively refines them with critic feedback and human inspection.Invalid or suboptimal outputs are regenerated up to three times.Source: §3.3.3 · p. 5

3.4 Dataset Statistics

  • Input schemas generally exceed generated outputs in depth and field count, ranging from one-level, one-field schemas to 22 levels and over 2K fields.Charts tend to have slightly deeper median schema hierarchies than the other image domains, whose CLIP image embeddings form distinct clusters.Source: Figure 3 · p. 5, §3.4 · p. 6, Figure 4 · p. 6

3.5 Evaluation Metrics

  • Evaluation uses AST comparison of nested dictionaries, with exact, fuzzy, or ignored primitive fields, and reports schema validation, field matching, and full structure matching accuracy.Fuzzy matches use normalized edit distance for strings and relative error for numbers; ignored fields are skipped when not required by the schema or directly related to the query.Source: §3.5 · pp. 6–7

4 Experiments and Results

The experiments benchmark visual structured-output performance across models, schema complexity, API generation, and training interventions. Training studies use data from the paper’s labeling pipeline and evaluate supervised fine-tuning and reinforcement learning with verifiable rewards.

4.1 Main Results

  • Gemini-2.5-Pro reaches nearly 98% schema validation accuracy; only it exceeds 70% Field Match (fuzzy), and no model exceeds 20% Full Match (fuzzy).GPT-5 has over 15% higher schema validation accuracy than GPT-4o.Source: §4.1 · p. 8

4.2.1 Metrics Correlation Analysis

  • SO-Bench aligns most strongly with BFCL, LiveBench (Coding), MMMU, and MIABench; Field Match correlates moderately with OCR understanding at approximately r = 0.5.The study reports no strong correlations with IFEval or RefCOCO.Source: §4.2.1 · p. 8

4.2.2 Schema complexity

  • As schema depth increases, schema validation and fuzzy field-match accuracy decline across models; frontier models retain over 95% schema validation accuracy beyond depth six.Intern3.5-VL (4B) shows a performance drop of around 40%.Source: §4.2.2 · p. 9, Figure 6 · p. 9

4.2.3 Inference with Structured Output API

  • Structured-output APIs slightly improve Schema Validation Accuracy for GPT-4o and GPT-4o-mini but tend to lower their Field Match Accuracy (Fuzzy); prompts generally outperform APIs for GPT-5 and Gemini.The comparison uses approximately 1.2K samples with schemas supported by both OpenAI and Gemini APIs.Source: §4.2.3 · p. 9, Table 3 · p. 9

4.3.1 Supervised Fine-tuning (SFT)

  • SFT on the 114K-example training set improves the 3B model as data scale grows, with no clear plateau; full-data performance is comparable to models 10 times larger.The training set uses the same labeling pipeline, except for human verification.Source: §4.3.1 · p. 10, §4.3 · p. 10, Figure 7 · p. 10
  • On real-schema SO-Bench samples, training on synthetic schemas performs much worse than training on real schemas or mixed data, suggesting a distribution shift.On synthetic-schema samples, the randomly sampled training subset performs best and the gaps are smaller.Source: §4.3.1 · p. 10, Figure 8 · p. 11
  • Training only on AriaUI data underperforms random-data training overall, especially on charts, where its shallower average schema depth may leave the model less prepared for deep schemas.The AriaUI subset averages depth 2.7, versus 2.9 for the random 20K subset; the AriaUI-trained model performs best on natural images.Source: §4.3.1 · pp. 10–11

4.3.2 Reinforcement Learning (RL)

  • RL training improves schema validation accuracy and modestly improves field matching over baseline, but remains well below SFT; adding RL to an SFT checkpoint yields very small gains.The reward penalizes invalid JSON, scales squared field-match accuracy by schema compliance, and is optimized on a 14K subset.Source: §4.3.2 · pp. 11–12, Table 4 · p. 11

5 Conclusion

SO-Bench provides a systematic framework for evaluating schema-grounded visual reasoning across diverse visual domains and JSON schemas. Experiments find persistent gaps in structural fidelity and show gains from targeted supervision, while the authors note limits in labels and metrics.

5 Conclusion

  • SO-Bench combines diverse visual domains, over 6.5K JSON schemas, and a unified protocol to evaluate multimodal LLM structured-output capability.The authors describe it as the first systematic framework for assessing schema-grounded visual reasoning.Source: §5 · p. 12
  • Even the strongest frontier models struggle with schema adherence and structural fidelity, exposing a gap between visual understanding and reliable structured generation.The paper reports that SFT and RLVR improve structured-output performance with targeted supervision.Source: §5 · p. 12

Discussions and Limitations

  • Possible label errors may cap measured performance, single-reference annotations can underestimate valid alternatives, and recall-focused FMA and FSMA may still score outputs with extra fields highly.The authors identify better metric design as future work.Source: Discussions and Limitations · p. 12

A Dataset

The dataset expands chart inputs by grouping related single-series charts into composite images for data generation.

A Dataset

  • The original Vistext evaluation set contains single-chart images with only one data series, making the task less challenging.Source: A Dataset · p. 17
  • The authors cluster charts in groups of 2–6 using chart-title bag-of-words features and TF-IDF, then render each group as one image with multiple subplots.Source: A Dataset · p. 17
  • They convert 383 charts into 99 composite images for the data-generation pipeline.Source: A Dataset · p. 17

B.1 Training Dataset

The 114K-example training corpus combines in-domain and out-of-domain image sources with real and synthetic JSON schemas.

B.1 Training Dataset

  • Training images come from HierText, AriaUI, and COYO; HierText is in-domain because its test split contributes to SO-Bench, while the other sources are out-of-domain.Source: §B.1 · p. 17
  • For each HierText and AriaUI image, the authors create examples using both synthetic and real JSON schemas.AriaUI is downsampled to 10K desktop, mobile, and web UI images; the full HierText training set contains 10K images.Source: §B.1 · p. 17
  • COYO is downsampled to 95K images across five categories, with 80% paired with real schemas and 20% with synthetic schemas.After critic-model filtering for low label quality, the corpus contains 114K training samples.Source: §B.1 · p. 17

B.2 Training hyperparameters

The SFT and RL experiments used distinct learning rates and batch sizes, with training lasting around three epochs for both.

B.2 Training hyperparameters

  • SFT training ran for around 3 epochs, with batch sizes of 64, 128, or 256 chosen according to training-set size.The learning rate was 2e-5 and maximum sequence length was 16k.Source: §B.2 · p. 17
  • RL used a learning rate of 3e-7, a batch size of 256, and 32 rollouts per sample.Advantages were computed with REINFORCE Leave One Out (RLOO), and training ran for around 3 epochs.Source: §B.2 · p. 17
  • SFT batch size and training steps were adjusted based on training-data size.The experiments used batch sizes among 64, 128, and 256.Source: §B.2 · p. 17

C Error Examples

The error examples show that failures can arise from schema-depth generalization, instruction following, dense text perception, and reasoning across an image.

C Error Examples

  • A plausible chart value inferred without an explicit unit can miss the fuzzy-match edit-distance threshold against the ground truth.The authors identify VLM-as-a-judge for fuzzy fields as a potential evaluation improvement.Source: C Error Examples · p. 17
  • Shallow schemas in AriaUI training did not generalize well to deep chart schemas, producing schema-invalid structures despite correct extracted values.The example receives zero scores on all metrics.Source: C Error Examples · p. 17
  • The full-data SFT model sometimes treated a schema title as a target field, making an otherwise partially correct output invalid.The authors identify instruction following as an area for improvement.Source: C Error Examples · p. 17
  • The full-data SFT model also missed discounts scattered across a page, despite schema validation and partial field-match credit.The authors point to image-wide visual reasoning as a direction for improvement.Source: C Error Examples · p. 18

D Ablation Studies

The section examines correlations between SO-Bench metrics and other benchmarks across open-source and proprietary models.

D Ablation Studies

  • The analysis compares SO-Bench metrics with other benchmarks using results from multiple open-source and proprietary models.The corresponding models appear in Table 1.Source: D Ablation Studies · p. 18
  • Pearson correlation coefficients for the benchmark comparisons are summarized in Table 2.Source: D Ablation Studies · p. 18

E Instruction Following Prompt

The prompt directs image-based extraction into the supplied JSON schema, with rules for required fields, constrained placeholders, and output formatting. Its appendix also specifies fuzzy field-matching evaluation.

E Instruction Following Prompt

  • The system and user prompts ask the model to extract image information according to the supplied schema and user intent, returning JSON only.The prompt requires all required fields, schema-compliant placeholders, and no comments, extra text, or undeclared fields.Source: E Instruction Following Prompt · pp. 22–23
  • For missing or unextractable required fields, the prompt prioritizes schema defaults, then const or enum values, before type-specific constrained rules.These default-handling instructions are applied across data generation and inference.Source: E Instruction Following Prompt · pp. 18, 20, 22
  • For strings, return exactly m hash characters when minLength=m≥1 or n when minLength=maxLength=n; numeric rules default to 0.With a minimum, choose the lowest valid value; if a maximum makes 0 invalid, choose the highest valid value under the bound.Source: E Instruction Following Prompt · p. 22
  • Array and object rules require recursively populated required fields and respect item-count and uniqueness constraints.Optional fields may be omitted or set to null when allowed; contradictory constraints receive the closest minimally violating placeholder.Source: E Instruction Following Prompt · p. 22
  • The prompt uses stable schema-order tie-breaking and requires valid JSON, while prohibiting hallucinated values and extra fields.The requested output is wrapped in a JSON code fence.Source: E Instruction Following Prompt · pp. 22–23
  • The fuzzy field-matching algorithm scores strings, numbers, lists, and dictionaries, while allowing per-field labels to ignore or relax matching.Its parameters include string similarity threshold 0.8 and relative numeric error threshold 0.05.Source: E Instruction Following Prompt · p. 23

Related papers

Loading 2511.21750v3…