PaperlayerView exact revision on arXiv

Source-linked AI summary

Youtu-Parsing-Omni: One Encoder, One Schema, All Modalities

Yunhang Shen, Zhangquan Chen, Xiaotian Li, Zhongpeng Cai, Qing Zhou, Zekai Zhang, Kun Zhang, Zezhong Wu, Jian Hu, Haihan Gao, Heting Gao, Xin Li, Yinsong Liu, Deqian Jiang, Di Yin, Haoyu Cao, Xing Sun, Bing Liu, Yanqiu Qu, Yunfei Wu, Kun Yin, Zhixiang Wei, Yongxiang Hua, Qianyu Li, Xiaoxing Hu, Peiyuan Zhang, Lijiang Li, Dong Xu, Zhenchuan Yang, Yiliang Jiang

arXiv:2610.39301v1cs.CV

TL;DR

Multimodal parsing remains fragmented across input types and often lacks grounded, structured outputs, limiting how agents trace claims to source regions. Youtu-Parsing-Omni uses one encoder and one schema across ten tasks; a single 5 B-parameter checkpoint leads on OmniParsingBench and achieves state-of-the-art document parsing, though dense structured content is expected to run at lower throughput than its predecessor.

  • Problem

    Parsers cover documents, images, audio, and video only partially, with different models and output formats and limited grounded structured parsing beyond standard documents.

    Source: §1 · p. 2

  • Method

    Youtu-Parsing-Omni combines one text-initialized perception encoder with OmniSchema and uses OSAD to train token-role objectives on its own parses against reference-conditioned targets.

    Source: §1 · p. 3, §2 · p. 4

  • Results

    A single 5 B-parameter checkpoint scores 96.96 Overall on OmniDocBench v1.6 and leads open-weight models on OmniParsingBench with 75.08 Avg., with its largest gains on audio and video.

    Source: §1 · p. 3, §7 · p. 24

  • Takeaways & Limitations

    The structured records provide spatial or temporal anchors that let language-model agents trace claims to their source regions or intervals.

    Source: §1 · p. 2, §7 · p. 24

    Because it decodes end-to-end and autoregressively without inference acceleration, throughput on dense structured content such as tables is expected to be lower than Youtu-Parsing.

    Source: §7 · p. 24

Abstract

from arXiv · show

We present Youtu-Parsing-Omni, an omni-modal parser that converts documents, images, audio, and video into structured records grounded in space and time, providing a perception layer for agents that read, locate, and cite evidence across modalities. Its design rests on one encoder for every modality and one schema for every task. Youtu-Omni-Encoder, initialized from a pre-trained language model to strengthen text reading, encodes all modalities with shared weights in a unified space-time coordinate system, and a few fusion layers let video frames attend to concurrent audio, bringing cross-modal fusion and temporal alignment into perception itself. OmniSchema unifies the outputs of all ten parsing tasks, allowing a single self-evolving post-training procedure, OmniSchema-Aware On-Policy Distillation (OSAD), to provide a shared token-role-based objective across tasks. In each round, the model is trained on its own rollouts against token-level targets from a reference-conditioned copy of itself, which is then refreshed with the trained weights, so that the parser and its supervision co-evolve. With a single 5 B-parameter checkpoint, Youtu-Parsing-Omni achieves state-of-the-art performance on OmniDocBench v1.6 (96.96 Overall) and leads all open-weight models on OmniParsingBench (75.08 Avg.), with its advantage concentrated on audio and video. It also achieves the best overall chart-parsing score (95.02) and the highest exact chemical-fingerprint agreement among compared systems, and matches a dedicated optical-music model in character-level transcription of typeset sheet music while staying far ahead of general-purpose systems. Code and model weights are publicly available.

1 Introduction

Youtu-Parsing-Omni addresses fragmented multimodal parsing with one encoder and one grounded output schema, enabling a single checkpoint to serve ten tasks. Its results lead on document and open-weight omni-modal benchmarks, with particularly large gains on audio and video.

1 Introduction

  • Parsing turns non-text inputs into structured records that locate content in space or time, letting language-model agents trace claims to source regions or intervals.The record can be read by text-only models without repeatedly re-encoding the raw visual or acoustic input.Source: §1 · p. 2
  • Existing parsers cover input types only partially, often relying on separate specialist models and output formats rather than one grounded record schema.Document parsing is strong on standard pages, while symbolic content such as chemical structures and sheet music remains largely domain-specific; broader parsing across images, audio, and video is less mature.Source: §1 · p. 2
  • The design addresses joint perception and optimization: separate towers lack in-perception fusion and shared time positions, while heterogeneous schema roles complicate shared task training.SFT uses reference prefixes, and scalar RLVR rewards provide sequence-level feedback; OSAD instead supervises token roles on model rollouts using a reference-conditioned copy refreshed after each round.Source: §1 · p. 3
  • One shared encoder and one schema support all ten parsing tasks, with visual and acoustic tokens sharing positional encoding and fusing inside the encoder.A few encoder layers let video frames attend to concurrent audio within short audio-visual windows.Source: §1 · p. 3, Figure 3 · p. 4
  • A single 5 B-parameter checkpoint scores 96.96 on OmniDocBench v1.6 and 75.08 on OmniParsingBench, leading open-weight models with its largest gains on audio and video.It also reaches 54.2 Tani@1.0 on chemical structures and 22.73 CER on typeset sheet music, on par with specialist LEGATO at 23.30 CER.Source: §1 · p. 3

2 Model

The model uses one text-initialized perception encoder for visual and audio inputs, with shared weights and space-time positions. Fusion layers allow concurrent video and audio content to interact within the encoder.

2 Model

  • Youtu-Parsing-Omni pairs a decoder-only language model with one perception encoder that reads text, images, audio, and video and emits text.The encoder uses shared weights and a common space-time positional encoding; only lightweight modality-specific stems and output heads remain.Source: §2 · p. 4, §2.1 · p. 4

2.1 Youtu-Omni-Encoder

  • Modality-specific stems tokenize image patches and log-mel audio, while a shared bidirectional Transformer contextualizes packed inputs before modality-specific mergers and projectors map them to decoder width.Most encoder parameters are shared across modalities.Source: §2.1 · pp. 4–5

2.2 Unified Positional Encoding and Audio-Visual Fusion

  • Vision tokens use frame time and patch coordinates, while audio tokens use time offsets on the same temporal axis; fusion windows let them attend to one another within videos.The model also retains independent per-frame or per-chunk windows, and training uses the fused route with probability p_fus below 1 from Stage 2 onward.Source: §2.2 · pp. 5–6, Figure 4 · p. 6

3 Omni-Modal Parsing Schema

OmniSchema represents ten parsing tasks across images, audio, and video as localized units paired with extracted content. Shared fields and explicit spatial or temporal anchors make outputs traceable to source regions or intervals.

3 Omni-Modal Parsing Schema

  • OmniSchema covers ten parsing tasks across document and natural images, audio, and video, representing content as units localized by spatial or temporal anchors.Each parsed element or segment can therefore be traced to its source region or time interval.Source: §3 · pp. 6–8
  • Each output is a JSON object with modality, localized elements or segments, and a global description, with additional fields depending on task.Images use spatial elements; audio and video use time segments, and video outputs include a structured report for text-rich video but leave it empty for natural video.Source: §3 · pp. 6–7
  • Image tasks encode document regions, chemical structures, sheet music, charts, flowcharts, geometric figures, or salient natural-image objects in task-specific content fields.Examples include SMILES strings for molecules, ABC notation for scores, Markdown tables for chart data, and Mermaid graphs for flowcharts.Source: §3 · pp. 7–8
  • Audio and video outputs divide input into time segments, with vocal audio carrying speaker, transcript, language, event labels, and captions, and nonvocal audio carrying event labels and an environment caption.Video segments also represent visible objects and text, actions, interactions, camera motion, events, and accompanying audio information.Source: §3 · pp. 7–8
  • OSAD distinguishes JSON structure tokens from content tokens, assigning separate objectives to syntax such as keys and punctuation and to field values.Structure tokens follow a constrained grammar, whereas content fields may admit multiple valid continuations.Source: §3 · p. 8

4 Training

Training first builds unified perception through a staged pre-training curriculum, then refines structured parsing with supervised and on-policy stages. OSAD adds token-role-specific supervision and refreshes its teacher as the student learns.

4.1 Pre-Training

  • Pre-training progressively expands trainable parameters, input scale, cross-modal interaction, and perturbation across four stages.The curriculum addresses alignment, native-resolution perception, language retention, and long-context inputs; Stage 1 uses 8 k–32 k-token sequences.Source: §4.1 · p. 8, Figure 5 · p. 9
  • Stage 0 aligns the encoder with a frozen decoder; Stages 1–3 then unfreeze all parameters, anneal the learning rate, and extend sequences to 128 k tokens.Stage 1 enables four fusion layers after separate modality alignment; Stage 2 uses a cosine decay over 512 B tokens and stochastic video routing, while Stage 3 supports up to 512 frames.Source: §4.1 · pp. 9–10

4.2 Post-Training

  • Post-training moves from SFT serialization to schema-routed RLVR sequence-level optimization and then OSAD token-adaptive supervision and self-evolution.SFT establishes OmniSchema output before the two on-policy stages learn from model-generated rollouts.Source: §4 · p. 8, §4.2 · p. 10

4.2.1 Supervised Fine-Tuning

  • SFT trains all parameters on prompt–response pairs to make complete, schema-conformant OmniSchema outputs the default response to parsing instructions.Its mixture covers all ten parsing tasks and includes general-purpose data to preserve conversational and instruction-following abilities.Source: §4.2.1 · p. 10

4.2.2 Schema-Routed RLVR

  • Schema-routed RLVR optimizes the policy on its own parses with GRPO, gating reward on decodability and a primary payload.When both gates pass, reward combines shared well-formedness with task-specific content accuracy; otherwise reward is zero.Source: §4.2.2 · pp. 10–11

4.2.3 OmniSchema-Aware On-Policy Distillation

  • OSAD refines the RLVR checkpoint with token-adaptive supervision and teacher refresh, updating the parser and its supervision across rounds.Content positions use symmetric JS divergence, structure positions use forward KL, and an all-token JS term is also applied; six rounds are run.Source: §4.2.3 · pp. 11–12
  • OSAD distills from admitted student rollouts on shared generated prefixes, while only the teacher receives the reference parse.The student trains LoRA adapters on frozen base weights; rollouts must pass JSON-validity, schema-conformance, box-syntax, and repetition checks.Source: §4.2.3 · p. 11
  • At each OSAD round boundary, learned adapters are merged into base weights to initialize the next teacher and student, while reference conditioning remains teacher-only.The teacher is fixed within each round and refreshed between rounds; the student receives fresh adapters.Source: §4.2.3 · p. 12, Figure 6 · p. 12

5 Evaluation

Across eight evaluation subsections, the paper tests one checkpoint on document, multimodal, video-understanding, symbolic-transcription, and ablation evaluations. Results show strong document and temporal parsing, alongside limitations in real-world recapture robustness and higher-level video reasoning.

5. Evaluation; 5.1 Setup

  • The evaluation uses one checkpoint and inference configuration; OmniParsingBench comparisons use a common implemented toolkit and exclude the camera-video subset and 136 unavailable-media samples.The toolkit repairs schema violations and uses corrected ground truth for some verified annotation errors; its scores are comparable across evaluated systems but not directly comparable with benchmark-author scores.Source: §5 · p. 13, §5.1 · pp. 13–14

5.2 OmniDocBench v1.6

  • 96.96 Overall is the highest OmniDocBench v1.6 score, while differences from TeleOCR on text, formulas, and tables are all within 0.21 points.Compared with Youtu-Parsing, gains are 4.8 TEDS on tables and 3.2 CDM on formulas; the same model handles ten tasks across broader modalities.Source: Table 2 · p. 14, §5.2 · p. 15, Figure 7 · p. 15

5.3 PureDocBench

  • 76.29 Avg3 is the highest end-to-end PureDocBench score, and Youtu-Parsing-Omni outperforms every end-to-end specialist across clean, digitally degraded, and physically recaptured tracks.It gains approximately 8 points over Youtu-Parsing on each track, but most systems still decline markedly on physically recaptured inputs.Source: §5.3 · p. 15, Table 3 · p. 16

5.4 OmniParsingBench

  • 75.08 Avg. makes Youtu-Parsing-Omni the strongest open-weight OmniParsingBench model, with advantages concentrated on temporal splits and the highest chart score, 95.02.It leads Logics-Parsing-Omni by 2.5 points on audio, 6.8 on natural video, and 10.3 on text-rich video, but trails on the static splits. Its chart Overall is 95.02; the model also has the highest chart Perception score, 97.49.Source: §5.4 · pp. 16–17

5.5 Video-MME-v2

  • On Video-MME-v2 without subtitles, group-score performance is on par with larger non-thinking models and exceeds the comparable Qwen3-Omni baseline at every level.The advantage is concentrated in L1 frame aggregation; on L2 and L3, which target temporal understanding and complex reasoning, the model is comparable to non-thinking baselines and has no similar advantage.Source: §5.5 · pp. 19–20, Table 11 · p. 19

5.6 ChemOCR and PDMX-Synth

  • 54.2 Tani@1.0 is the highest exact fingerprint-match rate on ChemOCR, while 22.73 CER matches specialized LEGATO’s typeset sheet-music character transcription.The model’s 74.6 Avg. Sim. is marginally below DeepSeek-OCR 2’s 74.9; on music, it is second to LEGATO at symbol and line levels, but far ahead of other general systems in CER.Source: §5.6 · p. 20, Table 12 · p. 20

5.7 Ablation Study

  • With two thirds of perception parameters, a randomly initialized unified encoder trails separate pre-trained towers by 1.3 Avg-6 points and 53.4 OCRBench points, but text initialization nearly closes the Avg-6 gap and leads by 77.2 OCRBench points.At the same endpoint, the text-initialized encoder comes within 0.4 points of the towers on Avg-6; the study uses a Qwen3-4B decoder and no parsing data.Source: §5.7 · pp. 20–21, Figure 8 · p. 21
  • OSAD contributes 1.75 of the 1.99 OmniDocBench v1.6 points gained after SFT, compared with 0.24 from RLVR.Across six rounds, the score rises from 95.21 to 96.96; OSAD gives reference-conditioned feedback at content positions and separately supervises JSON keys and delimiters.Source: §5.7 · p. 21, Figure 9 · p. 22

6 Related Work

Prior work motivates grounded structured records as an interface between multimodal evidence and agent reasoning, while identifying gaps in omni-modal parsing and perception. Youtu-Parsing-Omni combines a shared perception encoder and output schema with schema-aware, evolving distillation.

6.1 Parsing as a Perception Interface for Agents

  • OmniSchema represents entities, relations, and spatial or temporal anchors, retaining semantic content and source location for agent-facing interfaces.The paper positions this as a common representation across documents, images, audio, and video.Source: §6.1 · p. 22

6.2 Omni-Modal Parsing and Perception Encoders

  • Document parsers increasingly integrate layout and content prediction, but Youtu-Parsing extends encoder-centric parsing to grounded visual and auditory inputs.The related work distinguishes OCR pipelines, direct structured-output models, and approaches that allocate layout and recognition differently.Source: §6.2 · p. 22
  • Youtu-Parsing-Omni pairs OmniSchema with Youtu-Omni-Encoder, placing selective audio-visual interaction and temporal alignment before the decoder.Its shared backbone is text-initialized, uses common temporal positions, and confines cross-modal attention to selected layers and local windows.Source: §6.2 · p. 23

6.3 Post-Training for Structured Parsing

  • Schema-routed RLVR combines shared validity checks with task-specific content scores, while OSAD adds token-level feedback for heterogeneous structured fields.The related work distinguishes this from scalar-reward supervision alone.Source: §6.3 · p. 23
  • OSAD applies schema-aware token-role supervision to student-generated rollouts, with a reference-conditioned teacher refreshed from the trained student between rounds.It adapts supervision to content and structure tokens within a shared all-token objective.Source: §6.3 · pp. 23–24

7 Limitations and Conclusion

Youtu-Parsing-Omni provides one 5 B-parameter checkpoint for ten parsing tasks and reports leading results on two benchmarks. Its current end-to-end autoregressive decoding is expected to reduce throughput on dense structured content such as tables.

7 Limitations and Conclusion

  • Youtu-Parsing-Omni does not yet implement inference acceleration, and its end-to-end autoregressive decoding is expected to reduce throughput on dense content such as tables.The authors plan to add parallel decoding and related acceleration techniques in future work.Source: §7 · p. 24
  • One 5 B-parameter checkpoint serves all ten parsing tasks, scoring 96.96 on OmniDocBench v1.6 and leading open-weight models on OmniParsingBench with 75.08.The conclusion describes one inference configuration for the ten tasks.Source: §7 · p. 24

Appendix

  • The appendix provides the complete specification behind the self-contained main text.It includes post-training reward implementation, rollout sampling, and the OSAD round schedule.Source: Appendix · p. 31

A Data

The data span general-purpose and task-specific sources for ten parsing tasks, with document, image, audio, and video examples prepared through distinct annotation and filtering pipelines.

A Data

  • General-purpose training data are reused from five named language, vision, and audio datasets, while parsing datasets cover all ten tasks.The annotation-field tables document the structures used for document, natural-image, audio, and video parsing.Source: A Data · p. 31, Table 13 · p. 31, Table 14 · p. 33

A.1 Document Parsing

  • Document data combine full pages for layout and content parsing with specialized-element datasets, including rendered molecules, geometry, and music scores.Public pages are re-annotated and checked; synthetic pages and specialized examples are constructed from templates, graphs, or source data.Source: §A.1 · pp. 31–32

A.2 Natural Image Parsing

  • Natural-image annotations combine captions and verified entities with localization, region captions, and OCR checks; disputed or hallucinated elements can be removed.Entity boxes are localized independently by SAM-3 and a VLM, with a selection procedure when their boxes disagree.Source: §A.2 · p. 32

A.3 Audio Parsing

  • Audio training clips are segmented and transcribed with multiple systems, then filtered using language agreement and transcript disagreement thresholds.Fine-tuning retains chunks only when language identifiers agree and the disagreement score is at most 0.1; RL data receive further offline filtering.Source: §A.3 · p. 32

A.4 Video Parsing

  • Video annotations segment clips at scene or interface transitions, annotate visual and audio fields in parallel, and align them through segment timestamps.An LLM summarizes segments into global descriptions and, for text-rich video, structured reports.Source: §A.4 · p. 33

B Architecture Details

The appendix points readers to the full architectural specification and identifies video-parsing annotation fields as a separate structured reference.

B Architecture Details

  • The appendix lists architectural settings and encoder components beyond a standard Transformer, including two rotary variants not used in the final model.The complete settings are listed in Table 17.Source: B Architecture Details · p. 33
  • Table 16 maps video-parsing annotation fields to their semantics.Use the table to interpret the fields in the video parsing dataset.Source: Table 16 · p. 34

B.1 Model Specification

The specification pairs one shared multimodal encoder with modality-specific input stems and a text-pretrained decoder. Its positional encoding assigns common space-time axes across modalities while preserving the decoder’s 1D positions.

B.1 Model Specification

  • Youtu-Parsing-Omni covers all three modalities with one encoder, whereas other representative parsers use separate encoders per modality.Logics-Parsing-Omni also covers all three, but uses two towers.Source: §B.1 · p. 34, Table 18 · p. 35

B.2.1 Modality Stems

  • The thin vision and audio stems convert raw inputs to tokens of the shared-block width and together hold less than 3% of encoder parameters.Vision maps native-resolution 16 × 16 patches to single tokens; audio produces 13 tokens per second from log-mel spectrograms.Source: §B.2.1 · p. 34

B.2.2 Shared Transformer Block

  • The shared bias-free pre-norm Transformer uses bidirectional attention and per-token angle matrices for rotary positions.The angle matrices enable axis-factorized positions and layer-varying video-token rotations.Source: §B.2.2 · p. 36

B.2.3 Positional Encoding

  • Chunked axis-factorized RoPE partitions rotary slots among time, height, and width, with both vision and audio using a shared time axis.At d_rot = 64, the model assigns 21, 21, and 22 slots to those axes; preliminary runs favored this layout over an interleaved variant for videos up to 64 frames.Source: §B.2.3 · p. 36

B.2.4 Token Merger and Projector

  • Vision tokens are merged in 2 × 2 blocks after the shared encoder, reducing their count fourfold before projection to decoder width.For a 448 × 448 image, this yields 196 decoder tokens.Source: §B.2.4 · p. 36

B.3 Language Model

  • A text-pretrained 4.5 B-parameter decoder retains 1D positions because the encoder already represents spatial and temporal structure.Its 128 k context supports long outputs, and its text-checkpoint recipe is reused unchanged.Source: §B.3 · p. 37

C Post-Training Details

The appendix details schema-routed reward scoring and six-round OSAD training, then specifies evaluation procedures for document parsing benchmarks. Rollouts are filtered for validity and the OSAD objective averages losses over admitted instances.

C.1 Schema-Routed RLVR

  • The reward combines shared well-formedness with schema-specific content accuracy, optimized using GRPO with group-mean-centered advantages.Undecodable outputs and outputs without a primary payload receive zero reward.Source: §C.1 · p. 37
  • Well-formedness checks format validity, repetition, and incomplete coverage, while content accuracy scores grounded fields and, when available, descriptions.Task-specific grounded-field scorers cover charts, flowcharts, documents, natural images, and other structured fields.Source: §C.1 · p. 37

C.2 OmniSchema-Aware On-Policy Distillation

  • OSAD replaces sequence-level rewards with token-level supervision and iteratively trains a student on its rollouts against a frozen, reference-conditioned teacher.After each round, the learned student weights refresh the teacher for the next round.Source: §C.2 · p. 38, §D.1.1 · p. 39
  • OSAD admits rollouts only after JSON-validity, schema-conformance, box-syntax, and repetition checks, then averages losses across admitted instances.Rejected responses are masked rather than repaired, and an update is skipped only when no rollout is admitted.Source: §C.2 · p. 38
  • Across six OSAD rounds, content supervision is progressively emphasized, with learning rates reduced for rounds three through six.Content and structure weights change from (0.50, 0.10) in round one to (0.80, 0.08) in rounds three through six.Source: §C.2 · p. 38

D Evaluation Details

  • The appendix defines benchmark metrics, describes the OmniParsingBench evaluation toolkit, and lists the resulting evaluation set.The appendix also supplements post-training details with reward implementation, rollout sampling, distributed loss reduction, and the training schedule.Source: C Post-Training Details · p. 37, D Evaluation Details · p. 38

D.1.1 OmniDocBench

  • OmniDocBench evaluates document parsing on annotated text blocks, formulas, tables, and reading order using its official matching and metric pipeline.Metrics include normalized text and reading-order edit distances, rendered-formula character-level F1, and tree-edit-based table similarity.Source: §D.1.1 · p. 38

D.1.2 PureDocBench

  • PureDocBench compares clean, digitally degraded, and physically recaptured versions of identical pages to isolate image-condition effects on parsing scores.Overall averages text, formula, and table scores for each track; Avg3 averages the three track Overalls.Source: §D.1.2 · p. 39

D.1.3 OmniParsingBench

OmniParsingBench evaluates structured parsing across five modality and content splits using grounded-field perception, parse-based cognition, and parse-rate-adjusted Overall scores. The appendix also specifies evaluation procedures for chemical-structure recognition and sheet-music transcription.

D.1.3 OmniParsingBench

  • OmniParsingBench scores natural images, graphics, audio, natural video, and text-rich video by combining grounded-field Perception with parse-based Cognition and a parse-rate adjustment.Cognition uses an LLM judge reading only the parse, and the authors implement the evaluation toolkit because the benchmark lacks a complete one.Source: §D.1.3 · p. 39
  • Natural-image Perception measures recall for localized entities and text regions after matching and verification, while Cognition averages nine image-understanding dimensions.Targets are matched separately at IoU ≥0.5, with entity identity or text similarity checked by an LLM judge.Source: §D.1.3 · pp. 39–40
  • Graphics averages chart and geometric-figure scores, using RMS-F1 and Mermaid F1 for charts and canonicalized position, primitive, and relation scores for geometry.Chart and geometry Cognition scores cover their respective OCR/data, visual, logical, basic-element, and relation dimensions.Source: §D.1.3 · p. 40
  • Audio Perception averages temporal segmentation, event-label Key F1, and transcription scores, while Cognition averages information extraction and recognition.Temporal segments are matched by temporal IoU across thresholds 0.3, 0.5, and 0.7.Source: §D.1.3 · p. 40
  • Natural-video scoring combines temporal segmentation, semantically matched action or interaction phrases, transcription, and visual-description ratings with four Cognition dimensions.The visual component rates segment-pair descriptions at temporal IoU ≥0.5.Source: §D.1.3 · p. 40
  • Text-rich video scores ordered on-screen text after merging repeated adjacent screens and averages Cognition across parsing and report dimension groups.The benchmark’s leaderboard Avg. is the equal-weight mean of the five split Overall scores.Source: §D.1.3 · p. 40

D.1.4 ChemOCR

  • ChemOCR evaluates predicted SMILES on 990 scorable image–SMILES pairs using fingerprint Tanimoto similarity and exact-match agreement.Ten samples with unparseable reference SMILES are excluded from the 1,000-sample benchmark.Source: §D.1.4 · p. 41

D.1.5 PDMX-Synth

  • PDMX-Synth evaluates full-page optical music recognition on 800 rendered scores using normalized ABC edit distance at character, symbol, and line levels.The reported measures are CER, SER, and LER.Source: §D.1.5 · p. 41

D.2 Evaluation Set

The evaluation set excludes the camera-video subset because of annotation issues and reports the remaining sample scope. Natural video counts only the general subset.

D.2 Evaluation Set

  • 3,738 samples remain in the evaluation set after excluding the camera-video subset and 136 samples with unavailable media from 3,874.No sample IDs were added, and retained samples’ Cognition labels remain unchanged.Source: §D.2 · p. 41
  • Natural-video sample counts cover only the general subset in Table 19.Table 19 lists OmniParsingBench sample counts within the authors’ evaluation scope.Source: Table 19 · p. 41

E Additional Encoder Ablation Curves

The appendix compares two encoder-training choices using vision and ASR curves, then illustrates parses across document layouts and other modalities. The qualitative examples pair each input with parsed elements or segments and a rendered parse.

E Additional Encoder Ablation Curves

F Qualitative Examples

Related papers

Loading 2610.39301v1…