Source-linked AI summary
VERSE: Visual Embedding Reduction and Space Exploration. Clustering-Guided Insights for Training Data Enhancement in Visually-Rich Document Understanding
Ignacio de Rodrigo, Alvaro J. Lopez-Lopez, Jaime Boal
TL;DR
Synthetic training images are often judged by human-perceived realism, although that may not reflect whether they fit a model’s visual-embedding distribution. VERSE analyzes and explores VLM embeddings to assess task suitability and guide enrichment of error-prone regions. The paper reports performance gains that allow optimized on-premise models to match or sometimes surpass leading SaaS systems.
Problem
Synthetic-image quality is often judged by human-perceived realism, which does not necessarily indicate whether samples lie within a VLM’s real-data distribution in visual–semantic embedding space.
Source: §1 · p. 2Method
VERSE analyzes and visualizes VLM embedding structure to assess task potential, reveal relevant features, and identify error-inducing regions for training-data enrichment.
Source: §1 · pp. 2–3, §3 · p. 6Results
Takeaways & Limitations
The paper presents visual-embedding analysis as a way to assess model suitability and guide improvements to training data for visually-rich document understanding.
Source: §1 · p. 3
Abstract
from arXiv · showhide
This work introduces VERSE, a methodology for analyzing and improving Vision-Language Models applied to Visually-rich Document Understanding by exploring their visual embedding space. VERSE enables the visualization of latent representations, supporting the assessment of model feasibility. It also facilitates the identification of problematic regions and guides the generation of synthetic data to enhance performance in those clusters. We validate the methodology by training on the synthetic MERIT Dataset and evaluating on its real-world counterpart, MERIT Secret. Results show that VERSE helps uncover the visual features associated with error-prone clusters, and that retraining with samples containing these features substantially boosts F1 performance without degrading generalization. Furthermore, we demonstrate that on-premise models such as Donut and Idefics2, when optimized with VERSE, match or even surpass the performance of SaaS solutions like GPT-4 and Pixtral.
1. Introduction
VERSE shifts synthetic-image assessment from human judgments of realism to whether images occupy the target distribution in a model’s visual-embedding space. It uses that space to assess model suitability, locate error-prone clusters, and guide training-data enrichment; the paper reports that optimized on-premise models match or sometimes surpass leading SaaS systems.
1. Introduction
- VERSE evaluates synthetic images from the model’s perspective by examining whether their visual embeddings fall within the target distribution, rather than judging photorealism.The paper motivates this shift because visual realism does not necessarily indicate usefulness to the model.Source: §1 · p. 2, Figure 1 · p. 3
- VERSE analyzes VLM visual-embedding structure to assess task alignment, identify error-inducing regions, and guide dataset enrichment with representative samples.It also aims to reveal visual and structural features associated with model behavior and task performance.Source: §1 · p. 2
- The paper presents reduced-space visualization as a way to assess a model’s potential for a task and reveal problematic clusters and visual features to modify.These are the stated contributions of VERSE’s latent-space analysis.Source: §1 · p. 3
- VERSE produces a measurable performance boost, enabling on-premise models to match or sometimes surpass leading SaaS solutions such as GPT-4 and Pixtral [5].The methodology is validated on MERIT Dataset and its real-world counterpart, MERIT Secret, for Spanish key information extraction.Source: §1 · pp. 2–3
2. Related Work
Related work covers OCR-dependent and OCR-free model families, instruction-tuned multimodal systems, and datasets spanning document-analysis tasks. It also describes synthetic-data generation and the use of document-like benchmarks, while noting domain mismatch for digitally born documents.
Model architectures
- LayoutLM-family models develop multimodal document representations, while later variants add text-image alignment, multilingual support, and Word–Patch Alignment [8].LayoutXLM also released XFUND as a cross-lingual benchmark.Source: §2 · p. 4
- OCR dependence can constrain document models under complex layouts or degraded text, motivating spatial token reordering and OCR-free approaches such as Donut.DocParser and UDOP extend this direction with character-level discrimination and document inpainting capabilities.Source: §2 · p. 4
- Instruction-tuned multimodal systems combine visual encoders and language models, with examples including PaliGemma, LLaVA, and Idefics2 [16] [19].Idefics3 is also described as an instruction-aligned system trained on diverse image–text corpora [24].Source: §2 · pp. 4–5
Datasets
- VrDU datasets span entity recognition, token classification, multilingual benchmarks, receipts, layout analysis, and visual question answering.Digitally born scientific-article images can create a domain gap when models are applied to real-world documents.Source: §2 · p. 5
Data generation and VERSE
- Dataset-creation approaches include embedding-based retrieval and controlled data generation such as MERIT; Figure 2 outlines VERSE’s embedding reduction and feature/F1 overlays.The figure describes reduced embeddings as supporting interpretability and the overlays as supporting explainability.Source: §2 · pp. 5–6, Figure 2 · p. 6
3. Methodology
VERSE inspects, reduces, and visualizes VLM embeddings to assess task potential, reveal features in synthetic samples, and identify error-inducing features for enriched training data. Its validation data are real Spanish student transcripts labeled with document and environmental features.
3. Methodology
- VERSE uses embedding inspection, reduction, and visualization to assess task potential, uncover features in synthetic samples, and identify error-inducing features for enriched training data.The workflow is organized around interpretability and two explainability objectives.Source: §3 · p. 6
A. Validation dataset
- MERIT Secret contains 152 real-world Spanish student transcripts from 10 schools, captured with different devices and lighting conditions.Each school has its own template and formatting conventions, with some within-school variation.Source: A. Validation dataset · p. 7, Figure 3 · p. 7
- The validation samples carry human-observed labels for intrinsic document properties and environmental factors, alongside per-image F1 scores and model embeddings.Examples of environmental factors include wrinkles, shadows, and anonymization artifacts.Source: A. Validation dataset · p. 7
B. Models
The model setup uses Hugging Face VLMs with visual encoders, extracting embeddings from pretrained models and computing sample-level F1 after fine-tuning. The paper relates encoder training choices and pretraining data to the structure of visual embedding spaces.
B. Models
- The models are Hugging Face VLMs with visual encoders, and Table 1 summarizes their main features.The models use transformer architecture and process both text and images.Source: B. Models · p. 8, Table 1 · p. 8
- Training runs freeze the visual backbone so embedding comparisons and dimensionality reduction use a consistent visual semantic structure.The paper says fine-tuning otherwise changes the underlying visual representation.Source: B. Models · p. 8
- VERSE extracts embeddings from each sample’s off-the-shelf model and computes per-sample F1 with the fine-tuned model.Visual embeddings are taken from the last hidden state of the visual block.Source: B. Models · p. 8
- Donut and Idefics2 have unfrozen visual encoders and document-like pretraining, associated with richer, more structured embedding spaces than PaliGemma’s general-purpose pretraining.PaliGemma’s object-centric pretraining is described as less aligned with document-specific visual cues.Source: Table 1 · p. 8
C. Training dataset
The training data consists of Spanish-language MERIT subsets in multiple versions with differing visual richness. The versions include digital and rendered documents and samples from seven schools.
C. Training dataset
- Training uses Spanish-language subsets of MERIT vanilla and its augmented versions, which add visual information at different levels.The versions include digital samples and renderized documents.Source: C. Training dataset · p. 8, Figure 4 · p. 9
- Each MERIT version includes data from seven schools, whose visual, layout, and textual characteristics vary in alignment with MERIT Secret.The passage describes school-specific subsets and their relationship to the validation dataset.Source: C. Training dataset · p. 9, Figure 4 · p. 9
- VERSE iteratively fine-tunes each model using a selected MERIT version or dataset composition.Source: C. Training dataset · p. 9
D. Embedding reduction
VERSE reduces each image to a model-specific visual embedding, then reduces the high-dimensional representations to support human-interpretable visualization. The evaluated models use different visual encoders and projection mechanisms.
D. Embedding reduction
- VERSE averages the visual encoder’s [n×m×L] patch tensor into one [1×L] embedding per image, using frozen off-the-shelf models.Some models re-project visual representations to match the textual encoder’s dimensionality.Source: D. Embedding reduction · pp. 9–10
- Because visual embeddings have thousands of dimensions, dimensionality reduction is used to produce a human-interpretable representation.The resulting Reduced Embedding Spaces are illustrated for Idefics2 and LLaVA.Source: D. Embedding reduction · p. 10
- The evaluated models differ in visual encoders and embedding outputs; Idefics2 places its projection mechanism within the visual module.Table 3 summarizes these model characteristics; its caption notes that Idefics2 produces richer and more structured embeddings.Source: Table 3 · p. 10
E. Clustering
VERSE uses reduced visual embedding spaces to assess model suitability, inspect document-feature patterns, and locate low-performing clusters. Those findings guide targeted fine-tuning with zoom variation and samples containing problematic intrinsic features.
E. Clustering
- Reduced Embedding Spaces with coherent clusters support interpretability, while overlays of F1 scores or document attributes support explainability.The clustering procedure extracts one visual embedding per image, with encoder-specific projection mechanisms.Source: Figure 5 · p. 11, E. Clustering · p. 12
3.2. Interpretability and model feasibility
- VERSE identifies Idefics2 and Donut as suitable for VrDU, while PaliGemma and LLaVA are ruled out based on reduced-space structure and task potential.The analysis uses PCA and evaluates preservation of embedding neighborhoods alongside cluster structure.Source: §3.2 · pp. 12–13, Table 4 · p. 12
3.3. Explainability and exploration of the reduced embedding space (RES)
- In Donut and Idefics2, macrostructural properties form clearer clusters than zoom-related features, and lower-level details such as stamps or signatures have less influence on RES organization.The feature overlays include document layout, table properties, information blocks, and zoom-related measures.Source: §3.3 · pp. 13, 15, Figure 7 · p. 14, Figure 8 · p. 14
3.4. Explainability and model boosting
- Donut’s low-performing cluster A is associated with low zoom, while cluster B combines low zoom with dual-table layouts and alphanumeric grading.Cluster A’s two information blocks are already represented in training, whereas the identified intrinsic properties distinguish cluster B.Source: Figure 11 · p. 17, §3.4 · p. 17
3.4.1. Extrinsic document feature analysis: zoom level
- Training with zoom values distributed from 0.3 to 1.0 gives Donut its best results across the validation set and clusters A and B.For cluster A, a fixed zoom of 0.625 performs best among the tested fixed levels; the distributed range also introduces visual diversity.Source: §3.4.1 · p. 18, Figure 12 · p. 19
3.4.2. Intrinsic document features analysis
- Fine-tuning Donut on the booster set improves cluster B, but combining it with digital training partially forgets those gains while retaining comparable validation generalization.The booster set contains samples with the two identified intrinsic features and is outside MERIT’s default training partition.Source: §3.4.2 · pp. 18–19
- The rendered training set combined with the booster set produces Donut’s best-performing configuration across the validation set.The combination brings together dual-table and alphanumeric-grade features with the zoom variability introduced by rendering.Source: §3.4.2 · pp. 19–20, Figure 13 · p. 20
4. Results
VERSE evaluates model feasibility through embedding structure, uses problematic clusters to guide targeted fine-tuning, and examines which document properties shape those spaces. The resulting on-premise models reach performance comparable to API-based solutions, with Idefics2 reported to outperform GPT4-O.
4.1. Model feasibility
- Models with more structured embeddings, including Donut and Idefics2, achieve higher fine-tuned F1 scores than PaliGemma and LLaVA.The reported relationship is between embedding structure, Silhouette scores, and downstream F1.Source: §4.1 · p. 20, Table 5 · p. 21
4.2. Model Boosting
- VERSE targets features in low-performing clusters, and boosting improves conflictive regions without degrading already well-generalized regions.The identified features include alphanumeric scores, two-table layouts, and small visual attributes such as zoomed-out content.Source: §4.2 · p. 21, Table 6 · p. 21
- After VERSE optimization, on-premise models achieve performance comparable to API-based solutions, and Idefics2 outperforms GPT4-O.The comparison concerns the best-performing model configurations.Source: Table 7 · p. 22, §4.2 · p. 22
4.3. Explainability
- Macro-visual document properties shape RES organization, while extrinsic features such as zoom remain relevant to F1 variation and signatures, stamps, and noise show no observable influence.The reported structural properties include layout, information distribution, table columns and position, and vertical content blocks.Source: §4.3 · pp. 22–23
5. Discusion
VERSE links embedding-space patterns to model performance and training-data effects. The discussion also identifies limits in component interpretability and differences in visual embedding richness between models sharing an encoder.
5. Discusion
- Data augmentation disperses samples across the real-data manifold, correlating with substantial F1 improvements for Donut and Idefics2.The discussion contrasts this dispersion with samples scattering away from clusters defined by intrinsic and structural features.Source: §5 · p. 23
- Visual representation quality is critical for VrDU; degraded visual structure can pass degraded information to the textual decoder and harm downstream performance.The authors’ findings differ from Idefics2 authors’ emphasis on the textual decoder and align more with [44].Source: §5 · p. 23
- Principal components are polysemantic, but qualitative analysis identifies low zoom, double-table layouts, and alphanumeric grading in the lowest-performing clusters.Source: §5 · p. 23
- Idefics2 produces richer, more task-aligned visual embeddings than PaliGemma [16] despite their shared SigLIP-So400M encoder.The authors attribute the difference to more document-aligned pretraining and Idefics2’s internal MLP and Perceiver projection.Source: §5 · pp. 23–24
6. Conclusions and contributions
VERSE explores VLM visual embeddings to reveal how models structure visual information and to guide targeted training-data improvements. The paper reports improved low-performing regions without compromising generalization, and proposes future work to generate samples directly from problematic latent-space regions.
6. Conclusions and contributions
- Photorealistic synthetic data are not required for improvement; samples must populate embedding-space regions meaningful and useful to the model.Source: §6 · p. 24
- VERSE assesses whether VLM embeddings are sufficiently structured for VrDU, identifies error-inducing regions and features, and guides reinforcement of those features in training data.Source: §6 · p. 24
- Targeted VERSE training improves low-performing regions without compromising overall generalization.Source: §6 · p. 24
- Future work proposes sampling problematic reduced-space regions and reconstructing images from their latent representations with generative models.Source: §6 · p. 24
Data availability
The paper identifies the MERIT Dataset as its training resource and notes that MERIT Secret validation data are unavailable because the real samples are under an NDA. It also lists code and model training-session resources.
Data availability
- The MERIT Dataset training data are available on Hugging Face.Source: Data availability · p. 25
- MERIT Secret validation data are unavailable because they contain real samples covered by a Non-Disclosure Agreement.Source: Data availability · p. 25
- The authors list a GitHub repository for training, testing, and embedding extraction, plus WandB training projects for Donut, Idefics2, and LLaVA.Source: Data availability · p. 25
Appendix A.
Appendix A identifies the features extracted from MERIT Secret samples. Appendix B illustrates Idefics2 embedding patterns and reports that feature-targeted booster training improves problematic regions while retaining generalization.
Appendix A.
- Table A.8 describes features extracted from the MERIT Secret validation samples.Source: Appendix A · p. 25, Table A.8 · p. 26
Appendix B.
- Idefics2 has a more structured embedding space than Donut [13] and performs well on single-table clusters despite limited visual information.Source: Appendix B · p. 26
- Adding visual features shifts training samples toward clearer clusters, while augmentation disperses samples across the embedding space in a way correlated with better F1 scores.Source: Appendix B · p. 26
- Low zoom, dual-table layouts, and alphanumeric grading systems characterize persistent low-performing regions, with region B combining low zoom and double tables.Source: Appendix B · pp. 26–27
- The booster set scores slightly below zoom-only training in region B, but the two sets contribute distinct strengths and perform better together than either alone.Zoom training helps extract information from low-resolution tables, while the booster improves understanding of double-table layouts.Source: Appendix B · p. 27
- Including low-zoom, double-table samples in the booster set improves problematic region B and preserves consistent validation-set generalization.Source: Appendix B · p. 27, Figure B.15 · p. 28