Source-linked AI summary
Lot Machine: Multimodal Lot Extraction from Auction Catalogs
Mathias Zinnen, Alisha Mund, Sabine Lang, Lukas Hüttner, Thomas Gorges, Vincent Christlein
TL;DR
Historical auction catalogs are difficult to analyze at scale because variable layouts and subjective descriptions lack standardized lot-level representations. The paper evaluates a VLM extraction pipeline across commercial, institutional, and local deployments, finding that commercial models lead while privacy-preserving alternatives remain viable with strict schema enforcement. Results also show that metric scores and extraction difficulty depend on catalog and field semantics.
Problem
Variable formatting, implicit references, and subjective descriptors in historical auction catalogs make structured lot-level extraction and faithful evaluation difficult.
Method
The study benchmarks an end-to-end VLM pipeline on manually annotated German Sales lots across commercial APIs, institutional gateways, and quantized local models, using prompting and constrained decoding.
Results
Commercial models such as Mistral-OCR offer the highest accuracy, while institutional gateways and local models provide viable alternatives when constrained decoding enforces the output format.
Takeaways & Limitations
VLM-based extraction can unlock historical auction catalogs for large-scale provenance and art market analysis, but structural metrics may penalize semantically correct formatting variants.
Takeaways & Limitations
The schema and prompt design lacked direct specialist provenance or art-historical input and could not represent uncertainty or vagueness.
Abstract
from arXiv · showhide
For provenance research and art market studies, auction catalogs are an essential resource to trace specific objects over time and space. While historical auction catalogs follow established domain conventions, their internal formatting remains highly variable, and their large-scale analysis is currently restricted by the lack of machine-readable representations of the auction lots. We propose a pipeline to automatically extract structured lot-level metadata from German Sales, a large database of historical auction and sales catalogs from the 19th and 20th centuries. Using a manually annotated test set of representative catalog pages, we evaluate Vision-Language Models (VLMs) under varying prompt strategies and constrained decoding frameworks. To reflect the practical constraints faced by cultural heritage institutions, including budget, compute resources, and data privacy requirements, we benchmark the methods across different deployment modes ranging from commercial providers to locally hosted, quantized models. We find that commercial endpoints establish the performance ceiling, while institutional gateways offer a viable, privacy-preserving alternative. Local deployments remain feasible, but strictly require enforcing the output structure during generation to guarantee a valid JSON format. While varying degrees of human-in-the-loop correction are still necessary, this work demonstrates that a VLM-based pipeline can successfully unlock historical auction catalogs for large-scale automated analysis.
1 Introduction
German Sales catalogs contain valuable provenance and market information, but their variable layouts and unstructured representations limit large-scale analysis. The paper addresses this gap with a VLM-based pipeline evaluated across deployment strategies and schema-compliance methods.
- German Sales contains over 15,500 digitized auction and sales catalogs, mainly from German-speaking countries and published between 1901 and 1945.
- Generic OCR enables text search, but standardized lot-level records are needed to systematically analyze market trends, artist popularity, and object descriptions.
- Traditional multi-stage OCR and layout-parsing pipelines are effective for fixed-layout forms but fail on the variable layouts of German Sales catalogs.
- Variable formatting, implicit cross-references, and subjective descriptors make historical auction-lot extraction a semantic as well as technical problem.
- Cultural heritage institutions must balance extraction accuracy, infrastructure, expertise, budget, and data privacy when deploying VLMs.
- The study evaluates commercial APIs, institutional gateways, and quantized local models while comparing structural ANLS* with semantic rouge-1.
2 Methods & Materials
The study builds a manually reviewed benchmark and evaluates VLMs across commercial, institutional, and local deployment settings. It combines instruction-based prompting with constrained decoding to handle heterogeneous catalogs and enforce structured outputs.
- Dataset: 1,378 manually annotated lots across 152 pages from 5 representative catalogs form the benchmark dataset.Initial Mistral-OCR predictions were reviewed and corrected using systematic rules.
- Dataset: Mistral-OCR generated initial annotations, but its contribution to ground truth creates a possible optimistic bias in its ANLS* score.Every lot was reviewed and corrected, yet alignment with the model’s implicit structuring conventions cannot be ruled out.
- Dataset: The test corpus spans five auction catalogs with visual variation in page format and object type, including artworks, furniture, and vases.
- Target schema: The annotated examples expose semantic ambiguity in assigning descriptors such as “Barock” and in mapping furniture terms to schema fields.
- Deployment modes: The experiments represent commercial APIs, institutional gateways, and locally hosted quantized models selected for differing privacy, budget, and hardware constraints.Local deployments provide control over data flow and support custom constrained decoding, but require greater technical expertise.
- Prompting and schema adherence: Instruction-based prompting guides semantic focus, while constrained decoding passes the target schema directly to enforce valid JSON structure.Implementation uses native schema enforcement for Mistral and json_schema for Gemini, AcademicCloud, and local models.
- Prompting and schema adherence: Prompt rules address dotted leader lines, ditto marks, and forward references that complicate OCR and can trigger repetitive decoding behavior.Dotted lines caused Gemini models to enter infinite generative loops during preliminary evaluations.
3 Results and Discussion
The evaluation compares VLM extraction across deployment modes using structural, transcription, and latency metrics, then examines schema enforcement and historical-data ambiguities. Commercial and institutional systems generally outperform local models, while constrained decoding is essential for valid JSON in local deployments.
- Evaluation framework: The evaluation reports structural accuracy, transcription fidelity, and computational feasibility through ANLS*, CER, and latency in seconds per page.ANLS* evaluates structural and value-level accuracy, while CER isolates raw transcription fidelity.
- Overall comparison: 87% ANLS* makes Mistral-OCR the accuracy leader, with 0.03 CER and low cost, although annotation reuse may inflate its measured performance.The authors interpret Mistral-OCR’s score as potentially optimistic because its predictions helped generate the reviewed ground truth.
- Overall comparison: 0.75 ANLS* and 19.81 sec/p make Gemini-Flash weaker than Mistral-OCR but faster and effectively cost-free at this dataset scale.Gemma4-31B reaches 0.77 ANLS* but requires 159.34 sec/p, whereas InternVL3.5 is fastest at 17.3 sec/p with 0.71 ANLS* and 0.21 CER.
- Overall comparison: Local models remain feasible but trade accuracy or speed: InternVL3-Q8 reaches 0.61 ANLS* at 68.30 sec/p, while Qwen3.6 reaches 0.72 ANLS* and 0.16 CER at 78.96 sec/p.InternVL3-Q8 is roughly twice as slow as Mistral-OCR, while Qwen3.6 uses only 3B active parameters.
- Practical implications: Institutions can select Mistral-OCR for affordable accurate extraction, institutional gateways for privacy and scale constraints, or quantized local models for stricter privacy.Local deployments require constrained decoding, GPU-like acceleration, and manual verification remains recommended, especially for smaller models.
- Constrained decoding: System-level schema enforcement slightly improves ANLS* for most architectures, but prompt-only generation fails to produce syntactically valid JSON for locally deployed quantized models.InternVL3.5 is an exception, scoring 0.71 freely versus 0.69 with constrained decoding; constrained decoding remains an absolute requirement for local edge deployments.
- Catalog and field analysis: Historical field ambiguity affects evaluation: object_type is difficult, units can trigger ANLS* penalties, and creator formatting diverges between 0.81 ANLS* and 0.94 rouge-1.Name-order mismatches can preserve correct components while harming normalized representations needed for disambiguation and authority-file linking.
- Catalog and field analysis: Schema definitions and metric interpretation remain scope boundaries because historical fields are subjective and vary across catalog types and decades.The paper notes that refining structural guidelines may affect final accuracy metrics as much as using more capable models.
4 Limitations
The evaluation is limited by non-exhaustive model selection, practical deployment constraints, and schema choices made without direct provenance or art-historical expertise. It also omits comparisons with traditional extraction systems and fine-tuning.
- Deployment and evaluation scope: Model selection was non-exhaustive, while timeout errors for institutional Qwen access and the cost of Gemini-Pro constrained the evaluation.The rapidly changing model and API landscape may also change the reported performance measures.
- Schema and domain expertise: The target schema and prompts were designed without direct involvement from specialized provenance researchers or art historians.The authors therefore note that field definitions and instructions may lack domain-specific contextual nuance.
- Schema and domain expertise: The extraction schema cannot represent uncertainty or vagueness, which are considered important for digital provenance research.The authors propose multiple annotation and inter-annotator agreement analyses as future work.
- Missing baselines and training: The evaluation excludes traditional OCR-plus-parsing pipelines, text-only LLMs, and state-of-the-art layout understanding models such as PaddleOCR.It also relies on instruction-based prompting and constrained decoding with pre-trained VLM weights rather than fine-tuning.
- Missing baselines and training: Fine-tuning and complex multi-step pipelines were deliberately excluded to keep the approach realistic and maintainable for institutions lacking dedicated IT or machine-learning expertise.This design prioritizes immediate institutional usability over customized training and orchestration.
5 Conclusion
The paper presents a VLM pipeline that converts historical auction-catalog images into structured, searchable data and evaluates deployment strategies for institutional use. Commercial models achieved the highest accuracy, while local and gateway deployments remained viable with constrained decoding, despite subjectivity and metric sensitivity.
- Contribution: The pipeline extracts structured data from historical auction catalogs using VLMs, converting unstructured page images into searchable information.The authors connect this transformation to large-scale provenance research and art market analysis.
- Deployment findings: Commercial models such as Mistral-OCR offer the highest accuracy and are described as highly cost-effective.Institutional gateways and locally hosted models provide alternatives for privacy-sensitive or budget-constrained institutions.
- Evaluation findings: Strict metrics such as ANLS* can penalize minor formatting differences even when models understand the historical meaning of the text.Homogeneous art catalogs focused on paintings, drawings, and prints were more difficult than catalogs of everyday household items.
- Future directions: Future work will examine fine-tuning and compare the pipeline with traditional OCR and dedicated layout-understanding models.The authors also identify domain-expert collaboration as important for handling complex historical records.
- Broader applicability: The deployment findings are presented as transferable to museum and collection documents such as inventory cards and accession books with minimal schema and prompt adjustments.The structured data can also be integrated into multimodal search engines and linked with international provenance and art databases.
S1 Evaluation Including Unmatched Lots
The supplementary evaluation reports macro-level performance when unmatched, potentially hallucinated lots are retained. Unlike the primary evaluation, its structural accuracy and transcription metrics penalize structural over-generation across commercial, institutional, and local deployment modes.
- Evaluation setup: Unmatched lots are retained, so hallucinated or unassigned lot data is included in the reported performance.The primary evaluation instead applies deterministic post-processing to filter structural over-generations.
- Metrics: ANLS* and CER strictly penalize structural over-generation in this raw, unfiltered evaluation.The table therefore measures both structural accuracy and transcription fidelity before heuristic intervention.
- Evaluation setup: The supplementary table compares model configurations across commercial cloud APIs, institutional gateways, and local edge deployments.These are labeled Modes A, B, and C, respectively.