Source-linked AI summary
VaseVQA: Multimodal Agent and Benchmark for Ancient Greek Pottery
Jinchao Ge, Tengfei Cheng, Biao Wu, Zeyu Zhang, Shiya Huang, Judith Bishop, Gillian Shepherd, Meng Fang, Ling Chen, Yang Zhao
TL;DR
Expert-level analysis of ancient Greek pottery challenges current MLLMs because they have limited domain-specific knowledge and data. The paper introduces VaseVQA and trains VaseVL with supervised fine-tuning and reinforcement learning using verifiable, taxonomy-guided rewards. VaseVL consistently outperforms the SFT baseline on challenging question types, while the dataset remains bounded in its cultural and historical coverage.
Problem
Current MLLMs lack the domain-specific knowledge needed to combine fine visual cues with expert reasoning about ancient Greek pottery.
Source: §1 · p. 1Method
The paper introduces the VaseVQA benchmark and trains VaseVL with supervised fine-tuning followed by reinforcement learning guided by taxonomy-based, verifiable rewards.
Source: §1 · pp. 1–2, §4 · p. 4Results
VaseVL consistently outperforms the SFT baseline on challenging question types, while SFT improves factual recall but remains unstable on reasoning-intensive questions.
Source: §1 · p. 2Takeaways & Limitations
The results indicate that targeted reinforcement learning is a promising direction for adapting MLLMs to other domains requiring expert-level reasoning.
Source: §6 · p. 8VaseVQA focuses on ancient Greek pottery and may reflect annotation preferences and scholarly conventions, limiting its coverage of other cultural-heritage contexts.
Source: §8 · p. 9
Abstract
from arXiv · showhide
Understanding cultural heritage artifacts such as ancient Greek pottery requires expert-level reasoning that remains challenging for current MLLMs due to limited domain-specific data. We introduce VaseVQA, a benchmark of 31,773 images and 67,614 question-answer pairs across seven expert-defined categories, enabling systematic evaluation of expert-level cultural heritage understanding. Using this dataset, we explore effective training strategies for domain-specific reasoning. While supervised fine-tuning improves adaptation to domain knowledge, it struggles with deeper reasoning tasks. We propose VaseVL, which augments SFT with reinforcement learning using verifiable rewards. Experiments show that VaseVL consistently outperforms supervised baselines, especially on reasoning-intensive questions, highlighting the value of targeted reinforcement learning for cultural heritage visual question answering. Our code and dataset will be released at https://github.com/AIGeeksGroup/VaseVQA.
1 Introduction
The paper addresses the gap between general-purpose MLLMs and expert-level ancient Greek pottery analysis by introducing VaseVQA and VaseVL. VaseVL combines supervised fine-tuning with taxonomy-guided reinforcement learning, and the reported results favor it on challenging question types.
1 Introduction
- Ancient pottery analysis requires combining fine visual details with specialized knowledge, capabilities that current general-purpose MLLMs lack for reliable expert-level answers.Relevant knowledge is often confined to expert literature, museum archives, and specialized scholarship.Source: §1 · p. 1
- VaseVQA provides 31,773 images and 67,614 question–answer pairs across seven expert-defined question types with type-specific evaluation metrics.The benchmark is designed to assess lexical precision and semantic alignment in ancient Greek pottery understanding.Source: §1 · pp. 1–2
- VaseVL consistently outperforms the SFT baseline on challenging questions and improves compositional robustness, while SFT remains unstable on reasoning-intensive tasks.SFT improves factual recall and approaches ceiling performance on visually explicit categories, but does not reliably model deeper contextual relationships.Source: §1 · p. 2
- The paper proposes VaseVL, a two-stage model using taxonomy-aware reward shaping to leverage VaseVQA’s expert annotations beyond surface-level visual features.A rule-based reward guides reinforcement learning across Fabric, Technique, Shape, Provenance, Attribution, Date, and Decoration questions.Source: §1 · p. 2
2 Related Work
Existing vision-language evaluations largely target general tasks, while cultural-heritage analysis requires fine-grained visual and historically grounded reasoning. Cultural-heritage VQA resources are scarce, leaving classical artifacts such as ancient Greek vases underrepresented.
2 Related Work
- Generic vision-language evaluations do not adequately assess cultural-heritage analysis, which requires combining fine visual cues with historically and archaeologically grounded knowledge.The passage notes that current MLLMs are evaluated on broad vision-task suites, including retrieval, VQA, grounding, and dialogue.Source: §2 · pp. 2–3
- Public cultural-heritage VQA resources remain scarce, and existing examples focus on oracle bones or fragment reconstruction rather than classical artifacts such as ancient Greek vases.The text identifies RePAIR and HUST-OBS as examples of those narrower domains.Source: §2 · p. 3
- Table 1 summarizes characteristics of major VQA and visual-reasoning datasets, including VaseVQA.VaseVQA is designed to assess factual recall and expert-level reasoning under a type-aware evaluation protocol.Source: §2 · p. 3, Table 1 · p. 3
3 Data Collection
VaseVQA was constructed to represent ancient Greek culture through artifact images and aligned textual descriptions. Its collection combines institutional sources, research and expert text, and archaeologist-led annotation of artifact attributes and context.
3 Data Collection
- VaseVQA was designed to represent ancient Greek culture, with images and text aligned for vision-language tasks.The dataset description refers to train/test divisions and attribute-specific questions in a conversational Q&A format.Source: §3 · p. 3, Table 2 · p. 4
3.1 Image Collection
- Images gathered through collaborations with archaeological institutions, museums, and cultural-heritage centers include complete vases, fragments, and objects in burial contexts.The collection focuses on classical funerary vases and aims to capture materials, craftsmanship, and regional variation.Source: §3.1 · p. 3
3.2 Text Collection
- Textual data comes from academic papers, archaeological reports, and expert annotations, with descriptions of materials, motifs, and archaeological context structured to align with images.Source: §3.2 · p. 4
3.3 Annotation
- Archaeologists and cultural-heritage experts annotated material, patterns, excavation layers, radiocarbon estimates, techniques, use, and restoration marks.The annotation process is described as reflecting each artifact’s cultural and historical context in detail.Source: §3.3 · p. 4
4 Methods
VaseVL starts from an SFT model and applies GRPO reinforcement learning to improve reasoning while preserving foundational knowledge. Its rewards combine lexical and semantic signals, adapt weights by question type, and emphasize categories where SFT underperforms.
4 Methods
- VaseVL initializes reinforcement learning from an SFT model and uses GRPO to improve reasoning while preserving foundational knowledge.For each image-question pair, it samples candidate answers, computes type-conditioned rewards, and optimizes with a clipped PPO-style objective and KL regularization.Source: §4 · p. 4
- VaseVQA is split into training and test sets at a 4:1 ratio, with nearly identical question-type distributions across splits.Attribution and Decoration have longer average answers, reflecting more open-ended response requirements.Source: Table 3 · p. 5
4.1 Reward Design
- The reward combines keyword overlap for factual precision with embedding-based semantic similarity so paraphrases can retain credit.The keyword score resembles Jaccard similarity and penalizes omissions or hallucinated entities.Source: §4.1 · p. 5
4.2 Reward Shaping
- Question-type-conditioned weights balance lexical accuracy against semantic equivalence, while extra reward weight targets categories where SFT underperforms.The predefined challenging set includes numerical reasoning and multi-step factual queries.Source: §4.2 · pp. 5–6
5 Experiments
The experiments evaluate VaseVL across seven VaseVQA question types and compare it with general-purpose models and an SFT-only baseline. VaseVL surpasses larger baselines, while performance varies substantially by task and reward components contribute differently across tasks.
5 Experiments
- Experiments compare VaseVL with general-purpose MLLMs and an SFT-only model across the benchmark’s seven question types.The evaluation is organized to analyze expert-level reasoning by question type.Source: §5 · p. 6
5.1 Implementation details
- Training uses full-parameter SFT followed by task-specific RL, with the SFT model serving as the RL reference and informing selection of a shortcoming subset.The implementation describes a GRPO-style setup for RL.Source: §5.1 · p. 6
5.2 Evaluation Metrics
- Factual short-answer types use ANLS-based accuracy, Date uses a score combining range formatting, proximity, and year-range IoU, and Decoration uses BLEU@1.The metrics are tailored to the question type and its answer format.Source: §5.2 · pp. 6–7
5.3 Main Results
- VaseVL, with 3B parameters, surpasses larger general-purpose baselines on VaseVQA, where simply increasing model size does not yield gains.The comparison is reported in Table 4.Source: Table 4 · p. 7, §5.3 · p. 7
- Fabric and Technique reach near-perfect accuracy after SFT, but Date, Attribution, and Decoration remain harder, with VaseVL scoring 39.87% on Date, 60.83% on Attribution, and 9.82 BLEU@1 on Decoration.The reported pattern distinguishes rapid adaptation on low-level perception from persistent difficulty with high-level historical reasoning.Source: §5.3 · p. 7
5.4 Ablation Study
- VaseVL combines SFT and RL for 75.71% overall, retaining factual accuracy while raising Decoration to 9.82 BLEU@1 from 2.57 BLEU@1 with SFT alone.RL without prior SFT yields only marginal reasoning gains, including 11.50% on Attribute; Table 5 reports the ablation results.Source: Table 5 · p. 8, §5.4 · p. 8
- On VaseVQA-mini, removing the semantic score lowers Decoration from 6.21 to 3.24 BLEU-1, while the full reward framework achieves the highest overall score, 74.74%.The ablations also associate the keyword score with strict classification and shortcomings-aware amplification with difficult-task reasoning; Table 6 reports the comparison.Source: Table 6 · p. 8, §5.4 · p. 8
6 Conclusion
The conclusion presents VaseVQA and VaseVL as a benchmark and two-stage training framework for cultural-heritage analysis. It highlights targeted reward shaping and GRPO with KL regularization, and identifies evaluation on broader heritage datasets as future work.
6 Conclusion
- VaseVQA spans seven question types, and VaseVL uses a two-stage framework with reward shaping for cultural-heritage analysis.The authors describe this approach as moving beyond surface-level pattern recognition by aligning optimization with task-specific metrics and amplifying rewards for complex reasoning.Source: §6 · p. 8
- The conclusion states that GRPO with KL regularization preserves factual accuracy while enabling expert-level reasoning.Source: §6 · p. 8
- The authors present VaseVQA as a reproducible standard and call for future validation on a broader range of cultural-heritage datasets.Source: §6 · p. 8
7 Social Impact
The paper describes VaseVL as supporting archaeological analysis and public-facing uses of ancient Greek pottery collections.
7 Social Impact
- VaseVL is described as aiding archaeologists with classification and digital documentation, and supporting forgery detection and cultural preservation.Source: §7 · p. 8
- Museums and educators could use VaseVL to create learning experiences about ancient Greek pottery.Source: §7 · p. 8
- The paper presents the model as a way to connect AI and archaeology in support of preserving ancient Greek pottery’s historical legacy.Source: §7 · p. 8
8 Limitations
The benchmark focuses on ancient Greek pottery, and its coverage and annotations may limit how well models generalize to other cultural-heritage contexts.
8 Limitations
- VaseVQA focuses on ancient Greek pottery and cannot fully represent heritage artifacts across other regions, periods, and materials.Source: §8 · p. 9
- Question-answer distributions may reflect annotation preferences and scholarly conventions, potentially biasing model training.Source: §8 · p. 9
- The authors warn that models optimized on VaseVQA may overfit dataset-specific patterns rather than generalize to real-world archaeological or art-historical contexts.They propose broader artifact coverage, cultural traditions, and expert-annotation sources as future work.Source: §8 · p. 9
Appendix
The appendix outlines broad visual-task evaluation for current MLLMs and points to Table 7 for a summary of widely used visual-recognition datasets.
Appendix
- Current MLLM evaluation includes image classification, image–text retrieval, and object detection.Source: Appendix · p. 13
- The task suite also includes semantic segmentation and general-purpose VQA.Source: Appendix · p. 13
- Table 7 summarizes widely used visual-recognition datasets for MLLM evaluation.Source: Table 7 · p. 14