Source-linked AI summary
VaseMuseum: Digital Intelligent Museum for Ancient Greek Pottery
Jiazi Wang, Nonghai Zhang, Qiushi Xie, Zeyu Zhang, Yufeng Chen, Yang Zhao, Ling Shao, Hao Tang
TL;DR
Reliable VLM assistance for ancient Greek pottery remains difficult when interpretation requires specialized evidence and calibrated uncertainty. VaseMuseum addresses this with an inference-time multimodal agent that controls retrieved evidence and generated claims, improving citation validity, reducing hallucinations, and increasing neutrality under ambiguity.
Problem
Reliable VLM guidance for open-ended ancient Greek pottery questions remains difficult because appearance alone is insufficient.
Method
VaseMuseum combines 2D/3D museum interaction, multimodal reasoning, external knowledge acquisition, and inference-time source and response reliability controls.
Results
VaseMuseum improves citation validity, reduces hallucinations on knowledge-intensive queries, and produces more neutral responses under ambiguous evidence.
Takeaways & Limitations
Inference-time reliability control is a practical complement to stronger visual recognition for cultural-heritage deployment.
Takeaways & Limitations
Evidence quality depends on the availability, stability, and coverage of external web and museum sources, which may be incomplete or inconsistent.
Abstract
from arXiv · showhide
Vision-language models (VLMs) have made interactive digital museums increasingly feasible by connecting 3D digitization with natural-language artifact exploration. However, in cultural heritage domains such as ancient Greek pottery, reliable VLM assistance is limited by two challenges. First, open-ended interpretation requires grounding fine-grained 2D/3D visual evidence in specialized curatorial knowledge, yet the retrieval process may introduce weak sources and unverifiable references. Second, when the available evidence is incomplete, noisy, or ambiguous, VLMs often produce confident but unsupported answers instead of calibrated uncertainty. To address these challenges, we propose VaseMuseum, a lightweight and modular multimodal agent framework for intelligent digital museums of ancient Greek pottery. VaseMuseum combines an interactive virtual museum with VaseAgent, which supports both 2D images and 3D artifacts through multimodal perception, 3D-aware reasoning, external knowledge retrieval, and inference-time reliability control. Specifically, VaseAgent retrieves evidence from authoritative web and museum knowledge sources, and source-level control selects diverse and verifiable evidence before generation. Meanwhile, response-level control checks generated claims against the evidence pool and encourages neutral, evidence-bounded answers when support is insufficient or conflicting. Moreover, a training-free GRPO-style selection mechanism favors responses with valid references and calibrated confidence without updating the VLM backbone. Experiments in a realistic digital museum simulation show that VaseMuseum improves citation validity, reduces hallucinations on knowledge-intensive queries, and produces more neutral answers under ambiguity compared with search-enabled VLM baselines.
1 INTRODUCTION
VaseMuseum addresses the reliability challenges of VLM-based digital museum guides for ancient Greek pottery by controlling both retrieved evidence and answer uncertainty. It combines interactive 2D/3D artifact exploration with inference-time multimodal reasoning, source control, response control, and training-free response selection.
- Motivation: Digital museums combine 3D digitization, web visualization, and VLM interaction to make cultural artifacts inspectable and explainable.These technologies support preservation, access, education, and natural-language exploration beyond static catalog entries.
- Challenges: Reliable VLM museum guides remain difficult because questions require external historical or archaeological evidence, while retrieval and cultural-heritage evidence may be unreliable or incomplete.Low-quality pages, incomplete records, unverifiable citations, and disputed or underspecified evidence can lead to fluent but unsupported explanations.
- Framework: VaseMuseum connects interactive 2D/3D artifact exploration with a lightweight multimodal agent for ancient Greek pottery.Its virtual museum interface supports images and 3D artifacts, while VaseAgent performs perception, 3D-aware reasoning, knowledge retrieval, and answer verification at inference time.
- Reliability Control: VaseAgent controls evidence acquisition by retrieving authoritative web and museum sources and forming a compact, diverse evidence pool before generation.Source-level control suppresses unreliable retrieval results, while response-level control compares answers with collected evidence to improve uncertainty handling.
- Evaluation: Compared with search-enabled VLM baselines, VaseMuseum improves link validity, reduces hallucination on knowledge-intensive queries, and produces more neutral responses under ambiguous evidence.The evaluation covers visual-only, visual-plus-knowledge, and ambiguous questions, measuring answer accuracy, link validity, hallucination, and neutrality.
2 RELATED WORK
Prior work shows that general VLMs and existing cultural-heritage systems struggle with specialized 3D artifacts and open-ended museum questions. VaseMuseum combines an open-world virtual museum, 3D artifact handling without extensive retraining, internet-scale retrieval, and inference-time trust controls.
- General Vision–Language Models and VQA: General VLMs excel on broad VQA and captioning benchmarks but lack the specialized cultural knowledge needed for rare artifacts.Existing VQA benchmarks often fail to probe semantic understanding in art and cultural domains.
- 3D-Aware Vision–Language Methods: 3D-aware methods generate multiview captions, select informative views, or embed spatial geometry directly into VLMs.Examples include Cap3D, DiffuRank, and LLaVA-3D.
- Knowledge-Augmented Reasoning Agents: Knowledge-augmented agents combine planning, web search, tool use, and multimodal reasoning to ground answers in external evidence.VaseMuseum follows this direction but adds deterministic source and response control layers around each search round.
- Cultural Heritage and Digital Museum Systems: Digital museum systems increasingly provide interactive 3D reconstructions and AI-guided exhibits, but prior cultural-heritage demonstrations remain dataset-focused or statically programmed.VaseMuseum places an open-domain VQA agent inside a virtual gallery where visitors explore 3D pottery and ask questions answered with web evidence in real time.
- Research Gap and Positioning: VaseMuseum uniquely unifies an open-world virtual museum, complex 3D artifact understanding without extensive retraining, internet-scale retrieval, and inference-time trust measures.This combination distinguishes it from closed benchmark tasks, specialized-data-dependent 3D VLMs, and systems with unchecked answers or costly finetuning.
3 THE PROPOSED METHOD
VaseMuseum is a lightweight inference-time multimodal agent framework that combines virtual museum interaction, visual-language reasoning, external knowledge acquisition, and reliability control for ancient Greek pottery. VaseAgent grounds answers in visual evidence, selectively retrieves and verifies external sources, audits claims, and calibrates responses when evidence is insufficient or conflicting.
- Virtual museum interaction: VaseAgent supports open-ended artifact exploration through images, rendered views, and optional 3D meshes, covering questions about form, technique, iconography, chronology, provenance, and interpretation.The virtual museum is designed as an open-world environment rather than a closed VQA benchmark.
- Multimodal reasoning and retrieval: The reasoning core extracts pottery-specific visual cues and switches to iterative, tool-based retrieval when answers require knowledge beyond visible appearance.Retrieved web or museum evidence is fused with visual interpretation and conditioned on intermediate reasoning rather than single-shot retrieval.
- Source control: Source control filters malformed or inaccessible results, scores relevance and quality, removes redundancy, and selects at most Nsrc = 5 diverse evidence hits.Validity requires sval(h) ≥ 0.40 and a(h) ≥ 0.50; retained hits are normalized into evidence records for downstream verification.
- Response control: Response control decomposes drafts into claims, measures claim–evidence support, detects cross-source conflicts, and computes confidence from coverage, consistency, multi-source support, and evidence availability.When ψ < τconf with τconf = 0.52, VaseAgent enters uncertain mode and identifies unsupported or conflicting aspects; otherwise it returns normal-mode answers.
- GRPO-style selection: With additional inference budget, a training-free GRPO-style selector reranks stochastic controlled rollouts using link validity, claim–evidence support, neutrality, and conflict awareness.The selector does not update the VLM backbone and can be disabled under strict latency constraints.
4 EXPERIMENTS
Experiments show that VaseAgent’s reliability controls are most effective for knowledge-intensive museum queries requiring external evidence. Source filtering and response auditing improve citation validity, groundedness, neutrality, and hallucination resistance while preserving a balanced reliability profile.
- Evaluation protocol: Evaluation covers visual recognition, external knowledge grounding, and uncertainty-aware answering using shared visual inputs and queries across baseline methods.Search-enabled methods may invoke external tools, while GRPO-style selection samples K = 4 controlled trajectories by default.
- Main results: Search without reliability control increases hallucination and reduces groundedness, whereas VASEAGENT (Full) performs strongest on knowledge-intensive reliability.VASEAGENT (Full) obtains the lowest hallucination rate and highest groundedness and neutrality on V+K queries.
- Reliability metrics: Adding source and response control improves external link validity from 28.89 to 44.07, while the full GRPO-style selector further increases it to 60.00.Neutrality follows the same pattern under ambiguous evidence, indicating that reliability control discourages overconfident claims.
- Ablation: Ablations show that the full system has the strongest reliability profile, while accuracy-oriented selection can increase V+K accuracy at the cost of citation validity and neutrality.Source controls mainly affect link validity, whereas response controls mainly affect neutrality, addressing complementary failure modes.
- Ablation: Increasing GRPO-style group size K improves reliability but saturates beyond K = 4 while computational cost grows approximately linearly, motivating K = 4 as the default trade-off.The selector benefits from comparing more candidate trajectories, but additional candidates become less efficient after K = 4.
- Task-specific analysis: Task-specific and qualitative analyses find the full system most beneficial for V+K queries, where evidence retrieval and citation control directly affect answer quality.VaseAgent verifies retrieved evidence before generation and calibrates answers when support is insufficient, distinguishing observable attributes, supported facts, and uncertain interpretations.
5 LIMITATIONS AND FUTURE WORK
VaseMuseum remains limited by uneven external evidence, its focus on ancient Greek pottery, and reliability metrics that depend partly on operational definitions and human judgment. Future work will expand knowledge coverage and strengthen expert evaluation and agreement analysis.
- External web and museum evidence may be incomplete or inconsistent for specialized archaeological records, depending on source availability, stability, and coverage.
- The current implementation focuses on ancient Greek pottery, limiting the framework’s demonstrated artifact coverage.
- Reliability-oriented metrics, including link validity and neutrality, involve operational definitions and human judgment.
- Future work should incorporate curated institutional knowledge bases, broader heritage collections, and larger-scale expert evaluation with clearer agreement analysis.
6 CONCLUSION
VaseMuseum is an inference-time multimodal agent framework for trustworthy interaction with ancient Greek pottery in a virtual museum. It combines visual reasoning, external knowledge acquisition, evidence filtering, answer calibration, and training-free response selection to address reliability challenges in open-ended cultural-heritage dialogue.
- Framework: VaseMuseum provides an inference-time multimodal agent framework for trustworthy interaction with ancient Greek pottery in a virtual museum.The framework is presented as the paper’s central contribution.
- Framework: The framework combines visual reasoning with DeepResearch-style external knowledge acquisition for open-ended cultural-heritage dialogue.These capabilities support the acquisition and use of knowledge beyond visual input.
- Reliability control: Source-level evidence filtering, response-level answer calibration, and training-free GRPO-style response selection target reliability challenges without requiring model training.These controls operate at evidence, answer, and response-selection stages.