Source-linked AI summary
Tanawpamana: Benchmarking Vision-Language Models on Philippine Cultural Heritage Sites
Ivan Yuri De Leon, Hiroki Ouchi, Sakriani Sakti
TL;DR
Evidence on VLM recognition and knowledge of specific cultures remains limited, particularly for the underrepresented Philippines. Tanawpamana introduces a heritage-site dataset and benchmark, finding that models struggle more with fine-grained site discrimination and image-based geographic inference than with coarse recognition and text-based inference.
Problem
Whether VLMs can recognize and represent specific cultures remains poorly understood, while Philippine heritage is largely absent from evaluation.
Source: §1 · p. 1Method
Results
Models distinguish sites better across different heritage categories than visually similar sites within a category, and infer geographic area more accurately from names than images.
Source: §9 · p. 8Takeaways & Limitations
Evaluating visual recognition alongside cultural knowledge gives a more comprehensive view of VLM capabilities on underrepresented cultural domains.
Source: §9 · p. 8Mindanao is underrepresented in the dataset, and more sites, including less-documented ones, are needed to settle whether models recognize its heritage as well as elsewhere.
Source: §3.3 · p. 4, §7.1 · p. 7
Abstract
from arXiv · showhide
Vision-language models are increasingly used to identify and describe places, yet their recognition and knowledge of specific cultures remain poorly understood, particularly in underrepresented regions such as the Philippines. This study introduces the Tanawpamana Dataset, a repository of Philippine heritage site images with metadata based on the Talapamana registry of the National Commission for Culture and the Arts of the Philippines. We also introduce the Tanawpamana Benchmark, comprising 4,046 tasks designed to evaluate models' recognition and knowledge of heritage sites. Evaluating seven vision-language models, we find that they struggle more with fine-grained visual discrimination among similar heritage sites than with coarse recognition, and that their geographic knowledge is weaker when inferred from image inputs than from text.
1. Introduction
Evaluating VLMs on Philippine heritage matters as they increasingly shape how cultural places are interpreted, yet Philippine cultural heritage remains largely absent from evaluation. Tanawpamana addresses this gap with a heritage-image dataset and benchmark for recognition and site knowledge.
1. Introduction
- Philippine heritage remains largely absent from VLM evaluation, and existing resources do not assess recognition and knowledge of tangible Philippine heritage sites.Prior resources aggregate Philippine results, include few Philippine images, or do not target heritage landmarks.Source: §1 · pp. 1–2
- VLMs’ ability to recognize and retrieve information about Philippine heritage from visual and textual inputs therefore remains unclear.The paper frames culturally grounded datasets and benchmarks as necessary to assess this capability.Source: §1 · pp. 1–2
- The study introduces 499 Philippine heritage-site images with Talapamana-enriched metadata and a 4,046-task benchmark separating visual recognition from factual, geographic, and designation knowledge.The benchmark also pairs image- and text-based tests of geographic knowledge.Source: §1 · p. 2
2. Related Work
Related work distinguishes entity-level visual recognition from cultural knowledge and motivates evaluation that tests both. The Talapamana registry supplies institutionally recognized heritage classifications for grounding the dataset.
2.1. Cultural and Vision-Language Benchmarks
- Earlier benchmarks extend visual question answering toward multimodal reasoning, while cultural and entity-level work distinguishes identifying specific sites from understanding associated knowledge.Tanawpamana is designed to test both specific Philippine heritage-site identification and factual, geographic, and designation knowledge.Source: §2.1 · p. 2
2.2. Talapamana Registry
- Talapamana catalogs recognized cultural properties under six designation categories, whose levels of legal recognition and protection inform dataset classification.The registry-based labels reflect institutional judgments of cultural significance rather than model-derived or crowd-generated labels.Source: §2.2 · p. 2
3. Tanawpamana Dataset
Tanawpamana extends Talapamana’s registry of Philippine tangible immovable heritage with curated site images and standardized metadata. Its collection and distribution reflect both deliberate coverage choices and uneven online documentation.
3. Tanawpamana Dataset
- Tanawpamana pairs manually gathered representative images with geographic, category, and designation metadata derived from the Talapamana registry.The registry was filtered to Tangible Immovable Heritage sites in the Philippines; the dataset contains 499 publicly available images.Source: §3 · p. 2, §3.1 · p. 2
3.1. Data Collection
- Image collection prioritized four formal designations and exhaustively gathered images for those sites across Metro Manila, Luzon, Visayas, and Mindanao.Low-designation sites were then added to fill category–area gaps, using preliminary model labels that were later manually verified and corrected.Source: §3.1 · pp. 2–3
3.2.1. Heritage Category
- The manually assigned taxonomy has ten functional heritage categories, designed to support benchmark construction and visually similar distractors.When function is ambiguous, assignment prioritizes visual appearance because it determines distractor difficulty.Source: §3.2.1 · p. 3
3.2.2. Talapamana-Derived Features
- Metadata enrichment adds designation flags, expanded alternative names, standardized site names, and a QUESTION STEM field based on historical descriptions.The categories are a study-specific taxonomy, not part of the official NCCA classification.Source: §3.2.2 · p. 3, §3.2.1 · p. 3
3.2.3. Image Licensing
- Every image retains source licensing information; 375 images are CC BY-SA, 97 are public domain or CC0, and seven government works are for non-commercial research use.The collection also includes 19 CC BY images and one attribution-only image.Source: §3.2.3 · p. 3
3.3. Data Characterization
- The category–area distribution is nonuniform, reflecting varying levels of online documentation for heritage sites.Table 1 summarizes the site counts across categories and geographic areas.Source: §3.3 · p. 3, Table 1 · p. 4
- 97 of 499 sites (19.4%) are Religious Buildings, with 89.7% in the high-designation subset versus 41.3% of all sites.The collection strategy’s exhaustive inclusion of high-designation sites primarily drives this concentration.Source: §3.3 · pp. 3–4, Table 1 · p. 4
- Mindanao is underrepresented in image availability and designation: only 6 of 58 sites (10.3%) are high-designation, versus 36.6%–50.7% elsewhere.The dataset’s category–area collection shortfalls also occur primarily in Mindanao.Source: §3.3 · p. 4, Figure 3 · p. 4
4. Tanawpamana Benchmark
Tanawpamana evaluates vision-language models’ recognition of Philippine heritage sites and retrieval of related cultural knowledge. Its task suites use image–name matching and multiple-choice questions grounded mainly in heritage records.
4. Tanawpamana Benchmark
- The benchmark tests site identification and factual, geographic, and designation-related knowledge, but not culture’s interpretive or experiential dimensions.The authors regard entity-level recognition and knowledge as necessary, though not sufficient, conditions for cultural understanding.Source: §4 · p. 4
4.1. Task Suite
- The benchmark contains 4,046 multiple-choice tasks: 1,960 Recognition Suite tasks and 2,086 Knowledge Suite tasks.Items preserve metadata for analysis by heritage category, geographic area, and designation status.Source: §4.1 · p. 4, §4 · p. 4, Table 2 · p. 4
4.1.1. Recognition Suite
- Recognition tasks test image-to-name and name-to-image matching at easy cross-category and hard same-category difficulty levels.Same-category distractors require finer discrimination among visually similar sites than broad visual-category cues do.Source: §4.1.1 · p. 4
4.1.2. Knowledge Suite
- Knowledge tasks test historical-description matching and site identification from a heritage category and geographic area.General Information uses historical context; Short Description uses coarser categorical and regional cues.Source: §4.1.2 · pp. 4–5
- Other Knowledge Suite tasks test geographic knowledge from site names or images and recognition of sites associated with official designations.The paired location formats permit comparison of explicit geographic knowledge with inference from visual appearance.Source: §4.1.2 · p. 5
4.2. Benchmark Construction
- Except for General Information, tasks are deterministically built from TPTI; General Information questions use an LLM-assisted, rule-filtered pipeline and manual review.The 365 General Information items were manually reviewed for specificity, factual grounding, and a single correct answer.Source: §4.2 · p. 5
- Recognition distractors are randomly sampled by difficulty, while location questions use the same four geographic options for text and image variants.The Miscellaneous category is excluded from hard recognition tasks because it is not a consistent visual category.Source: §4.2 · p. 5
5. Evaluation
The evaluation tested seven vision-language models in a zero-shot multiple-choice setting and scored their final answer choices. It also defined separate aggregate scores for easy and hard recognition tasks and treated invalidly formatted responses as incorrect.
5.1. Models
- Seven proprietary, open-weight, and legacy vision-language models were evaluated zero-shot, with optional reasoning modes disabled.Models were instructed to provide a sentence of reasoning, but only the extracted final answer was scored.Source: §5.1 · p. 5
5.2. Scoring
- Multiple-choice accuracy compared each extracted final answer label with the ground truth; generated reasoning traces were retained for error analysis, not scoring.The evaluation stored the complete response as well as the extracted answer.Source: §5.2 · p. 5
- Separate Recognition (Hard) and Recognition (Easy) aggregate scores summarize performance at each difficulty level using the respective Image-to-Name and Name-to-Image accuracies.The supplied scoring text identifies the component accuracies for each level.Source: §5.2 · p. 5
5.3. Error Parsing
- Responses without a valid answer in the required format were scored incorrect; most parsing failures involved rejecting the options or overly verbose responses.The maximum output length was 1,024 tokens.Source: §5.3 · p. 6
6. Results
Across the seven task clusters, models generally scored above chance, but performance varied by task. Recognition was harder with similar-site distractors, while designation knowledge and image-based geographic reasoning were particularly challenging.
6.1. Recognition
- Gemini-3.5-Flash led both recognition tasks; hard-task accuracy ranged from 58.5% to 92.9%, versus 90.0–99.2% on easy tasks.On Recognition (Hard), Gemini-3.5-Flash scored 92.9%, compared with the next-best 69.4%; hard-task accuracy correlated with easy-task accuracy (r = 0.95).Source: §6.1 · p. 6
- Recognition accuracy varied little by geographic area: hard-task averages ranged from 66.7% in Luzon to 72.3% in the Visayas, with no detectable disadvantage for Mindanao.The authors caution that Mindanao has comparatively fewer sites.Source: §6.1 · p. 6
- High-designation sites scored 61.9% versus 74.1% for low-designation sites on Recognition (Hard), but category standardization narrowed the gap to 5.7 points.The authors attribute 6.6 points, or 53.7% of the original gap, to category composition.Source: §6.1 · p. 6
6.2. Knowledge
- Gemini-3.5-Flash led all five Knowledge tasks, while open-weight and proprietary models did not consistently separate by model type.Llama-4-Maverick averaged 66.0%, on par with GPT-5.4-Mini at 66.1%; individual model differences mattered more than model type.Source: §6.2 · p. 6
- Designation was the hardest Knowledge task: accuracy ranged from 34.3% to 59.1%, and only Gemini-3.5-Flash exceeded 50%.The authors suggest periodic updates to official status may partly contribute to this difficulty.Source: §6.2 · p. 6
- Every model scored lower on Location (Image) than Location (Text), with declines ranging from 7.6 to 24.0 percentage points.The tasks test the same site-area knowledge but provide the site as an image or name, respectively.Source: §6.2 · p. 6
6. Results
- Models performed above chance on nearly all tasks, with easier recognition outperforming hard recognition and designation yielding the lowest overall performance.Figure 4 summarizes accuracy across the Recognition and Knowledge suites.Source: §6 · p. 6, Figure 4 · p. 7
7. Discussion
The discussion finds that fine-grained visual discrimination, rather than coarse recognition, is a major heritage-recognition challenge, while geographic knowledge is more reliably retrieved from site names than images. Regional comparisons and error analysis qualify these findings: Mindanao parity is not established, and recognition and knowledge failures reflect different error patterns.
7.1. Regional Representation Gaps
- On Recognition (Hard), Mindanao and other sites both average 69.1%, but limited power and publicly available imagery prevent interpreting this as regional parity.Mindanao contributes 52 sites versus 429 elsewhere; the authors conclude that more Mindanao sites, including less-documented ones, are needed to settle the question.Source: §7.1 · p. 7
7.2. Fine-Grained Discrimination as a Bottleneck
- Every model performs worse on Recognition (Hard) than Recognition (Easy), indicating that same-category visual similarity—not coarse category recognition—is the central discrimination challenge.Gemini-3.5-Flash’s smaller Easy-to-Hard drop suggests its advantage lies in more precise visual discrimination rather than broader landmark familiarity.Source: §7.2 · p. 7
7.3. The Gap Between Visual and Textual Geographic Knowledge
- Heritage-site geographic area is retrieved substantially more reliably from a site name than from an image, suggesting geographic knowledge is more textually associated than visually grounded.The authors describe modality alignment as an open challenge distinct from increasing factual knowledge capacity.Source: §7.3 · pp. 7–8
7.4. Error Analysis
- In Gemini-3.5-Flash’s sampled errors, recognition failures stemmed chiefly from visually similar-site confusion, while knowledge failures more often misattributed attributes to heritage entities.Location (Image) errors typically followed visual misidentification; text-based location errors instead assigned the site to the wrong area.Source: §7.4 · p. 8
8. Limitations
The limitations concern how sites and labels are defined and documented, as well as the benchmark’s coverage and evaluation scope. These constraints affect interpretation of category-based recognition, time-sensitive designation results, and comparisons with people or regionally adapted models.
8.1. Tanawpamana Dataset
- Tanawpamana’s heritage categories are benchmark-oriented groupings rather than official classifications, and some site assignments are inherently ambiguous.The NCCA Talapamana registry does not provide a functional taxonomy for computer vision applications.Source: §8.1 · p. 8
- The dataset is a time-stamped snapshot of the Talapamana registry, so designation fields may no longer reflect current site status.The registry was accessed in January 2026 and is periodically updated.Source: §8.1 · p. 8
8.2. Tanawpamana Benchmark
- Each site has only one image, so recognition performance may partly reflect viewpoint, lighting, resolution, or framing rather than site familiarity.The authors suggest adding images per site to separate image-specific factors from site-level recognition.Source: §8.2 · p. 8
- The evaluation covers general-purpose models, not regionally adapted models such as SEA-LION or Aya Vision.The authors identify evaluating such models as future work to test whether regional training improves recognition and knowledge.Source: §8.2 · p. 8
- One researcher assigned heritage categories, and inter-annotator agreement was not assessed, so Recognition (Hard) difficulty may partly reflect category design.Hard-task distractors are sampled using these categories.Source: §8.2 · p. 8
- Designation labels may become outdated as Talapamana changes, so future comparisons should refresh them or exclude the Designation task.Official designation status can change as the registry is updated.Source: §8.2 · p. 8
- Without a human performance baseline, the benchmark cannot show how model recognition and knowledge compare with human performance.Source: §8.2 · p. 8
9. Conclusion
The paper introduces an image-and-metadata dataset and benchmark for Philippine heritage sites, then reports challenges in fine-grained recognition and image-based geographic inference. The findings highlight model limitations and the value of evaluating visual recognition alongside cultural knowledge in underrepresented domains.
9. Conclusion
- Tanawpamana combines Philippine heritage-site images with Talapamana-derived information and evaluates visual recognition and cultural knowledge; its dataset reflects Mindanao’s underrepresentation in heritage documentation.Source: §9 · p. 8
- Models recognize sites more successfully against different-category distractors than visually similar same-category sites, and infer geographic area more accurately from a site name than an image.Source: §9 · p. 8
- The findings highlight current VLM limitations on culturally specific heritage sites and the value of evaluating both visual recognition and cultural knowledge.Source: §9 · p. 8
11. Generative AI Use Disclosure
The authors report using Claude for manuscript proofreading and editing and Claude Code as a coding assistant. They state that AI was not used for research design or analysis unless otherwise stated, and that authors independently verified quantitative results, claims, and figures.
11. Generative AI Use Disclosure
- Claude assisted with proofreading and editing, and Claude Code with coding; authors independently verified quantitative results, claims, and figures.The disclosure states that no AI-generated content was used for research design or analysis unless otherwise stated.Source: §11 · p. 9