PaperlayerView exact revision on arXiv

Source-linked AI summary

PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding

Sławomir Dadas, Michał Perełkiewicz, Rafał Poświata, Małgorzata Grębowiec, Bartłomiej Jaworski, Izabela Woźniakowska

arXiv:2608.21853v1cs.CL

TL;DR

Multimodal models lack comprehensive evaluation in non-English cultural and linguistic contexts. PUMA addresses this gap with a 900-task Polish benchmark spanning cultural understanding and practical processing across images, audio, and documents; results show strong frontier performance in visual tasks but persistent challenges in audio and broader multimodal handling.

  • Problem

    Multimodal capabilities have not been comprehensively evaluated in cultural and linguistic contexts beyond English, despite models increasingly supporting images, audio, and documents.

  • Method

    PUMA is a manually crafted benchmark of 900 Polish-language tasks covering images, audio, documents, cultural QA, ASR, OCR, and structured extraction, evaluated with deterministic rules.

  • Results

    Current multimodal models differ substantially on Polish inputs: frontier commercial systems perform strongly in visual QA and document understanding, while audio understanding, especially non-speech sound and music interpretation, remains challenging.

  • Takeaways & Limitations

    PUMA provides a reproducible, cost-efficient framework for comparing culturally grounded and practical multimodal capabilities in Polish.

  • Takeaways & Limitations

    Rule-based evaluation requires structured tasks with clearly defined answers and verification rules, limiting highly open-ended tasks.

Abstract

from arXiv · show

Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understanding and generation have been extensively studied, multimodal data processing capabilities, particularly in the context of cultures and languages other than English, have not yet been evaluated comprehensively. In this paper, we propose PUMA (Polish Unified Multimodal Assessment), a novel benchmark of 900 hand-crafted tasks designed to probe the limits of multimodal models in the Polish cultural and linguistic context. The dataset evaluates both cultural understanding and practical skill in processing text, images, audio, and visually rich documents. Our extensive evaluation of frontier commercial models, open-weights models, and specialized smaller systems highlights a significant performance gap. While top commercial models achieve high scores in visual question answering, most models struggle with complex audio or document understanding. We open-source our evaluation framework to advance localized multimodal AI research.

1 Introduction

PUMA addresses the need to evaluate culturally grounded multimodal understanding beyond text, focusing on Polish language and culture across images, audio, and documents. It combines cultural question answering with practical multimodal processing tasks and releases evaluation software for reproducible research.

  • Motivation: Cultural context is necessary for interpreting many utterances, because literal language understanding alone can produce superficial or incorrect interpretations.Relevant context includes history, traditions, literature, art, social life, and popular culture.
  • Motivation: Existing cultural benchmarks have primarily focused on text, motivating multimodal evaluation that combines linguistic information with images, documents, and other media.
  • Benchmark: PUMA contains 900 hand-crafted tasks evaluating Polish linguistic and cultural competencies across images, audio, and documents.Each modality contains three categories targeting distinct knowledge areas or practical competencies.
  • Benchmark: The benchmark combines culturally grounded question answering with practical tasks including ASR, OCR, and structured information extraction.
  • Evaluation and resources: The authors evaluate more than a dozen open-weight and commercial multimodal models, over 50 additional VLMs on visual tasks, and smaller specialized ASR and OCR systems.
  • Evaluation and resources: The evaluation framework supports multimodal question-dataset construction and model evaluation, and is released as open-source software.

2 Related Work

Related benchmarks evaluate cultural knowledge across languages, regions, and modalities, but PUMA targets Polish cultural grounding while jointly covering images, audio, and documents. Its task range extends from culturally grounded QA to practical multimodal processing and structured extraction.

  • Cultural benchmarks: Cultural benchmarks assess everyday knowledge, regional culture, cultural dimensions, and cultural reasoning across diverse languages and regions.Examples include BLEnD, CulturalBench, CDEval, SeaEval, and MAPS.
  • Region-specific benchmarks: Region-specific datasets cover languages and cultures including Arabic dialects, Indonesian regions, Basque, Lao, Romanian, Korean, and Polish.
  • Multimodal benchmarks: Multimodal visual benchmarks test culturally diverse content, including visual question answering and robustness across linguistic rephrasings and visual modalities.Examples include CulturalVQA, SEA-VQA, ALM-bench, CVQA, and BLEnD-Vis.
  • Polish benchmarks: The Polish version of MMBench consists of questions translated from English and therefore emphasizes general knowledge rather than Polish cultural knowledge.
  • PUMA’s scope: PUMA jointly covers images, audio, and documents while combining culturally grounded QA with practical tasks such as language-specific OCR, ASR, and structured information extraction.

3 PUMA

PUMA organizes 900 tasks into nine categories across images, audio, and documents, covering cultural QA and practical processing skills. Its deterministic grading uses strict binary scores, auxiliary soft scores, and specialized verification rules.

  • Benchmark structure: PUMA contains 900 tasks arranged as nine categories of 100 tasks across images, audio, and documents.Six categories use question answering, while three assess ASR, OCR, and structured document extraction.
  • Audio categories: The audio categories test speech transcription, culturally grounded speech understanding, and recognition of non-speech sounds and music related to Poland.Recordings may include interference, background noise, complex language, dialects, ambient sounds, instruments, animals, jingles, and machinery.
  • Document categories: Document categories evaluate OCR from challenging images, visually rich document question answering, and extraction into schema-defined JSON.OCR examples include handwriting, low-quality scans, and tables whose structure must be preserved.
  • Grading: The benchmark’s strict score is binary, while soft scores award fractional credit for partial satisfaction and serve only as auxiliary metrics.For multi-rule questions, every verification rule must be satisfied; the final benchmark score is the mean strict score.
  • Task-specific metrics: The framework includes four QA verification-rule types and three specialized metrics for ASR, OCR, and structured extraction.
  • Question-answering verification: Question-answering verification normalizes Polish responses by removing nonalphanumeric characters, lowercasing letters, and lemmatizing words before matching.Annotators can disable normalization for specific rules.
  • Task-specific metrics: Structured extraction receives a correct score only when corresponding fields contain identical values, with a soft score based on the proportion of matching field values.

4 Evaluation

PUMA evaluates multimodal models across Polish visual, audio, and document tasks, revealing strong commercial performance in some areas but substantial weaknesses in structured document extraction and audio understanding.

  • Around 64% was the best score among tested OpenAI model pairs, below Gemini mainly in audio understanding and document-structure extraction.
  • Structured extraction is difficult because even minor JSON-format errors cause failed responses, producing high score variance.
  • Qwen3.5 models perform strongly in other document categories but struggle to extract structured data from documents.
  • ASR results: Whisper-Large-V3 performs only slightly worse than GPT-4o-Transcribe and GPT-Audio on Polish ASR, while high-resource-language performance does not necessarily predict less popular-language performance.
  • OCR results: Claude-Fable-5 achieves the best OCR result, while dedicated OCR models score significantly lower under the strict metric.

5 Conclusions

PUMA is a manually created Polish benchmark covering cultural question answering and practical multimodal skills, evaluated through deterministic rules. Results show substantial variation across models, with persistent challenges in non-speech audio and music interpretation.

  • PUMA contains manually created image, audio, and document tasks covering Polish linguistic and cultural understanding plus practical multimodal skills.
  • Deterministic rule-based verification enables reproducible and cost-efficient evaluation without external judges.
  • Frontier commercial models perform strongly in visual question answering and document understanding, but multimodal models still differ substantially on Polish inputs.
  • Audio understanding, especially non-speech sound and music interpretation, remains challenging for all evaluated systems.
  • English-centric or text-only benchmarks do not reliably capture limitations exposed by local language, cultural grounding, and realistic document and audio inputs.

Limitations

PUMA’s evaluation is constrained by its rule-based task design, selected modality and temporal coverage, and annotator composition. These choices support consistency and reproducibility but limit openness, recency, modality breadth, and representational coverage.

  • Rule-based evaluation: Rule-based evaluation excludes many highly open-ended tasks, although it enables fast, low-cost, consistent, deterministic, and reproducible assessment.Question-answering categories use rules, while ASR, OCR, and Structured Extraction use task-specific metrics.
  • Modality coverage: Video is absent because current model support is limited, though the framework can support future video extensions.Excluding video preserves the number of models that can be evaluated.
  • Temporal coverage: Most questions avoid rapidly changing information and short-lived trends, limiting coverage of very recent phenomena.Some Contemporary Life tasks include knowledge up to 2025.
  • Cultural diversity: Manual curation provides a flexible foundation for cultural coverage, but defining and measuring cultural diversity remains challenging.Annotators introduced diverse topics to reduce strong underrepresentation of specific areas.
  • Annotation process and consistency: The relatively homogeneous annotator group may limit perspectives specific to other generations, despite improving annotation consistency.Annotators shared similar age, experience, and language-model evaluation backgrounds.
  • Annotation process and consistency: Consensus review improved dataset consistency but provides no quantitative evidence that independent annotators would construct comparable tasks.Three annotators reviewed questions and verification criteria using predefined acceptance rules; standard agreement measures were difficult to apply.

Ethical considerations

The benchmark uses mostly public media, manually written original questions, and limited personal-data exposure. Its release strategy withholds the full dataset to reduce leakage, while the publication used AI tools only for language editing.

  • Most media files come from public sources, with a small number created by the authors or drawn from private collections.
  • Questions were manually written by the annotation team and are original.
  • Personal data in the dataset is limited to information about well-known public figures.
  • Only a subset of the dataset will be publicly released, while the full benchmark remains private to reduce data-leakage risk.A public leaderboard is planned and will be updated with new models, balancing transparency with benchmark integrity.
  • AI tools were used only for language editing, grammatical correction, and minor stylistic improvements, not for core scientific content or benchmark creation.

E Extended discussion of results

The extended analysis identifies audio and document understanding as major challenges, while showing that recording quality and speaker count affect speech tasks differently and refusals do not explain the full model gap.

  • Refusals: Eliminating refusals would improve OpenAI models’ scores but would not close their overall gap to Gemini-3.1-Pro, particularly in audio.GPT-Audio had refusals across Speech QA, Sound & Music QA, and ASR, while GPT-5.4 and GPT-5.5 also refused some image questions.
  • Audio understanding: 56% was the best model score in Sound & Music QA, the most challenging category, with hallucinated interpretations causing many errors.GPT-Audio refusals were also a substantial error source, while some models showed limited non-speech audio understanding.
  • Document understanding: All six examined models scored lower on scans and photographs than on born-digital documents, especially in Structured Extraction.Document date had no consistent effect, while receipts and invoices averaged 56.1% and handwriting 60.5%.
  • Audio understanding: 72.2% for single-speaker recordings versus 47.1% for multiple-speaker recordings shows that speaker count substantially affects ASR transcription.Speech QA scores were nearly identical across speaker counts, indicating less impact on general recording understanding.
  • Audio understanding: High-quality audio improved ASR and Speech QA scores by approximately 16-17 percentage points, but not Sound & Music QA.The analysis therefore identifies recording fidelity as less central for non-speech audio understanding.

F Error analysis

The error analysis separates genuine model errors, refusals, and rule-related false negatives, finding that deterministic verification occasionally rejects valid answers but has limited overall impact.

  • Error classification: Each zero-scored answer was classified as a model error, refusal, or rule-related false negative.The audit covered six QA categories and 600 questions per representative run.
  • Model-specific patterns: Gemini-3.1-Pro produced rule-related cases in History & Culture and Geography & Environment, while other runs showed cases across additional QA categories.The GPT model pair also produced substantially more refusals than the other evaluated systems.
  • Rule-related errors: 3-6 out of 600 questions per run were rule-related false negatives, corresponding to approximately 0.5-1.0% of QA tasks.These were semantically correct answers rejected by incomplete or overly restrictive verification rules.
  • Rule-related errors: 17 rule-related false negatives were identified in total, confirming that deterministic verification can reject valid answers occasionally.Most failed responses were genuine model errors rather than verification-rule failures.

G Evaluation on visual categories

Visual-category evaluation was necessary because few models support text, audio, and vision simultaneously; results show strong performance from leading families and useful smaller document-capable models.

  • Evaluation scope: Few models simultaneously support text, audio, and vision, so the evaluation of over 50 models focused on six vision-based categories.The results are broken down by model family in Table 5.
  • Model-family results: Gemini, GPT, and Claude families were strongest, followed by Kimi, Qwen3.5/3.6, and Gemma 4 models.This ordering was consistent with the previously discussed full-benchmark results.
  • Evaluation metrics: Table 5 reports strict scores and soft scores, highlighting the best score in each column and underlining the highest score within each model group.Soft scores are shown as smaller gray numbers in parentheses.
  • Dataset construction: The benchmark was manually crafted by six native Polish-speaking annotators using unified guidelines for cultural grounding, difficulty balance, diversity, and rule-based compatibility.The guidelines required multimodal analysis and encouraged multiple valid answer formulations.

H.1 Review and Revision Phase

The review and revision phase used native Polish-speaking reviewers and model evaluations to identify and correct task and verification problems, improving dataset consistency and automatic-evaluation reliability.

  • Review process: Three native Polish-speaking annotators reviewed task design, content quality, and evaluation criteria after initial annotation.The review targeted issues rather than calibrating question difficulty.
  • Review process: Reviewers used evaluations from Gemini-3.1-Pro, Gemini-3-Flash, GPT-5.4, and GPT-Audio to detect ambiguities, inconsistencies, and verification failures.Identified issues were discussed and resolved through iterative refinement.
  • Revision criteria: The guidelines required unambiguous rule-based verification, Polish cultural grounding where appropriate, multimodal-input analysis, and avoidance of duplicate questions.These constraints shaped both task inclusion and revision decisions.
  • Revisions: 28 tasks were removed and replaced, while 57 tasks received substantial revisions to prompts or evaluation conditions.Minor editorial corrections were applied continuously.
  • Outcome: The review phase improved dataset consistency, alignment with benchmark objectives, and the reliability of automatic evaluation.

H.2 Data Sources

The benchmark records provenance for source files and describes how those sources are distributed across categories and domains.

  • Annotators attach each source file’s original URL, author or creator, and license as provenance metadata.
  • Figure 5 summarizes source-file distributions across benchmark categories and frequently represented source domains.
  • Counts may exceed category totals because some files have multiple sources, including examples documented in Appendix J.

I Annotation app

The authors developed a multilingual application to coordinate multimodal benchmark construction, manage tasks and models, monitor annotation progress, and define evaluation criteria.

  • The dedicated application supports creating, managing, reviewing, and evaluating multimodal tasks while recording all annotator edits.
  • The application currently supports nine languages, including Polish, English, German, French, Dutch, Portuguese, Spanish, Italian, and Russian.
  • Its modules cover model configuration, annotator statistics, question management, and task editing.
  • Models are configured in structured JSON with properties such as endpoints, maximum tokens, required prefixes, and category assignments.
  • Statistics tools track weekly task completion, current-week task creation, category and tag filters, personalized task views, and searchable task content.
  • Task editing supports image, audio, and text inputs, metadata such as licenses and sources, questions or commands, and verification conditions for correct answers.

J Sample questions

The sample questions span Polish history, culture, geography, contemporary life, audio transcription, OCR, and sound-based cultural understanding, with task-specific answer checks.

  • Geography and environment: Another geography task asks models to identify a medieval fortress, its builders, and the year it was taken over by the Kingdom of Poland.
  • History and culture: History and culture questions ask models to identify Polish video games from screenshots and provide titles in a specified order.
  • Contemporary life: Contemporary-life questions require identifying soups from photographs and returning names in a prescribed numbered-list format.
  • Geography and environment: Geography questions test ordering Polish mountain peaks by elevation and cities from north to south, with explicit answer sequences or lists.
  • Audio understanding: Audio examples include general ASR, a Silesian-dialect monologue with statement verification, and a recording about Jan Czochralski’s invention.
  • Sound and music QA: Sound-and-music questions connect Polish folk instruments and animal sounds with regions associated with those cultural practices.
  • Other multimodal tasks: Additional examples ask about sheep and breeding regions, apply OCR to Polish text, and identify the Polish ruler whose mint struck a photographed coin.
Loading 2608.21853v1…