PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding

1 Introduction

• Cultural context matters because literal understanding alone can yield superficial or incorrect interpretations when models lack familiarity with cultural references (Hershcovich et al., 2022).As multimodal models develop, evaluations need to test whether they combine linguistic information with images, documents, and other media, beyond earlier text-focused cultural benchmarks (Pawar et al., 2025).Source: §1 · p. 1
• PUMA comprises 900 hand-crafted Polish-language tasks across images, audio, and documents, combining culturally grounded question answering with practical multimodal processing skills.The practical tasks include ASR, OCR, and structured information extraction; the authors describe PUMA as the first Polish-language benchmark to jointly evaluate image, audio, and document understanding in one framework.Source: §1 · p. 1
• The authors evaluate over a dozen commercial and open-weight multimodal models and over 50 additional visual language models on visual tasks.They also compare large multimodal models with smaller specialized models on ASR and OCR.Source: §1 · pp. 1–2
• The work releases open-source software for building multimodal question datasets and running evaluations.The authors developed the software as part of their benchmark work.Source: §1 · p. 2

Evaluation approaches

• LLM evaluation commonly uses deterministic checks for closed-ended tasks, human or LLM-judge assessment for open-ended generation, and programmatic rules for specific capabilities.The cited work notes that LLM judges scale well but can exhibit evaluator bias, while human evaluation is cost-intensive but highly aligned.Source: §2 · p. 2

Cultural and multilingual benchmarks

• Cultural and multilingual benchmarks include BLEnD (Myung et al., 2024), CulturalBench, and Polish PLCC (Dadas et al., 2025), which uses rule-based evaluation.PUMA uses a PLCC-inspired rule-based approach for QA categories, extended to multimodality.Source: §2 · p. 2

Multimodal benchmarks

• Prior multimodal work covers culturally diverse visual understanding, generation, and audio; Polish PCVB focuses mainly on identifying cultural objects and their country of origin.Available descriptions of PCVB are limited, while the paper characterizes PUMA as using more varied task types.Source: §2 · p. 2

Polish benchmarks

• The Polish version of MMBench (Statkiewicz et al., 2026) translates English questions and emphasizes general rather than Polish-culture-grounded knowledge.Source: §2 · p. 3

PUMA’s scope

• PUMA spans visual and speech QA, Polish OCR and ASR, document understanding, and structured information extraction across three modalities.The paper contrasts this task range with benchmarks described above that cover narrower scopes.Source: §2 · p. 3

3 PUMA

• PUMA contains nine categories of 100 tasks each: six assess knowledge and reasoning through question answering, while three test practical skills in ASR, OCR, and structured extraction.Evaluation is deterministic and requires no external judge.Source: §3 · p. 3
• A custom application supported question creation, editing, model integration, verification, evaluation, and log review; subsequent review removed 28 questions and substantially modified 57.Annotators reviewed evaluation logs from selected frontier multimodal models and raised issues about question content or verification criteria.Source: §3 · p. 3

3.1.1 Image categories

• Image categories cover Polish history and culture, contemporary life and pop culture, and geography and the environment.Their scope includes art and historically significant people or objects, modern media and social issues, and natural landmarks, infrastructure, cities, and socioeconomic topics.Source: §3.1.1 · pp. 3–4

3.1.2 Audio categories

• Audio categories test speech transcription, speech understanding, and recognition and interpretation of non-speech sounds related to Poland.ASR samples range from about a minute to several minutes and may include interference, background noise, complex language, or dialects; speech QA also probes intentions, emotions, and broader cultural knowledge.Source: §3.1.2 · p. 4

3.1.3 Document categories

• Document tasks assess OCR, question answering over Polish documents, and structured information extraction into JSON.OCR includes handwritten text, low-quality scans, and tables; document QA often requires interpreting visually rich material such as tables, charts, or infographics.Source: §3.1.3 · p. 4

3.2 Grading

• The benchmark uses strict binary scores as its primary metric and auxiliary soft scores for partial correctness; a response must satisfy every rule attached to a question.The final benchmark score is the mean of strict scores, reported as the percentage of correct responses.Source: §3.2 · p. 4

3.2.1 Verification rules for question answering

• Question-answering verification normalizes Polish text by default and applies include, exclude, regex, or order rules to responses.Rules can define phrase or pattern requirements, and soft scores measure partial satisfaction; normalization removes punctuation, lowercases text, and lemmatizes words, but can be disabled per rule.Source: §3.2.1 · pp. 4–5

3.2.2 Verification rules for other tasks

• ASR and OCR use task-specific metrics, while structured extraction checks whether returned JSON matches a reference object.ASR strict scoring requires WAcc above 0.9; OCR combines text WAcc and table TEDS scores by character count and applies a 0.9 strict-score threshold. Structured-extraction soft scores reflect the ratio of matching field values.Source: §3.2.2 · pp. 5–6

4 Evaluation

• Evaluation began at temperature 0, raised it by 0.3 when reasoning models entered repetition loops, and used maximum image-token budgets where available.Other generation parameters followed model-author defaults; the maximum image budgets were intended to preserve detail, especially for document categories.Source: §4 · p. 6

4.1 Main results

• Nearly 80% was Gemini-3.1-Pro’s highest average score across categories, while the best tested OpenAI model pair scored around 64%.The OpenAI pairs fell short primarily in audio understanding and document structure extraction; the paper attributes some lower knowledge-category scores to refusals.Source: §4.1 · p. 7
• Sound & music QA was the hardest category, with a highest score of 56%, while strict metrics and JSON-format errors also depressed scores and increased variance in structured extraction.Qwen3.5 models were particularly weak at extracting structured document data despite strength in other document-related categories.Source: §4.1 · p. 7

4.2 ASR and OCR evaluation

• The ASR and OCR comparison tested small specialized systems against larger multimodal models, adding output normalization because most specialized models do not support instructions.Normalization adapted formats such as numbers and tables to expected output; soft BN denotes raw soft scores before normalization.Source: §4.2 · pp. 7–8
• ElevenLabs achieved the best ASR result, Gemini 2.5 and 3.1 were similar, and Whisper-Large-V3 was only slightly worse than GPT-4o-Transcribe and GPT-Audio.Newer open-weight ASR models lagged behind Whisper-Large-V3, suggesting that performance on high-resource languages may not predict performance on less popular ones.Source: Table 2 · p. 7, §4.2 · p. 8
• Claude-Fable-5 achieved the best OCR result; Gemini models also scored highly, while dedicated OCR models scored substantially lower under the strict metric.Chandra-OCR-24 was the best dedicated model on the strict metric, while four other OCR models exceeded 70% on the soft metric.Source: §4.2 · p. 8, Table 3 · p. 8

5 Conclusions

• PUMA is a manually created benchmark of image, audio, and document tasks covering culturally grounded question answering and practical skills.It targets multimodal language models in the Polish linguistic and cultural context.Source: §5 · p. 8
• Frontier commercial models perform strongly in visual question answering and document understanding, but non-speech sound and music interpretation challenge all evaluated systems.The findings suggest that English-centric or text-only benchmarks alone cannot reliably assess multimodal AI progress.Source: §5 · p. 8
• Deterministic rule-based verification enables reproducible, cost-efficient evaluation without external judges.Source: §5 · p. 8

• Rule-based evaluation supports fast, low-cost, consistent, and largely deterministic assessment, but requires structured questions and limits highly open-ended tasks.Question answering uses rule-based evaluation; ASR, OCR, and Structured Extraction use task-specific metrics.Source: Limitations · p. 9
• PUMA omits video because current multimodal model support is limited, reducing the number of evaluable models, although its framework can support future video extensions.Source: Limitations · p. 9
• Most questions avoid rapidly changing information and short-lived trends, improving long-term stability while limiting coverage of recent phenomena.Some Contemporary Life tasks include knowledge up to 2025.Source: Limitations · p. 9
• Manual curation and annotator topic selection aim to avoid strong underrepresentation across cultural domains, but cultural diversity remains difficult to define and measure.The authors describe the current approach as a flexible foundation for future extension.Source: Limitations · p. 9
• Annotators were relatively homogeneous in age and experience, potentially limiting generational perspectives; consensus review by three annotators improved consistency but did not show that independent annotators would construct comparable tasks.Standard agreement measures were considered difficult to apply because annotators created complete multimodal tasks rather than assigning labels to shared examples.Source: Limitations · p. 9

Ethical considerations

• Most media come from public sources, personal data is limited to well-known public figures, the full benchmark remains private, and AI tools aided language editing only.Only a subset will be publicly released, alongside a regularly updated leaderboard.Source: Ethical considerations · p. 9

A Verification rules in detail

• Unlike question-answering categories, ASR, OCR, and structured extraction use one universal category-wide prompt specifying the task and response format.Source: A Verification rules in detail · p. 16

A.1 Automatic speech recognition (ASR)

• ASR uses WAcc = max(0, 1 − WER), with strict passing at 0.9 and soft scores equal to WAcc after case-insensitive alphanumeric-word comparison.For task-only systems that may output digits, a Gemini 3 Flash normalizer converts numeric values to words to match reference transcripts.Source: §A.1 · p. 16

A.2 Optical Character Recognition (OCR)

• OCR combines text WAcc with table TEDS; cell substitutions account for colspan and rowspan mismatches and give partial credit for text similarity, while strict acceptance requires 0.9.Text-block permutations accommodate ambiguous reading order; tables are matched independently of order, size-weighted, and missing tables score zero.Source: §A.2 · pp. 16–17

A.3 Structured extraction

• Structured extraction uses per-record JSON schemas and permits only flat objects of primitive fields or primitive arrays; invalid JSON scores zero, while valid outputs receive field-value similarity scores.Numbers and booleans must match exactly; strings are normalized, and array elements are order-independent with unmatched objects scoring zero and optimal object alignment.Source: §A.3 · pp. 18–19

B Image categories

• Polish image categories test cultural knowledge using a single image and question prompt, combining multiple images into a collage when reasoning across them is required.Instructions specify answer format, supporting instruction-following evaluation and rule-based parsing.Source: B Image categories · p. 19

B.1 History and culture

• History-and-culture items cover Polish heritage, traditions, landmarks, art, artifacts, and material culture; text requiring character recognition belongs in document categories, while food requires historical context.Contemporary food images are excluded; general library views and historical book covers are exceptions to the exclusion of written texts.Source: §B.1 · pp. 19–20

B.2 Contemporary life; B.3 Geography and environment

• Polish image categories cover contemporary post-1989 mass and digital culture, everyday visual practices and cuisine, as well as Polish geography, ecosystems, native species, landmarks, climate, geology, and infrastructure.Source: §B.2 · p. 20, §B.3 · p. 20

C.1 Automatic speech recognition

• Audio tasks span ASR, Speech QA, and Sound & Music QA; mean durations are 82 seconds overall, 119 for ASR, 73 for Speech QA, and 54 for Sound & Music QA, with ASR testing noisy, multi-speaker, dialectal, unclear, or low-quality speech.ASR uses one universal prompt and includes both clear studio recordings and amateur recordings with interference or difficult diction.Source: C Audio categories · pp. 20–21, §C.1 · p. 21

C.2 Speech question answering

• Speech QA ranges from Polish speech comprehension to culturally contextualized quote attribution, voice-based identification, pop-culture recognition, reasoning, and analysis of intent, emotion, argumentation, and dialogue structure.Questions may also test dialect identification and linguistic correctness, including finding and correcting errors in recorded speech.Source: §C.2 · pp. 21–22

C.3 Sound and music question answering

• Sound and music QA covers non-speech audio tied to Poland, including music, instruments, animal and ambient sounds, and institutional, brand, or television jingles; speech understanding cannot be required.A sound need not itself be Polish-related if the question unambiguously supplies the Polish context.Source: §C.3 · p. 22

D Document categories

• Document tasks use graphical document fragments containing text and elements such as tables, charts, photos, diagrams, or handwriting, tagged by document type, creation period, and image quality.Types range from books, newspapers, and official documents to advertisements, receipts, presentations, natural-environment text, and web or UI screenshots.Source: D Document categories · p. 23

D.1 Optical character recognition

• OCR tests faithful transcription and table preservation on difficult images, including handwriting, low resolution, blur, poor lighting, perspective distortion, and degraded paper.Evaluation checks transcription fidelity as well as preservation of table structure and content.Source: §D.1 · p. 23

D.2 Document QA

• Document QA tests text recognition, interpretation, and inference across textual and graphical elements, emphasizing multi-level relationships while avoiding unnecessary reliance on external knowledge.Examples typically show a document fragment, often a single page, with a prompt; OCR uses a shared instruction, while other document tasks have individual prompts.Source: D Document categories · p. 23, §D.2 · p. 24

D.3 Structured extraction

• Structured extraction maps document content into user-schema JSON, covering practical records, explicit structures, narratives, and fields distributed across text and visual elements.Each task pairs a document image and extraction instruction with a defined JSON schema and ground-truth object.Source: §D.3 · p. 24

Refusals of OpenAI models

• OpenAI models were the only evaluated systems for which refusals noticeably affected results, and removing them would not close the overall gap to Gemini-3.1-Pro, particularly in audio.GPT-Audio had 15 refusals in Speech QA, 10 in Sound & Music QA, and 3 in ASR; GPT-5.4 had 11, 5, and 1 in Contemporary Life, History & Culture, and Geography & Environment, versus GPT-5.5’s 7, 3, and 1, respectively.Source: E Extended discussion of results · pp. 24–25

Challenges in non-speech audio understanding

• Sound & Music QA was the most challenging category, with the best model achieving only 56%, and most errors arising from hallucinated interpretations.The discussion reports unsatisfactory performance across evaluated models, consistent with strongest-model scores of approximately 50% on MMAU (Sakshi et al., 2025); GPT-Audio refusals also contributed substantially, while Voxtral sometimes failed basic sound discrimination.Source: E Extended discussion of results · p. 25

Generation temperature

• When reasoning models return empty answers after repetitive loops, the evaluation restarts tasks with temperature raised by 0.3 until a valid response is obtained.The risk increased as model size decreased and occurred regularly for smaller Qwen3.5 variants and smaller Gemma 4 models; this behavior was rare for frontier and large open-weight models.Source: E Extended discussion of results · p. 25

Strict and soft scoring

• Strict scores binarize task success, while soft scores preserve metric values for partially correct responses; the paper reports both.OCR and ASR use a conservative 0.9 threshold for success; soft-score distributions differ across task types, so cross-category comparisons require caution.Source: E Extended discussion of results · p. 25

Document categories

• All six representative models scored lower on scans and photographs than on born-digital documents, while document date had no consistent effect.Books averaged 78.7%, receipts and invoices 56.1%, and handwriting 60.5%; handwriting performance varied substantially across models, and document-type comparisons are limited by small groups.Source: E Extended discussion of results · p. 25

Audio categories

• High-quality audio improved ASR and Speech QA scores by approximately 16–17 percentage points, while multiple speakers reduced ASR from 72.2% to 47.1%.Speech QA scores were nearly identical for single- and multiple-speaker recordings, and Sound & Music QA did not show the recording-quality pattern, suggesting fidelity is not the primary limit on non-speech audio understanding.Source: E Extended discussion of results · pp. 25–26

Error analysis

• The audit manually inspected every zero-scored response from four representative runs, classifying failures as model errors, refusals, or rule-related false negatives.The runs were Gemini-3.1-Pro, Gemini-3-Flash, MiMo-V2.5, and the GPT-5.5 + GPT-Audio model pair; each covered 600 questions across six QA categories.Source: F Error analysis · p. 26
• Gemini-3.1-Pro had three rule-related cases in History & Culture and Geography & Environment; the GPT pair had six across three categories.Gemini-3.1-Pro's cases comprised two in History & Culture and one in Geography & Environment. The GPT pair's comprised three in Geography & Environment, two in History & Culture, and one in Contemporary Life.Source: F Error analysis · p. 26
• Gemini-3-Flash had four rule-related cases across four categories, while MiMo-V2.5 had four across Sound & Music QA, History & Culture, and Speech QA.Flash had one case each in Contemporary Life, Document QA, Geography & Environment, and Sound & Music QA; MiMo had two in Sound & Music QA and one each in History & Culture and Speech QA.Source: F Error analysis · p. 26
• Rule-related false negatives affected 3–6 of 600 questions per run (0.5–1.0%), totaling 17 cases, suggesting limited impact on reported scores.The audit examined four representative runs across six question-answering categories.Source: F Error analysis · p. 26
• Most failed responses were genuine model errors, while the GPT model pair produced substantially more refusals than the other evaluated systems.The audit classified failures as genuine model errors, refusals, or semantically correct answers rejected by incomplete or overly restrictive rules.Source: F Error analysis · p. 26
• An LLM-as-a-judge alternative could avoid rule-based evaluation, but may introduce judge errors and biases, prompt and version sensitivity, and higher evaluation costs.The paper cites Panickssery et al. (2024), Shi et al. (2025b), and Tan et al. (2025) on these challenges.Source: F Error analysis · p. 26

G Evaluation on visual categories

• Because few models support text, audio, and vision together, the evaluation focused on six vision-based categories, covering over 50 models.The paper says this allowed assessment of the larger set of Polish-supporting vision-language models.Source: G Evaluation on visual categories · p. 26
• In visual-category evaluation, Gemini, GPT, and Claude families are strongest, followed by Kimi, Qwen3.5/3.6, and Gemma 4 models; Table 5 reports strict and soft scores.The soft scores are shown as smaller gray numbers in parentheses.Source: G Evaluation on visual categories · p. 26, Table 5 · p. 27
• Smaller Gemma 4 and Qwen3.5 models around 30B parameters scored notably well on document categories, while Kimi models showed good image understanding but were much larger.Kimi-K2.5/K2.6 exceed 1 trillion parameters, and Kimi-K3 has 2.8 trillion parameters; the smaller Gemma and Qwen models were described as suitable for local document processing.Source: G Evaluation on visual categories · pp. 26, 28
• Grok models underperformed in image processing, most Mistral models struggled with documents, and Mistral-Medium-3.5 showed a noticeable improvement.The paper notes Grok’s image results were below expectations given its Polish text capabilities (Dadas et al., 2025).Source: G Evaluation on visual categories · p. 28
• InternVL-3.5 performed below Mistral and Llama 4, Polish LLaVa models lacked document-processing capability, and Phi-4-Multimodal-Instruct ranked last.The paper characterizes InternVL-3.5 (Wang et al., 2025a) as offering poor Polish support, and Polish LLaVa (Statkiewicz et al., 2026) as doing slightly better on knowledge-based categories.Source: G Evaluation on visual categories · p. 28

H Annotation process

• Six native Polish-speaking annotators created the benchmark from mid-October 2025 to the end of March 2026, averaging around 40 new tasks per week with roughly balanced contributions.The team comprised four men and two women, all in their thirties or forties and resident in Poland since birth.Source: H Annotation process · p. 28
• Unified guidelines required unambiguous questions compatible with rule-based verification, concise answers, flexible acceptance of valid formulations, and systematic testing against model responses.For closed questions, annotators combined inclusion and exclusion criteria.Source: H Annotation process · p. 28
• Tasks incorporated Polish history, public figures, traditions, language, and everyday life, balancing recognition with reasoning and contextual interpretation while iteratively refining difficulty and tag coverage.Annotators added questions from less represented areas as the dataset grew, aiming to avoid strongly underrepresented subcategories.Source: H Annotation process · p. 28

H.1 Review and Revision Phase

• Three native Polish annotators reviewed task design, content quality, and evaluation criteria to improve dataset consistency, alignment with benchmark objectives, and automatic-evaluation reliability.The review followed predefined acceptance criteria.Source: §H.1 · pp. 28–29
• Annotators used multimodal-model evaluation logs to identify and iteratively resolve inconsistencies, ambiguities, and failure cases, without changing questions to calibrate difficulty.The models included Gemini-3.1-Pro, Gemini-3-Flash, GPT-5.4, and GPT-Audio.Source: §H.1 · p. 28
• Acceptance criteria required unambiguous rule-based verification, Polish cultural grounding for image and Sound & Music QA questions, and questions that required multimodal-input analysis.Questions were also screened for duplication and ephemeral knowledge; language- or speech-only questions were allowed in the remaining categories.Source: §H.1 · pp. 28–29
• 28 tasks were removed and replaced, while 57 underwent substantial revisions to prompts or evaluation conditions.Minor stylistic and typo corrections were also applied throughout the process.Source: §H.1 · p. 29

H.2 Data Sources

• Annotators could record each source file’s URL, original creator, and license as provenance metadata.Source: §H.2 · p. 29
• Figure 5 reports source-file distributions by category and frequent domains; “other” groups rarer domains, “private” marks annotator-created files, and counts may exceed category totals when files have multiple sources.Appendix J questions 3, 4, and 5 provide examples of questions with multiple sources.Source: §H.2 · p. 29

I Annotation app

• The application supports multimodal benchmark construction and evaluation, records all annotator edits, and currently supports nine languages.Adding languages requires changes only to the lemmatizer and default prompts for ASR, OCR, and Structured Extraction.Source: I Annotation app · p. 29
• The app’s modules cover model configuration, annotator statistics, question management, and editing; model settings use structured JSON and can be assigned to specific question categories.Model parameters include endpoints, token limits, and required prefixes, and models can be added, modified, or deleted.Source: I Annotation app · p. 29
• Annotator statistics and task-management views track weekly activity and dataset progress, while category, tag, user, and content filters support browsing and workflow monitoring.The task overview lists each question with its category, input, tags, annotator, and creation date, and supports task management actions.Source: I Annotation app · pp. 29–30
• The task editor lets users define image, audio, or text inputs, add provenance metadata, formulate category-specific prompts, and specify conditions for automatic answer validation.Separate response blocks show each selected model’s output and validation result; Figure 6 depicts the task-edit view.Source: I Annotation app · p. 30, Figure 6 · p. 31

History and culture; contemporary life; geography and environment

• Other image questions ask users to identify a medieval fortress and its builders and takeover year, name pictured soups, or order Polish mountain peaks by height.The fortress verification specifies the Teutonic Order and 1457; the mountain answer is A, B, D, C.Source: J Sample questions · p. 32

Contemporary life

• A visual question asks readers to identify Polish video games from screenshots and return titles in A–D order: Cyberpunk, Polanie, Wiedźmin, and This War of Mine.The prompt requests titles without additional comments.Source: J Sample questions · p. 32

Geography and environment

• A map-based task asks users to name marked cities and sort them from north to south, returning only the ordered list.The verification order is Gdańsk, Szczecin, Białystok, Warszawa, Wrocław, Katowice, Kraków.Source: J Sample questions · p. 33

Audio

• Audio examples include general Polish speech recognition and questions about a Silesian-dialect monologue, including which statements about its advice are true.The monologue task expects the letters B, C, and F.Source: J Sample questions · pp. 33–34

Structured extraction

• Document tasks include OCR, extracting products discounted by at least 30% into JSON, and returning fields from a sample identity document as JSON.The promotional-flyer schema specifies product name and prices before and after the discount.Source: J Sample questions · pp. 34, 36

J Sample questions

• The record includes the holder’s name, birth details, PESEL number, citizenship, issuing authority, identity-card number, and issue and expiry dates.For example, it lists the name as KOWALSKI, the PESEL number as 81010200131, and the expiry date as 02.03.2025.Source: J Sample questions · p. 36