Source-linked AI summary
Evaluating Polish linguistic and cultural competency in large language models
Sławomir Dadas, Małgorzata Grębowiec, Michał Perełkiewicz, Rafał Poświata
TL;DR
LLM evaluation of Polish requires attention to cultural context as well as language, yet no existing dataset focuses on this combined competence. The paper introduces a 600-question benchmark and evaluates more than 30 models; commercial models retain a significant advantage, while Polish-focused models can match much larger multilingual models in cultural understanding.
Problem
No dataset yet focuses on evaluating Polish linguistic and cultural competence, including understanding references whose meanings depend on cultural context.
Source: §1 · pp. 2–3, §1.2 · p. 4Method
The authors introduce 600 manually crafted questions across six Polish culture, language, and knowledge categories and evaluate over 30 open-weight and commercial LLMs.
Source: §1.1 · p. 3, §2 · p. 5Results
Commercial models maintain a significant advantage over open-weight models, while Polish-focused models such as Bielik can match much larger multilingual models in cultural understanding.
Source: §4 · p. 10Takeaways & Limitations
The authors conclude that training models focused solely on Polish is meaningful, as small models such as Bielik can match much larger multilingual models in cultural understanding.
Source: §4 · p. 10
Abstract
from arXiv · showhide
Large language models (LLMs) are becoming increasingly proficient in processing and generating multilingual texts, which allows them to address real-world problems more effectively. However, language understanding is a far more complex issue that goes beyond simple text analysis. It requires familiarity with cultural context, including references to everyday life, historical events, traditions, folklore, literature, and pop culture. A lack of such knowledge can lead to misinterpretations and subtle, hard-to-detect errors. To examine language models' knowledge of the Polish cultural context, we introduce the Polish linguistic and cultural competency benchmark, consisting of 600 manually crafted questions. The benchmark is divided into six categories: history, geography, culture & tradition, art & entertainment, grammar, and vocabulary. As part of our study, we conduct an extensive evaluation involving over 30 open-weight and commercial LLMs. Our experiments provide a new perspective on Polish competencies in language models, moving past traditional natural language processing tasks and general knowledge assessment.
1 Introduction
Evaluating LLMs is difficult because they handle varied tasks and generate free-form responses whose correctness is harder to assess than structured outputs. This paper addresses a gap in Polish-language evaluation by introducing a benchmark for linguistic and cultural competence.
1 Introduction
- Evaluating LLMs requires a multifaceted approach, and assessing correctness in free-form text is harder than checking structured outputs.Models tackle varied tasks and may generalize to tasks unseen during training.Source: §1 · pp. 1–2
- Underrepresentation of Polish in pre-training can leave models without cultural context, making references to history, folklore, literature, or pop culture difficult to interpret correctly.The paper also illustrates morphological, lexical, word-formation, and language-switching errors in a Polish response.Source: §1 · pp. 2–3, Figure 1 · p. 2
1.1 Contributions
- The authors introduce 600 manually prepared questions across six categories and evaluate more than 30 open-weight and commercial LLMs.The benchmark also varies question forms, including fact combination, knowledge aggregation, sorting, and filtering.Source: §1.1 · p. 3
1.2 Related work
- LLM evaluation includes closed-ended, model-as-judge, human, and rule-based approaches, each with distinct constraints such as cost, judge dependence, or task-specific rules.Rule-based checks can verify specified answer content, while human evaluation can assess alignment with preferences.Source: §1.2 · pp. 3–4
- To the authors’ knowledge, no existing Polish dataset focuses on LLMs’ linguistic and cultural competence, while other-language benchmarks have begun to appear.Existing Polish efforts include adaptations of English benchmarks and leaderboards aggregating older NLP, generative, or medical-exam datasets.Source: §1.2 · p. 4
2 The benchmark
The benchmark uses handcrafted, deterministically verifiable questions to assess knowledge of Polish culture, tradition, and language across six categories. Its grading normalizes responses and awards credit only when every question condition is met.
2 The benchmark
- 600 handcrafted questions are divided evenly across six categories: culture and tradition, art and entertainment, geography, history, grammar, and vocabulary.The questions range from widely known material to region-specific culture and ethnic minorities.Source: §2 · pp. 5–6
- Questions vary in difficulty and form, with about half closed-ended and the rest usually requiring a sentence containing a specific fact or facts.Instructions commonly request short, precise answers without additional commentary.Source: §2 · p. 5
2.1 Grading
- A question earns either one point or zero: all its conditions must be satisfied, and the overall score is the percentage of correct answers.Partial credit is not possible.Source: §2.1 · pp. 6–7
- Responses are generated at temperature 0 without a system prompt, then normalized by lowercasing, removing non-alphanumeric characters, and lemmatizing before condition checks.Lemmatization lets the evaluator match different forms of a word in Polish.Source: §2.1 · p. 7
- Grading conditions can require inclusion, exclusion, ordering, or regex matches, with configurable alternatives and occurrence limits.Regex checks use the unnormalized response and can require matched strings to be valid Polish words.Source: §2.1 · pp. 7–8
3 Evaluation
The evaluation compares commercial and open-weight models on the Polish competency benchmark, finding a commercial lead and a strong result from Polish-focused training. Results also show that performance on this benchmark differs from other Polish benchmarks and often does not improve across model versions.
Evaluation setup
- More than 30 commercial and open-weight models were evaluated, and evaluation of GPT-4o cost less than $1 according to the pricing cited in the paper.The results report category scores and an overall score.Source: §3 · p. 8
The effectiveness of training on Polish data
- More than 40 points separated Bielik-2.3 from its Mistral base, and the 11B-parameter Polish-focused model ranked second among open models.Bielik-2.3 was pre-trained on a Polish corpus and fine-tuned on Polish instructions.Source: §3 · p. 8
Open vs. Commercial LLMs
- 83% for Gemini-Exp-1206 made it the top model; Claude Sonnet 3.5 and GPT-4o also passed 80%, while the best open-weight model scored approximately 12 points lower.Table 1 reports category and total scores; the prose identifies the leading models and comparison.Source: §3 · p. 8, Table 1 · p. 9
Comparison with other Polish benchmarks
- Qwen-2.5 models, usually highly rated on other Polish benchmarks, performed below expectations here, where existing benchmarks often emphasize general knowledge or classic NLP tasks.The authors argue that those benchmarks do little to test models’ actual ability to use and understand Polish.Source: §3 · pp. 8–9
Results in relation to model size
- Smaller models generally lost quality on this knowledge-based task; the authors relate this to information density and the challenge of compressing multilingual knowledge.The authors identify Bielik as the exception among smaller models.Source: §3 · pp. 9–10
Comparison with older models
- Most model versions released in 2024 did not improve their benchmark scores; Claude Sonnet showed the most progress, especially from 3.0 to 3.5.Mistral-Large, Bielik, and GPT-4o’s latest versions scored slightly below earlier versions; Llama 3 also improved during the year.Source: Figure 3 · p. 10, §3 · p. 10
4 Conclusions
The benchmark assesses LLMs’ understanding of Polish cultural context and finds commercial models still significantly ahead of open-weight models. Polish-focused training can yield strong results even in smaller models, while the benchmark remains challenging and offers a way to track progress.
4 Conclusions
- Commercial models retain a significant advantage over open-weight models, and the best models answer about 80% of questions correctly.The authors describe the benchmark as challenging for modern LLMs.Source: §4 · p. 10
- Polish-focused training is meaningful: a small model such as Bielik can match much larger multilingual models on cultural understanding.The authors also report consistent development in both commercial and open-weight model groups.Source: §4 · p. 10
- The benchmark leaves considerable room for improvement and can be used to track future progress in models’ understanding of Polish.Source: §4 · p. 10