PaperlayerView exact revision on arXiv

Source-linked AI summary

WikiOriginQA: Automated Construction of a QA Dataset with Knowledge Origins for Analyzing Cultural Bias

Kento Haneda, Yosuke Kishinami, Ryo Fujii, Keisuke Sakaguchi, Makoto Morishita

arXiv:2610.30677v1cs.CL

TL;DR

Cultural-knowledge datasets often depend on labor-intensive human annotation, limiting scalability. WikiOriginQA automatically infers knowledge origins from Wikipedia article creation dates and generates QA data; experiments find LLM performance tends to favor Anglophone cultures, with results suggesting the importance of knowledge propagation into the Anglosphere.

  • Problem

    Existing cultural-knowledge datasets require considerable human effort, making expansion across cultures and languages difficult.

    Source: §1 · p. 1, §2 · p. 2

  • Method

    WikiOriginQA automatically identifies knowledge origins by comparing Wikipedia article creation dates across language editions and uses the extracted knowledge to construct QA data.

    Source: §1 · p. 1, §3.1 · p. 2

  • Results

    Experiments found that LLMs tend to perform better on questions based on English articles or knowledge propagated into English, suggesting bias toward Anglophone cultures.

    Source: §4.2 · p. 3, §5.1 · p. 4

  • Takeaways & Limitations

    The results suggest that knowledge propagation into the Anglosphere is important for LLM knowledge acquisition.

    Source: §4.2 · p. 3, §5.1 · p. 4

    The automated approach cannot guarantee the absolute reliability associated with human annotation, and its extraction is not perfectly accurate.

    Source: Limitations · p. 5

Abstract

from arXiv · show

Existing datasets on cultural knowledge have been typically created by human annotators familiar with each culture. However, the process requires considerable human effort, presenting a significant challenge to scalability. To address this challenge, we introduce WikiOriginQA, a QA dataset automatically constructed by objectively representing the origins of knowledge through the creation dates of different Wikipedia language editions. Experimental results with the created dataset confirm that LLM performance tends to be biased toward Anglophone cultures. Furthermore, the results suggest the importance of knowledge propagation into the Anglosphere. These findings demonstrate the validity of our automated approach, enabling cultural bias evaluations with superior scalability across diverse languages without extensive manual annotation.

1 Introduction

Cultural-bias evaluation needs broad multilingual coverage, but human assignment of knowledge to cultures makes existing datasets difficult to scale. WikiOriginQA uses Wikipedia article creation dates to identify knowledge origins automatically, and experiments find LLMs tend to favor Anglosphere cultures.

1 Introduction

  • Existing cultural-bias datasets require culture-familiar experts to assign knowledge manually, limiting expansion across languages.Manual judgments also raise concerns about subjectivity.Source: §1 · p. 1
  • WikiOriginQA identifies knowledge origins from article creation dates across Wikipedia language editions, based on the hypothesis that culture-specific knowledge appears first in its associated language.For example, Japanese-culture knowledge is expected to appear first in Japanese Wikipedia.Source: §1 · p. 1
  • Experiments using WikiOriginQA found that LLMs tend to be biased toward Anglosphere cultures.The authors present this as evidence of the effectiveness of fully automated datasets for evaluating cultural bias.Source: §1 · p. 1

2 Related Work

Prior cultural-knowledge benchmarks report Anglocentric biases, but their human collection and annotation can limit scale and introduce subjectivity. WikiOriginQA uses Wikipedia timestamps to address a gap in an existing objective approach: capturing knowledge that has propagated across language editions.

2 Related Work

  • Existing cultural datasets report Anglocentric bias, while their collection and annotation require substantial human effort and can raise subjectivity concerns.The scalability challenge is especially acute for low-resource languages.Source: §2 · p. 2
  • KoLasSimpleQA [Jiang et al., 2025] extracts items found only in one Wikipedia language edition, but can miss knowledge later added to other editions.The authors note this may skew its dataset toward excessively niche information.Source: §2 · p. 2
  • WikiOriginQA uses Wikipedia article timestamps to remain robust to knowledge propagation across language editions.The authors motivate timestamps with prior findings linking language, culture, and Wikipedia article development.Source: §2 · p. 2
  • Prior studies describe language-linked patterns in Wikipedia: culturally salient topics and people are more likely to appear in related language editions, and editor communities influence article content.The cited examples include Roy et al. (2020), Eom et al. (2015), and Miquel-Ribé and Laniado (2018).Source: §2 · p. 2

3 WikiOriginQA

WikiOriginQA is a short-answer dataset for evaluating cultural bias in LLMs. It infers knowledge origins from Wikipedia creation dates, generates and filters QA pairs, and builds datasets for eight selected languages plus English.

3 WikiOriginQA

  • The dataset construction process produces short-answer QA data designed to evaluate cultural biases in LLMs.The section introduces WikiOriginQA as the paper's dataset.Source: §3 · p. 2

3.1 Extraction of Knowledge for QA Data

  • Creation dates from Wikipedia dumps are compared across language editions linked by Wikidata entity IDs to infer each item's originating language sphere.The comparison covers 342 languages.Source: §3.1 · p. 2

3.2 Generation of QA Data

  • An LLM generates questions from each entity's article, iteratively refines them, translates them into English, and filters them using seven criteria.Questions are retained only if they receive maximum scores on every criterion.Source: §3.2 · pp. 2–3

3.3 Target Languages and Dataset Details

  • The authors selected eight languages from 75 candidates and sampled 1,000 items per language, also creating English questions for comparison.Selection considered regionality and Wikipedia resource availability.Source: §3.3 · p. 3
  • A manual Japanese case study found that 96 of 108 questions (88.9%) originated from Japan, supporting the validity of the extraction method.The authors also report that the dataset captures knowledge propagated to other languages.Source: §3.3 · p. 3

4 Experiment

Across most models, questions based on English articles achieved higher accuracy, and items with English articles tended to score above the overall questions for their source language. Similar patterns appeared in source-language and manually verified Japanese data; the error-analysis table also reports high true-negative rates for automatic evaluation.

4.1 Settings

  • The study evaluated GPT-5, LLM-jp, and Qwen3 models using exact-match accuracy and character-level F1.GPT-5 used low reasoning effort with other parameters at default.Source: §4.1 · p. 3

4.2 Results

  • English-article questions achieved higher accuracy for most models, and items with English articles tended to outperform each language’s overall questions.The English-article analysis was not conducted for ka, ps, and uk because each had fewer than 10 such questions.Source: §4.2 · p. 3, Figure 2 · p. 4
  • Table 4 reports high true-negative rates as evidence for the validity of automatic evaluation.Source: Table 4 · p. 4
  • Source-language QA pairs and manually verified Japanese data showed patterns similar to the translated data.For Japanese, the results mirrored those from the automatically generated dataset.Source: §4.2 · p. 4

5 Analysis

Results suggest an Anglosphere-oriented performance skew, with higher accuracy on questions based on English articles or knowledge propagated into English. Error checks found that most examined incorrect answers were genuinely incorrect, though some correct responses were misclassified, including through translation and romanization issues.

5.1 Result Analysis

  • LLMs achieved higher accuracy on questions based on English articles or knowledge propagated into English, suggesting an Anglosphere-oriented bias.Afrikaans questions were relatively accurate, possibly because English is also an official language in South Africa and Afrikaans knowledge may be more readily translated into English.Source: §5.1 · p. 4
  • Further investigation is needed to separate English-resource availability from intrinsic question difficulty as sources of the performance gap.Source: §5.1 · p. 4

5.2 Error Analysis

  • Manual checks found that most examined Japanese exact-match errors were reasonably judged incorrect, but some correct answers were classified as incorrect.Some misclassifications involved English translation or kanji romanization, and question construction and answer evaluation remain future work.Source: §5.2 · p. 4

6 Conclusion

The conclusion presents WikiOriginQA as a fully automated, scalable dataset for evaluating cultural bias using Wikipedia article creation dates. Experiments revealed a performance skew toward the Anglosphere, and the authors expect the approach and dataset to support more culturally inclusive language models.

6 Conclusion

  • WikiOriginQA is constructed through a fully automated method intended to make cultural-bias evaluation highly scalable.The approach identifies knowledge origins using Wikipedia article creation dates and is presented as addressing scalability and subjectivity issues in human-annotated datasets.Source: §6 · p. 4
  • Experiments across LLMs revealed a performance skew toward the Anglosphere, highlighting a persistent risk of excluding non-English cultures.Source: §6 · p. 4
  • The authors expect the approach and dataset to help develop more culturally inclusive language models.Source: §6 · p. 5

Limitations

The study’s limitations include restricted applicability to languages spanning multiple regions, incomplete knowledge coverage in some languages, and constraints on dataset scale and reliability.

Limitations

  • Eight languages were selected as an illustrative sample from 75 languages that are official in exactly one country; this selection does not limit the method’s scalability.Regional diversity and other factors informed the selection.Source: Limitations · p. 5
  • Methods based on a language–region connection are difficult to apply directly to widely spoken languages such as French or Spanish.Using Wikipedia editors’ IP addresses was considered as an alternative but not adopted over sustainability and scalability concerns.Source: Limitations · p. 5
  • For low-resource languages or underdeveloped Wikipedia editions, native cultural items documented only in English or other major languages can be missed, limiting recall.Items documented first in a specific language ahead of all others strongly imply a connection to that language’s cultural sphere, supporting high-precision extraction.Source: Limitations · p. 5
  • Over 800,000 Japanese and over 300,000 Ukrainian items were created earlier than in any other language edition, but the dataset sampled only a subset.The dataset is relatively small because of financial constraints.Source: Limitations · p. 5
  • Fully automated construction cannot guarantee the absolute reliability typically associated with human annotation.The authors report that extraction is imperfect, but does not critically affect cultural-bias evaluation for Japanese questions; a manually reviewed sample of QA pairs was judged high quality.Source: Limitations · p. 5

B Details of QA Data Generation

The QA generation process uses source-language Wikipedia entries to generate questions, then accepts multiple forms of an entity’s name as correct answers.

B Details of QA Data Generation

  • Given a source-language Wikipedia entry, the LLM is instructed to generate a question whose answer is the entry’s entity.The question is iteratively refined by the LLM to improve quality.Source: B Details of QA Data Generation · p. 7
  • Correct answers include the entity title in its original language and its English translation, with English Wikipedia titles and Wikidata aliases added when available.The listed aliases are treated as equally valid answers.Source: B Details of QA Data Generation · p. 7

C Details of Results

Table 5 reports detailed QA performance for questions derived from each source language.

C Details of Results

  • Table 5 presents detailed QA performance for questions derived from each source language.The supplied passage identifies the table’s scope but does not provide its results.Source: C Details of Results · p. 7

E Results of Manually Verified Data

Manually verified Japanese QA results mirrored the overall dataset findings; the appendix also reports filtering’s improvement to Japanese extraction accuracy and gives examples of response errors.

E Results of Manually Verified Data

G Example of Incorrect Answer by GPT-5

H Examples Erroneously Judged as Incorrect

J Filtering Extracted Knowledge Using LLM

Loading 2610.30677v1…