Source-linked AI summary
No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding
Vynska Amalia Permadi, Xingwei Tan, Nafise Sadat Moosavi, Nikos Aletras
TL;DR
Cultural QA benchmarks often rely on single-hop questions, leaving complex cultural reasoning insufficiently assessed. This paper introduces ID-MoCQA, a bilingual Indonesian multi-hop dataset generated from single-hop questions and validated through human and LLM review. Evaluation shows that models struggle with culturally appropriate multi-hop reasoning and often favor prominent cultural information over situation-specific practices.
Problem
Most cultural QA benchmarks use single-hop questions, limiting assessment of whether LLMs connect cultural clues, context, and appropriate practices.
Method
The paper transforms single-hop Indonesian cultural questions into two-hop bilingual questions across six clue types and validates them through expert annotation and LLM-as-a-judge filtering.
Results
Evaluation across ten open-weight and frontier LLMs shows persistent difficulty with multi-hop cultural questions, including a tendency to select well-known information regardless of situational suitability.
Takeaways & Limitations
ID-MoCQA provides a benchmark for testing cultural reasoning beyond factual recall and for studying models’ preference toward prominent culture over local practices.
Takeaways & Limitations
Zero-shot chain-of-thought produces inconsistent gains, including negative results for some model-task-language combinations, so it does not uniformly solve cultural reasoning.
Abstract
from arXiv · showhide
Understanding culture requires reasoning across context, tradition, and implicit social knowledge, far beyond recalling isolated facts. Yet most culturally focused question answering (QA) benchmarks rely on single-hop questions, which may allow models to exploit shallow cues rather than demonstrate genuine cultural reasoning. In this work, we introduce ID-MoCQA, the first large-scale multi-hop QA dataset for assessing the cultural understanding of large language models (LLMs), grounded in Indonesian traditions and available in both English and Indonesian. We present a new framework that systematically transforms single-hop cultural questions into multi-hop reasoning chains spanning six clue types (e.g., commonsense, temporal, geographical). Our multi-stage validation pipeline, combining expert review and LLM-as-a-judge filtering, ensures high-quality question-answer pairs. Our evaluation across state-of-the-art models reveals substantial gaps in cultural reasoning, particularly in tasks requiring nuanced inference. ID-MoCQA provides a challenging and essential benchmark for advancing the cultural competency of LLMs.
1 Introduction
ID-MoCQA addresses the limits of single-hop cultural QA by transforming culturally grounded questions into multi-hop reasoning tasks. The resulting Indonesian benchmark combines bilingual, human-verified data with broad model evaluation that exposes persistent cultural-reasoning challenges.
- Motivation: Single-hop cultural benchmarks often allow answers to be retrieved from one fact or cue, limiting evaluation of interconnected cultural reasoning.Multi-hop QA instead requires combining evidence and connecting cultural context to appropriate practices.
- Approach: The framework transforms culturally grounded single-hop questions into two-hop instances requiring models to identify relevant cultural context before selecting context-appropriate practices.An LLM adds an intermediate reasoning step linked to the original context, followed by expert and LLM-as-a-judge validation.
- Dataset: 15,590 multi-hop questions cover Indonesian culture across six clue types and two languages, English and Indonesian.The dataset is described as the first large-scale multihop cultural QA resource focused on a single national context.
- Evaluation: Extensive evaluation across open, frontier, and region-specific LLMs reveals persistent limitations in cultural multi-hop reasoning.The evaluation establishes ID-MoCQA as a benchmark for future research on cultural reasoning.
2 Related Work
Prior work defines cultural competence as context-appropriate communication and action, while cultural QA resources increasingly cover diverse languages and regions. However, existing benchmarks remain largely focused on cultural knowledge and single-hop evaluation rather than multi-hop reasoning.
- Cultural competence: Cultural competence involves communicating and acting appropriately across communities and contexts, including knowing when, how, and to whom communication is suitable.Acceptability depends on social relationships, status, and context.
- Cultural QA benchmarks: Existing cultural benchmarks span many countries, languages, and socio-cultural topics, including BLEnD, NativQA, and regional resources for Indonesian, Korean, Arabic, and global settings.These resources broaden cultural representation and evaluation coverage.
- Indonesian resources: Indonesian benchmarks address narratives, commonsense, causal reasoning, linguistic variation, and province-specific cultural practices.IndoCulture covers 11 provinces across six Indonesian islands, with distinct ethnic, linguistic, and religious practices.
- Generation and gap: LLM-based generation has been used for large-scale cultural QA and planned multi-hop reasoning, but prior cultural applications described here do not establish the proposed benchmark’s multi-hop focus.The related resources include CulturePark, ID-CSQA, NativQA, and WikiQA-IS.
3 Multi-hop QA Generation Framework
The framework expands province-specific IndoCulture questions into bilingual multi-hop questions by hiding the province behind one of six cultural clue types. It then applies structured generation and validation procedures to produce a large candidate set for quality screening.
- Framework overview: The framework has two components: LLM-guided automatic QA expansion and multi-stage validation combining human expertise with LLM verification.These components are designed to ensure dataset reliability.
- Source selection: The source data are high-quality IndoCulture questions labeled for province-specific cultural elements, with selected province names serving as first-hop links.The selection targets practices unique to particular provinces, such as Tor-tor dance in North Sumatra.
- Multi-hop transformation: Six clue types—commonsense, comparison, entity, geographical, intersection, and temporal—turn province information into indirect reasoning clues.Models must infer the referenced province before answering the original cultural question.
- Bilingual generation: The bilingual process removes direct province mentions, converts the original context into a question, and integrates an indirect first-hop clue into the final question.The generated English and Indonesian versions preserve culture-specific terms while combining both reasoning steps.
- Validation: Manual verification sampled 3,000 randomly selected instances and classified them into quality categories based on issues affecting factuality, clarity, or language.The reported categories include OK and Minor, with further categories continuing beyond the supplied passage.
- Scale: Applying the process to 1,847 IndoCulture questions across six clue types and two languages yielded 22,164 instances.The questions were generated with Claude-3.7-Sonnet at temperature = 1.
4 Dataset Validation
ID-MoCQA uses manual review, LLM judging, structural checks, translation, and human assessments to validate generated cultural multi-hop questions. The final dataset contains 15,590 bilingual instances, with quality and difficulty varying across question types.
- Initial Quality Assessment: 57.07% of sampled questions were rated OK, while 26.20% contained significant errors during manual verification.The sample comprised 3,000 randomly selected instances in both languages.
- LLM-as-a-Judge: Human agreement averaged Cohen’s κ=0.54 across question types, ranging from 0.35 to 0.75.GEOGRAPHICAL questions had the highest agreement, whereas INTERSECTION questions had the lowest, reflecting differing annotation consistency by category.
- LLM-as-a-Judge: The LLM judge achieved 0.78 precision and 0.82 recall against human labels, identifying most acceptable questions but varying more on problematic instances.Three frontier models assessed factual accuracy, structural coherence, and linguistic quality; majority-acceptable items were retained and any single significant judgment triggered rejection.
- Question Structure Verification: Less than 1% of questions were removed during final structural verification for failing multi-hop requirements.The process checked copied answer-option phrases, invalid province references, sequential reasoning, and cultural-question alignment.
- Final Dataset: 44.8% of questions were rated Hard, 25.9% Moderate, and 29.2% Easy, while the final dataset contained 15,590 instances split evenly across Indonesian and English.COMPARISON was the smallest category, with 730 instances per language, because factual inaccuracies reduced verification success.
5 Experimental Setup
The experiments evaluate frontier, multilingual open, and Indonesia-focused regional models on ID-MoCQA. Each question requires identifying a province from clues before selecting the correct province-specific cultural answer.
- Models: The evaluation includes GPT-5, DeepSeek-V3, Claude-3.7-Sonnet, Gemma2-27B-Instruct, Llama, Qwen, Merak-7B, and SeaLLM-7B.The models cover frontier systems, multilingual open models, and region-specific Indonesian models.
- Task Design: Each question requires open-ended province identification followed by selection among three final-answer options.The options come from the original IndoCulture dataset, while prompts request both reasoning stages.
- Human Evaluation: Three native Indonesian-speaking university graduates answered all 7,795 Indonesian questions without external tools.Participants first identified the target province and then selected one of three options.
6 Results and Analysis
Human and model results show that province identification is substantially easier than selecting culturally appropriate final answers, with performance varying by model, language, and clue type. Qualitative errors further indicate a preference for prominent cultural knowledge over situationally appropriate practices, while chain-of-thought gains remain inconsistent.
- Human and Model Performance: 70.0% human multi-hop accuracy versus 95.1% first-hop accuracy reveals a 25.1-point gap between locating provinces and answering culturally.Human multi-hop performance ranged from 66.6% to 75.3%.
- Human and Model Performance: Frontier models exceed humans by over 10% in Indonesian, with GPT-5 and Claude-3.7-Sonnet leading both languages and DeepSeek-V3 following closely.The comparison is reported for multi-hop accuracy.
- Geographic Differences: Models score 86% versus humans’ 84% on familiar provinces, but on less central provinces models maintain 77% while humans drop to 65%.The paper attributes this pattern to differences in geographic knowledge coverage.
- Model Size and Language: Larger models perform better in Indonesian, whereas smaller models show inconsistent language preferences and lose performance on multi-hop questions despite target-language specialization.Merak-7B and SeaLLM-7B fall from roughly 53% on IndoCulture single-hop questions to 51.14% and 50.97% on ID-MoCQA.
- Clue-Type Variation: No clue type is universally easiest: models exhibit distinct strength profiles, and several types shift between strongest and weakest across languages.Examples include GPT-5’s shift from COMPARISON to TEMPORAL and Merak-7B’s weakness on INTERSECTION in English.
- Human and Model Performance: Frontier models exceed 96% first-hop accuracy but score 18%–23% lower across both reasoning steps, while correct-first-hop but incorrect-second-hop cases occur six to ten times more often.Both-step failures remain below 1.2%, and the reverse error pattern remains under 3%.
- Chain-of-Thought Results: Zero-shot CoT produces mixed gains, averaging 2.67% in English and 2.63% in Indonesian for GPT-5, with negative improvements for some model–task–language combinations.GPT-5 reaches gains of up to 4.00% on GEO-GRAPHICAL and COMMONSENSE in English and 3.51% on INTERSECTION in Indonesian.
- Qualitative Analysis: Models favor well-documented practices over contextually appropriate ones, including ceremonial or communal interpretations when everyday or locally specific practices are correct.The errors include selecting kuah beulangong over sate matang, 8th-month rituals over mee boh kayee, and free sharing over market sales.
7 Conclusion
The paper introduces a framework for converting single-hop Indonesian cultural questions into multi-hop questions and releases the resulting bilingual ID-MoCQA dataset. Evaluation across ten open-weight and frontier LLMs finds persistent difficulty with contextual cultural reasoning and a tendency to select prominent rather than situation-appropriate information.
- Conclusion: The framework expands single-hop cultural questions into multi-hop questions targeting Indonesian culture.The transformation preserves a cultural focus while adding reasoning over an intermediate province-identification step.
- Conclusion: ID-MoCQA contains 15,590 multi-hop questions in Indonesian and English.The dataset is presented as the result of the proposed framework.
- Conclusion: Evaluation across ten open-weight and frontier LLMs shows persistent difficulty with multi-hop cultural questions.The conclusion identifies contextual cultural reasoning as an ongoing challenge.
- Conclusion: Models tend to select well-known cultural information even when it is unsuitable for the specific situation.The paper identifies debiasing and preference-tuning as future directions.
A Multi-Hop Question Prompt Guidelines
The prompt guidelines convert cultural contexts into bilingual two-hop questions without explicit geographic shortcuts, using clue-specific instructions and added verification for complex clue types. Comparison and intersection prompts receive specialized procedures to ensure unique province identification.
- A.1 Sample Full Prompt: Context conversion preserves the original topic while turning each statement into a province-specific cultural question without naming the province.The guidelines require answerability from the provided options.
- A.1 Sample Full Prompt: Multi-hop generation requires indirect cultural clues to identify the province before answering the final cultural question.Cities and regencies are also excluded as direct or indirect geographic references.
- A.1 Sample Full Prompt: Entity clues connect a province through one exact, uniquely associated person, historical figure, or cultural artifact.Geographical features, compound entities, translations, and synonyms are disallowed.
- A.1 Sample Full Prompt: The prompts require both English and Indonesian versions of every question.
- A.2 Clue Types and Structural Templates: The six clue types are commonsense, comparison, entity, geographical, intersection, and temporal, each with tailored transformation guidelines and examples.
- A.2 Clue Types and Structural Templates: Comparison prompts verify comparative claims against empirical data, including values, sources or years, and uniqueness before finalization.Failed verification triggers claim revision and re-verification.
- A.2 Clue Types and Structural Templates: Intersection prompts verify two conditions separately, then intersect their province sets so the result identifies exactly one target province.The first condition must identify multiple provinces, while the second narrows the result.
B LLM-as-a-Judge Evaluation Criteria
The LLM-as-a-judge process evaluates generated questions across provincial accuracy, reasoning structure, answer quality, clarity, and language quality. Revision rules specifically remove geographic shortcuts and option copying while preserving authentic cultural terms.
- Evaluation Criteria: The judge scores eight criteria from 0 to 2, for a maximum of 16 points.A score of 2 indicates the highest quality and 0 the lowest.
- Evaluation Criteria: The first four criteria assess provincial clue accuracy, conciseness, cultural alignment, and reasoning structure.
- Evaluation Criteria: The remaining criteria assess reasoning necessity, answer alignment, clarity, and language quality.
- Evaluation Criteria: Answer discrimination checks that both reasoning steps are required, while answer quality checks option compatibility and reasoning alignment.
- Issue Detection and Revision: The process detects copied option text and province names used as location references, then requires rewording or indirect references.Cultural terms such as “Rumoh Aceh” are preserved rather than treated as geographic shortcuts.
- Structural Integrity: Structural revision enforces province identification followed by a cultural question, indirect references, preserved cultural terms, and natural language.
D Manual Evaluation Guideline
ID-MoCQA is manually evaluated for linguistic and cultural naturalness, answerability through province identification, and cognitive difficulty. Three native Indonesian graduate students perform these assessments without external sources or AI assistance.
- Three native Indonesian graduate students assess question quality and difficulty using naturalness, multi-hop answering, and cognitive-difficulty evaluations.They receive the cultural context, bilingual questions, and three answer options in both languages.
- Naturalness is rated as Natural, Acceptable, or Unnatural on a 3-point scale.The criteria cover fluency, grammar, cultural accuracy, authenticity, minor issues, and major errors.
- Answering follows two steps: identify the referenced Indonesian province, then select the correct option from A, B, or C.The province is inferred from the provided cultural clues before the answer choice is selected.
- Cognitive difficulty is labeled Easy, Moderate, or Hard according to reasoning complexity and the rarity of required cultural knowledge.Hard questions involve specialized regional knowledge, uncommon facts, or complex multi-hop reasoning.
E Semantic Analysis
The analysis extracts linguistic and cultural features from English questions while preserving Indonesian cultural terms, and evaluates models with structured zero-shot and chain-of-thought prompts. The section also reports annotation disagreements, provincial variation, human error patterns, and model consistency.
- Semantic Analysis: English questions are analyzed with GPT-4o-mini at temperature=0 to extract lexical, semantic, entity, temporal, cultural-term, and word-count features.English is used because current LLMs provide more reliable part-of-speech tagging and named entity recognition, while Indonesian cultural terms remain unchanged across languages.
- Semantic Analysis: Models are evaluated in zero-shot settings with temperature=0, using prompts that request either structured province-and-answer outputs or step-by-step reasoning.GPT-5 uses its default temperature of 1.0 because it does not support temperature control.
- Semantic Analysis: Human annotator disagreements concern judgment criteria such as temporal and statistical specificity, with reported cases resolved through majority vote.Table 12 presents examples of these disagreements and their resolutions.
- Semantic Analysis: East Nusa Tenggara has 226 questions and the highest culture-specific-term density at 0.92 terms per question, while West Sumatra has 1,072 questions and 0.65 terms per question.The passage associates these densities with differences in reliance on local terminology and cultural familiarity.
- Semantic Analysis: Human error rates are highest for West Sumatra, South Sulawesi, and Papua, while West Java and Bali have the lowest reported province-level error rates.Food, Wedding, and Art topics account for approximately 15–16%, 15–16%, and 12–16% of errors, respectively.
G.5 Multi-Hop Reasoning Error Patterns
Multi-hop errors increase as model performance decreases, with weaker models accumulating mistakes across reasoning chains. Models often identify the province correctly but apply incorrect cultural reasoning, and leading models share systematic biases on some topics.
- G.5 Multi-Hop Reasoning Error Patterns: 28–30-point gaps appear in several 27B–72B models between reasoning outcomes, while smaller models show 16–35-point variation across both languages.Some smaller models also produce substantial rates of incorrect first-hop but correct second-hop and both incorrect outcomes.
- G.5 Multi-Hop Reasoning Error Patterns: First-hop-to-both-correct gaps widen as performance decreases, spanning 18–23 points for frontier models, 28–30 for 70B models, and 16–35 for smaller models.The analysis suggests weaker models accumulate errors across the reasoning chain rather than failing at only one step.
- G.5 Multi-Hop Reasoning Error Patterns: GPT-5, Claude-3.7-Sonnet, and DeepSeek-V3 often select the same wrong answer, especially on death ceremonies, traditional games, and art forms.These topics have higher same-wrong-answer rates than daily activities.
- G.5 Multi-Hop Reasoning Error Patterns: Models average 96.5% province-identification accuracy but still struggle with cultural reasoning within correctly identified contexts.In the bajapuik tradition, models recognize matrilineal markers yet incorrectly expect the groom’s family to pay uang japuik instead of the bride’s family.