Source-linked AI summary
UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languages
Pranjal A. Chitale, Varun Gumma, Sanchit Ahuja, Prashant Kodali, Manan Uppadhyay, Deepthi Sudharsan, Sunayana Sitaram
TL;DR
Multilingual synthetic data remains underexplored, and translation-heavy datasets can lack cultural grounding and diverse task coverage. UPDESH pairs translated reasoning data with culturally grounded generation, and fine-tuned models consistently outperform existing baselines across NLU and NLG evaluations. The paper also reports that Wikipedia-based grounding has cultural-representation limits where documentation is insufficient.
Problem
Method
Results
UPDESH fine-tuned models consistently outperform existing baselines across NLG and NLU evaluations in 13 Indic languages.
Source: §5.1 · pp. 5–6Takeaways & Limitations
The authors conclude that culturally grounded, bottom-up synthetic data can potentially bridge resource gaps and that UPDESH’s results support its utility across NLU and NLG tasks.
Source: §6 · p. 8Reliance on Wikipedia may leave underrepresented communities’ cultural customs and nuances insufficiently documented, potentially favoring well-documented urban practices over rural or minority contexts.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Developing culturally grounded multilingual AI systems remains challenging, particularly for low-resource languages. While synthetic data offers promise, its effectiveness in multilingual and multicultural contexts is underexplored. We investigate bottom-up synthetic data generation using large open-source LLMs (>= 235B parameters) grounded in language-specific Wikipedia content, complementing dominant top-down translation-based approaches from English. We introduce Updesh, a high-quality large-scale synthetic instruction-following dataset comprising 9.5M data points across 13 Indian languages and English, encompassing diverse reasoning and generative tasks. Comprehensive evaluation using automated metrics and 10K human assessments confirms high data quality. Downstream evaluations performed by fine-tuning models on various datasets and assessing performance across 13 diverse multilingual datasets and model comparative evaluations, demonstrate that models trained on Updesh consistently obtain significant improvements on NLU, NLG evaluations. Finally, through ablation studies and cultural evaluations, we show that context-aware, culturally grounded data generation is essential for effective multilingual AI development.
1 Introduction
The paper frames multilingual AI as an equity challenge: English-centric development and limited non-English data leave language and cultural nuance underrepresented. It introduces UPDESH, a 9.5M-example culturally grounded dataset, and reports consistent NLU and NLG improvements from fine-tuning on it.
Motivation
- English-centric pretraining and translation-only fine-tuning can miss linguistic nuance and cultural context, limiting equitable access to multilingual and multicultural AI.The cited work identifies more than 2.4K languages—93.87% of the world’s languages—spoken by 1.2B people as severely underrepresented online.Source: §1 · p. 1
Research gap
- Multilingual synthetic data remains under-evaluated: fluency, correctness, and diversity checks alone are insufficient, so downstream fine-tuning and benchmarking are needed to establish utility.The authors note that English-centric quality assumptions may not transfer to multilingual and multicultural settings.Source: §1 · p. 1
Contribution
- UPDESH contains 9.5M culturally grounded synthetic instruction examples, and fine-tuned models consistently improve NLU and NLG performance across languages.The authors also report improvements in cultural evaluations and offer design considerations for multilingual and multicultural data generation.Source: §1 · p. 1
2 Related Work
Prior instruction-tuning work uses English synthetic-data pipelines and multilingual translation, templating, or hybrid approaches. The paper identifies gaps in cultural grounding, task diversity, advanced reasoning, and long-context or multi-turn coverage.
English instruction fine-tuning
- English instruction-tuning work ranges from large task collections and LLM-generated synthetic data to conversational, curation-heavy, and explanation-tuning approaches.The review describes FLAN’s more than 1.8K tasks and ORCAAGENT-INSTRUCT’s 25.8M synthetic pairs.Source: §2 · pp. 1–2
Multilingual IFT datasets
- Multilingual instruction-following datasets use translation, template-driven synthesis, or hybrids that combine approaches such as crowdsourcing, templating, and translation.The reviewed efforts span 52- and 70-language datasets as well as collections covering 65 and 20 Indian languages.Source: §2 · p. 2
Data generation strategies
- Alternative generation strategies use diverse web content for self-augmentation and self-curation, including synthesizing instructions from documents and iteratively rewriting responses.The review says these approaches often outperform pure distillation.Source: §2 · p. 2
Limitations of prior multilingual IFT
- Translation-heavy multilingual instruction datasets often emphasize basic instruction following, underrepresent advanced reasoning, and provide limited cultural grounding.The review also notes context loss and subtle errors from sentence-level translation, limited culturally specific content, and short-context, single-turn coverage.Source: §2 · p. 2
3 Data Generation
UPDESH combines translated reasoning data with open-domain generation grounded in Indic-language Wikipedia and curated cultural material. Filtering removes low-quality generations, while the reported drop rates are below 2% for most subsets.
Dataset design
- UPDESH covers 13 Indic languages and combines reasoning and open-domain generation as complementary instruction-following subsets.The reasoning subset uses translation for language- and culture-agnostic tasks, while open-domain generation targets culturally aware, natural, locally grounded output.Source: §3 · pp. 2–3
Reasoning data
- The reasoning subset translates seven ORCAAGENT-INSTRUCT subsets and the ORCAMATH math subset into 13 Indic languages using LLAMA-3.1-405B-INSTRUCT.The generated translations undergo quality checks.Source: §3 · p. 3
Generation strategy
- QWEN3-235B-A22B outperformed LLAMA-3.1-405B-INSTRUCT on generative tasks and complex instruction following, which the authors link to stronger reasoning traces.For English, reasoning data is reused from ORCAAGENT-INSTRUCT and ORCAMATH, while the generative subset is created with the same pipeline as for Indian languages.Source: §3 · pp. 3–4
Open-domain generative data
- Open-domain generation constructs questions from unlabelled target-language content and uses LLM-generated answers, with Wikipedia serving as the grounding source.Some of the eight generative task categories require two LLM inference phases.Source: §3 · p. 3
Cultural curation
- The cultural content curation traverses India- and state-specific Wikipedia categories 2–3 levels deep, sampling 26.8K artifacts for multi-hop question-answer generation.The categories yield region-specific material on festivals, cuisine, traditional arts, architecture, and religious practices.Source: §3 · pp. 3–4
Data filtering
- Filtering applies INDICLID language identification at a 0.75 confidence threshold and caps word repetition ratio at 0.75.The authors use standard threshold-based filtering rather than anomaly detection.Source: §3 · p. 4
- Below 2% of data was dropped for most subsets, with higher filtering for Urdu FS-COT due to repetition and for Assamese translation due to Bengali outputs.Table 1 reports aggregate filtering statistics across the 13 Indic languages and English.Source: Table 1 · p. 3, §3 · p. 4
4 Dataset Quality Analysis (Q-A)
The quality analysis checks reasoning-data translation fidelity and assesses generative data with native-speaker and LLM evaluations. Human ratings were overwhelmingly nonzero, but agreement between human and LLM judges varies across evaluation dimensions.
4.1 Q-A for Reasoning Data
- Backtranslation of 4,096 samples per subset and language yielded consistently high ChrF scores, which the authors interpret as confirming translation quality for the reasoning data.LLAMA-3.3-70B-INSTRUCT backtranslated samples to English for comparison with their original sources.Source: §4.1 · p. 4
4.2 Q-A for Generative Data
- Across 10K individual ratings of generative data, only 27 were zero, indicating uniformly high data quality under the evaluation rubrics.Native-speaker annotators and GPT-4O used identical protocols and task-specific multidimensional metrics on a 0–2 Likert scale.Source: §4.2 · pp. 4–5
- LLM-judge agreement varies by metric: it is stronger for objective checks such as toxicity detection and weaker for culturally nuanced linguistic plausibility and repetition in long dialogues.The authors caution that toxic or problematic examples are expected to be rare or absent because the data comes from benign prompts and topics.Source: §4.2 · p. 5
5 Downstream Tasks Evaluation
Across benchmark and cultural evaluations, UPDESH-tuned models outperform the tested multilingual baselines, with strong NLG results and the most dataset wins. The results also show that dataset composition and training sequence length affect performance, and that UPDESH transfers to languages absent from training.
5 Downstream Tasks Evaluation
- The downstream evaluation uses likelihood-scored multiple-choice NLU, generation tasks such as translation and summarization, and comparative evaluations of model win rates.Fine-tuning comparisons use Llama-3.1-8B and Phi-4-14B against AYA-COLLECTION, INDICALIGN, and BACTRIAN-X.Source: §5 · p. 5, Table 2 · p. 5
5.1 Results
- UPDESH-tuned Llama and Phi-4 models consistently outperform BACTRIAN-X, INDICALIGN, and AYA-COLLECTION across NLG and NLU tasks in 13 Indic languages.The advantage is particularly pronounced in NLG and spans both higher-resource languages such as Hindi and Bengali and lower-resource languages such as Assamese and Odia.Source: §5.1 · pp. 5–6, Figure 3 · p. 6
- UPDESH records the most dataset wins—7 for Llama3 and 8 for Phi-4—while its NLG scores lead and fine-tuned models cluster more closely on NLU.The win counts summarize the 12 evaluated tasks.Source: Figure 4 · p. 6, §5.1 · p. 6
- On Flores languages absent from training, UPDESH leads NLG scores on both architectures: 24.93 for Llama-3-8B and 24.10 for Phi-4.The evaluation covers 16 out-of-domain Indic languages.Source: §5.1 · pp. 6–7, Figure 5 · p. 7
- Isolating the generative subset slightly improves NLU, including +3.78 for Llama, but reduces NLG quality; training on reasoning alone lowers both, including −37.06 for Phi-4 NLG.The ablation compares reasoning (R), generation (G), and combined R+G training.Source: §5.1 · p. 7, Table 3 · p. 8
- Phi-4 trained with a 32K context window achieves the most balanced profile, with strong NLU, boosted NLG, and the highest number of dataset wins (7).The authors identify training sequence length as an important design consideration.Source: §5.1 · p. 7, Table 4 · p. 8
5.2 Comparative Cultural evaluations (ELO Rankings)
- UPDESH-32K leads the Phi-4 cultural evaluation with an ELO rating of 1696, ahead of IndicAlign Cleaned (1659) and standard UPDESH (1607).Because of evaluation cost, this analysis covers Phi-4 checkpoints only.Source: §5.2 · pp. 7–8, Table 5 · p. 8
- UPDESH-32K also combines the highest ELO score (approximately 1700) with leading NLU average (approximately 0.55), NLG average (approximately 35), and dataset wins.The reported ELO scores show a positive correlation with the automated benchmark metrics.Source: §5.2 · p. 8
6 Conclusion
The work builds UPDESH as a 9.5M-example, 13-language instruction-tuning dataset through a culturally grounded, bottom-up pipeline. Its evaluations find broad NLU and NLG gains across tasks and models, while offering design considerations and research artifacts for future work.
6 Conclusion
- UPDESH is a 9.5M-example, 13-language instruction-tuning dataset built with a culturally grounded, bottom-up pipeline.The pipeline is evaluated for data generation, quality, and downstream tasks.Source: §6 · p. 8
- Across tasks and models, UPDESH outperforms baselines in both NLU and NLG settings.The conclusion also reports considerable uplift in cultural evaluations.Source: §6 · p. 8
- The paper distills design considerations for multilingual and multicultural synthetic-data generation and plans to release the dataset, evaluation protocols, and analyses.Source: §6 · p. 8
7 Limitations
The authors identify limitations in LLM-based quality evaluation, Wikipedia’s coverage of cultural knowledge, and specialized benchmarking for Indic-language capabilities.
7 Limitations
- LLM evaluators align strongly with humans on toxicity detection (96-98%) but less on fluency assessment and persona consistency (45-60%).The authors advise caution when relying solely on LLM-based quality estimates and call for more calibrated evaluators and robust frameworks.Source: §7 · p. 9
- Reliance on Wikipedia may leave cultural customs and nuances of underrepresented Indian communities insufficiently documented.The authors warn that this could produce incomplete representations favoring documented urban practices over rural or minority contexts.Source: §7 · p. 9
- Few Indic-language benchmarks assess cultural aspects and long-context or multi-turn capabilities, complicating systematic evaluation.The authors also note that their real-world-query comparative evaluation uses an LLM evaluator; human evaluation could provide finer-grained quality assessment.Source: §7 · p. 9
8 Ethical Considerations
The paper reports ethics oversight and quality checks for its data work, while describing annotator recruitment and compensation.
8 Ethical Considerations
- Data annotation was conducted by a third-party vendor and approved by the authors’ institutional review board and the vendor.Source: §8 · p. 9
- A quality-control process combined GPT-4o evaluation with human verification; annotators flagged 1 of 500 samples (0.2%) for potentially problematic content.The authors report high concordance between automated and human judgments and reduced baseline incidence of such content from state-of-the-art LLM generation.Source: §8 · p. 9
- Annotators were recruited through an external services company, and those assigned to each data point were native speakers of its language.The paper reports compensation of $2 per data point and points to Table 6 for demographic summaries.Source: §8 · p. 9, Table 6 · p. 9
- The authors state that a detailed reproducibility statement is provided in Appendix A.Source: §8 · p. 9
A Reproducibility Statement
The appendix describes released materials for reproducing the pipeline and sets out broader design considerations for multilingual and multicultural synthetic data. These include foundation-model and seed-data selection, grounding, quality assessment, evaluation, and native-speaker involvement.
A Reproducibility Statement
- The planned release includes code, datasets, evaluation materials, raw model scores, and human annotations, with instructions and training configurations for reproduction.The code is organized into seven components covering translation, Wikipedia-grounded generation, filtering, quality evaluation, training, downstream evaluation, and plotting.Source: A Reproducibility Statement · p. 15
B.1 Design Considerations
- The framework’s design considerations address data generation, quality metrics, downstream evaluation, and native-speaker involvement, and may generalize beyond SFT data.The framework covers pre-training, SFT, RLHF, and evaluation, while the paper’s focus is SFT.Source: §B.1 · p. 15, Base model capability & Seed data… · p. 16
Base model capability & Seed data selection
- Select foundation models using target-language benchmark performance or related-language and overall multilingual proxies, while weighing licensing, cost, and availability.Seed data should span diverse sources and tasks, prioritizing culturally relevant knowledge, norms, and values.Source: Base model capability & Seed data… · p. 15
- The design discussion contrasts translation and backtranslation with retrieval-augmented generation from curated, native-speaker-authored web content.It notes that translation-based approaches can introduce translationese and that bottom-up web-grounded approaches remain underexplored in multilingual contexts.Source: Base model capability & Seed data… · pp. 15–16
B.2 Task Descriptions
UPDESH organizes reasoning and generative tasks into distinct categories, with evaluation criteria tailored to each task type. The section outlines criteria for creative writing, reasoning, dialogue, summarization, and translation.
B.2 Task Descriptions
- Tables 7 and 8 organize UPDESH tasks by category, with Table 8 also listing synthesis methods, phases, and model configuration.Task descriptions are provided in Tables 7 and 8.Source: §B.2 · p. 16, Table 7 · p. 17, Table 8 · p. 17
Creative Writing
- Creative-writing evaluation covers instruction adherence, fluency, and narrative coherence.The criteria assess constraint-following, readability, and consistency of plot, character development, and themes.Source: Creative Writing · p. 18
Reasoning Tasks
- Reasoning-task evaluation checks answer adequacy, context and instruction adherence, readability, and problematic content or cultural relevance.The criteria include correctness and completeness, fidelity to supplied context, and culturally sensitive responses.Source: Reasoning Tasks · p. 18
Multi-turn Dialog
- Multi-turn dialogue is evaluated for persona and topic adherence, linguistic plausibility, repetition, toxicity, and following meta-instructions.The criteria cover whether the dialogue maintains its role and focus while remaining natural and non-repetitive.Source: Multi-turn Dialog · p. 18
Summarization
- Summarization criteria cover essential-point coverage, factual accuracy, concision, coherence, and preservation of source style and tone.Evaluation considers both faithful representation and brevity relative to the source.Source: Summarization · p. 18
Translation
- Translation criteria assess semantic and fluency correctness, domain appropriateness, style and tone, and completeness.The criteria include suitable terminology and avoiding omissions or additions.Source: Translation · p. 18
B.4 Results from the quality evaluations of the synthetic data
Human evaluators consistently gave UPDESH data scores of 2 across evaluated tasks and languages, while human–LLM agreement varied by evaluation type. The appendices also report generation and training settings and compare model results.
B.4 Results from the quality evaluations of the synthetic data
- Expert human evaluators consistently gave scores of 2 across tasks and languages, while largest human–LLM disagreements involved regional-language plausibility and long-context evaluation.Agreement was considerable for toxicity and problematic-content detection; LLMs also performed reasonably well on cultural relevance, but diverged on finer multilingual and multicultural judgments.Source: §B.4 · p. 19
B.5 Generation Hyperparameters
- Data synthesis used nucleus sampling with top_p = 0.95 and temperature = 1.0.Source: §B.5 · p. 19
B.6 Training Hyperparameters
- Training hyperparameters for the experiments are provided in Table 9.Source: §B.6 · p. 19, Table 9 · p. 19
B.7 Data Quality Assesment of Reasoning Data
- The quality assessment includes LLM and expert evaluations across five synthetic tasks, human–LLM confusion matrices, and task- and language-level agreement plots.Table 10 reports backtranslation ChrF scores for the reasoning subset.Source: Table 10 · p. 19, Figure 7 · p. 20, Figure 8 · p. 20
B.8 Detailed Results
- UB 32K consistently outperforms baselines across the NLU average, NLG average, and dataset wins shown in Figure 11.The figure places UB 32K in the top-right quadrant for all three metrics.Source: Figure 11 · p. 23
C Cultural Evaluation Framework
The cultural evaluation framework tests model knowledge using community-collected questions across Indian languages and high-impact domains. Sample prompts illustrate task-specific rubrics for creative writing, multi-turn dialogue, and reasoning.
C Cultural Evaluation Framework
- The cultural evaluation uses 4,399 community-collected queries across 11 Indian languages and four domains, with 91,982 pairwise checkpoint comparisons.The domains are healthcare, education, finance, and legal; the questions were collected with a third-party non-profit.Source: C Cultural Evaluation Framework · p. 24
C.1 Sample Prompts Used for Evaluation
- The sample prompts apply established evaluation rubrics to representative tasks.The section provides prompts and scoring criteria for creative writing, dialogue, and reasoning.Source: §C.1 · p. 25
C.1.1 Creative Writing - Instruction Adherence
- Creative-writing instruction adherence is judged by format and constraint compliance, creative intent, completeness, and stylistic and tonal fit.The sample rubric uses scores from 1 to 5 and requests a score with a brief explanation.Source: §C.1.1 · p. 25
C.1.2 Multi-turn Dialog - Persona Adherence
- Multi-turn persona evaluation checks consistent voice and believable interactions, including character consistency and role-appropriate knowledge.The sample rubric uses scores from 1 to 5.Source: §C.1.2 · p. 27
C.1.3 Reasoning Tasks - Answer Adequacy
- Reasoning answer-adequacy evaluation checks direct relevance, clear reasoning, correctness, and completeness.The sample rubric uses scores from 1 to 5.Source: §C.1.3 · p. 28