Source-linked AI summary
PingPong: A Natural Benchmark for Multi-Turn Code-Switching Dialogues
Mohammad Rifqi Farhansyah, Hanif Muhammad Zhafran, Farid Adilazuarda, Shamsuddeen Hassan Muhammad, Maryam Ibrahim Mukhtar, Nedjma Ousidhoum, Genta Indra Winata, Ayu Purwarianti, Alham Fikri Aji
TL;DR
Existing code-switching dialogue benchmarks are underexplored, outdated, and increasingly misaligned with everyday multilingual communication. PINGPONG introduces a human-authored, multi-party benchmark spanning five language combinations and three downstream tasks, while evaluations reveal persistent limitations in current language models.
Problem
Existing code-switching dialogue benchmarks are underexplored, increasingly outdated, and less reflective of everyday multilingual communication.
Method
PINGPONG uses manually crowdsourced conversations from fluent multilingual speakers across five language combinations, supporting Question Answering, Summarization, and Topic Classification.
Results
Evaluations of open-source and proprietary language models reveal persistent limitations in modeling code-switched, multi-party conversations, with regionally designed models generally outperforming general-purpose multilingual models.
Takeaways & Limitations
PINGPONG identifies current NLP limitations and provides a foundation for developing more robust, inclusive systems for the world’s multilingual majority.
Takeaways & Limitations
The benchmark covers four under-studied languages across five combinations and three geographic regions, so its scope is not exhaustive and remains extendable.
Abstract
from arXiv · showhide
Code-switching is a widespread practice among the world's multilingual majority, yet few benchmarks accurately reflect its complexity in everyday communication. We present PingPong, a benchmark for natural multi-party code-switching dialogues covering five language-combination variations, some of which are trilingual. Our dataset consists of human-authored conversations among 2 to 4 participants covering authentic, multi-threaded structures where replies frequently reference much earlier points in the dialogue. We demonstrate that our data is significantly more natural and structurally diverse than machine-generated alternatives, offering greater variation in message length, speaker dominance, and reply distance. Based on these dialogues, we define three downstream tasks: Question Answering, Dialogue Summarization, and Topic Classification. Evaluations of several state-of-the-art language models on PingPong reveal that performance remains limited on code-switched inputs, underscoring the urgent need for more robust NLP systems capable of addressing the intricacies of real-world multilingual discourse.
1 Introduction
PINGPONG introduces a benchmark for natural, multi-party code-switching dialogues that addresses limitations in existing resources through broad language coverage, diverse conversational structure, and evaluation across three downstream tasks.
- Motivation: PINGPONG targets underexplored code-switched dialogue, while existing benchmarks are increasingly outdated and less reflective of everyday code-switching.The benchmark is motivated by insufficient evaluation of multilingual dialogue systems and limitations in prior datasets.
- Benchmark Design: PINGPONG contains authentic interactions among 2 to 4 speakers, including dialogues from 17 to 189 turns with long-distance replies to earlier messages.These properties are intended to capture the fluid dynamics of multilingual communication in conversational settings.
- Benchmark Design: The benchmark spans five language combinations across high-resource, low-resource, and diverse-script settings.This broad coverage supports evaluation across varied multilingual scenarios.
- Evaluation Tasks: PINGPONG evaluates Question Answering, Dialogue Summarization, and Topic Classification under code-switching conditions.Together, these tasks assess language understanding and generation across multiple aspects of multilingual dialogue.
- Evaluation Tasks: Experiments with open-source and proprietary LLMs reveal persistent limitations in modeling code-switched, multi-party conversations.The evaluation framework is designed to expose challenges for current systems.
2 Related Work
Existing code-switching datasets cover many NLP tasks but remain limited in release sustainability, linguistic diversity, language coverage, and conversational realism. Their evaluation tasks also often fail to reflect contemporary NLP challenges.
- Dataset limitations: Existing code-switching datasets face limitations in public availability, linguistic diversity, and language or regional coverage.Several datasets are not publicly released, most focus on bilingual settings, and many languages remain under-explored.
- Dataset limitations: Most existing benchmarks focus exclusively on bilingual code-switching, with only one dataset explicitly addressing trilingual scenarios.
- Conversational realism: Many datasets do not capture spontaneous, interactive code-switching in real-world conversational settings.
- Evaluation scope: Existing evaluation tasks are frequently limited to traditional benchmarks that no longer reflect contemporary NLP challenges.
3 PINGPONG Dataset
PINGPONG is built from manually collected, native-speaker conversations across five language combinations, including trilingual settings. Its dialogues support question answering, summarization, and topic classification while preserving natural multi-party and code-switching structure.
- Data collection: PINGPONG covers five language combinations, including Indonesian–English, three trilingual combinations, Hausa–English, and Algerian Arabic–Standard Arabic–French.Native speakers collect dialogues and curate downstream tasks for each language combination.
- Conversational Dialogue Collection: Participants hold 15-minute conversations on assigned topics and may code-switch at word, sentence, paragraph, or utterance levels.Discord replies allow participants to respond directly to specific utterances while preserving dialogue structure.
- Downstream tasks: PINGPONG supports question answering, dialogue summarization, and multi-label topic classification across predefined topic categories.Topics include science and technology, entertainment, society and culture, education, and daily life.
- Dataset statistics: Dialogue statistics measure linguistic complexity using Code-Mixing Index and Switch Point Fraction.CMI measures mixing intensity, while SPF measures the frequency of transitions between languages.
- Dataset analysis: Human dialogues show participation imbalance, multi-threaded replies, and highly variable utterance lengths, unlike more monotonous machine-generated dialogues.Machine-generated conversations tend toward rigid turn-taking, uniform speaker participation, consistent utterance lengths, and immediate linear replies.
4 Experimental Setup
The experiments evaluate diverse model families across PINGPONG’s three tasks using task-specific prompts, shot settings, reasoning configurations, output constraints, and metrics. Models span English-centric, multilingual, regional, instruction-tuned, and reasoning-oriented systems.
- Models: The study evaluates models across question answering, dialogue summarization, and topic classification.Models are categorized by linguistic coverage and include base, reasoning-oriented, and instruction-tuned variants.
- Models: The evaluated models include English-centric, multilingual, and region-specific systems covering the study’s target languages.
- Reasoning behavior: The experiments compare reasoning-enabled behavior with settings where models generate explicit reasoning traces through prompting.Qwen3 provides native thinking traces, while other models can be prompted to produce reasoning steps.
- Prompting: Prompting varies the number of in-context examples, with QA using zero- and one-shot settings and summarization using zero-, one-, and three-shot settings.Examples are drawn from dialogues outside the evaluation set.
- Output format: Models are instructed to return JSON outputs, and selected configurations include reasoning traces for analyzing model logic.
- Evaluation metrics: QA and topic classification use accuracy, while dialogue summarization uses ROUGE, METEOR, CHRF++, and BERTScore, with ROUGE-L reported in the main paper.
5 Results and Analysis
Performance is generally poor across PINGPONG’s downstream tasks, although regionally designed models generally outperform general-purpose multilingual models. Reasoning usually helps, while few-shot prompting provides inconsistent benefits outside summarization, and language-level comparisons remain difficult to interpret.
- Overall Results: Most evaluated models perform poorly across PINGPONG’s downstream tasks, indicating substantial challenges for current systems.
- Overall Results: Regionally designed models generally outperform general-purpose multilingual models.The authors attribute the observed benefit to specialization for particular linguistic and cultural characteristics.
- Limitations: Non-parallel dialogue content prevents reliable conclusions about whether any language is inherently more difficult than another.Observed performance differences remain tied to the specific conversation content of each language.
- Reasoning Behavior: Reasoning-enabled configurations generally improve performance across the benchmark’s tasks.The authors associate this pattern with additional computation and internal verification from reasoning traces.
- Prompting: Few-shot examples do not consistently improve QA or topic classification, but they significantly improve dialogue summarization.
- Reasoning Behavior: Reasoning improves answerable questions, whereas unanswerable cases benefit only from explicit Qwen3 reasoning traces and may decline with implicit reasoning.Few-shot effects are model-dependent, ranging from marginal gains to no improvement or slight regressions.
6 Conclusion
PINGPONG captures authentic human multilingual discourse across five language combinations and shows that current state-of-the-art models still struggle with natural, multi-party code-switching.
- PINGPONG captures authentic complexity in human multilingual discourse across five language combinations.
- Human-authored conversations among 2 to 4 participants include multi-threaded dynamics and varied speaker dominance often missing from synthetic corpora.
- The dataset analysis finds greater naturalness and structural diversity than machine-generated alternatives, particularly in message length and reply distances.
- PINGPONG evaluates Question Answering, Dialogue Summarization, and Topic Classification to address gaps in prior benchmarks.
- State-of-the-art models still struggle with the intricacies of natural, multi-party code-switching.
Limitations
The benchmark covers four complementary under-studied languages across five combinations and three geographic regions, but does not capture every code-switching variation.
- The study covers 4 complementary under-studied languages across 5 language combinations spanning 3 geographic regions.
- The benchmark’s scope is not exhaustive across all code-switching variations in its five language combinations.
- The authors describe the dataset as a scalable foundation that can be extended to additional languages and regions.
- The study uses a relaxed CMI formulation because reliable tools for advanced CMI metrics are unavailable for the under-studied languages.
Ethical Considerations
The benchmark’s data-collection materials organize recruitment, language assessment, dialogue construction, and downstream annotation procedures for multilingual participants and tasks.
- Ethical Considerations: Annotators receive compensation above the local minimum wage and detailed instructions, with demographic collection conducted using informed consent.
- Ethical Considerations: Annotators are instructed not to use offensive language, although some instances may be missed and later reported cases may prompt benchmark updates.
- Data Collection: Recruitment forms collect demographic information, self-reported language use, and language-assessment responses for annotator selection.
- Data Collection: Language assessment asks respondents to compose a short paragraph using a specified language combination to simulate natural code-switching behavior.
- Annotation: Guidelines cover dialogue construction, Question Answering, Dialogue Summarization, and Naturalness scoring to support consistent dataset creation.
F Quantitative Results
The quantitative-results section organizes findings across experimental configurations, including Question Answering comparisons of reasoning settings and prompting strategies.
- The section provides a comprehensive overview of quantitative findings across all experimental configurations.
- Question Answering: Question Answering results compare reasoning-enabled with non-reasoning configurations.
- Question Answering: Question Answering results also compare zero-shot with one-shot prompting strategies.
- Question Answering: Table 8 reports per-language-combination accuracy statistics for reasoning versus non-reasoning approaches.
- Question Answering: Table 9 reports per-language-combination accuracy statistics for zero-shot versus one-shot approaches.
F.2 Text Summarization
The text summarization experiments examine reasoning mechanisms and prompt shot size, with detailed results reported across five language combinations. They also compare human-written and machine-generated conversations.
- The text summarization evaluation examines the role of reasoning mechanisms and prompt shot size.
- Reasoning vs. Non-Reasoning: Reasoning and non-reasoning approaches are compared across five language combinations using detailed metric tables.The comparisons cover ID-EN, JV-ID-EN, SU-ID-EN, HA-EN, and ar_DZ-FR.
- 0-shot vs. Few-shot: Zero-shot and few-shot approaches are compared across the same five language combinations for text summarization.The reported metrics include ROUGE, METEOR, CHRF++, and BERTScore measures.
- Human-written and machine-generated conversations are compared using overall quantitative statistics.
G Additional Results
The additional results describe unanswerable-question categories, evaluation metrics across language combinations and prompting conditions, and annotation and prompt guidelines. They also include comparisons between human-written and machine-generated conversational text.
- Question Answering annotation targets five unanswerable categories: Negation, Antonym, Entity-Swap, Mutual-Exclusion, and Impossible-Condition.Three language combinations produced up to five unanswerable questions per instance: ID-EN, JV-ID-EN, and SU-ID-EN.
- Text summarization tables report ROUGE-1 through ROUGE-4, METEOR, CHRF++, and BERTScore precision, recall, and F1 across five language combinations.
- Additional summarization tables compare zero-shot with few-shot approaches across the same five language combinations.
- The results include full quantitative statistics comparing human-written and machine-generated conversational patterns.
- The appendix provides annotator guidelines for Question Answering, Dialogue Summarization, and human-versus-machine naturalness judgments.
- Prompt templates specify task instructions for Question Answering, Dialogue Summarization, Topic Classification, and machine-generated conversational text.The Question Answering and Dialogue Summarization templates include optional few-shot components.