PaperlayerView exact revision on arXiv

Source-linked AI summary

ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning

Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, Thien Huu Nguyen

arXiv:2304.05613v1cs.CLcs.AI

TL;DR

The paper addresses the lack of comprehensive, public evaluation of ChatGPT across non-English languages and diverse NLP tasks. It evaluates ChatGPT zero-shot across seven tasks and 37 resource-diverse languages, finding generally worse performance than supervised state-of-the-art models. The results support further multilingual research and consideration of task-specific models.

  • Problem

    Comprehensive, public, and independent evidence on ChatGPT’s reliability across non-English languages and diverse NLP tasks remains limited.

  • Method

    The study evaluates ChatGPT zero-shot across seven NLP tasks and 37 languages spanning high-, medium-, low-, and extremely-low-resource categories.

  • Results

    ChatGPT exhibits significantly worse performance than state-of-the-art supervised models for most considered NLP tasks in different languages.

  • Takeaways & Limitations

    The findings support developing or using task-specific models for multilingual NLP applications.

  • Takeaways & Limitations

    The current work does not cover all languages or tasks and evaluates ChatGPT only in the zero-shot setting, preventing comparisons with several other multilingual LLMs.

Abstract

from arXiv · show

Over the last few years, large language models (LLMs) have emerged as the most important breakthroughs in natural language processing (NLP) that fundamentally transform research and developments in the field. ChatGPT represents one of the most exciting LLM systems developed recently to showcase impressive skills for language generation and highly attract public attention. Among various exciting applications discovered for ChatGPT in English, the model can process and generate texts for multiple languages due to its multilingual training data. Given the broad adoption of ChatGPT for English in different problems and areas, a natural question is whether ChatGPT can also be applied effectively for other languages or it is necessary to develop more language-specific technologies. The answer to this question requires a thorough evaluation of ChatGPT over multiple tasks with diverse languages and large datasets (i.e., beyond reported anecdotes), which is still missing or limited in current research. Our work aims to fill this gap for the evaluation of ChatGPT and similar LLMs to provide more comprehensive information for multilingual NLP applications. While this work will be an ongoing effort to include additional experiments in the future, our current paper evaluates ChatGPT on 7 different tasks, covering 37 diverse languages with high, medium, low, and extremely low resources. We also focus on the zero-shot learning setting for ChatGPT to improve reproducibility and better simulate the interactions of general users. Compared to the performance of previous models, our extensive experimental results demonstrate a worse performance of ChatGPT for different NLP tasks and languages, calling for further research to develop better models and understanding for multilingual learning.

1 Introduction

The paper addresses limited comprehensive evaluation of ChatGPT beyond English by testing multilingual performance across diverse tasks and languages. Its results indicate substantial weaknesses relative to supervised models and persistent English bias.

  • Scope and motivation: Existing multilingual evaluations remain limited in their tasks, languages, settings, and evaluation-data scale, leaving no comprehensive public picture of ChatGPT beyond English.Prior examples covered only a few languages or small evaluation samples.
  • Scope and motivation: The study evaluates ChatGPT across 37 languages and seven NLP tasks in a zero-shot setting without human-provided examples.The tasks include NLI, question answering, common sense reasoning, POS tagging, NER, relation extraction, and summarization.
  • Main findings: ChatGPT’s zero-shot performance is generally worse than supervised state-of-the-art models across most tasks and languages, with usually very large gaps.This pattern holds across high-, medium-, low-, and extremely-low-resource languages.
  • Main findings: ChatGPT generally performs better in English than other languages, especially on higher-level tasks requiring complex reasoning.Differences can be substantial for some tasks and lower-resource languages.
  • Main findings: English prompts can improve ChatGPT’s performance even when task inputs are intended for other languages, further indicating an English bias.The finding reinforces concerns about relying on multilingual training alone for non-English NLP applications.

2 Related Work

Related work covers evaluations of ChatGPT, multilingual language-model development, and the contrast between language-specific and multilingual modeling directions.

  • ChatGPT evaluation: Prior ChatGPT studies compare it with state-of-the-art systems across summarization, translation, question answering, extraction, classification, grammatical error detection, and stance detection.Other evaluations examine robustness, logical consistency, and broader societal impacts.
  • Multilingual NLP: Multilingual NLP includes encoder-only, decoder-only, and encoder-decoder language models derived from the Transformer framework.Examples include BERT and RoBERTa, GPT, and BART and T5.
  • Multilingual NLP: Research on non-English languages follows language-specific modeling and multilingual modeling trained on combined data as two major directions.The language-specific direction targets one language, while the multilingual direction develops shared models across languages.
  • Multilingual NLP: Multilingual language models have enabled a new generation of models that significantly boost NLP performance across different languages.This work positions ChatGPT’s multilingual evaluation within that broader development.

3 Methodology

The methodology evaluates ChatGPT on diverse multilingual NLP tasks using zero-shot prompting and resource-stratified languages. It compares results with supervised state-of-the-art references and tests prompt-language choices.

  • Evaluation design: The study evaluates seven tasks across diverse languages spanning high-, medium-, low-, and extremely-low-resource categories.The tasks are POS tagging, NER, relation classification, NLI, QA, common sense reasoning, and summarization.
  • Prompting: ChatGPT is evaluated zero-shot by concatenating a natural-language task description with each input, while supervised state-of-the-art results provide reference points.The setting reflects general users’ inability to fine-tune ChatGPT for downstream tasks.
  • Prompting: The study uses single-stage prompting with simple task descriptions rather than two-stage zero-shot chain-of-thought prompting.Other prompting strategies are reserved for future work.
  • Language stratification: Table 1 classifies languages by their CommonCrawl data ratios as high resource above 1%, medium resource above 0.1%, low resource above 0.01%, or extremely low resource below 0.01%.The table also lists language codes and first- and second-speaker counts.
  • Prompting: For non-English targets, task descriptions are evaluated in both English and the target language to assess multilingual prompting choices.Non-English descriptions are automatically translated from the designed English descriptions.

4 Part-of-Speech Tagging

The study evaluates ChatGPT’s zero-shot multilingual POS tagging with structured prompts across 17 XGLUE-POS languages. ChatGPT outperforms supervised XLM-R in most languages, while English prompts are generally competitive with language-specific prompts.

  • Dataset and task: POS tagging labels each word’s syntactic information and is evaluated on XGLUE-POS test sets covering 17 languages with more than 15K samples.The labels are derived from Universal Dependencies treebanks.
  • Prompt design: ChatGPT’s POS prompt concatenates a task description, output-format note, and a list of sentence words to facilitate word-label alignment.The output uses tuples pairing each input word with its predicted POS tag.
  • Results: ChatGPT outperforms supervised XLM-R on 13 of 17 languages for multilingual POS tagging.Performance is measured by POS-tag accuracy.
  • Results: English prompts generally perform better than or competitively with language-specific prompts across languages.ChatGPT also achieves higher accuracy than English on some languages, including French and Spanish.

5 Named Entity Recognition

The study evaluates ChatGPT for multilingual NER on MultiCoNER using zero-shot prompts and compares it with the supervised DAMO system. ChatGPT substantially underperforms DAMO across all 11 languages, with especially weak extraction for group and creative-work entities.

  • Dataset and task: MultiCoNER evaluates multilingual NER across 11 languages, six entity types, and short sentences from Wikipedia, web questions, and user queries.The entity types are PER, LOC, CORP, CW, GRP, and PROD, using the WNUT 2017 taxonomy.
  • Prompt design: ChatGPT’s NER prompt follows the POS structure but explicitly requests BIO-format tags for each word, which improves performance despite making the task more challenging.The prompt contains a task description, output-format note, and input sentence.
  • Results: ChatGPT significantly underperforms supervised DAMO across all 11 MultiCoNER languages, with F1 scores below 40% in every language.DAMO retrieves relevant Wikipedia context and uses an XLM-RoBERTa-large model with a CRF layer.
  • Results: ChatGPT’s extraction is especially poor for group-of-people and creative-work entities, whose F1 scores are below 15%.The label-wise analysis reports precision, recall, F1, and spurious-prediction percentages.

6 Relation Extraction

The evaluation measures ChatGPT’s multilingual relation extraction using SMiLER and compares zero-shot prompting with a supervised multilingual baseline. ChatGPT generally trails mT5-IL substantially across languages, while prompt-language effects are limited and language-dependent.

  • SMiLER covers 14 languages and 36 relation types, with test sets exceeding 12K samples for evaluation.
  • ChatGPT is prompted with a task description, output-format note, input text, and two entity mentions, while the relation types are explicitly provided.
  • mT5-IL significantly outperforms ChatGPT across languages for relation extraction, except for Swedish and Ukrainian.
  • The performance gap reaches up to 15% in average F1 score across languages.
  • Language-specific prompts generally match or improve on English prompts, with Ukrainian favoring English prompts.

7 Natural Language Inference

The study evaluates multilingual natural language inference on XNLI with zero-shot ChatGPT prompts in English or target languages. ChatGPT performs substantially below supervised mT5-XXL, and English task descriptions consistently work better.

  • XNLI evaluates entailment, contradiction, and neutral relations across English and 14 other languages.
  • Each prompt concatenates a task description, premise, hypothesis, and multiple-choice question.
  • ChatGPT performs significantly worse than mT5-XXL across languages by large margins.
  • The performance gaps between ChatGPT and mT5-XXL appear smaller for high-resource languages.
  • English task descriptions produce significantly higher accuracy than target-language descriptions across all considered languages.

8 Question Answering

The question-answering evaluation uses multilingual XQuAD prompts designed to elicit short answer spans. ChatGPT’s zero-shot performance is below supervised mT5-XXL, with English prompts and English data yielding better results than target-language settings.

  • XQuAD contains 240 paragraphs and 1,190 question-answer pairs in English and ten translated languages.
  • The prompt includes an instruction to return short answers directly extracted from the input passage because ChatGPT tends to generate long responses.
  • Performance is evaluated with exact match and F1 using the SQuAD scorer after removing final periods from responses.
  • ChatGPT’s zero-shot performance is significantly worse than supervised mT5-XXL in every evaluated language.
  • English performance is significantly better than performance in other languages, and English prompts generally outperform target-language prompts.

9 Common Sense Reasoning

The study tests multilingual common-sense reasoning with translated multiple-choice datasets and compares ChatGPT against supervised models. ChatGPT generally underperforms outside English, although language-specific prompts can help in some high-resource languages.

  • Common-sense reasoning uses multiple-choice questions requiring selection among answer choices.
  • X-CSQA contains English data and translations into 15 other languages, while Wikipedia Cloze QA adds multilingual retrieved knowledge.
  • On X-CSQA, ChatGPT performs worse than supervised TRT across all non-English languages when using English task descriptions.
  • For high-resource languages in X-CSQA, language-specific prompts generally outperform English prompts, except for Chinese.
  • IndicNLPSuite results confirm that supervised IndicBERT generally performs better than ChatGPT across the evaluated languages.

10 Summarization

The paper evaluates ChatGPT’s zero-shot multilingual summarization across selected XL-Sum languages using English and language-specific prompts, comparing it with supervised mT5-XXL. ChatGPT performs substantially worse, tends to generate longer summaries, and has lower success rates for lower-resource languages.

  • Dataset and setup: ChatGPT is evaluated on XL-Sum’s abstractive news summarization task, which covers summaries in 44 languages.The experiments select 12 languages spanning high-, medium-, low-, and extremely low-resource categories.
  • Dataset and setup: The evaluation uses ROUGE-1, ROUGE-2, and ROUGE-L on test datasets for 12 selected languages.The reported test-set sizes are provided in Table 10.
  • Evaluation and prompting: The prompts specify both the summarization task and the required output language, because ChatGPT may otherwise respond in English to non-English inputs.The language-specific prompt concatenates a task description with an output-language instruction.
  • Evaluation and prompting: ChatGPT’s summarization performance is compared with mT5-XXL using English and language-specific prompts.mT5-XXL is described as a supervised model trained on language-specific XL-Sum data and reported as state of the art for the dataset.
  • Results: ChatGPT performs consistently worse than mT5-XXL across languages, generates lengthier summaries, and has lower success rates for lower-resource languages.The paper identifies excessive summary length as a possible contributor to poorer ROUGE performance.

11 Discussion

The discussion finds that ChatGPT underperforms supervised models on most multilingual tasks, with POS tagging as a notable exception. English task descriptions often outperform language-specific prompts, while the evaluation remains limited in languages, tasks, models, settings, and metrics.

  • Overall findings: ChatGPT performs significantly worse than state-of-the-art supervised models for most considered NLP tasks across different languages.The authors note that training and API costs may make smaller task-specific models more practical for the considered tasks.
  • Task-specific findings: POS tagging is an exception, with ChatGPT achieving competitive or better performance than supervised models, especially with English prompts.The reported gains include Thai, Vietnamese, Bulgarian, Hindi, and Urdu.
  • Task-specific findings: The authors hypothesize that ChatGPT has stronger grammatical and low-level syntactic abilities than semantic reasoning abilities in multilingual settings.They relate POS tagging’s low-level syntactic focus to the greater difficulty of more complex semantic analysis.
  • Language-resource effects: Resource level alone does not consistently predict performance, since low- and extremely low-resource languages can match or exceed higher-resource languages on some tasks.Examples include Urdu outperforming Vietnamese and Thai in POS tagging and Bengali outperforming Chinese in NER under English prompts.
  • Prompt effects: English task descriptions outperform language-specific prompts on most problems and languages, possibly because English is dominant in ChatGPT’s training data.The finding raises whether English is optimal for prompting different target languages.
  • Limitations: The study is limited by incomplete language and task coverage, evaluation only in zero-shot learning, comparisons with few multilingual LLMs, and reliance on NLP-task performance metrics.Suggested future metrics include robustness, bias, toxicity, accessibility, development cost, and interpretability.

12 Conclusion

The paper presents a broad zero-shot evaluation of ChatGPT’s multilingual NLP abilities across seven tasks and 37 languages spanning multiple resource levels.

  • Conclusion: The evaluation covers seven NLP tasks and 37 diverse languages with high-, medium-, low-, and extremely low-resource levels.The tasks are POS tagging, NER, relation extraction, NLI, question answering, common sense reasoning, and summarization.
Loading 2304.05613v1…