PaperlayerView exact revision on arXiv

Source-linked AI summary

Zero-Shot Cross-Lingual Summarization via Large Language Models

Jiaan Wang, Yunlong Liang, Fandong Meng, Beiqi Zou, Zhixu Li, Jianfeng Qu, Jie Zhou

arXiv:2302.14229v4cs.CLcs.AI

TL;DR

The paper addresses limited evidence about LLM performance on cross-lingual summarization, where a model must summarize across languages. It evaluates prompt-guided zero-shot approaches and finds GPT-4 competitive with fine-tuned mBART-50, while open-source models show limited ability.

  • Problem

    LLM performance on cross-lingual summarization remains underexplored, despite the task’s role in producing summaries of documents in another language.

    Source: §1 · pp. 1–2

  • Method

    The authors evaluate zero-shot CLS prompts across LLMs and datasets, using interactive prompts to shorten preliminary summaries from conversational models.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Across three CLS datasets, GPT-4 achieves state-of-the-art zero-shot CLS performance and is competitive with fine-tuned mBART-50; interactive prompting improves ChatGPT and GPT-4 performance.

    Source: §1 · p. 2, §3.3 · p. 5, §5 · p. 8

  • Takeaways & Limitations

    The authors propose CLS as a testbed for assessing whether multilingual and bilingual LLMs can follow composite instructions and combine translation with summarization.

    Source: §1 · p. 2

    Experiments cover only En⇒Zh and En⇒De, both considered high-resource directions; performance on low-resource languages remains unexplored.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Given a document in a source language, cross-lingual summarization (CLS) aims to generate a summary in a different target language. Recently, the emergence of Large Language Models (LLMs), such as GPT-3.5, ChatGPT and GPT-4, has attracted wide attention from the computational linguistics community. However, it is not yet known the performance of LLMs on CLS. In this report, we empirically use various prompts to guide LLMs to perform zero-shot CLS from different paradigms (i.e., end-to-end and pipeline), and provide a preliminary evaluation on the generated summaries. We find that ChatGPT and GPT-4 originally prefer to produce lengthy summaries with detailed information. These two LLMs can further balance informativeness and conciseness with the help of an interactive prompt, significantly improving their CLS performance. Experimental results on three widely-used CLS datasets show that GPT-4 achieves state-of-the-art zero-shot CLS performance, and performs competitively compared with the fine-tuned mBART-50. Moreover, we also find some multi-lingual and bilingual LLMs (i.e., BLOOMZ, ChatGLM-6B, Vicuna-13B and ChatYuan) have limited zero-shot CLS ability. Due to the composite nature of CLS, which requires models to perform summarization and translation simultaneously, accomplishing this task in a zero-shot manner is even a challenge for LLMs. Therefore, we sincerely hope and recommend future LLM research could use CLS as a testbed.

1 Introduction

The introduction frames zero-shot cross-lingual summarization as an underexplored use of LLMs and evaluates prompt-guided approaches. It reports strong GPT-4 performance alongside persistent difficulty for open-source models.

1 Introduction

  • Across three CLS datasets, GPT-4 achieves the best zero-shot performance, but remains below fine-tuned mBART-50 on ROUGE and BERTScore.The experiments cover news, how-to guides, and dialogue in En⇒Zh and En⇒De.Source: §1 · p. 2
  • Case studies indicate that ChatGPT and GPT-4 can capture source-document core ideas and produce fluent, concise target-language summaries.Source: §1 · p. 2
  • BLOOMZ, ChatGLM-6B, Vicuna-13B, and ChatYuan show limited zero-shot CLS ability, highlighting the challenge of combining translation and summarization end to end.The authors suggest CLS as a testbed for evaluating whether multilingual and bilingual LLMs can follow composite instructions and combine abilities.Source: §1 · p. 2
  • The report presents an initial study of LLMs on CLS, a task for producing a target-language summary of a document written in another language.The introduction notes that LLMs’ performance on CLS had not yet been explored.Source: §1 · pp. 1–2

2 Methodology

The methodology compares prompt designs for zero-shot CLS and evaluates a range of LLMs. It also uses interactive prompting for conversational models to make preliminary summaries more concise.

2.1 Cross-Lingual Summarization Prompts

  • Three prompt paradigms are used: direct target-language summarization, translate-then-summarize, and summarize-then-translate.The two chain-of-thought prompts specify staged operations, but the target-language summary is still generated in a single turn.Source: §2.1 · pp. 2–3
  • An interactive prompt asks conversational LLMs to shorten a preliminary summary after a direct, translate-then-summarize, or summarize-then-translate prompt.The resulting processes are denoted Dir+Inta, TS+Inta, and ST+Inta.Source: §2.1 · p. 3

2.2 Large Language Models

  • The study explores Davinci-003, ChatGPT, GPT-4, BLOOMZ, ChatGLM-6B, Vicuna-13B, and ChatYuan; only conversational LLMs receive the interactive prompt for zero-shot CLS evaluation.BLOOMZ is an open-source multilingual LLM with 176B parameters and supports 59 languages.Source: §2.2 · pp. 3–4

3 Experiments

Across three CLS datasets, GPT-4 is the strongest zero-shot LLM, while interactive prompting improves conversational LLM results and summary concision. Open-source models lag, and ChatGPT-based evaluation favors GPT-4 on most aspects but does not consistently reward interactive prompting.

3.1 Experimental Setup

  • The evaluation covers CrossSum, WikiLingua, and XSAMSum in selected English-to-Chinese and English-to-German directions, using ROUGE and BERTScore against fine-tuned mBART-50.ChatGPT, GPT-4, and Vicuna-13B were evaluated on 50 randomly sampled test documents per dataset because manual evaluation was time-consuming.Source: §3.1 · p. 4

3.2 Implementation Details

  • Vicuna-13B was evaluated only on XSAMSum because its demo truncates long inputs; ChatGLM-6B and ChatYuan were evaluated only English-to-Chinese.The mBART-50 baseline was fine-tuned separately on each CLS dataset for 10 epochs.Source: §3.2 · pp. 4–5

3.3 Main Results

  • GPT-4 achieves the best zero-shot CLS performance among LLMs, though it remains slightly below fine-tuned mBART-50 on automatic metrics.Davinci-003, ChatGPT, and GPT-4 are also reported as competitive with the fine-tuned baseline.Source: Table 2 · p. 5, §3.3 · pp. 5–6
  • Chain-of-thought prompts outperform the direct prompt, while open-source LLMs perform poorly on longer CrossSum and WikiLingua documents and trail GPT-4 on XSAMSum.The authors attribute the broader challenge to CLS requiring summarization and translation simultaneously.Source: §3.3 · pp. 5–6
  • 183.7 vs. 66.4 tokens for ChatGPT (Dir vs. Dir+Inta) on CrossSum shows the interactive prompt's effect on summary length.The authors report that interactive prompting helps balance informativeness and concision while improving zero-shot CLS ability.Source: §3.3 · pp. 5–6, Table 3 · p. 6

3.4 LLM-based Evaluation

  • Human comparisons with fine-tuned models may be biased because their summary styles differ and evaluators can recognize which system produced them.The authors therefore call for more carefully designed human evaluation.Source: §3.4 · pp. 6–7
  • On XSAMSum, GPT-4 achieves the best ChatGPT-judged performance in most aspects; GPT-4 (Dir) scores 64.0, 48.6, 67.6, and 67.0 across the four aspects.The authors report that these GPT-4 scores are significantly better than fine-tuned mBART-50; interactive prompting does not show the same consistent gains under LLM-based evaluation as under automatic metrics.Source: Table 4 · p. 7, §3.4 · p. 8
  • ChatGPT scored generated summaries reference-free on coherence, relevance, consistency, and fluency, using aspect-specific instructions illustrated by Figure 4.The instructions were inspired by SummEval, and the evaluation used the gpt-3.5-turbo API at temperature 0.Source: §3.4 · pp. 7–8, Figure 4 · p. 8

4 Related Work

Prior CLS work developed pipeline and end-to-end approaches, but had not explored LLMs’ CLS performance. Related LLM studies examined other NLP tasks, including translation and question answering.

4.1 Cross-Lingual Summarization

  • Earlier CLS work moved from translation-then-summarization or the reverse to end-to-end models, but had not examined LLM performance on CLS.The reviewed end-to-end approaches include multi-task learning, knowledge distillation, resource enhancement, and pre-training.Source: §4.1 · p. 8

4.2 Large Language Models

  • Studies of LLMs examined downstream tasks such as machine translation, code-mixed text generation, and knowledge-based question answering.Other cited work used ChatGPT or GPT-4 as an evaluation metric for task-specific models.Source: §4.2 · p. 8

5 Conclusion and Future Work

The report finds that Davinci-003, ChatGPT, and GPT-4 can perform zero-shot cross-lingual summarization competitively with fine-tuned mBART-50, while open-source models generally show limited end-to-end ability. It proposes few-shot learning as future work.

5 Conclusion and Future Work

  • Davinci-003, ChatGPT, and GPT-4 achieve competitive zero-shot CLS results against fine-tuned mBART-50.Source: §5 · p. 8
  • BLOOMZ, ChatGLM-6B, Vicuna-13B, and ChatYuan generally show limited ability to perform CLS end to end.The report says the challenge of zero-shot CLS remains.Source: §5 · p. 9
  • The authors propose exploring few-shot learning to further use LLMs for CLS.Source: §5 · p. 9

Limitations

The report describes its evaluation as a lower bound on model CLS performance and identifies limits in language coverage, model comparisons, and evaluation methodology.

Limitations

  • The evaluation gives only a lower threshold of model CLS performance, and better prompts may improve results.Source: Limitations · p. 9
  • Experiments cover only En⇒Zh and En⇒De, both high-resource directions, leaving low-resource CLS performance unexplored.The authors conjecture that limitations in ChatGPT’s low-resource translation ability might also apply to CLS.Source: Limitations · p. 9
  • LLM comparisons are not rigorous because decoding strategies differ, and the authors identify this as a major source of limited soundness.They propose future human evaluation for further analysis.Source: Limitations · p. 9
Loading 2302.14229v4…