Source-linked AI summary
XFEVER: Exploring Fact Verification across Languages
Yi-Chen Chang, Canasai Kruengkrai, Junichi Yamagishi
TL;DR
Fact-verification models require large labeled datasets, while building them for each language is costly. This paper introduces XFEVER by translating FEVER into multiple languages and evaluates zero-shot and translate-train approaches; it reports effective translate-train learning with high-quality machine translations and reduced miscalibration from symmetric-divergence consistency regularization.
Problem
Building large labeled fact-verification datasets from scratch for each language is costly, limiting how readily models can be built for other languages.
Source: §1 · p. 1Method
The paper translates FEVER claims and evidence into five target languages to create XFEVER and evaluates zero-shot and translate-train learning scenarios.
Source: §1 · p. 1, §3.2 · p. 4Results
Translate-train learning with high-quality machine-translated data can be effective, and consistency regularization with symmetric divergence measures can help reduce miscalibration.
Source: §6.2.3 · p. 7, §7 · p. 8Takeaways & Limitations
Multilingual pre-trained language models substantially improve Japanese and Chinese zero-shot accuracy over monolingual models when target-language training data are unavailable.
Source: §6.2.1 · p. 6Translate-train learning assumes that machine-translated data are available for training.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
This paper introduces the Cross-lingual Fact Extraction and VERification (XFEVER) dataset designed for benchmarking the fact verification models across different languages. We constructed it by translating the claim and evidence texts of the Fact Extraction and VERification (FEVER) dataset into six languages. The training and development sets were translated using machine translation, whereas the test set includes texts translated by professional translators and machine-translated texts. Using the XFEVER dataset, two cross-lingual fact verification scenarios, zero-shot learning and translate-train learning, are defined, and baseline models for each scenario are also proposed in this paper. Experimental results show that the multilingual language model can be used to build fact verification models in different languages efficiently. However, the performance varies by language and is somewhat inferior to the English case. We also found that we can effectively mitigate model miscalibration by considering the prediction similarity between the English and target languages. The XFEVER dataset, code, and model checkpoints are available at https://github.com/nii-yamagishilab/xfever.
1 Introduction
The paper asks whether fact-checking models for other languages can be built without the cost of creating large labeled datasets from scratch. It introduces XFEVER, built by translating FEVER, and defines zero-shot and translate-train evaluation scenarios.
1 Introduction
- Deep models require massive labeled datasets, while building a FEVER-scale database from scratch for each language is time-consuming and costly.The paper frames this as the central barrier to building fact-checking models for other languages.Source: §1 · p. 1
- XFEVER extends FEVER by machine-translating claims and evidence into Spanish, French, Indonesian, Japanese, and Chinese, with professionally translated texts available for additional evaluation.The languages cover several language families, including Japanese, described as an isolated language.Source: §1 · p. 1
- The paper hypothesizes that claim–evidence relations should remain consistent across languages when translations preserve the original information.This consistency motivates translating FEVER claim–evidence pairs for cross-lingual verification.Source: §1 · p. 1, Figure 1 · p. 2
- Zero-shot learning trains on English only, whereas translate-train learning builds a multilingual model using English and machine-translated target-language data.The paper also evaluates prediction or representation similarity between English and target-language examples as part of the translate-train loss.Source: §1 · pp. 1–2
2 Related Work
Related work situates automated fact-checking in a pipeline from evidence retrieval to verdict prediction and reviews existing fact-checking datasets and cross-lingual benchmarks. It identifies textual entailment and multilingual fact-checking as relevant neighboring tasks.
Automated fact-checking
- Automated fact-checking is described as retrieving relevant documents, selecting evidence sentences or paragraphs, and predicting a verdict from that evidence.The verdict prediction step tests whether evidence entails or contradicts the claim.Source: §2 · p. 2
Fact-checking datasets
- FEVER contains 185,445 claims checked against Wikipedia, while Wiki-FactCheck has 124K examples and VitaminC tests sensitivity to subtle factual changes.Changes in evidence over time make sensitivity to subtle factual differences relevant to fact-checking.Source: §2 · p. 2
Datasets for cross-lingual understanding tasks
- Cross-lingual benchmarks include XTREME and XGLUE; XNLI is a related professionally translated 15-language inference corpus, and X-FACT contains 31,189 real-world claims.XNLI and XFEVER are described as artificially created datasets with similar data distributions across languages.Source: §2 · pp. 2–3
3 The XFEVER dataset
XFEVER extends FEVER claim–evidence pairs into six languages and provides machine-translated data alongside a small professionally translated evaluation set. It supports zero-shot and translate-train learning scenarios.
3.1 Overview
- XFEVER uses the FEVER version limited to claims requiring single-sentence evidence and covers English plus Spanish, French, Indonesian, Japanese, and Chinese.English is the source language; the five others are target languages.Source: §3.1 · p. 3
- DeepL translated the English data into five target languages, and test-6h contains 600 randomly selected claim–evidence pairs translated and verified by professional translators.The test set was created to analyze whether translation methods affect prediction accuracy.Source: §3.1 · p. 3
- Translated pairs retain the original labels: supported, refuted, or not enough information.Source: §3.1 · p. 3
3.2 Two scenarios
- Zero-shot learning trains with English training and development data only, then evaluates on test data in all languages.Source: §3.2 · pp. 3–4
- Translate-train learning uses machine-translated data and trains on the training and development sets in all languages simultaneously, with evaluation matching the zero-shot scenario.Source: §3.2 · p. 4
4 Cross-lingual fact verification
The paper formulates cross-lingual fact verification as classification with a pre-trained language model and an MLP, then compares English-only zero-shot training with training on English and translated data. Translate-train methods mix examples or explicitly pair originals with translations to promote prediction consistency.
4 Cross-lingual fact verification
- A PLM encodes concatenated claim-evidence input, and an MLP maps its representation to label logits for cross-lingual fact verification.The model applies softmax to obtain predicted probabilities.Source: §4 · p. 4
4.1 Zero-shot learning scenario
- Zero-shot training uses only the original English data, with a multilingual PLM expected to support application to other languages.The paper defines original data as nontranslated English data.Source: §4.1 · p. 4
4.2 Translate-train learning scenario
- Translate-train assumes machine-translated data are available and trains on English plus the five target languages.The target languages are Spanish, French, Indonesian, Japanese, and Chinese.Source: §4.2 · p. 4
4.2.1 Non-parallel training
- Non-parallel training mixes available examples and reshuffles them each epoch, sampling each example from English or translated data.The mixed set contains N×(|T|+1) examples.Source: §4.2.1 · p. 4
4.2.2 Parallel training
- Parallel training explicitly pairs original and translated examples to encourage consistent predictions, with a loss combining both cross-entropy terms and regularization.The translated example is randomly selected from the translated data after epoch-level reshuffling.Source: §4.2.2 · p. 4
5 Consistency regularization
The paper explores consistency regularization between predictions for original and translated examples. It considers divergences between their predicted distributions, including asymmetric and symmetric choices.
5 Consistency regularization
- The paper examines prediction- and representation-based regularization, and relates prediction consistency to the confidence penalty.It positions these as alternatives to the specific consistency-regularization forms in prior work (Zheng et al., 2021; Yang et al., 2022).Source: §5 · pp. 4–5
5.1 Prediction consistency
- KL divergence pushes the translated example’s predicted distribution toward the original, under the hypothesis that the original prediction is more accurate.The paper also considers symmetric Jeffreys divergence, which pushes both distributions toward each other.Source: §5.1 · p. 5
- Jensen–Shannon divergence is another symmetric option, and unlike KL and Jeffreys divergence, it is bounded.Source: §5.1 · p. 5
Relationship between prediction consistency and confidence penalty
The paper connects prediction consistency to confidence penalties that discourage overconfident predictions, and separately applies consistency constraints to intermediate representations. It considers both penultimate-layer and PLM-feature representations, with distance-based measures.
Relationship between prediction consistency and confidence penalty
- A confidence penalty adds negative entropy to cross-entropy, encouraging a smoother, less peaky predicted distribution.The model maximizes prediction entropy to minimize this penalized loss.Source: Relationship between prediction… · p. 5
- Prediction-consistency regularization intrinsically introduces a confidence penalty, and the resulting loss includes negative entropy terms that should help reduce overconfidence.The paper illustrates this connection using parallel training with Jeffreys divergence.Source: Relationship between prediction… · p. 5
5.2 Representation consistency
- Representation consistency can be imposed before the logits at either the penultimate layer or the PLM feature output.The paper denotes the original and translated representations as h and h̃.Source: §5.2 · p. 6
- MSE penalizes squared Euclidean distance, while cosine distance compares representation direction without regard to vector magnitudes.For cosine distance, the vectors are normalized to unit length.Source: §5.2 · p. 6
6 Experiments
Experiments compare multilingual models and translated-data training for cross-lingual fact verification, while examining calibration and translation quality. They find gains in zero-shot performance and calibration, with similar accuracy on machine- and human-translated test data.
6.1 Training details
- The experiments compare multilingual and monolingual PLMs, including models pretrained on data covering all six XFEVER languages, and train for up to ten epochs with early stopping.The listed training settings include batch size 32 and λ=1 for consistency regularization unless otherwise specified.Source: §6.1 · p. 6
6.2.1 Effect of multilingual PLMs in zero-shot learning
- In zero-shot learning, RoBERTa-large→XLM-R-large improved Japanese and Chinese accuracy by +43% and +45.6%, respectively.Multilingual PLMs also yielded average gains over their monolingual counterparts.Source: §6.2.1 · p. 6
6.2.2 Effect of translate-train learning on performance improvement
- With machine-translated data, non-parallel Jnp training works for most target languages, while regularization and representation choices have less effect on accuracy.Combining prediction and representation consistencies improves mBERT accuracy, but the effects diminish with XLM-R-large.Source: §6.2.2 · p. 6
6.2.3 Effect of consistency regularization in reducing miscalibration
- Symmetric J and JS divergence significantly reduce ECE by encouraging high-entropy predictions for original and translated examples.The authors recommend prediction consistency with J or JS when lower ECE is preferred, despite slight accuracy differences among regularizers.Source: §6.2.3 · p. 7
6.2.4 Performance comparison of human- and machine-translated data
- Accuracy differences between machine- and human-translated test-6h data average only 0.3∼0.7%.The authors attribute the small discrepancies to DeepL’s accurate translations and conclude translate-train works with high-quality translated data.Source: §6.2.4 · pp. 7–8
7 Conclusion
XFEVER addresses the shortage of non-English fact-verification benchmarks with a dataset for cross-lingual verification. The paper reports effective translate-train learning with high-quality machine-translated data and reduced miscalibration from symmetric-divergence consistency regularization.
7 Conclusion
- The authors introduce XFEVER to address the lack of non-English fact-verification benchmarks.False claims can spread across languages, and online claims may cause real-world harm.Source: §7 · p. 8
- Translate-train learning with high-quality machine-translated data can be effective, and symmetric-divergence consistency regularization can help reduce miscalibration.Source: §7 · p. 8
- Future work will examine training with only a few examples when large machine-translated datasets are unavailable.Source: §7 · p. 8
- The authors also plan to expand human-translated XFEVER data to more languages, especially low-resource ones.Source: §7 · p. 8
A Additional results
Additional experiments examine the consistency-regularization setting for J divergence with XLM-R-large. The authors report that λ = 1 works well, suggest a possible explanation, and heuristically reduce λ to 0.25.
A Additional results
- λ = 1 works well with J divergence and XLM-R-large in preliminary experiments.Source: A Additional results · p. 10
- A plausible explanation is that J divergence penalizes the loss more heavily than other divergence measures.Source: A Additional results · p. 10
- The authors heuristically reduce λ to 0.25 for J divergence to alleviate the issue.Tables 8 and 9 report XLM-R-large test-set accuracy and ECE scores, respectively.Source: A Additional results · p. 10
B Machine vs. human translations
The passages identify test-set examples of machine- and human-translated texts and describe two XLM-R-large test-set tables: one for accuracy and one for ECE.
B Machine vs. human translations
- Table 8 reports XLM-R-large accuracy scores on the test set; its caption defines Pred, Repr, feat, and penu.Source: Table 8 · p. 11
- Table 9 reports XLM-R-large test-set ECE scores, for which lower is better.Source: Table 9 · p. 11
- Table 10 presents claim-and-evidence examples from six languages in XFEVER’s test-6h set, comparing DeepL machine translations with professional human translations.Source: B Machine vs. human translations · p. 10, Table 10 · p. 11