PaperlayerView exact revision on arXiv

Source-linked AI summary

I Wish I Would Have Loved This One, But I Didn't -- A Multilingual Dataset for Counterfactual Detection in Product Reviews

James O'Neill, Polina Rozenshtein, Ryuichi Kiryo, Motoko Kubota, Danushka Bollegala

arXiv:2104.06893v2cs.CLcs.AIcs.LG

TL;DR

Labeled counterfactual-detection resources were limited to English, and cue-phrase-based dataset selection can introduce selection bias. The paper builds a professionally annotated English, German, and Japanese Amazon-review dataset and trains classifiers on it. The models show robustness to selection bias, while machine translation performs poorly for cross-lingual classification, indicating a need for language-specific datasets.

  • Problem

    Existing labeled counterfactual datasets were limited to English, and cue-phrase-based sentence selection can introduce selection bias.

    Source: §1 · p. 2

  • Method

    The authors annotate Amazon product-review sentences in English, German, and Japanese and train counterfactual classifiers using different encoders and classification algorithms.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 5

  • Results

    The models are relatively robust to clue-phrase selection bias, whereas machine translation yields poor cross-lingual classification performance.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that cross-lingual counterfactual detection requires language-specific datasets; their English data is compatible with SemEval-2020 and can be merged to improve classifier accuracy.

    Source: §1 · p. 2, §5 · p. 9

    The authors suspect that social biases encoded in pretrained mBERT and XLM-RoBERTa may propagate into, and could be amplified during, CFD model training.

    Source: §6 · p. 9

Abstract

from arXiv · show

Counterfactual statements describe events that did not or cannot take place. We consider the problem of counterfactual detection (CFD) in product reviews. For this purpose, we annotate a multilingual CFD dataset from Amazon product reviews covering counterfactual statements written in English, German, and Japanese languages. The dataset is unique as it contains counterfactuals in multiple languages, covers a new application area of e-commerce reviews, and provides high quality professional annotations. We train CFD models using different text representation methods and classifiers. We find that these models are robust against the selectional biases introduced due to cue phrase-based sentence selection. Moreover, our CFD dataset is compatible with prior datasets and can be merged to learn accurate CFD models. Applying machine translation on English counterfactual examples to create multilingual data performs poorly, demonstrating the language-specificity of this problem, which has been ignored so far.

1 Introduction

Prior labeled counterfactual-detection datasets covered only English, while product reviews and other languages remained unaddressed. This paper introduces a professionally annotated multilingual review dataset and reports results on model robustness, dataset compatibility, and machine translation.

1 Introduction

  • Existing labeled counterfactual datasets were limited to English, leaving product reviews and additional languages uncovered.Prior datasets covered tweets and finance, politics, and healthcare.Source: §1 · pp. 1–2
  • The paper annotates Amazon product-review sentences in English, German, and Japanese, using professional linguists and a balanced counterfactual class.At least 10% of samples in each language represent the positive counterfactual class.Source: §1 · p. 2
  • Cue-phrase selection can introduce selection bias, but contextualized masked-language-model classifiers are reported to be robust against it.The paper compares models trained with and without masking clue phrases and also tests adding semantically similar sentences without clue phrases.Source: §1 · p. 2
  • Merging the English dataset with SemEval improves counterfactual-classifier accuracy, while machine-translated English examples produce poor cross-lingual models.The authors interpret the translation result as indicating language-specificity in counterfactual detection.Source: §1 · p. 2
  • Contextualized sentence embeddings outperform noncontextualized representations, supporting the authors’ conclusion that counterfactual detection is context-sensitive.Source: §1 · p. 2

2 Related Work

Prior counterfactual research includes both identifying real-world counterfactual statements and augmenting data with counterfactual examples. Existing datasets and modeling approaches provide context for this paper’s focus on e-commerce reviews.

2 Related Work

  • Counterfactual data augmentation aims to reduce reliance on spurious associations, unlike this paper’s task of identifying existing counterfactuals for search filtering.Kaushik et al. (2020) and Gardner et al. (2020) are cited in this contrast.Source: §2 · pp. 2–3
  • Earlier CFD datasets covered news or social media, were English-only, and used crowdsourced binary labels; e-commerce reviews were not covered.The paper positions its multilingual product-review dataset as addressing these gaps.Source: §2 · p. 3
  • High-performing SemEval systems commonly used pretrained language models and ensembles, while traditional SVM and random-forest methods had less success.Systems also varied in their additional CNN, recurrent, or parsing components; prior work addressed imbalance with pseudo-labelling and multi-sample dropout.Source: §2 · p. 3

3 Dataset Curation

The dataset operationalizes counterfactuals using linguistic patterns, then extends collection and annotation across English, German, and Japanese product reviews. Its design addresses clue-phrase selection bias and aligns the resulting dataset with prior counterfactual corpora.

3 Dataset Curation

  • The curation adopts a definition of counterfactuals as hypothetical changes to past experience and extends annotation guidance to German and Japanese with professional linguists.The Japanese guidelines required rewriting to address syntactic and orthographic differences.Source: §3 · pp. 3–4

3.1 Data Collection

  • The two-stage collection combines clue-phrase matches with clue-free sentences selected for similarity to clustered multilingual-BERT embeddings.The first iteration uses verified-purchase reviews containing exactly one clue phrase; the second seeks roughly as many clue-free sentences for annotation.Source: §3.1 · p. 4

3.2 Annotation

  • At least 90% of sentences required agreement from two professional linguists, with a third linguist resolving the remaining disagreements.A sentence was labelled counterfactual if any part contained a counterfactual.Source: §3.2 · p. 5

3.3 Dataset Statistics

  • EN-ext contains 1/10 counterfactuals, compared with about 1/5 in EN; just 76 of 4,977 added sentences were positive, while DE and JP contain 69.1% and 9.5%, respectively.German clues had higher precision than English clues despite similar list sizes; Japanese had the largest clue list but the lowest precision, and neither DE nor JP used the second collection iteration.Source: §3.3 · p. 5

3.4 Comparison with Existing Datasets

  • The dataset aligns with prior corpora in definition, positive-example proportion, and construction keywords, enabling standalone use or combination with them.It adds e-commerce reviews and German and Japanese coverage; professional linguists produced clue lists and supervised annotation.Source: §3.4 · p. 5

4 Evaluations

The evaluations test clue-phrase selection bias, cross-dataset compatibility, machine-translation transfer, and alternative sentence encoders and classifiers. Combined training performs best across measures, while translated test data underperforms in-language baselines.

4.1 Selection Bias due to Clue Phrases

  • For mBERT, masking clue phrases does not significantly change performance overall, indicating its sentence embeddings generalize beyond the phrases used to select annotation candidates.For English, the no-mask setting shows a slight drop across evaluation measures relative to masking, which the authors attribute to overfitting on clue phrases.Source: §4.1 · p. 6, Table 4 · p. 7

4.2 Cross-Dataset Adaptation

  • Training on Comb, which merges EN-ext with SemEval, yields the best performance across all measures.Models trained on EN and EN-ext perform poorly on SemEval, while SemEval-trained models perform relatively well on EN and EN-ext.Source: §4.2 · p. 7, Table 6 · p. 7

4.3 Cross-Lingual Transfer via MT

  • MCC for machine-translation-based models is significantly lower than for the corresponding in-language baselines on German and Japanese.The study concludes that translating test data is not an alternative to annotating counterfactual datasets for a new target language.Source: §4.3 · pp. 7–8, Table 7 · p. 8

4.4 Sentence Encoders and Classifiers

  • Transformer classifiers are least affected by clue masking, while traditional BoN models show the largest performance drops.BoN models’ drops are equal on clue-based EN and enriched EN-ext, suggesting reliance on clue words in both datasets.Source: §4.4 · p. 8
  • XLM-RoBERTa performs best among the sentence embedding methods, except on Japanese, where an SVM with BoN features performs best.The paper suggests Japanese-specific tokenization may help lexicalized BoN models relative to language-independent subtokenization.Source: Table 8 · p. 8, §4.4 · p. 9

5 Conclusion

The paper concludes that its Amazon-review counterfactual dataset covers English, German, and Japanese, and that English data is compatible with SemEval-2020. It also reports robustness to clue-phrase selection bias and poor cross-lingual classification from machine-translated examples.

5 Conclusion

  • The annotated Amazon-review dataset covers counterfactual statements in English, German, and Japanese.The conclusion highlights the multilingual scope of the dataset.Source: §5 · p. 9
  • The English dataset is compatible with SemEval-2020, and the datasets can be used together to train better CFD models.Source: §5 · p. 9
  • CFD models are relatively robust to selection bias from clue phrases, but machine-translated English examples perform poorly in cross-lingual classification.The authors conclude that language-specific CFD datasets are needed.Source: §5 · p. 9

6 Ethical Considerations

The ethical discussion describes the released review sentences and considers possible bias in both the data and pretrained encoders. It notes that subjective review bias is not filtered and that encoder biases could propagate into, or be amplified by, trained CFD models.

6 Ethical Considerations

  • The dataset uses sentences selected from a publicly available Amazon review collection, and the authors manually checked them for customer-sensitive information.Source: §6 · p. 9
  • Subjective opinions in product reviews can be socially biased, and the authors do not filter out such biases.Source: §6 · p. 9
  • Biases encoded in mBERT and XLM-RoBERTa could propagate into CFD models and may be amplified during training, particularly if training counterfactuals also contain such biases.The authors plan to evaluate social biases in CFD models in future work.Source: §6 · p. 9

Supplementary Materials

The supplement describes the fine-tuned multilingual BERT classifier and the Matthews Correlation Coefficient (MCC) used for evaluation. It outlines the model's input representations, classification setup, and configuration.

A Fine-tuned multilingual BERT for counterfactual classification

B Matthews Correlation Coefficient

C Extended version of Table 8

The extended table section identifies the metrics reported in Table 9.

C Extended version of Table 8

D Examples of Incorrect Predictions

The appendix gives examples of transformer misclassifications and documents model hardware and configuration details. It also describes the predefined transformer configurations and the use of manual development-data searches for hyperparameters.

D Examples of Incorrect Predictions

  • Table 10 presents examples of reviews misclassified by transformer models and identifies which remaining models also misclassified each review.B denotes mBERT, XR XLM-RoBERTa, and X XLM without embedding.Source: D Examples of Incorrect Predictions · p. 13

E Hardware Used

  • The transformer, RNN, and CNN models were trained on a GeForce NVIDIA GTX 1070 GPU with 8GB GDDR5 memory.Source: E Hardware Used · p. 13

F Model Configuration and Hyperparameter Settings

  • Predefined transformer configurations include hidden-layer group counts, feed-forward intermediate size, and maximum position-embedding length.For BERT-base, the sentence representation is the final hidden state of the first [CLS] token, passed through a fine-tuned linear layer and sigmoid classifier.Source: F Model Configuration and… · p. 13

F.1 Transformer Model Hyperparameters

Transformer pretraining hyperparameters were retained, while a manually selected subset of fine-tuning hyperparameters was tested. Fine-tuning was computationally bounded, with all language models taking less than two and a half hours.

F.1 Transformer Model Hyperparameters

  • The original pretraining hyperparameter settings were unchanged, with model configurations summarized in Table 11.Source: §F.1 · p. 13
  • For fine-tuning transformer models, researchers manually tested combinations of a subset of hyperparameters: learning rates {50−4, 10−5, 50−5}, batch sizes {16, 32, 128}, warmup proportions {0, 0.1}, and Adam’s ϵ.Source: §F.1 · p. 13
  • Fine-tuning all language models with a sentence classifier took less than two and a half hours; BERT averaged 184.13 seconds per epoch at batch size 16.The BERT estimate uses 2,682 training samples; the stated worst case of 50 epochs is 2 hours and 30 minutes.Source: §F.1 · p. 13

F.2 Baseline Hyperparameters

The appendix specifies SVM, decision-tree, and random-forest baseline settings. It identifies Table 9 as reporting F1, Matthew’s Correlation Coefficient, and accuracy for the CFD models.

F.2 Baseline Hyperparameters

  • The SVM used an RBF kernel, tested C values {0.01, 0.1, 1}, autotuned γ, inverse-frequency class weights, and Platt scaling for AUC probabilities.Platt scaling is attributed to Platt et al. (1999).Source: §F.2 · p. 13
  • The decision-tree and random-forest baselines used 20 trees without a depth restriction, a minimum split size of 2, and Gini importance.Source: §F.2 · p. 13
  • Table 9 reports F1, Matthew’s Correlation Coefficient, and accuracy for the different CFD models.Source: Table 9 · p. 14

G Further Details on the Datasets

The appendix details dataset category counts and clue-phrase statistics across English, German, and Japanese. The Japanese dataset spans 18 product categories, while the clue-word tables define class-specific counts and frequencies.

G Further Details on the Datasets

  • The category breakdown format reports total review sentences, counterfactual examples, and non-counterfactual examples; the PC row lists 1,297 total, 873 counterfactual, and 424 non-counterfactual examples.Source: G Further Details on the Datasets · p. 14
  • Dataset JP covers 18 product categories, with category-level totals and counterfactual and non-counterfactual example counts.The listed categories include Automotive, Camera, Electronics, PC, Toys, and Wireless.Source: G Further Details on the Datasets · p. 14
  • Clue-phrase tables cover English, German, and Japanese; their statistics include positive and negative example counts and class-specific percentages or fractions.The English and Japanese captions also define fdata as clue-word frequency in the dataset.Source: G Further Details on the Datasets · p. 14, Table 13 · p. 16, Table 14 · p. 16

Related papers

Loading 2104.06893v2…