PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification

Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, Kilian Weinberger

arXiv:1606.01614v5cs.CL

TL;DR

Low-resource languages often lack the labeled data required by supervised sentiment classifiers. The paper proposes ADAN, which transfers sentiment knowledge from English using shared features, a sentiment classifier, and an adversarial language discriminator. On Chinese and Arabic, ADAN outperforms the evaluated baselines and state-of-the-art cross-lingual systems.

  • Problem

    Most languages have limited or no labeled sentiment data, creating a need for sentiment classification methods that use resource-rich source-language data.

  • Method

    ADAN uses a shared feature extractor feeding a sentiment classifier and language discriminator, with bilingual word embeddings supporting cross-lingual transfer without a highly engineered machine translation system.

  • Results

    ADAN outperforms several baselines and state-of-the-art cross-lingual methods on Chinese and Arabic sentiment classification, including when randomly initialized embeddings are used.

  • Takeaways & Limitations

    ADAN can transfer sentiment classification from English to Chinese and Arabic with little or no annotated TARGET data and without relying on a full machine translation system.

  • Takeaways & Limitations

    ADAN is implicitly dependent on bilingual word embeddings, which are often trained using a parallel corpus.

Abstract

from arXiv · show

In recent years great success has been achieved in sentiment classification for English, thanks in part to the availability of copious annotated resources. Unfortunately, most languages do not enjoy such an abundance of labeled data. To tackle the sentiment classification problem in low-resource languages without adequate annotated data, we propose an Adversarial Deep Averaging Network (ADAN) to transfer the knowledge learned from labeled data on a resource-rich source language to low-resource languages where only unlabeled data exists. ADAN has two discriminative branches: a sentiment classifier and an adversarial language discriminator. Both branches take input from a shared feature extractor to learn hidden representations that are simultaneously indicative for the classification task and invariant across languages. Experiments on Chinese and Arabic sentiment classification demonstrate that ADAN significantly outperforms state-of-the-art systems.

1 Introduction

Cross-lingual sentiment classification addresses the lack of labeled data in most languages by leveraging resources from a source language. ADAN uses adversarial training and bilingual word embeddings to avoid highly engineered machine translation systems, and it outperforms the evaluated alternatives.

  • Most languages have limited or no labeled sentiment data, unlike resource-rich languages such as English.
  • Cross-lingual sentiment classification transfers knowledge from a resource-rich SOURCE language to a TARGET language with little or no task-specific annotation.Previous methods commonly use bilingual lexica, parallel corpora, or machine translation resources.
  • ADAN uses language-adversarial training with bilingual word embeddings instead of requiring a highly engineered machine translation system.It also requires one to two orders of magnitude less parallel data than machine translation systems.
  • ADAN achieves statistically significantly better results than the compared baseline and state-of-the-art cross-lingual systems.The comparisons include source-only, domain-adaptation, machine-translation, and distillation approaches.
  • In the semi-supervised setting, ADAN continues to outperform alternatives given the same amount of TARGET supervision.Its performance is also robust to the choice of bilingual word embeddings.
  • Wasserstein-distance minimization yields better performance and smoother training than standard gradient-reversal-layer training.

2 The ADAN Model

ADAN learns shared representations that support source-language sentiment prediction while suppressing language information, using labeled SOURCE and unlabeled TARGET text. Its Wasserstein-based adversarial training is designed to improve stability and cross-lingual generalization.

  • Architecture: F is trained to produce features useful for sentiment classification but uninformative for identifying the input language.Q attempts to distinguish SOURCE from TARGET, while F learns language-invariant features.
  • Training setup: ADAN trains with labeled SOURCE data and unlabeled TARGET data, sending both languages through Q but only labeled SOURCE examples through P.The trained feature extractor and sentiment classifier are then applied to TARGET texts at test time.
  • Architecture: ADAN uses a shared feature extractor F with sentiment classifier P and language discriminator Q as two network branches.F maps input sequences into a shared feature space; P predicts sentiment, while Q scores whether text comes from SOURCE or TARGET.
  • Input representations: Pre-trained bilingual word embeddings place similar words from different languages closer together, but high-quality embeddings often require a parallel corpus.The paper notes that this dependency is lighter than the parallel-data requirements of MT systems, and randomly initialized embeddings can still outperform some BWE-based baselines.
  • Adversarial training: ADAN replaces gradient-reversal training with Wasserstein-distance minimization between SOURCE and TARGET feature distributions.Q approximates the Wasserstein objective, assigning higher scores to SOURCE instances and lower scores to TARGET instances.
  • Adversarial training: Compared with ADAN-GRL, Wasserstein training is reported to be more stable for hyperparameter selection and avoids delicate tuning of the alternating-update parameter k.ADAN-GRL can degrade when k is improperly set, whereas the proposed training removes the need for delicate k tuning.

3 Experiments and Discussions

The experiments evaluate ADAN for cross-lingual sentiment classification from English into Chinese and Arabic, with Stanford CoreNLP used for tokenization.

  • Experimental setting: ADAN is evaluated with English as SOURCE and Chinese and Arabic as TARGET languages.The experiments address sentiment classification in both target languages.
  • Experimental setting: Tokenization for all experimental data is performed using Stanford CoreNLP.

3.1 Data

The data setup combines a large labeled English sentiment corpus with Chinese and Arabic resources, while labeled TARGET data are used for validation or evaluation rather than ADAN training.

  • English data: The English source dataset contains 700k balanced Yelp reviews with ratings from 1 to 5, split into 650k training and 50k validation reviews.
  • Chinese data: The Chinese resources include 10k balanced hotel reviews used as a validation set because ADAN does not require labeled Chinese training data.
  • Arabic data: The Arabic sentiment dataset contains 1200 social-media sentences, split into 600 validation and 600 test examples with three sentiment labels.

3.2 Cross-Lingual Sentiment Classification

ADAN is evaluated against source-only, domain-adaptation, machine-translation, and cross-lingual distillation baselines for cross-lingual sentiment classification. It substantially outperforms the MT baseline on Chinese and Arabic and achieves new state-of-the-art performance over both CLD-KCNN variants.

  • Bilingual resources: All systems use bilingual word embeddings to map source and target words into a shared feature space, except CLD-based systems, which use a parallel corpus.Traditional bag-of-words features cannot directly represent the different source and target vocabularies.
  • Train-on-SOURCE-only baselines: The train-on-source-only baselines comprise logistic regression and the nonadversarial DAN model, both trained on labeled English data and transferred using bilingual word embeddings.The DAN baseline is the nonadversarial portion of ADAN.
  • Machine Translation baselines: ADAN substantially outperforms the machine-translation baseline on both Chinese and Arabic without annotated target-language data.The MT pipeline translates target text into English and applies the better source-only classifier.
  • Cross-lingual Text Classification baselines: ADAN significantly outperforms both CLD-KCNN variants and achieves new state-of-the-art performance.The comparison is against cross-lingual distillation methods that use soft source predictions on parallel data to train a target model.
  • Cross-lingual Text Classification baselines: ADAN’s direct adversarial adaptation can be more effective than chaining three adaptation steps in CLDFA-KCNN, despite not explicitly separating language and domain variation.The source and target monolingual data come from different domains, suggesting ADAN may bridge both variation sources in one step.
  • Supervised SOURCE accuracy: ADAN achieves 58.7% accuracy on English in the 5-class English-Chinese setting and 75.6% in the 3-class English-Arabic setting.These supervised source accuracies provide a comparison for ADAN’s transferred target performance.

3.3 Analysis and Discussion

The analysis examines ADAN’s semi-supervised behavior, representations, bilingual embeddings, feature extractors, and training stability. Across these studies, ADAN benefits from added target supervision, narrows cross-lingual representation gaps, remains robust across embeddings and architectures, and outperforms ADAN-GRL.

  • 3.3.1 Semi-supervised Learning: ADAN better utilizes added Chinese supervision than DAN baselines, remaining superior even with 12,800 labeled Chinese reviews.The performance margin decreases as more target supervision is added.
  • 3.3.2 Qualitative Analysis and Visualizations: BWEs alone leave English and Chinese representations separated, while ADAN’s joint feature extractor dramatically reduces their Averaged Hausdorff Distance.The separation persists after removing stop words, suggesting language divergence beyond content differences.
  • 3.3.2 Qualitative Analysis and Visualizations: ADAN aligns Chinese samples more closely with confident English sentiment clusters, including a highlighted Chinese 5-star review near the positive English cluster.The baseline leaves many Chinese samples away from dense English clusters.
  • 3.3.3 Impact of Bilingual Word Embeddings: ADAN with random word embeddings outperforms DAN and mSDA baselines using BWEs and matches the LR+MT baseline, while BWEs further improve ADAN.With alternative BilBOWA embeddings, all systems perform slightly lower, but ADAN still outperforms other baselines.
  • 3.3.4 Feature Extractor Architectures: More sophisticated feature extractors improve ADAN’s performance at the cost of slower running time, confirming compatibility beyond DAN.The framework is reported to work with other feature-extractor choices.
  • 3.3.5 Training Stability: Across a grid search over k and λ, ADAN shows higher accuracy and greater stability than ADAN-GRL on the Chinese development set.The result suggests reduced sensitivity to adversarial-training hyperparameter tuning.

3.4 Implementation Details

The implementation uses a three-layer fully connected feature extractor and two-layer classifier and discriminator, with ReLU activations, batch normalization in P and Q, and Adam optimization.

  • 3.4 Implementation Details: ADAN uses three fully connected ReLU layers in F and two layers in both P and Q, with 900 hidden units per layer.Batch normalization is applied in P and Q but not in F.
  • 3.4 Implementation Details: F and P are jointly optimized with Adam at learning rate 0.0005, while Q uses a separate Adam optimizer with the same rate.

4 Related Work

Related work frames cross-lingual sentiment classification as a response to scarce non-English labeled data and situates ADAN among adversarial and cross-lingual NLP methods.

  • 4 Related Work: Cross-lingual sentiment classification addresses the lack of high-quality labeled data in many non-English languages, including Chinese and Arabic.
  • 4 Related Work: Prior work includes adversarial training for generative models and cross-lingual NLP, while ADAN is presented as the first adversarial neural network for cross-lingual NLP.The cited cross-lingual applications include text classification and cross-lingual word embeddings.

5 Conclusion and Future Work

The paper concludes that ADAN transfers English sentiment knowledge to Chinese and Arabic with little or no target annotation, outperforming several competing systems. Future work extends the framework to other adaptation tasks and beyond a single source-target pair.

  • 5 Conclusion and Future Work: ADAN outperforms domain adaptation models, a competitive machine-translation baseline, and state-of-the-art cross-lingual text-classification methods on Chinese and Arabic.
  • 5 Conclusion and Future Work: Randomly initialized embeddings still yield encouraging performance, and labeled target data can be incorporated for more competitive results.
  • 5 Conclusion and Future Work: Future work will apply language-adversarial training to other NLP adaptation and phrase-level tagging tasks, and consider settings beyond a single source and target.
Loading 1606.01614v5…