PaperlayerView exact revision on arXiv

Source-linked AI summary

TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter

Xinyang Zhang, Yury Malkov, Omar Florez, Serim Park, Brian McWilliams, Jiawei Han, Ahmed El-Kishky

arXiv:2209.07562v3cs.CL

TL;DR

General-domain PLMs are not tailored to noisy social-media text, and their pre-training typically omits social engagement information. TwHIN-BERT combines text-based and social objectives in a multilingual Tweet model. It outperforms previously released language models on semantic and social-engagement prediction tasks.

  • Problem

    General-domain PLMs may not capture noisy Tweet language well, and their pre-training does not use social engagement logs.

    Source: §1 · p. 1

  • Method

    TwHIN-BERT is a multilingual Tweet language model trained with masked language modeling and a contrastive social objective using socially similar Tweets.

    Source: §1 · p. 2, §2.1.3 · p. 3

  • Results

    TwHIN-BERT outperforms previously released language models on semantic and social-engagement prediction tasks.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The authors release TwHIN-BERT and curated hashtag-prediction and social-engagement benchmark datasets for further social-media NLP research.

    Source: §1 · p. 2, §5 · p. 7

    Compared models were trained on different amounts of data, from different time periods and linguistic distributions.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Pre-trained language models (PLMs) are fundamental for natural language processing applications. Most existing PLMs are not tailored to the noisy user-generated text on social media, and the pre-training does not factor in the valuable social engagement logs available in a social network. We present TwHIN-BERT, a multilingual language model productionized at Twitter, trained on in-domain data from the popular social network. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision, but also with a social objective based on the rich social engagements within a Twitter heterogeneous information network (TwHIN). Our model is trained on 7 billion tweets covering over 100 distinct languages, providing a valuable representation to model short, noisy, user-generated text. We evaluate our model on various multilingual social recommendation and semantic understanding tasks and demonstrate significant metric improvement over established pre-trained language models. We open-source TwHIN-BERT and our curated hashtag prediction and social engagement benchmark datasets to the research community.

1 INTRODUCTION

General-domain language models may miss the informal diction, abbreviations, emojis, and topical tokens of Tweets, while existing Twitter pre-training largely repeats general-domain methods. TwHIN-BERT combines text and social-engagement signals, and is evaluated on multilingual social and semantic tasks.

1 INTRODUCTION

  • Tweets’ informal diction, abbreviations, emojis, and hashtags can challenge general-domain PLMs, while prior Twitter models replicate general-domain pre-training.Tweets are short user-generated messages, and their topical tokens include hashtags.Source: §1 · p. 1
  • Tweet engagements provide social context and co-engaged Tweets can connect content that a text-only objective may struggle to relate.The introduction also identifies social recommendation and information-diffusion prediction as relevant applications.Source: §1 · p. 1
  • TwHIN-BERT combines masked language modeling with a contrastive social objective, mining socially similar Tweet pairs from a heterogeneous network of engagement logs.The model is trained on 7 billion Tweets in over 100 languages, including 1 billion with social engagement logs.Source: §1 · p. 2
  • TwHIN-BERT achieves state-of-the-art performance in the reported evaluations, with a major advantage on social tasks.The evaluation covers social recommendation and semantic understanding tasks.Source: §1 · p. 2
  • The authors release TwHIN-BERT and two Tweet benchmarks for hashtag prediction and social engagement prediction.The two datasets focus on language and social aspects, respectively.Source: §1 · p. 2

2 TWHIN-BERT

TwHIN-BERT mines socially similar Tweets from engagement-network embeddings, then trains a Tweet language model with social and masked-language objectives before downstream fine-tuning.

2 TWHIN-BERT

  • TwHIN-BERT’s pipeline mines socially similar Tweet pairs, trains with joint social and masked-language objectives, and then fine-tunes for downstream tasks.The mined pairs provide social self-supervision for pre-training.Source: §2 · p. 2, Figure 2 · p. 3

2.1 Mining Socially Similar Tweets

  • Social similarity is defined by Tweets co-engaged by similar users, and TwHIN embeddings make that similarity searchable through approximate nearest neighbors.TwHIN fuses heterogeneous engagement types, including Favorites, Replies, and Retweets, into a shared vector space.Source: §2.1 · p. 2

2.1.2 Embedding TwHIN Nodes.

  • TwHIN represents users and Tweets as a directed bipartite engagement graph, then scores edges with relation-specific user translations and Tweet-vector dot products.The graph contains approximately 200 million users, 1 billion Tweets, and over 100 billion edges.Source: §2.1 · p. 2, §2.1.2 · p. 2

2.2 Pre-training Objectives

  • Tweet representations are produced by Transformer encoding and CLS pooling; ANN search over engagement embeddings retrieves socially similar Tweets for training.For billion-scale search, the authors use a FAISS index with product quantization to fit embeddings on a 16-A100-GPU node.Source: §2.2 · p. 3, §2.1.3 · p. 3

2.1.3 Mining Similar Tweet Pairs.

  • The social objective applies NT-Xent to socially similar Tweet pairs with in-batch negatives, and the overall pre-training loss combines it with masked language modeling.The hyperparameter λ balances the social and language losses.Source: §2.1.3 · p. 3

2.3 Pre-training Setup

  • Pre-training uses 7 billion Tweets across 100 languages and 100 billion engagement records, with training staged from text-only initialization to joint social and MLM training.Each stage runs for 500K steps; the text-only stage uses 6 billion Tweets and the joint stage uses 1 billion Tweets.Source: §2.3 · p. 4

3 EXPERIMENTS

Experiments compare TwHIN-BERT with established language models and an MLM-only ablation across social-engagement prediction and Tweet classification.

3.1 Evaluated Methods

  • Baselines span general-domain multilingual and Twitter PLMs plus an MLM-only ablation; published baselines differ in data quantity, timing, and language.TwHIN-BERT base and large models use the same corpus.Source: §3.1 · p. 4

3.2 Social Engagement Prediction

  • TwHIN-BERT significantly improves social-engagement prediction over baselines, and contrastive social pre-training lifts results over the MLM-only ablation.Table 1 reports HITS@10 across high-, mid-, and low-resource languages and their average.Source: §3.2 · p. 5, Table 1 · p. 5

3.3 Tweet Classification

  • TwHIN-BERT significantly beats same-size baselines on multilingual hashtag prediction; its large model is slightly better than or on par with base overall.On the English dataset, it also outperforms BERTweet, while the MLM-only ablation performs similarly but slightly worse.Source: §3.3 · p. 6, Table 3 · p. 6
  • On external classification benchmarks, TwHIN-BERT matches or beats multilingual baselines, and the large model beats all baselines.BERTweet fares better than the base model; TwHIN-BERT is on par with or slightly better than the MLM-only ablation.Source: §3.3 · p. 6, Table 4 · p. 6

3.4 Varying Downstream Supervision

  • TwHIN-BERT gains persist across downstream-supervision levels, with an even larger relative advantage at two labeled examples per class.Figure 4 compares Macro-F1 on English and Japanese hashtag-prediction datasets by labeled examples per class.Source: §3.4 · pp. 6–7, Figure 4 · p. 7

3.5 Feature-based Classification

  • In feature-based classification, TwHIN-BERT beats baselines by a wide margin on all evaluated languages.Table 5 reports Macro-F1 on hashtag-prediction datasets.Source: §3.5 · p. 7, Table 5 · p. 7

4 RELATED WORKS

Prior language-model work largely used text-only objectives on general-domain corpora, while Twitter-specific models mostly adapted those methods to Tweets. This work instead combines Twitter in-domain data with social-based objectives and draws on scalable heterogeneous network embeddings.

4 RELATED WORKS

  • PLMs mostly use text-only objectives on general-domain corpora, whereas this work trains on Twitter data with both text-based and social-based objectives.The related-work discussion contrasts this approach with prior work focused on general-domain corpora.Source: §4 · p. 7
  • Twitter language models such as BERTweet, TimeLMs, and XLM-T mostly adapt general-domain methods by substituting Tweets as training data.TwHIN-BERT additionally uses social engagement signals to enhance Tweet representations.Source: §4 · p. 7
  • Other approaches add knowledge-graph entities and relations, document metadata, or hyperlinks, while this work uses social engagements for text relevance.Source: §4 · p. 7
  • For multi-typed, multi-relational networks, this work uses knowledge graph embeddings, described as scalable and flexible across node and edge types.The passage contrasts these with network embedding techniques that are difficult to scale to very large networks.Source: §4 · p. 7

5 CONCLUSIONS

TwHIN-BERT combines masked language modeling with a social objective and outperforms previously released language models on semantic and social engagement prediction tasks. The paper also reports full engagement and hashtag prediction results and releases the model for academic research.

5 CONCLUSIONS

  • TwHIN-BERT outperforms previously released language models on semantic and social engagement prediction tasks.The full social engagement results are reported in Table 6, and full hashtag prediction results in Table 7.Source: §5 · p. 7, Table 6 · p. 8, Table 7 · p. 8
  • The multilingual Tweet model uses a standard masked language modeling objective and a contrasting social objective.Source: §5 · p. 7
  • The authors release TwHIN-BERT to the academic community for further research in social media NLP.Source: §5 · p. 7

A DISTRIBUTION OF LANGUAGES IN TRAINING DATASET

The appendix describes language identification and resource-level definitions for the pre-training dataset. It also notes that language variants may share an ISO language code.

A DISTRIBUTION OF LANGUAGES IN TRAINING DATASET

B HYPERPARAMETERS FOR PRE-TRAINING AND FINE-TUNING

The appendix specifies that unlisted pre-training architecture and hyperparameters follow RoBERTa [27]. Fine-tuning settings were selected on development data, with distinct handling for base and large models.

B HYPERPARAMETERS FOR PRE-TRAINING AND FINE-TUNING

C EVALUATION METRICS FOR EXTERNAL CLASSIFICATION BENCHMARKS

External classification benchmarks use different recommended metrics depending on the dataset.

D ENGAGEMENT PREDICTION RESULTS ON ADDITIONAL LANGUAGES

Engagement prediction results cover all available evaluation languages, with example counts varying by language availability.

E HASHTAG PREDICTION RESULTS ON ADDITIONAL LANGUAGES

Hashtag prediction results cover available evaluation languages, but data availability varies and Russian is excluded because the XLM-T baseline fails on some dataset characters.

Loading 2209.07562v3…