PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantics-aware BERT for Language Understanding

Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, Xiang Zhou

arXiv:1909.02209v3cs.CL

TL;DR

Existing language models mainly use plain contextual features and rarely incorporate explicit contextual semantic clues for richer language representations. SemBERT integrates semantic role labels with BERT text representations through parallel encoding and semantic integration for downstream NLU tasks. SemBERT is evaluated on 11 benchmark datasets, achieves new state-of-the-art performance on SNLI, and obtains significant gains on the GLUE benchmark and SQuAD 2.0.

  • Problem

    Existing language models mainly use plain contextual features and rarely incorporate explicit contextual semantic clues for richer language representations.

  • Method

    SemBERT integrates semantic role labels with BERT text representations through parallel encoding and semantic integration for downstream NLU tasks.

  • Results

    SemBERT is evaluated on 11 benchmark datasets, achieves new state-of-the-art performance on SNLI, and obtains significant gains on the GLUE benchmark and SQuAD 2.0.

  • Takeaways & Limitations

    Explicit contextual semantics can be effectively integrated with pre-trained language representations in a lightweight, unified model for improved NLU performance.

  • Takeaways & Limitations

    The approach’s language coverage depends on available semantic role labeling treebanks or effective unsupervised SRL methods.

Abstract

from arXiv · show

The latest work on language representations carefully integrates contextualized features into language model training, which enables a series of success especially in various machine reading comprehension and natural language inference tasks. However, the existing language representation models including ELMo, GPT and BERT only exploit plain context-sensitive features such as character or word embeddings. They rarely consider incorporating structured semantic information which can provide rich semantics for language representation. To promote natural language understanding, we propose to incorporate explicit contextual semantics from pre-trained semantic role labeling, and introduce an improved language representation model, Semantics-aware BERT (SemBERT), which is capable of explicitly absorbing contextual semantics over a BERT backbone. SemBERT keeps the convenient usability of its BERT precursor in a light fine-tuning way without substantial task-specific modifications. Compared with BERT, semantics-aware BERT is as simple in concept but more powerful. It obtains new state-of-the-art or substantially improves results on ten reading comprehension and language inference tasks.

1 Introduction

SemBERT addresses the limitation of language models that primarily use plain contextual features by integrating explicit semantic-role information with BERT representations. It is evaluated across 11 NLU benchmarks, achieving state-of-the-art on SNLI and gains on GLUE and SQuAD 2.0.

  • 1 Introduction: Existing pre-trained language models achieve strong NLU results but mainly use plain contextual features and rarely incorporate explicit contextual semantic clues.The paper argues that relying only on language modeling restricts the power of pre-trained representations, despite their ability to implicitly encode some semantics.
  • 1 Introduction: SemBERT enriches BERT’s plain contextual representations with explicit contextual semantics from multiple predicate-specific argument sequences.Semantic role labeling captures who did what to whom, when, and why, matching the contextual semantic analysis shared by NLU tasks.
  • 1 Introduction: The model combines an off-the-shelf semantic role labeler, parallel text and semantic-role encoders, and a semantic integration component for downstream representations.Semantic-role labels are mapped to embeddings and integrated with language-model text representations to form a joint representation.
  • 1 Introduction: SemBERT obtains new state-of-the-art on SNLI and significant gains on GLUE and SQuAD 2.0 across 11 benchmark datasets.The evaluation covers natural language inference, question answering, semantic similarity, and text classification; ablations identify explicit semantics as essential for improvement.

2 Background and Related Work

Contextual language models address the limitations of context-independent word embeddings and improve performance across major NLU benchmarks, but their semantics remain largely implicit. This motivates incorporating explicit predicate–argument semantics extracted through semantic role labeling.

  • Background: Contextual models such as ELMo, GPT, BERT, and XLNet extend word representations by encoding sentence-level context rather than relying on a single context-independent vector per word.ELMo improves question answering, sentiment analysis, and named entity recognition, while BERT is effective on GLUE, MultiNLI, and SQuAD (Peters et al. 2018; Devlin et al. 2018).
  • Motivation: Despite strengthening language representations, implicit semantics may be insufficient for natural language understanding, as illustrated by semantically incomplete answer spans generated by BERT on SQuAD.This observation motivates directly introducing explicit semantics into contextual representations.
  • Semantic Representations: Formal semantic resources such as FrameNet and PropBank represent meaning through predicate–argument structures, with PropBank more widely implemented in computational linguistics.For example, semantic roles identify a predicate and its arguments, including agent, theme, recipient, and temporal adjuncts.
  • Semantic Role Labeling: Semantic role labeling parses predicate–argument structures, and recent end-to-end neural systems jointly perform argument identification and classification.He et al. (2017) introduced a deep highway BiLSTM with constrained decoding that was selected as the basic semantic role labeling system.

3 Semantics-aware BERT

SemBERT augments BERT with explicit contextual semantics from pre-trained semantic role labeling while retaining BERT’s subword-based contextual representations. It aligns word-level semantic embeddings with word-level representations derived from BERT subwords, then concatenates them for downstream tasks.

  • 3 Semantics-aware BERT: SemBERT concatenates the aligned contextual word representations and distilled semantic embeddings into a joint representation for downstream tasks.The framework is designed to handle multiple sequence inputs while preserving a light integration with the BERT backbone.
  • 3 Semantics-aware BERT: SemBERT uses a pre-trained semantic role labeler to annotate each sentence with multiple PropBank-style predicate-argument structures [Palmer, Gildea, and Kingsbury 2005].The labeler is not fine-tuned within the framework.
  • 3 Semantics-aware BERT: BERT’s transformer encodes tokenized word pieces into contextual embeddings, while semantic label sequences are mapped to vectors and encoded with a BiGRU for each predicate.The semantic labels remain word-level and therefore correspond to the original sentence length.
  • 3 Semantics-aware BERT: A fully connected layer aggregates the representations from multiple predicate-related label sequences into a refined semantic embedding.The resulting semantic embedding integrates multidimensional semantic signals from the predicate-specific structures.
  • 3 Semantics-aware BERT: Because BERT uses subwords while semantic labels use words, SemBERT groups each word’s subwords and applies a CNN with ReLU and max pooling to recover word-level representations.The paper selects CNN for speed and reports that preliminary experiments found it performed better.

4 Model Implementation

SemBERT uses a pre-trained SRL module to provide explicit semantic labels and combines them with BERT representations through a lightweight implementation. Its task-specific prediction layers remain essentially unchanged from BERT, while span-based reading comprehension uses start/end logits and null-answer thresholding.

  • Representation Fusion: Concatenation is the best-performing mechanism among the tested methods for combining contextual semantic and language-model representations.Summation, multiplication, and attention mechanisms were also evaluated.
  • Semantic Role Labeling: SemBERT uses a pre-trained SRL module with Viterbi decoding and BIO constraints to predict semantic roles, producing 104 labels including O for non-argument words.The SRL implementation achieves 84.6% F1 on English OntoNotes v5.0 for the CoNLL-2012 shared task.
  • Task Adaptation: The fused representation h is transformed to a lower dimension and passed to essentially unmodified BERT-style prediction layers for classification, regression, and span-based MRC.Classification uses CrossEntropy loss, whereas regression uses Mean Square Error loss.
  • Span-Based Reading Comprehension: For span-based reading comprehension, start and end logits define each candidate span score as s_i + e_j, and the highest-scoring span with j ≥ i is selected.The model compares the best non-null span against the pooled first-token null span s_null = s_0 + e_0.
  • Span-Based Reading Comprehension: A non-null answer is predicted only when the best span score exceeds s_null + τ, with τ selected on the development set to maximize F1.Candidate span scores are normalized by softmax.

5 Experiments

SemBERT is evaluated on ten NLU benchmarks spanning inference, reading comprehension, semantic similarity, and classification. It substantially improves over BERT and prior state-of-the-art systems while retaining BERT’s lightweight fine-tuning procedure.

  • Reading Comprehension: SemBERT boosts BERT on both Exact Match and F1 on the SQuAD 2.0 test set, surpassing published systems and approaching a few unpublished leaderboard models.The comparison uses single models; SemBERT∗ LARGE additionally uses synthetic self-training for data augmentation.
  • GLUE benchmark: SemBERT substantially improves over BERT and outperforms previous state-of-the-art models across the GLUE benchmark datasets.Because SemBERT uses BERT as its backbone with the same evaluation procedure, the reported gains are attributed to explicit contextual semantics; it remains a lightweight single model.
  • Analysis: Explicit semantics particularly improves performance on small datasets such as RTE, MRPC, and CoLA, where large annotated training sets are unavailable.The evaluation covers ten datasets across natural language inference, machine reading comprehension, semantic similarity, and text classification.
  • Natural Language Inference: SemBERT achieves a new state of the art on SNLI and outperforms all ensemble models by a large margin.Both SemBERT and BERT are single models fine-tuned from pre-trained models.

6 Analysis

The analysis shows that explicit semantic information improves answer quality, while span segmentation contributes marginally and performance remains robust to labeler errors. SemBERT also appears adaptable across languages and diverse domains.

  • Ablation study: The ablation study evaluates key method factors on SNLI and SQuAD 2.0 development sets, including direct SRL–BERT embedding concatenation against SemBERT’s deep semantic processing.Table 6 reports the corresponding analysis, but the supplied passage does not provide its cell values.
  • Ablation study: A modest maximum of predicate-argument structures m performs better than larger settings across the tested range of 1 to 5.This finding is reported in the influence study summarized by Table 7.
  • Prediction analysis: SemBERT extracts more semantically accurate and complete answers than baseline BERT, producing more exact-match answers on SQuAD 2.0 examples.The comparison suggests explicit semantics guide the model toward meaningful predictions.
  • Prediction analysis: 83.69(EM)/87.02(F1) from SRL-derived segmentation tags shows span segmentation works but is only marginally beneficial versus the complete architecture.The analysis motivates SRL signals as cues for span relationships and semantic-role segmentation.
  • Robustness and adaptation: 87.93, 87.31, and 87.24 SQuAD F1 scores result when 0%, 20%, and 40% of SRL labels are randomly corrupted, respectively.The authors attribute this robustness to concatenating BERT hidden states with SRL representations.
  • Robustness and adaptation: SemBERT can be adapted to other languages because SRL is fundamental, with seven CoNLL 2009 SRL treebanks and unsupervised SRL available where treebanks are absent.Experiments on diverse GLUE and SQuAD domains indicate the method continues to work despite out-of-domain concerns.

7 Conclusion

The paper proposes Semantics-aware BERT, a novel architecture for fine-grained language representation. Across diverse NLU tasks, it outperforms the strong BERT baseline and surpasses all published works considered.

  • 7 Conclusion: Semantics-aware BERT surpassed all published works across the concerned NLU tasks.The evaluated tasks span natural language inference, question answering, machine reading comprehension, semantic similarity, and text classification.
  • 7 Conclusion: Experiments showed Semantics-aware BERT’s superiority over the strong baseline BERT across a wide range of NLU tasks.The evaluation covered natural language inference, question answering, machine reading comprehension, semantic similarity, and text classification.
  • 7 Conclusion: The paper proposes a novel semantics-aware BERT network architecture for fine-grained language representation.
Loading 1909.02209v3…