Automated Concatenation of Embeddings for Structured Prediction

Xinyu Wang; Yong Jiang; Nguyen Bach; Tao Wang; Zhongqiang Huang; Fei Huang; Kewei Tu

Source-linked AI summary

Automated Concatenation of Embeddings for Structured Prediction

Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, Kewei Tu

arXiv:2010.05006v4Oct 10, 2020cs.CLcs.AIcs.LG

TL;DR

Selecting useful embedding combinations for structured prediction becomes difficult as candidate embedding types proliferate and the best selection varies by task. ACE automates this search with a controller optimized by reinforcement learning, and it outperforms strong baselines, achieving state-of-the-art performance across 6 tasks and 21 datasets with fine-tuned embeddings.

Problem
The best embedding concatenation varies by task and candidate collection, while the growing number of embedding types makes manual selection increasingly difficult.
Method
ACE uses an NAS-inspired controller that samples embedding concatenations and updates candidate beliefs with task-model accuracy rewards using reinforcement learning.
Results
ACE outperforms strong baselines and achieves state-of-the-art performance with fine-tuned embeddings across 6 structured prediction tasks and 21 datasets.
Takeaways & Limitations
ACE provides an efficient and practical way to search for strong task-specific embedding representations, using a single GPU for only a few GPU-hours.
Takeaways & Limitations
In some Russian and Turkish datasets, unavailable language-specific embeddings are replaced with corresponding English embeddings, making those search spaces only approximately comparable.

Abstract

from arXiv · show

Pretrained contextualized embeddings are powerful word representations for structured prediction tasks. Recent work found that better word representations can be obtained by concatenating different types of embeddings. However, the selection of embeddings to form the best concatenated representation usually varies depending on the task and the collection of candidate embeddings, and the ever-increasing number of embedding types makes it a more difficult problem. In this paper, we propose Automated Concatenation of Embeddings (ACE) to automate the process of finding better concatenations of embeddings for structured prediction tasks, based on a formulation inspired by recent progress on neural architecture search. Specifically, a controller alternately samples a concatenation of embeddings, according to its current belief of the effectiveness of individual embedding types in consideration for a task, and updates the belief based on a reward. We follow strategies in reinforcement learning to optimize the parameters of the controller and compute the reward based on the accuracy of a task model, which is fed with the sampled concatenation as input and trained on a task dataset. Empirical results on 6 tasks and 21 datasets show that our approach outperforms strong baselines and achieves state-of-the-art performance with fine-tuned embeddings in all the evaluations.

1 Introduction

ACE automates the search for task-specific concatenations of embedding types through an NAS-inspired controller and reinforcement learning. It is designed to provide accurate representations efficiently and achieves strong results across structured prediction tasks.

Method: A controller samples embedding concatenations according to estimated candidate effectiveness, then uses task-model accuracy as a reward to update its belief.
Motivation and contribution: ACE searches for better word representations rather than better model architectures.
Method: ACE uses a novel search space and reward function to guide reinforcement-learning-based search for embedding concatenations.
Efficiency and practicality: ACE achieves high accuracy without retraining the task model, unlike approaches that typically require retraining during neural architecture search.
Efficiency and practicality: ACE finds strong word representations on a single GPU with only a few GPU-hours, while many NAS approaches require dozens or thousands of GPU-hours.
Results: ACE outperforms strong baselines and achieves state-of-the-art accuracy with fine-tuned embeddings across 6 structured prediction tasks and 21 datasets.

2 Related Work

Prior work developed diverse non-contextualized and contextualized embeddings and applied neural architecture search to automate model design. These lines of work motivate searching embedding representations while highlighting the substantial cost and flexibility of broader architecture searches.

Embeddings: Non-contextualized word and character embeddings, along with contextualized embeddings such as ELMo, Flair, and BERT, have been widely used in NLP tasks.
Neural architecture search: Recent neural architecture search methods automate neural architecture design because manually designing strong task-specific architectures requires substantial effort and expertise.
Neural architecture search: NAS search spaces define which architectures can be discovered, and global spaces can provide considerable freedom.
Neural architecture search: Reinforcement learning and evolutionary algorithms are common NAS strategies, with prior reinforcement-learning methods using RNNs or Markov Decision Processes to generate architectures.

3 Automated Concatenation of Embeddings

ACE searches over embedding concatenations by having a controller sample candidate combinations and using task-model accuracy as reinforcement-learning feedback. Its reduced search space, shared task-model parameters, and candidate-sensitive reward function support efficient iterative optimization.

3.3 Searching in the Space: ACE repeatedly has a controller sample an embedding concatenation and a task model train on it, returning development accuracy as the controller’s reward.The controller updates its parameters from the reward and samples a new concatenation.
3.1 Task Model: The task model uses concatenated word representations for sequence-structured and graph-structured prediction, implemented with BiLSTM-CRF and BiLSTM-Biaffine models.The word-representation matrix contains one concatenated vector for each input word.
3.2 Search Space Design: Each embedding candidate is represented as an independent node with a fixed operation, yielding 2^L−1 possible nonempty combinations.The absence of connections between candidate nodes substantially reduces the search space.
3.2 Search Space Design: ACE shares task-model parameters across searched concatenations by masking embedding inputs, while pretrained embedding weights remain fixed except for character embeddings.After selecting the best concatenation, unused candidates and corresponding weights can be removed for a lighter task model.
3.3 Searching in the Space: The controller independently samples each embedding-selection variable as a Bernoulli decision and optimizes expected reward with policy gradients using one sampled selection per step.Sampling one selection makes gradient estimation practical when the exact expectation is intractable.
3.3 Searching in the Space: ACE’s reward function estimates each candidate’s contribution from accuracy differences across previously searched concatenations and discounts comparisons with larger Hamming distance.This is intended to reduce misleading attribution when many candidates change simultaneously.
3.3 Searching in the Space: Search begins with all candidates concatenated, then iteratively samples a combination, trains and evaluates the task model, updates the controller, and stores the result.Repeated combinations retain their higher observed validation score, while the previous combination and all-zero selection are excluded.

4 Experiments

Experiments evaluate ACE across six structured prediction tasks using varied datasets, embedding candidates, baselines, and fine-tuning settings. ACE is compared with all-embedding concatenation, random search, and strong fine-tuned models.

Tasks and datasets: ACE is evaluated on six tasks spanning named entity recognition, POS tagging, chunking, aspect extraction, syntactic dependency parsing, and semantic dependency parsing.The experiments include both sequence-structured and graph-structured outputs, as well as syntactic and semantic tasks.
Embedding configurations: For English datasets, the embedding search space contains 2^11−1=2047 nonempty combinations of contextualized, word, and character embeddings.The candidates include ELMo, Flair, BERT, GloVe, fastText, character embeddings, multilingual Flair, M-BERT, and XLM-R.
Baselines: ACE is compared with concatenating all embeddings and with random search, using the same maximum iteration count for the random-search baseline.The all-embedding baseline lets the task model learn each candidate’s contribution through the linear-layer weights.
State-of-the-art comparisons: The experiments also compare ACE with state-of-the-art systems using task-fine-tuned transformer embeddings, including XLM-R for four tasks and XLNet for dependency parsing.ACE uses fine-tuned transformer embeddings as candidates in these comparisons rather than only pretrained versions.
Results: ACE with fine-tuned embeddings achieves state-of-the-art performance in all reported test sets and is stronger than the compared fine-tuned models.These results are reported for NER, POS tagging, chunking, aspect extraction, syntactic dependency parsing, and semantic dependency parsing.

5 Analysis

ACE searches embedding concatenations more efficiently than random search and benefits from its reward design. It also outperforms alternative weighting and ensemble approaches.

5.1 Efficiency of Search Methods: ACE is consistently stronger than random search on best development accuracy for CoNLL English NER.The accuracy of the current selection gradually increases and stabilizes as more concatenations are sampled.
5.2 Ablation Study on Reward Function Design: Both the discount factor and binary vector |a_t − a_i| improve development and test performance in reward-function comparisons.The ablation samples 2,000 CoNLL English NER training sentences and trains the controller for 50 steps.
5.3 Comparison with Embedding Weighting & Ensemble Approaches: ACE outperforms All, All+Weight, Ensembledev, Ensembletest, and Random, while the other approaches have no clear winner across datasets.The results support concatenating embeddings because representations can incorporate information from multiple embeddings more directly than model voting.

6 Discussion: Practical Usability of ACE

ACE addresses the computational cost of concatenating multiple embeddings by removing embeddings that are not useful for a task. Its models may also guide weaker models through knowledge distillation.

6 Discussion: Practical Usability of ACE: ACE removes embeddings that are not very useful, making it more practical than concatenating all embeddings.Concatenating multiple embeddings is commonly used to improve structured-prediction accuracy but can be computationally costly.
6 Discussion: Practical Usability of ACE: ACE models can guide weaker structured-prediction models through knowledge distillation toward models that are both stronger and faster.

7 Conclusion

ACE automatically searches for effective embedding concatenations using reinforcement learning and a novel reward function. With fine-tuned embeddings, it achieves state-of-the-art performance across 6 tasks and 21 datasets.

7 Conclusion: ACE uses a simple search space and reinforcement learning with a novel reward function to guide embedding-concatenation search.The reward incorporates changes in embedding concatenations and is reported as stronger than simpler alternatives.
7 Conclusion: ACE outperforms strong baselines and achieves state-of-the-art performance in 6 tasks over 21 datasets with fine-tuned embeddings.

A Detailed Configurations

The experiments use task-specific metrics and separate optimization configurations for sequence-structured and graph-structured tasks. Controller hyperparameters are tuned by grid search, and training cost varies with task and dataset size.

Evaluation: NER, chunking, and aspect extraction use F1; POS tagging uses accuracy; DP uses UAS and LAS; SDP uses labeled F1.
Task Models and Controller: Sequence-structured tasks use SGD with batch size 32, learning rate 0.1, development-based annealing, and a 150-epoch limit.
Task Models and Controller: Graph-structured tasks use Adam with learning-rate annealing every 5,000 iterations and a 300-epoch maximum.
Task Models and Controller: Dependency parsing applies maximum spanning trees during testing, and task-model hyperparameters are fixed.
Task Models and Controller: Controller learning rate and discount factor are selected by grid search, with 0.1 and 0.5 performing best on the development set.Controller parameters start at zero so candidates are selected evenly during the first two time steps.
Task Models and Controller: Training the controller for 30 steps on CoNLL English NER takes 45 GPU hours on one Tesla P100 GPU.The paper presents this as an acceptable practical training time.
Sources of Embeddings: The embedding sources used in the experiments are listed in Table 7.

B Additional Analysis

This analysis replaces document-level transformer representations with sentence-level representations when document context is unavailable, and compares ACE with fine-tuned embeddings.

Sentence-level representations are used for BERT and XLM-R because document contexts may be unavailable in some applications.
ACE further improves the accuracy of fine-tuned embedding models.

B.3 Retraining

The retraining analysis evaluates whether searched embedding concatenations remain effective when task models are trained from scratch. Retrained models outperform using all embeddings, but often underperform ACE.

Retrained models outperform All on every evaluated task, supporting the effectiveness of ACE’s searched embedding concatenations.
Retrained models are competitive with ACE in SDP and chunking but inferior to ACE in three other tasks.
The authors suggest ACE may benefit because each search step initializes the model from the trained model of the previous step.

B.4 Effect of Embeddings in the Searched Embedding Concatenations

The best embedding concatenation varies across structured-output, semantic or syntactic, and monolingual or multilingual settings. Several task-specific selection patterns nevertheless emerge.

Selection patterns: BERT appears in every best concatenation for syntactic sequence-structured tasks, while Flair, M-Flair, word, and XLM-R appear in syntactic graph-structured tasks.
Selection patterns: All multilingual NER concatenations select M-BERT, whereas M-BERT is rarely selected for multilingual aspect extraction.
Selection patterns: Monolingual Flair is always selected in NER, while XLM-R is selected more frequently in multilingual tasks than in monolingual sequence-structured tasks.
Selection patterns: Table 13 reports selection percentages by embedding candidate, sequence- versus graph-structured tasks, semantic versus syntactic tasks, and language setting.