PaperlayerView exact revision on arXiv

Source-linked AI summary

Natural Language Processing Advancements By Deep Learning: A Survey

Amirsina Torfi, Rouzbeh A. Shirvani, Yaser Keneshloo, Nader Tavaf, Edward A. Fox

arXiv:2003.01200v4cs.CLcs.AIcs.LG

TL;DR

NLP needs data-driven methods to analyze human language as linguistic data and computational resources expand. This survey categorizes deep learning architectures, representations, tasks, applications, datasets, and evaluations in NLP. It reports that surveyed models advance multiple NLP areas, including named entity recognition, where one cited model achieved an F-score of 93.5 on CoNLL03.

  • Problem

    NLP seeks to help machines understand, process, and analyze human language, while expanding linguistic data increases demand for automated semantic analysis.

  • Method

    The survey categorizes deep learning architectures, representations, datasets, NLP tasks, applications, challenges, opportunities, and evaluations across the field.

  • Results

    An F-score of 93.5 on the CoNLL03 dataset is reported for one named entity recognition model that outperformed other recent models.

  • Takeaways & Limitations

    The survey provides educational resources intended to help students and researchers understand NLP and advance its integration with deep learning.

Abstract

from arXiv · show

Natural Language Processing (NLP) helps empower intelligent machines by enhancing a better understanding of the human language for linguistic-based human-computer communication. Recent developments in computational power and the advent of large amounts of linguistic data have heightened the need and demand for automating semantic analysis using data-driven approaches. The utilization of data-driven strategies is pervasive now due to the significant improvements demonstrated through the usage of deep learning methods in areas such as Computer Vision, Automatic Speech Recognition, and in particular, NLP. This survey categorizes and addresses the different aspects and applications of NLP that have benefited from deep learning. It covers core NLP tasks and applications and describes how deep learning methods and models advance these areas. We further analyze and compare different approaches and state-of-the-art models.

I. INTRODUCTION

NLP connects human language with computers, while deep learning has become increasingly central as computational power and linguistic data expand. This survey reviews deep learning architectures, NLP applications, datasets, challenges, and evaluations across broad categories.

  • NLP bridges natural languages and computers by enabling machines to understand, process, and analyze human language.
  • Increasing computational power, abundant big data, and successful results in computer vision and speech recognition have accelerated data-driven deep learning in NLP.
  • Deep learning has been applied to NLP tasks including part-of-speech tagging, named entity recognition, and semantic role labeling through supervised and unsupervised learning.
  • The survey builds on narrower prior reviews by covering architectures, applications, challenges, opportunities, and evaluations of deep learning for NLP.
  • Deep learning research in NLP includes recurrent, convolutional, and recursive neural networks, with the survey focusing on essential models and seminal publications.

B. Motivation for Deep Learning in NLP

NLP requires effective representations of complex language, but traditional feature engineering is costly and task-specific. Deep learning offers data-driven approaches for learning robust representations from labeled and unlabeled data.

  • Neural language modeling addresses the curse of dimensionality by learning distributed word representations and probability functions for word sequences.
  • NLP progress depends on representing language effectively, yet language representation is especially complex compared with domains such as computer vision.
  • Traditional NLP methods rely on time-consuming, application-specific handcrafted features, whereas supervised deep learning learns more general data-driven representations.
  • Unsupervised feature learning uses unlabeled data to derive low-dimensional representations from high-dimensional data spaces.
  • The potential superiority of deep learning motivates comprehensive analysis of its methods and architectures in NLP applications.

III. CORE CONCEPTS IN NLP

NLP systems depend on compact representations of linguistic elements and models that learn from context. The section introduces distributed representations, sentence-level encoding, and context-based word prediction.

  • NLP systems need learned word representations because atomic symbol representations do not directly encode informative properties.
  • Text features may represent characters, words, sentences, or other linguistic elements, with compact representations generally preferred over sparse ones.
  • Doc2vec learns fixed-length representations for variable-length sentences and documents, while skip-thought uses an encoder-decoder to generate surrounding sentences.
  • One-hot encoding produces high-dimensional sparse vectors without meaningful proximity between correlated words such as “ocean” and “water.”
  • CBOW predicts a word from nearby context and is typically trained as a pre-trained model for more sophisticated NLP tasks.

3) Word-Level Embedding:

The survey presents learned word embeddings as distributed representations that place semantically related words near one another, then describes sequence-to-sequence models for generating structured outputs.

  • Word-level representations: Word embeddings learn distributed representations in which semantically related words become correlated, improving generalization over sparse bag-of-words vectors.Contextual word usage supports similar representations for semantically correlated words.
  • Character-level representations: Character-level CNN embeddings use lower-level language elements, reduce model size, and can alleviate out-of-vocabulary problems, but may weakly capture semantic and syntactic structure.Subword methods are discussed as a higher-level alternative for out-of-vocabulary instances.
  • Seq2seq framework: Seq2seq models represent both inputs and outputs as sequences and support applications including translation, summarization, and speech conversion.The common architecture uses an encoder to produce an intermediate representation and a decoder to generate outputs.
  • Seq2seq training: Teacher Forcing trains seq2seq models with ground-truth output sequences using maximum-likelihood cross-entropy loss.The model parameters are optimized during training, and generated tokens are selected after optimization.
  • Seq2seq decoding: Beam search selects the top K candidate outputs at each step before choosing a final sequence, rather than relying only on argmax selection.The survey notes that training-time improvements to beam search are outside its scope.
  • Evaluation: Task-specific metrics such as ROUGE, BLEU, and METEOR evaluate generated sequences against ground-truth outputs.ROUGEL uses the largest common substring between the reference and generated output.

C. Reinforcement Learning in NLP

The survey describes reinforcement learning as a response to exposure bias and the mismatch between seq2seq training objectives and non-differentiable evaluation metrics, while noting substantial NLP-specific challenges.

  • Motivation: Seq2seq training faces exposure bias and inconsistency between training-time objectives and test-time measurements.Teacher Forcing uses ground-truth tokens during training, whereas testing relies on model-generated tokens.
  • Motivation: Exposure bias can induce error growth during test-time sequence generation.Scheduled sampling addresses this by gradually replacing ground-truth inputs with model-generated samples after pretraining.
  • Reinforcement-learning approaches: Reinforcement learning techniques can address both exposure bias and the mismatch between cross-entropy training and metrics such as ROUGE or METEOR.Policy-gradient and actor-critic methods are identified among commonly used approaches in deep-learning NLP.
  • Actor-critic models: Actor-critic training uses an actor to generate outputs and a critic to estimate performance, returning feedback that improves subsequent actions.In NLP, an action is typically the next output token and the state is the decoder output state.
  • Actor-critic models: NLP actor-critic models face very large action spaces, because vocabularies commonly contain 50K to 150K tokens compared with fewer than 20 actions in simple Atari games.The large action space complicates action selection and slows training.
  • Datasets and evaluation: Benchmark datasets support assessment against existing methods, but creating suitable, sufficient, and accurate datasets is application-dependent and complicated.The survey distinguishes real-world, synthetic, and toy datasets and emphasizes separate training and testing portions.

V. DEEP LEARNING FOR NLP TASKS

The survey reviews deep-learning applications across NLP tasks, including tagging, parsing, and semantic role labeling, alongside benchmark datasets and state-of-the-art model comparisons.

  • Overview: The applications section presents representative NLP tasks and frames evaluation through task-specific metrics and benchmark datasets.The survey directs readers to common evaluation metrics for NLP systems.
  • Part-Of-Speech Tagging: Part-of-speech tagging labels words by grammatical category and commonly uses the WSJ corpus, which contains over a million tokens.The WSJ corpus is widely used to assess POS-tagging performance.
  • Part-Of-Speech Tagging: POS-tagging research combines word- and character-level representations, convolutional networks, recurrent networks, bidirectional LSTMs, and CRF layers.CharWNN joins word- and character-level representations using CNNs.
  • Parsing: Constituency parsing assigns syntactic structure to sentences, with neural approaches learning from vector representations and morphological embeddings.The survey states that deep neural models outperformed traditional algorithms in parsing.
  • Parsing: Dependency parsing represents sentence structure through relations among words rather than phrasal elements and phrase-structure rules.Neural transition systems, biaffine classifiers, and bidirectional LSTMs are among the reviewed approaches.
  • Semantic Role Labeling: Semantic role labeling identifies predicate-argument structure, characterizing who did what to whom, and also how, where, and when.The survey reviews approaches using explicit or implicit syntactic information.

3) Semantic Role Labeling:

The survey contrasts syntactic and syntactic-agnostic approaches to semantic role labeling, then reviews deep-learning architectures and benchmark results across classification and information extraction tasks.

  • Semantic Role Labeling: Semantic role labeling research includes joint predicate-argument prediction, new word representations, self-attention, and syntax-informed models.LISA is described as leveraging syntactic information for SRL.
  • Benchmarks: The reviewed task comparisons use OntoNotes for SRL, AG News for classification, and CoNLL-2003 for named entity recognition.The cited tables specify F1 for SRL and NER and classification accuracy for AG News.
  • Text Classification: Text classification assigns predefined categories to words, sentences, or documents for organization and further analysis.Reviewed architectures include CNNs, RNNs, dynamic k-max pooling, and combined recurrent-convolutional models.
  • Information Extraction: Information extraction converts unstructured sources such as social-media posts and online news into structured information.The surveyed subtasks include named entity recognition, relation extraction, coreference resolution, and event extraction.
  • Information Extraction: Named entity recognition locates and categorizes entities in context using CNN, RNN, and hybrid bidirectional LSTM-CNN architectures.The survey reports an F-score of 93.5 on CoNLL03 for the cited leading model.
  • Information Extraction: Relation extraction identifies semantic relationships between entity pairs using recursive, convolutional, tree-structured LSTM, and related neural models.These approaches learn compositional, lexical, sentence-level, or tree-structured representations.

3) Coreference Resolution:

The survey describes coreference resolution as identifying mentions that refer to the same entity, then situates related sentiment-analysis tasks and deep-learning approaches within NLP applications.

  • Coreference Resolution: Coreference resolution identifies mentions in context that refer to the same entity, such as “car,” “Camry,” and “it.”
  • Sentiment Analysis: Sentiment analysis extracts subjective information and is commonly organized into document-, sentence-, and aspect-level tasks.
  • Aspect-level Sentiment Analysis: Aspect-level analysis combines aspect extraction with sentiment classification because different targets within one text may receive different opinions.
  • Aspect-level Sentiment Analysis: Deep-learning approaches for aspect-based sentiment analysis include attention-based LSTMs, sentence-pair classification, BERT post-training, and pretrained embeddings.

2) Neural Machine Translation:

Neural machine translation uses end-to-end sequence models that generate outputs recurrently and increasingly incorporate attention, hierarchical representations, and pretrained or adversarial components.

  • Neural Machine Translation: End-to-end neural machine translation avoids extensive preprocessing and word alignments by shifting emphasis toward network structure.
  • Sequence-to-sequence models: Recurrent encoder-decoder models read input tokens and generate output tokens until an end-of-sentence token determines the sequence length.
  • Recurrent Continuous Translation Models: RCTM builds hierarchical sentence representations that move from lower-layer n-grams toward whole-sentence representations through convolutional layers.
  • Recurrent Continuous Translation Models: RCTM II estimates target-sentence length and achieves better perplexity on WMT datasets than other existing machine translation systems.
  • Attention-based translation: Attention addresses fixed-length encoder-decoder bottlenecks by allowing each decoding step to use a different context vector derived from useful source segments.
  • Subsequent developments: Later systems address rare words, multilingual translation, noisy inputs, and forgetting using residual layers, BERT, mBART, adversarial training, and dynamic gating.

F. Question Answering

Question answering seeks specific answers inferred from available documents rather than merely retrieving documents, and its development spans symbolic, statistical, and deep-learning systems.

  • Question Answering: Question answering is a fine-grained form of information retrieval that seeks specific answers inferable from available documents.
  • Question Answering: The survey reviews QA history, deep-learning breakthroughs, and visual question answering as related areas of study.
  • Early QA systems: An early baseball QA system combined question read-in, dictionary lookup, syntactic analysis, content analysis, and relevance estimation.
  • Statistical QA systems: IBM’s statistical QA system included question-answer type classification, query expansion and retrieval, named-entity processing, and answer selection.

2) Question answering in the era of deep learning:

Deep learning supports question answering by encoding questions, answers, and multimodal inputs while using matching, memory, attention, and coattention mechanisms. The section also surveys extractive and abstractive document summarization, including their modeling advances and evaluation limitations.

  • Question answering: Deep-learning question-answering systems encode question–answer pairs into fixed-length vectors and calculate their matching degree with a nonlinear tensor layer.The approach avoids relying solely on distance measures such as cosine correlation.
  • Question answering: Dynamic Memory Networks process input-question-answer triplets and form episodic memories to answer relevant questions.Their modules encode inputs and questions, then iteratively use attention to focus on relevant input parts.
  • Question answering: Visual Question Answering answers natural-language questions about images using neural models and spatial attention over image regions.The VQA dataset contains over 250K images, 760K questions, and around 10M answers.
  • Document summarization: Extractive summarization selects salient sentences, whereas abstractive summarization generates sentences that may contain novel words.Extractive outputs preserve the author’s expression but may be long or overlapping; abstractive outputs are shorter but hard to train.
  • Document summarization: Summarization research advanced from ranking-based extraction and attention-based sequence-to-sequence headline generation toward graph attention, transformers, pointer-generators, and copying mechanisms.Attention-based encoder-decoder models can generate repeated phrases, motivating methods that avoid unknown tokens and copy from input articles.
  • Document summarization: ROUGE and BLEU can undervalue abstractive summaries that paraphrase reference sentences with different words, motivating better automated evaluation methods.ROUGE primarily counts shared n-grams between sentences.

H. Dialogue Systems

Dialogue systems support human–computer interaction through task-based and non-task-based designs. Deep learning is being used to improve performance, generalizability, and robustness, while task-based systems commonly organize language understanding, dialogue management, and response generation.

  • Dialogue system types: Dialogue systems are task-based or non-task-based, and this section focuses on their linguistic components rather than speech technology.Automatic Speech Recognition and language-to-speech components may exist but are outside the discussion.
  • Motivation: Deployed dialogue systems often depend on expensive handcrafted features whose generalizability to other domains and functionalities is problematic.Recent attention therefore focuses on deep learning for performance, generalizability, and robustness.
  • Task-based systems: A task-based dialogue system typically includes Natural Language Understanding, a Dialogue Manager, and Natural Language Generation.NLU interprets the user utterance, the DM selects a semantic response, and NLG produces the resulting utterance.
  • Task-based systems: Task-based systems require predefined tasks with manually crafted states and associated slots, limiting their usefulness for other tasks.This design creates a scope boundary around the tasks anticipated during construction.
  • Task-based systems: Deep reinforcement learning and end-to-end frameworks have produced promising results for task-oriented dialogue performance, domain adaptation, and dialogue generation.End-to-end systems replace the traditional pipeline with a single module that handles external databases.

2) Non-task-based Systems:

Non-task-based dialogue systems aim for natural conversation and include retrieval-based and generative chatbots. Retrieval systems select responses from candidates, while generative systems create responses from scratch but face greater risks of grammatical and conceptual errors and out-of-corpus failures.

  • Non-task-based systems: Non-task-based dialogue systems aim to enable natural conversation between machines and humans.Chatbots generally use either retrieval-based or generative methods.
  • Retrieval-based methods: Retrieval-based models select responses from candidate sets and can provide more concise, fluent, and accurate outputs.Their core operation is query-response matching, commonly formulated as an information-retrieval search problem.
  • Generative methods: Generative models produce responses from scratch without predefined responses, typically mapping an input query to a target response with sequence-to-sequence models.They can respond when suitable outputs are absent from the corpus.
  • Generative methods: Generative models are more prone than retrieval-based models to grammatical and conceptual mistakes.This is a stated trade-off of generating responses rather than selecting them.
  • Open-domain challenge: Large conversation corpora do not directly solve the challenge of generating proper responses to out-of-corpus conversations.The difficulty arises when queries lack matching content in the training data despite the wide range of plausible human queries.
  • Open-domain challenge: Evaluating natural conversation requires characterizing and measuring conversational properties and incorporating that knowledge into dialogue systems.One qualitative criterion is whether generated utterances can be distinguished from natural human dialogues.
  • Open-domain challenge: Commercial systems such as Siri and Alexa still heavily rely on handcrafted features, while data-driven dialogue machines remain difficult to design and train.The stated difficulties include natural-language complexity, framework design, and complex data sources.
  • Conclusion: The survey presents a categorized account of deep-learning NLP research and emphasizes its educational value for understanding notable works and advancing NLP integration.The authors note that rapid progress may make some cited references dated, although newer publications may build on them.
Loading 2003.01200v4…