Source-linked AI summary
Natural Language Processing: State of The Art, Current Trends and Challenges
Diksha Khurana, Aditya Koli, Kiran Khatter, Sukhdev Singh
TL;DR
NLP addresses the challenge of representing and analyzing human language computationally across diverse applications. The paper synthesizes NLP levels, NLG phases, historical development, applications, trends, and challenges, concluding with examples of current and emerging uses. Its scope also includes privacy concerns in chatbot systems and the need for further refinement and evaluation in a proposed conceptual framework.
Problem
NLP must represent and analyze human language despite ambiguity, language barriers, and the difficulty of preserving meaning across languages.
Method
The paper reviews NLP levels, describes NLG phases and components, and surveys the field’s history, applications, trends, and challenges.
Results
The paper presents a state-of-the-art overview of NLP applications and current trends and challenges across areas including translation, information extraction, summarization, medical applications, and question answering.
Takeaways & Limitations
NLP is increasingly used to make information accessible, summarize large volumes of data, and support applications such as multilingual translation and sentiment analysis.
Takeaways & Limitations
The chatbot example raises privacy concerns because administrators may access full conversation transcripts containing customers’ personal banking information.
Abstract
from arXiv · showhide
Natural language processing (NLP) has recently gained much attention for representing and analysing human language computationally. It has spread its applications in various fields such as machine translation, email spam detection, information extraction, summarization, medical, and question answering etc. The paper distinguishes four phases by discussing different levels of NLP and components of Natural Language Generation (NLG) followed by presenting the history and evolution of NLP, state of the art presenting the various applications of NLP and current trends and challenges.
1. Introduction
NLP combines artificial intelligence and linguistics to help computers understand human language, with applications spanning language analysis and generation. The field addresses linguistic ambiguity and includes tasks such as translation, summarization, and entity recognition.
- NLP combines artificial intelligence and linguistics to make computers understand statements or words written in human languages.
- Natural language processing is broadly divided into Natural Language Understanding and Natural Language Generation.These components address understanding and generating text.
- Linguistics contributes analyses of sound, word formation, sentence structure, meaning, and language use.
- NLP includes automatic summarization, co-reference resolution, discourse analysis, machine translation, morphological segmentation, named entity recognition, optical character recognition, and part-of-speech tagging.The passage identifies machine translation, named entity recognition, and optical character recognition as tasks with direct real-world applications.
- Ambiguity is a major NLP problem, arising across lexical, morphological, and syntactic levels and requiring knowledge of the complete sentence for resolution.
2. Levels of NLP
The paper presents levels of language as a way to represent NLP and generate text through content planning, sentence planning, and surface realization. It also introduces linguistic levels including sound and word formation.
- The levels of language represent NLP and support text generation through content planning, sentence planning, and surface realization.
- Phonology concerns the systematic arrangement and organization of sounds in language.
- Morphology studies words through morphemes, the smallest units of meaning, including prefixes, roots, suffixes, and grammatical endings.The paper illustrates this with the decomposition of “precancellation” into pre, cancella, and -tion.
3. Lexical
The lexical level analyzes individual words and assigns context-sensitive interpretations, including probable parts of speech and semantic representations.
- Lexical processing assigns each word a part-of-speech tag, selecting the most probable tag from its sentence context when alternatives exist.
- Lexical processing also represents word meaning, with the form varying according to the semantic theory used by the NLP system.
- The next level examines words in a sentence to uncover grammatical structure and structural dependency relationships.Grammar and a parser are required, although not every NLP application needs a full parse.
5. Semantic
Semantic processing determines possible sentence meanings by combining word-level meanings and resolving ambiguity among words with multiple senses.
- Semantic processing determines possible sentence meanings from interactions among word-level meanings.
- It can disambiguate words with multiple senses by using their sentence-level interactions.
6. Discourse
Discourse-level NLP analyzes text beyond individual sentences by connecting them into a meaningful whole. It includes anaphora resolution and discourse/text structure recognition, while pragmatic interpretation uses situational and world knowledge.
- Discourse-level NLP analyzes units longer than a sentence as connected text rather than as isolated sentences.
- Anaphora resolution replaces semantically stranded pronouns with the entities they refer to.
- Discourse/text structure recognition identifies sentence functions that contribute to a meaningful representation of the text.
- Pragmatic interpretation explains meaning not literally encoded in text by using situational context and world knowledge about intentions, plans, and goals.
3. Natural Language Generation
Natural Language Generation produces meaningful language from internal representations through goal identification, planning, and realization. Its components organize content and linguistic resources into fluent text or voice output within a modeled situation.
- NLG produces meaningful phrases, sentences, and paragraphs from an internal representation.
- NLG proceeds through identifying goals, planning how to achieve them, and realizing the plans as text.
- A speaker or application supplies intentions, while a generator renders them into fluent phrases relevant to the situation.
- Components and Levels of Representation: Content selection and textual organization determine which information is included and how it is structured.
- Components and Levels of Representation: Linguistic resources provide words, idioms, and syntactic constructs that support realization as actual text or voice output.
- Application or Speaker: The application or speaker models the situation, stores history, structures relevant content, and selects propositions it knows.
4. History of NLP
NLP evolved from early machine-translation and question-answering research through grammar- and logic-based systems toward statistical processing and information extraction. Recent work emphasizes unsupervised and semi-supervised learning.
- Machine-translation research began in the late 1940s, initially emphasizing Russian and English alongside other languages such as Chinese.
- After the 1966 ALPAC report judged machine translation unproductive, later production systems eventually supplied output to customers.
- AI-influenced NLP systems in the 1960s included BASEBALL, LUNAR, and Winograd SHRDLU, which increased linguistic and task-processing sophistication.
- Computational grammar theory became active in the early 1980s, linked with logic-based approaches to meaning, beliefs, intentions, emphasis, and themes.
- By the end of the 1980s, general-purpose sentence processors and discourse theories supported extended discourse, alongside growing practical resources and tools.
- Statistical language processing, information extraction, and automatic summarization became major focuses in the 1990s, while recent research emphasizes unsupervised and semi-supervised learning.
5. Related Work
Related NLP work spans sentiment, tagging, chunking, entity recognition, emotion detection, semantic role labeling, and event discovery. These studies address multilingual, informal, noisy, and domain-specific language conditions.
- NLP research tools include sentiment analyzers, POS taggers, chunking, named entity recognition, emotion detection, and semantic role labeling.
- Sentiment Analysis: Sentiment analysis extracts topic-specific features and sentiment, using lexicons and pattern databases to rate documents from -5 to +5.
- Parts of Speech Tagging: POS taggers classify words into categories such as nouns, adjectives, and verbs, but procedures effective for European languages may not transfer efficiently to Asian or Middle Eastern languages.
- Chunking: Chunking labels sentence segments such as noun and verb phrases, with reported systems achieving around 94.3% F1 on the CoNLL 2000 shared task.
- Named Entity Recognition: Informal Internet language degrades standard NER performance, while annotated in-domain and out-of-domain data improves performance compared with standard tools.
- Emotion Detection: Emotion detection categorizes mixed-language social-media statements into six emotion groups while identifying ambiguous words and lexical categories.
- Semantic Role Labeling: Semantic role labeling assigns roles to verb arguments through parsing, argument identification, and classification stages.
- Event Discovery: A graphical model extracted event records from noisy social-media feeds by aggregating information across multiple messages.
6. Applications of NLP
NLP applications span translation, categorization, information extraction, summarization, dialogue, and medicine, using language analysis to make information more accessible and useful.
- Machine Translation: Machine translation seeks to preserve sentence meaning, grammar, and tense when converting text between languages.Statistical systems learn correspondences from parallel data, while neural and deep-learning approaches have also been introduced.
- Text Categorization: Text categorization assigns incoming documents to predefined categories, supporting indexing, routing, and email spam filtering.Spam filters distinguish unwanted from legitimate messages using content, headers, blacklists, rules, permissions, or challenge-response protocols.
- Information Extraction: Information extraction identifies entities and phrases such as names, places, events, dates, times, and prices for focused search and summarization.Extracted segments can populate specific search fields and improve the presentation and matching of results.
- Knowledge Discovery: Knowledge-discovery techniques include POS tagging, chunking, stop-word removal, stemming, phrase indexing, and word-sense disambiguation.These techniques support summaries, databases, keyword identification, and predefined text classification, although acquiring relations between extracted terms remains difficult.
- Summarization: Text summarization addresses information overload while preserving meaning and includes single- or multi-document, generic or query-focused, supervised or unsupervised variants.BSTM, FGB, and TAOS represent model families for multi-document summarization using term, sentence, document, or topic associations.
- Dialogue and Medicine: Dialogue systems currently use phonetic and lexical processing in narrow applications, while broader language-level integration is presented as a route toward fully automated dialogue.NLP is also used in medicine to extract and summarize symptoms, drug dosage, and response data for clinical and biomedical applications.
7. Approaches
NLP approaches range from symbolic representations and generative models to discriminative and probabilistic sequence models, with applications spanning language identification, translation, and speech recognition.
- Symbolic Approach: The rationalist or symbolic approach encodes linguistic knowledge directly in rules or other representations and assumes some knowledge is innate.It aims to provide machines with fundamental knowledge and reasoning mechanisms resembling aspects of human language processing.
- Generative and Discriminative Models: Generative models use deeper language knowledge to match an unknown speaker’s language, whereas discriminative methods distinguish languages with fewer knowledge requirements.Discriminative methods can accommodate more features, while generative models may become troublesome when many features are used.
- Hidden Markov Models: Hidden Markov Models represent transitions among finite hidden states that generate observable output symbols and support inference, pattern matching, and training.Inference estimates likely state sequences for observed outputs, while training estimates state-transition and output probabilities from data.
- Hidden Markov Models: HMMs are extensively used in speech recognition by matching output sequences to phoneme sequences, and they also support bioinformatics tasks such as multiple-sequence alignment.The paper presents HMMs as applicable beyond speech recognition.
- Research Scope: NLP research covers common tasks such as word segmentation and translation alongside areas including infant-learning segmentation and opinion-versus-fact document identification.The discussed research also includes Bayesian methods for designing algorithms.
8. NLP in Talk
Recent NLP deployments make business, customer, biomedical, and multilingual information more accessible through document analysis, chatbots, network analysis, and speech translation.
- Business and Compliance: RAVN’s GDPR Robot analyzes business documents to search, retrieve, flag, classify, and report potentially sensitive personal data.It also identifies personal data requiring attention and supports reports concerning data to delete or secure.
- Customer Chatbots: Capital One’s Eno is a natural-language SMS chatbot that lets customers ask questions about savings and other information through text.The service is presented as a human-like conversational environment using a direct customer interface.
- Data Chatbots: A separate chatbot trend aims to return intelligible answers by learning semantic relations and inferences rather than merely displaying filtered data pages.The proposed interaction begins with questions about business changes and progressively performs filtering and formulation automatically.
- Biomedical Research: The MTM framework study extracted concepts from 142 Medline abstracts with MetaMap, analyzed pairwise co-occurrence, and constructed a concept network.Medication adherence was the most studied drug-therapy problem and co-occurred with patient-centred self-management interventions.
- Speech Translation: Waverly Labs’ Pilot combines speech recognition, machine translation, and machine learning to translate spoken and written languages through connected earpieces.Users can participate in group conversations, although the described experience includes a few-second lag that the company intends to reduce.