PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

Katikapalli Subramanyam Kalyan

arXiv:2310.12321v1cs.CL

TL;DR

The survey addresses the need for a comprehensive account of rapidly advancing GPT-3 family large language models and their research applications. It synthesizes foundation concepts and more than 350 papers across model development, tasks, domains, languages, capabilities, robustness, and evaluation. The paper concludes with future research directions and presents the synthesis as a resource for academic and industry readers.

  • Problem

    Rapid growth of GPT-3 family large language models creates a need for a comprehensive survey of research progress across multiple dimensions.

  • Method

    The paper synthesizes more than 350 research papers while reviewing foundations, GLLM models, applications, capabilities, robustness, evaluation, and future directions.

  • Results

    The survey covers GPT-3 family models across downstream tasks, specific domains, multiple languages, data labelling, data augmentation, robustness, and evaluation.

  • Takeaways & Limitations

    The survey serves as a resource for academic and industry readers to stay updated on research related to GPT-3 family large language models.

  • Takeaways & Limitations

    GPT-3 family models can generate harmful text, limiting their scalable deployment for public use.

Abstract

from arXiv · show

Large language models (LLMs) are a special class of pretrained language models obtained by scaling model size, pretraining corpus and computation. LLMs, because of their large size and pretraining on large volumes of text data, exhibit special abilities which allow them to achieve remarkable performances without any task-specific training in many of the natural language processing tasks. The era of LLMs started with OpenAI GPT-3 model, and the popularity of LLMs is increasing exponentially after the introduction of models like ChatGPT and GPT4. We refer to GPT-3 and its successor OpenAI models, including ChatGPT and GPT4, as GPT-3 family large language models (GLLMs). With the ever-rising popularity of GLLMs, especially in the research community, there is a strong need for a comprehensive survey which summarizes the recent research progress in multiple dimensions and can guide the research community with insightful future research directions. We start the survey paper with foundation concepts like transformers, transfer learning, self-supervised learning, pretrained language models and large language models. We then present a brief overview of GLLMs and discuss the performances of GLLMs in various downstream tasks, specific domains and multiple languages. We also discuss the data labelling and data augmentation abilities of GLLMs, the robustness of GLLMs, the effectiveness of GLLMs as evaluators, and finally, conclude with multiple insightful future research directions. To summarize, this comprehensive survey paper will serve as a good resource for both academic and industry people to stay updated with the latest research related to GPT-3 family large language models.

1 INTRODUCTION

The introduction traces NLP’s evolution from rule-based and feature-engineered systems to transformer-based pretrained models and GPT-3 family LLMs. It positions this survey as a comprehensive review of GLLMs across models, tasks, capabilities, robustness, evaluation, and future directions.

  • In-context learning: LLMs use in-context learning to treat NLP tasks as conditional text generation, avoiding task-specific fine-tuning and large numbers of labelled instances.Prompts can include task descriptions, test inputs, and optionally a few examples.
  • NLP evolution: NLP evolved from laborious rule-based systems and feature engineering toward transformers that capture long-term dependencies through fully attention-based processing.Traditional models also struggled to leverage GPUs fully because of sequential processing.
  • From PLMs to LLMs: Pretrained language models use transformers and self-supervised learning, but unlike LLMs, they generally require task-specific training to generalize to unseen tasks.LLMs emerged to address this limitation through broader task generalization without task-specific training.
  • Survey scope: The survey reviews GPT-3, InstructGPT, Codex, ChatGPT, and GPT-4, covering more than 350 relevant research papers collected through citation, venue, and database searches.The collection spans papers published online from June 2020 to September 2023 and was manually reviewed after duplicate removal.
  • Survey scope: Its coverage spans foundation concepts, downstream tasks, specific domains, multilingual scenarios, data labelling and augmentation, robustness, evaluation, and future research directions.The survey is intended to support academic and industry readers tracking research on GPT-3 family large language models.

2 FOUNDATION CONCEPTS

Foundation concepts trace NLP from traditional deep learning through transformers, transfer learning, self-supervised learning, pretrained language models, and large language models. This progression addresses costly labelled-data requirements and enables adaptation or generalization with less task-specific supervision.

  • Traditional Deep Learning Models: Traditional deep learning models include MLPs, CNNs, RNNs, LSTMs, GRUs, and sequence-to-sequence architectures, but face limitations in sequence understanding, computation, long-term dependencies, and sequential processing.MLPs struggle with semantic relationships; CNNs capture local features but not long-term dependencies; recurrent models process tokens sequentially and can suffer vanishing gradients.
  • Transformers: Transformers use self-attention to process sequences without recurrence and became a preferred architecture for pretrained and large language models.Transformer encoder and decoder layers use attention, feedforward networks, embeddings, residual connections, and normalization to build contextual representations and generate outputs.
  • Transfer Learning: Transfer learning reuses knowledge from a source task or domain for a related target task or domain, avoiding training from scratch and potentially improving target performance.Unlike multitask learning, transfer learning learns the source task first and then transfers its knowledge to the target task.
  • Self-Supervised Learning: Self-supervised learning automatically generates labels from pretraining tasks and uses unlabelled data, providing an alternative to large-scale supervised pretraining with expensive annotations.It supports learning universal knowledge and addresses annotation costs, generalization errors, and spurious correlations associated with supervised learning at scale.
  • Large Language Models: Large language models scale model size, pretraining corpus, and computation, then use in-context learning to perform many tasks without task-specific fine-tuning.Their pretraining supplies universal language knowledge, while emerging abilities support remarkable performance across many NLP tasks from prompts containing task descriptions and optionally a few examples.

3 GPT-3 FAMILY LARGE LANGUAGE MODELS

The GPT-3 family evolved through scaling, broader pretraining data, code-focused fine-tuning, and alignment methods toward chat-optimized models such as ChatGPT and GPT-4.

  • GPT evolution: GPT-1 introduced OpenAI’s transformer-based pretrained language model and the “pretrain and fine-tune” paradigm.GPT-1 contained 117M parameters.
  • GPT evolution: GPT-2 expanded pretraining to the 40B-token Web-Text corpus and four model sizes up to 1.5B parameters.Perplexity continued decreasing as model size increased, without saturation at 1.5B parameters.
  • GPT evolution: GPT-3 scaled model size, pretraining data, and computation, reaching 175B parameters and learning from webpages, Wikipedia, and books.This scaling enabled GPT-3 to handle unseen tasks without task-specific training.
  • GPT-3 limitations: GPT-3’s limitations included weak complex reasoning, difficulty following instructions, and occasional harmful text generation.Its pretraining did not include code data, and its corpus contained misinformation, abuse, hate, and sexism.
  • Alignment and chat optimization: A pretraining–inference mismatch causes GPT-3 to sometimes fail to understand prompts and continue generating likely next words instead.Pretraining predicts the next word from previous words, whereas in-context learning conditions generation on the prompt.
  • Alignment and chat optimization: GPT-3.5 models addressed these limitations through code-data fine-tuning and supervised fine-tuning or reinforcement learning from human feedback.ChatGPT and GPT-4 further addressed the lack of chat optimization, with ChatGPT introduced in November 2022.

4 PERFORMANCE OF GLLMS IN DOWNSTREAM TASKS

Research on GLLMs spans text classification and information extraction, showing that prompting and task-specific frameworks can substantially affect performance, while results vary against fine-tuned models.

  • Text Classification: GLLMs are evaluated on text classification tasks including sentiment, stance, intent, mental health, hate speech, misinformation, paraphrase, and inference.Studies use zero-shot and few-shot settings with varied prompting strategies.
  • Text Classification: Advanced prompting strategies generally outperform direct prompting by generating intermediate outputs that guide the final answer.One-shot chain-of-thought prompting achieved near-state-of-the-art stance-detection results, while emotion-enhanced chain-of-thought improved mental-health analyses.
  • Text Classification: ChatGPT generally outperforms traditional CNN and RNN models but still lags behind task-specific fine-tuned models in text classification.Some studies nevertheless report GLLMs outperforming fine-tuned and domain-specific models.
  • Information Extraction: Information-extraction studies cover entity, relation, and event tasks, with prompting and multi-stage decomposition improving performance.ChatIE’s two-stage multi-turn question-answering framework outperformed vanilla ChatGPT by more than 18 points.
  • Information Extraction: Code pretraining and code-style prompts improve coding-related performance, with Codex outperforming GPT-3 and moderately sized fine-tuned models.The reported explanation is that code pretraining encodes structured code information useful for the task.
  • Information Extraction: GPT-3 family models can achieve state-of-the-art information-extraction results, although fine-tuned domain models can remain stronger on particular tasks.Fine-tuned PubMedBERT outperformed GPT-4 on one entity-extraction comparison with an F1-score of 86.47.

4.3 Question Answering

GLLMs are applied to question answering across domains and formats, with performance improved by external knowledge and prompting strategies but remaining uneven on complex or specialized questions.

  • Question Answering challenges: ChatGPT’s complex open-domain failures involve understanding, factual accuracy, specificity, and logical reasoning.External knowledge, knowledge-recall cues, and logical-reasoning guidance improve answer accuracy.
  • Domain evaluations: 86.8%: ChatGPT correctly answered 131 of 151 bariatric-surgery questions.The study presents ChatGPT as an additional information resource alongside healthcare professionals.
  • Domain evaluations: GPT-4 achieved the best finance-domain question-answering results, outperforming ChatGPT, BloombergGPT, FinQANet, OPT, and BLOOM.The cited comparison includes both domain-specific and general language models.
  • Prompting strategies: Multiple-choice prompts helped GLLMs achieve near-state-of-the-art results across 20 datasets by supplying answer options with each question.Cloze prompts omit answer options and were identified as a reason for weaker multiple-choice performance.
  • Scope of Question Answering: Research evaluates GLLMs on tabular, knowledge-based, code, multi-document, conversational, educational, healthcare, legal, finance, and scientific question answering.Most studies focus on English, with some work in Portuguese and Japanese.
  • State-of-the-art results: GLLMs can reach state-of-the-art question-answering performance, including a visual question-answering approach that exceeded existing methods by over 8 points.The approach used GPT-3 as an implicit, unstructured knowledge source.

4.4 Machine Translation

The survey examines GPT-3 family models for machine translation across languages, domains, and granularities, emphasizing prompting strategies and comparisons with neural and commercial systems.

  • GLLMs have been investigated for machine translation across multiple languages, domains, and sentence-, paragraph-, and document-level settings.
  • A two-step prompting strategy with a Japanese-to-Chinese pre-edit scheme improves translation quality by more than 35%.
  • ChatGPT outperforms other evaluated LLMs across 102 languages and 202 translation directions but lags behind NLLB in most directions.
  • GLLM translation performance is generally satisfactory but not consistently on par with commercial systems, although some document-level approaches surpass them.

4.6 Dialogue Tasks

The survey covers GLLMs across dialogue understanding, generation, summarization, tracking, and evaluation, showing that prompting and instruction-tuning data strongly affect outcomes.

  • GLLMs have been explored for dialogue summarization, question answering, emotion understanding and generation, state tracking, generation, discourse analysis, and evaluation.
  • ChatGPT performs better in dialogue state tracking than spoken language understanding, with gains from multi-turn prompts and richer slot-filling details.
  • ChatGPT performs better in emotion dialogue generation than emotion dialogue understanding.
  • GPT-3-generated summaries yield comparable performance to human-generated summaries, while mixed training summaries perform better than either source alone.
  • Selecting few-shot examples with a scoring system reduces dialogue summarization failures by 11%, while evaluator performance depends on instruction-tuning dataset diversity and relevance.

4.9 Coding Tasks

The survey finds promising but uneven GLLM performance across code generation, repair, and testing, with outcomes shaped by prompting, programming language, benchmark, and evaluator strength.

  • GLLMs have been applied to coding tasks across many programming languages, with most studies focusing on Python and Java.
  • ChatGPT generates correct code 65.2% of the time on HumanEval, versus a maximum of 46.3% for the other compared tools.
  • ChatGPT outperforms novice programmers but is outperformed by experienced programmers, while GPT-3.5 exceeds Bard by more than 37% on Java code generation.
  • Codex achieves 80% HumanEval unit-test coverage but both Codex and ChatGPT achieve less than 2% coverage on SF110.
  • Across coding tasks, GLLM performance is promising but still lags behind state-of-the-art results.
  • ChatRepair resolves 114 bugs on Defects4j 1.2 and 48 on Defects4j 2.0, exceeding previous best results by 15 and 17 bugs, respectively.
  • Advanced prompting strategies such as chain-of-thought improve ChatGPT code-generation capabilities.

4.10 Multimodal AI Tasks

The survey reviews GPT-3 family models in multimodal systems spanning image, video, audio, and language combinations, including both single-task and multi-task applications.

  • Multimodal GLLM research combines image, video, or audio inputs with language for understanding and generation tasks.
  • Applications include action generation, visual question answering, radiology report generation, named entity recognition, text-to-video, layout, and text-to-image generation.
  • DirecT2V uses GPT-4 as a frame-level director whose descriptions guide a text-to-image model in generating video frames.
  • Control-GPT combines GPT-4-generated TikZ sketches and diffusion models for controllable text-to-image generation.
  • Visual ChatGPT and related systems combine ChatGPT with visual foundation models or convert video into text stories for multiple-task understanding.
  • GPT-3 family models also support multimodal data labeling, instruction generation, data augmentation, prompt editing, and evaluation.

4.11 Machine Learning Tasks

GLLMs are explored as alternatives to conventional AutoML and as tools for planning and plan extraction. Reported results are promising, but planning remains sensitive to prompts and difficult for spatial or numerical reasoning.

  • Machine learning automation: GLLMs are being investigated as alternatives to AutoML for automating machine learning tasks.The motivation is to reduce trial-and-error, reuse prior experience, and improve interpretability limitations associated with AutoML.
  • Machine learning automation: HuggingGPT uses task planning, model selection, task execution, and response generation to solve language, vision, and speech tasks.The authors reported promising results across these three modalities.
  • Machine learning automation: MLCopilot retrieves relevant knowledge and examples before a GLLM generates results from the task description.Its two stages are an offline experience-pool construction stage and an online retrieval-and-generation stage.
  • Planning: GLLMs are used both for direct planning and for extracting structured plans from natural-language workflows.Extracted plans can subsequently be used by automated systems.
  • Planning: GPT-3 outperforms existing state-of-the-art models in some plan-extraction cases, whereas GPT-3.5 models are poor standalone planners.The models are better at extracting plans than planning independently, remain prompt-sensitive, and struggle with spatial or numerical reasoning.

5 PERFORMANCE OF GLLMS IN SPECIFIC DOMAINS

GLLMs show strong but uneven performance across healthcare, legal, and finance domains. Domain-specific models can outperform them, while results vary substantially by task, prompting, and domain expertise.

  • Domain motivation: Domain-specific NLP is challenging because specialized terminology and abbreviations make general-domain modeling insufficient.General-domain models may remain limited even after fine-tuning because fine-tuning alone cannot provide enough domain knowledge.
  • Domain motivation: GLLMs achieve impressive general-domain results, motivating evaluation of their capabilities in specific domains.The survey covers healthcare, legal, and finance applications among other domain-specific settings.
  • Healthcare: 49.25% average accuracy is reported for ChatGPT on a medical question test set, with stronger performance in general medicine than in specific ophthalmology subareas.Other studies report that ChatGPT can generate accurate diagnosis lists but remains less accurate than physicians.
  • Healthcare: Domain-specific pretrained models outperform GPT-3 on three healthcare NLP tasks despite being much smaller.The comparison covered natural language inference, question answering, and text classification.
  • Healthcare: GPT-4 outperforms LED and ranks first in the MEDIQA-Chat 2023 clinical dialogue summarization shared task.The evaluation used automatic metrics including BERTScore and ROUGE, with Instructor selecting similar few-shot examples.
  • Legal: ChatGPT performs poorly on legal text classification, while advanced prompts perform worse or no better than simple prompts on some legal multiple-choice questions.In one legal evaluation, ChatGPT performed just above the passing threshold.
  • Finance: ChatGPT and GPT-4 perform well on finance question answering but lag behind specialized or fine-tuned models on other financial tasks.Evaluated tasks include news classification, sentiment analysis, entity extraction, and question answering.
  • Finance: GPT-4 outperforms ChatGPT in few-shot financial relation extraction, and learning-based example retrieval outperforms learning-free retrieval.The result highlights the importance of example selection in few-shot in-context learning.

6 MULTILINGUAL PERFORMANCE OF GLLMS

Multilingual evaluations show that GLLM performance is biased toward English and degrades for low-resource languages, especially those using non-Latin scripts. New benchmarks and prompting methods support broader evaluation, but substantial task-specific improvement remains necessary.

  • Multilingual scope: GPT-3’s pretraining corpus covers around 90 languages, but more than 90% of its text is English.This imbalance motivates evaluation beyond English datasets, particularly for low-resource languages.
  • Multilingual scope: Multilingual GLLM studies evaluate tasks including tagging, extraction, inference, question answering, summarization, reasoning, generation, and sentiment analysis.The surveyed evaluations span multiple non-English languages and zero- or few-shot settings.
  • Observed performance: ChatGPT performance degrades for low-resource languages, particularly languages using non-Latin scripts.It also tends to misclassify abusive content as hate speech for non-English non-protected groups.
  • Benchmarks: MEGA provides multilingual evaluation across 16 datasets covering 70 languages, while reported results show GPT-4 outperforming ChatGPT.Performance is promising on some tasks but leaves substantial room for improvement on others.
  • Prompting: English prompting can improve GLLM performance on non-English languages, while evaluations confirm ChatGPT’s bias toward English.The reported evaluation covered seven tasks and more than 30 languages.
  • Evaluation: GPT-4-based multilingual evaluators tend to favor high scores and should therefore be used carefully.This caveat concerns their use for evaluating natural language generation in multilingual settings.

7 DATA LABELLING AND DATA AUGMENTATION ABILITIES OF GLLMS

GLLMs are studied for reducing the cost and effort of data labelling and for generating synthetic training data. Results indicate benefits in some settings, but performance depends on the task, annotation comparison, and preservation of labels during augmentation.

  • Data labelling: Human annotation is accurate but expensive, laborious, and time-consuming, motivating GLLM-based data labelling.GLLMs receive detailed instructions and labelled examples to annotate data.
  • Data labelling: GLLM data-labelling studies cover classification, extraction, summarization, question generation, and other NLP tasks across several languages.Most surveyed work focuses on English datasets, with fewer studies in French, Spanish, Italian, and Basque.
  • Data labelling: PubMedBERT outperforms GPT-3.5 and GPT-4 by 6 and 5 F1 points, respectively, for adverse drug reaction extraction.PubMedBERT achieves results comparable to the state-of-the-art model after fine-tuning on GPT-3.5-labelled data.
  • Data labelling: 60.9 average accuracy is reported for ChatGPT across five social-computing datasets.The evaluated tasks include stance detection, hate speech detection, bot detection, and sentiment analysis.
  • Data labelling: ChatGPT exceeds MTurk crowd-workers by 25 accuracy points, costs approximately 30 times less, and has higher intercoder agreement.These findings come from zero-shot annotation across four text-classification tasks.
  • Data augmentation: Existing augmentation methods often lack diversity and struggle to guarantee accurate labels.Text augmentation can operate at character, word, or sentence level, including deletion, replacement, back translation, and paraphrasing.
  • Data augmentation: ChatGPT-generated paraphrases improve fine-tuned pretrained-model performance on medication event classification and medical identification tasks.Other studies report that larger GPT-3 variants with inline annotations perform best for paraphrase-based entity-extraction augmentation.
  • Data augmentation: Synthetic GLLM data is used for implicit hate-speech detection, multilingual tweet-intimacy prediction, named entity recognition, and relation classification.In healthcare tasks, models fine-tuned on synthetic data outperform zero-shot ChatGPT by a large margin.

8 DETECTING GLLM GENERATED TEXT

GLLM adoption has increased concerns about misuse, motivating research on detecting machine-generated text across domains. Existing detectors can perform well in some settings but remain vulnerable to paraphrasing and other attacks.

  • Motivation: Human-like fluency makes GLLM-generated text difficult to distinguish from human-authored text, creating risks of misuse in news, education, and research.Reported misuse includes fake news, academic fraud, and fabricated research papers.
  • Research scope: Detection research spans scientific, academic, healthcare, news, legal, social-media, and financial text.The survey reviews approaches across multiple application domains.
  • Detection approaches: Existing detection tools range from logistic-regression classifiers to pretrained-language-model classifiers, but benchmark performance is often unsatisfactory.MGTBench found that most detectors performed unsatisfactorily, while the ChatGPT and LM detectors were exceptions.
  • Trained detectors: More than 98% accuracy was achieved by an LSTM classifier with word2vec embeddings for identifying GPT-3-generated scientific abstracts.This classifier outperformed the other evaluated classifiers.
  • Trained detectors: More than 97% accuracy was reported by fine-tuned RoBERTa and T5 classifiers on ChatGPT-generated paraphrased text.The OpenGPTText dataset contains ChatGPT-generated paraphrases.
  • Robustness: Paraphrase attacks caused a drop of more than 65% in DetectGPT accuracy, showing that existing detectors are not robust to common transformations.Other evaluated attacks included synonym replacement and writing-style modification.

9 ROBUSTNESS OF GLLMS

GLLMs can achieve strong zero- and few-shot performance, including state-of-the-art results, but their robustness varies substantially across adversarial and out-of-distribution settings. Research evaluates these weaknesses and explores prompting and benchmark-based remedies.

  • Zero- and few-shot performance: InstructGPT achieved state-of-the-art results using only 16 examples on four text-classification datasets.The result used an advanced prompting strategy.
  • Evaluation scope: Robustness research examines out-of-distribution instances, adversarial prompts, and adversarial inputs across numerous tasks.Evaluated tasks include semantic parsing, code generation, table question answering, multiple-choice question answering, and text-to-SQL generation.
  • Adversarial inputs: GLLMs such as GPT-3 and Codex showed better robustness than fine-tuned models on table question answering with adversarial inputs.The RobuT benchmark also found that GLLM-generated adversarial inputs can enhance adversarial robustness.
  • Cross-task robustness: Across 21 datasets covering nine natural-language-understanding tasks, GPT-3 and GPT-3.5 were robust on some tasks but experienced performance degradation on others.The evaluation used adversarial text transformations from TextFlint.

10 GLLMS AS EVALUATORS

GLLM-based evaluators address limitations of human and reference-based automatic evaluation by using instructions and examples to assess generated outputs. Studies report useful human-score correlations, but prompt sensitivity and positional bias remain important limitations.

  • Evaluation motivation: Human evaluation is time-consuming, expensive, difficult to scale, inconsistent, and not reproducible, while existing automatic metrics often require reference outputs.N-gram metrics also correlate poorly with human scores because they do not capture semantic meaning.
  • GLLM-based evaluation: GLLM-based metrics can correlate strongly with human scores even without reference outputs.Their in-context-learning and instruction-tuning abilities support evaluation that imitates human evaluator behavior.
  • Applications: GLLM evaluators have been studied for code generation, style transfer, summarization, dialogue, translation, story generation, paraphrasing, and text-to-image synthesis.The survey summarizes frameworks across these natural-language-generation tasks.
  • Task-specific findings: ChatGPT achieved good correlations with human judgments for text style transfer, with separate prompts producing the best results for fluency, content, and style.GEMBA uses GPT-based evaluation for translation and allows references to be optional.
  • Task-specific findings: For abstractive summarization, Likert scaling implemented as a multiple-choice question produced the best and most stable zero-shot evaluation results.The study compared Likert scaling with head-to-head comparisons.
  • Limitations: GLLM evaluation is prompt-sensitive, and positional bias can make candidate-response order significantly influence results.Multiple-evidence calibration and balanced-position calibration reduced bias and improved correlation with human judgments.

11 FUTURE RESEARCH DIRECTIONS

The survey identifies strong task performance alongside unresolved concerns about robustness, safety, cost, domain transfer, context length, contamination, hallucination, and multilingual performance. It points to improved prompting, detection, efficiency, domain adaptation, and evaluation as future priorities.

  • Overview: GLLMs achieve state-of-the-art results in some tasks, but their brittleness to out-of-distribution inputs and adversarial prompts remains a research challenge.The survey calls for broader robustness studies and improved methods.
  • 11.2 Red Teaming: ChatGPT and GPT-4 generated answers to more than 60% of harmful queries in one red-teaming study.The survey links this behavior partly to toxic, biased, and noisy pretraining text and calls for stronger safety alignment.
  • 11.3 State-Of-The-Art Results Across NLP Tasks: Advanced prompting enabled InstructGPT to achieve state-of-the-art results with 16 examples, while GPT-RE and other approaches extended state-of-the-art performance to additional tasks.The survey presents these developments as progress toward reducing task-specific labeling and fine-tuning.
  • 11.4 Robust Approaches to Detect GLLM Generated Text: Reliable detectors robust to paraphrasing, synonym replacement, and writing-style modification are needed because current approaches are often unsatisfactory and attack-sensitive.The stated goal is to reduce misuse of GLLMs in illegal activities.
  • 11.5 Reduce Inference Costs: GPT-4 inference can cost a small business more than $21,000 monthly, motivating prompt adaptation, caching, batching, and LLM approximation.Batch prompting reduced Codex inference cost across ten datasets with little or no performance degradation.
  • 11.6 Enhance Performance in Domain-Specific NLP Tasks: BioBERT outperformed GPT-3 in few-shot biomedical tasks despite being 514 times smaller, while ChatGPT performed poorly on legal text classification datasets.Domain terminology, abbreviations, and complex language structures make specialized tasks challenging.
  • 11.7 Handle Limited Context Length: GLLM context lengths range from 2049 to 32,768 tokens, limiting processing of long documents and conversations.Selective context filters less useful input content to improve question answering and summarization on lengthy instances.
  • Evaluation concerns: Data contamination is difficult to assess because proprietary GLLMs generally do not disclose their training corpora.The concern is the presence of downstream test instances in pretraining data.

12 CONCLUSION

The survey reviews GPT-3 family large language models across multiple research dimensions and synthesizes findings from more than 350 recent papers. It also identifies future research directions and serves as a resource for academic and industry readers.

  • The survey covers more than 350 recent research papers on GPT-3 family large language models.
  • It examines GLLM performance across downstream tasks, specific domains, and multiple languages.
  • It reviews GLLMs’ data labelling, data augmentation, data generation, robustness, and evaluator capabilities.
  • The survey concludes with future research directions and aims to help academic and industry readers stay updated on GLLM research.
Loading 2310.12321v1…