PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer to GPT-5: Advancements, Limitations, and Future Directions in Large Language Model Design

Harsidak Singh Banwait, Chander Prabha

arXiv:2610.34281v1cs.CLcs.LG

TL;DR

Thorough taxonomies of LLMs remain limited, despite growing work on their development and societal impact. This review constructs a reproducible metadata corpus and taxonomy through systematic search and curation. It describes scaling-related reasoning gains alongside diminishing performance gains at larger model and dataset scales.

  • Problem

    Thorough taxonomies of LLMs remain limited, while understanding their effects as social communication tools requires examining content production, moderation, and engagement.

    Source: §II · p. 2

  • Method

    The review builds a reproducible LLM metadata corpus and taxonomy through systematic searching, screening, standardized extraction, rule-based mapping, and dual validation.

    Source: §III · p. 2

  • Results

    The review reports that increasing dataset and model complexity accompanied improved reasoning, while performance gains diminish as models and datasets grow larger.

    Source: §II · p. 1, A. Technical and Computational… · p. 2

  • Takeaways & Limitations

    The paper aims to provide a repeatable chronology and curated metadata corpus for examining LLM development and its challenges.

    Source: §II · p. 1

    Quantization below 8 bits can significantly reduce model performance during deployment.

    Source: D. Training and Fine-Tuning Limitations · p. 4

Abstract

from arXiv · show

Large Language Models (LLMs) have revolutionized Artificial Intelligence (AI) by their extraordinary capabilities, like contextual understanding, few-shot/zero-shot learning, code generation, ability to reason and problem-solving. Recent studies have shown immense work on LLMs. Yet, there is limited work on the thorough taxonomy of LLMs. This study would showcase a detailed classification of LLMs, starting with the earlier models like BERT and leading to LLMs like GPT, Gemini, Claude and Grok. We would delve into topics like datasets used to train LLMs, computational power used during training, their hardware specifications, architecture and parameters. Further, we would also look at the limitations, computational challenges and open problems as well. This study would primarily focus on the timeline of LLMs, signifying their growth over the period. Findings reveal a clear pattern: models that optimize the ratio between architectural complexity and data quality exhibit stronger reasoning stability and better alignment, without proportionally increasing the computational load. This desired balance between scaling and sustainability is the defining characteristic of the current generation of AI.

I. INTRODUCTION

The introduction traces LLM development from the Transformer and scaling to varied architectures and training choices, while the literature review highlights the need to understand LLMs’ effects as social communication tools.

I. INTRODUCTION

  • The Transformer’s self-attention enabled large-scale pretraining, while parameter growth and vast unlabeled datasets were associated with improved performance and in-context learning [2].In-context learning allows tasks to be completed with few or no examples at inference without changing model parameters.Source: §I · p. 1
  • LLMs are classified by parameter size, architecture, and input modality; architecture, pretraining data, and fine-tuning also shape their real-world usefulness, flexibility, and efficiency [4].The introduction notes concerns about bias, misinformation, and AI-generated content entering later training corpora.Source: §I · p. 1

II. LITERATURE REVIEW

  • The review aims to provide a repeatable chronology and curated metadata corpus while discussing reliability, transparency, and challenges spanning hallucination, bias, interpretability, and ethical alignment.It describes increasing dataset and parameter scales alongside improved reasoning capabilities.Source: §II · p. 1
  • Research on LLMs as social communication tools should examine their content production, moderation, and user engagement before deployment, including effects on the wider information ecosystem.The passage also identifies the challenge of monitoring AI-generated content at large scale.Source: §II · p. 2

III. METHODOLOGY

The methodology builds a reproducible LLM metadata corpus and taxonomy through systematic searching, screening, extraction, rule-based mapping, and dual annotation.

III. METHODOLOGY

  • The review combines a structured literature and model-card search, explicit screening, standardized CSV metadata extraction, rule-based taxonomy mapping, and dual-annotation validation.The authors state that data, extraction scripts, and plotting code are publicly available.Source: §III · p. 2

A. Search Strategy

  • Searches covered vendor and model-card pages, scholarly repositories, and proceedings, targeting LLM topics and terms from 2017 through 2024.About 4,000 raw records were retrieved; the most relevant papers were analyzed from March 3 to November 2, 2025.Source: A. Search Strategy · p. 2
  • The search included topics such as in-context learning, RLHF, and LoRA, alongside terms including Transformer, LLM, GPT, BERT, and LLaMA.The sources included OpenAI, Meta, Hugging Face, arXiv, IEEE Xplore, ACL Anthology, major conference proceedings, and Google Scholar.Source: A. Search Strategy · p. 2

B. Selection Criteria

The selection criteria prioritize primary model sources with quantitative metadata or historical significance, and the taxonomy classifies models by modality, architecture, and parameter scale.

B. Selection Criteria

  • Eligible sources introduce or evaluate a model and report quantitative metadata, or document historically significant releases; tutorials, duplicates, and toy models are excluded unless historically significant.Screening proceeds through title and abstract review, full-text assessment, and metadata extraction.Source: B. Selection Criteria · p. 2

C. Taxonomy Construction

  • The taxonomy distinguishes text, vision-and-text multimodal, and audio inputs; encoder-only, decoder-only, and encoder-decoder architectures; and five parameter-size bands.The bands range from small models below 100M parameters to HXL models above 100B.Source: C. Taxonomy Construction · p. 2

IV. HISTORICAL TIMELINE

The historical account moves from statistical and recurrent models to Transformer-based and multimodal systems, while identifying persistent technical, ethical, and social limitations.

IV. HISTORICAL TIMELINE

  • Earlier statistical n-gram models had limited contextual windows, while RNNs and LSTMs contributed to language-model development but relied on stepwise computation.The passage identifies that computation as a barrier to deployment.Source: §IV · p. 2
  • The 2017 Transformer addressed the deployment barrier of stepwise recurrent computation, and GPT’s autoregressive design supports high-quality text generation [6].In-context learning, human-feedback alignment, and multimodal development are presented as complementary advances [8].Source: §IV · p. 2

V. CHALLENGES, LIMITATIONS, AND OPEN PROBLEMS

  • LLM limitations span knowledge representation, factual accuracy, the training-deployment pipeline, and ethical and social risks.These concerns qualify the technical progress described in the timeline.Source: §V · p. 2

A. Technical and Computational Challenges

LLMs pose computational, energy, and storage demands that challenge use on low-resource hardware and deployment. The section also notes that performance gains diminish as models and datasets grow, and identifies tables summarizing LLM architecture specifications.

A. Technical and Computational Challenges

  • LLMs require substantial computational power, powerful GPUs, substantial RAM, engineering work, and high energy use, making low-end hardware difficult to use.Source: A. Technical and Computational… · p. 2
  • GPT-3's 175-billion-parameter FP16 weight matrices require approximately 350 GB of storage, challenging deployment, replication, and large-scale testing.Source: A. Technical and Computational… · p. 2
  • As models and datasets grow, performance gains begin to diminish, a slowdown termed sub-scaling behaviour.Traditional scaling laws predict improvement from increasing model size and data quality, but more recent studies suggest those gains slow at larger scales.Source: A. Technical and Computational… · p. 2
  • Table I summarizes LLM architectures and their specifications, while Table II summarizes current multimodal LLM architectures.Source: Table I · p. 3, Table II · p. 3

B. Ethical, Societal, and Safety Issues

LLMs raise ethical and societal concerns involving biased outputs, privacy and data security, and potential misuse to generate misinformation and manipulate public opinion. The paper also notes concerns about the rapid proliferation of these models.

B. Ethical, Societal, and Safety Issues

C. Knowledge, Accuracy, and Generalisation Issues

The section describes weaknesses in the integrity and applicability of learned knowledge: fluent outputs can be contextually wrong, general-purpose models may lack depth in specific fields, and inaccurate results can persist despite accurate training and quality data.

C. Knowledge, Accuracy, and Generalisation Issues

  • LLMs can produce fluent, grammatically correct but contextually wrong outputs, undermining confidence in their reliability.Source: C. Knowledge, Accuracy, and… · p. 4
  • General-purpose training can leave models short of deep problem-solving for particular fields, potentially because field-specific data is limited; small datasets and minimal retraining may improve domain-relative performance.Source: C. Knowledge, Accuracy, and… · p. 4
  • Even accurate training techniques and quality training data do not prevent LLMs from producing inaccurate results.Source: C. Knowledge, Accuracy, and… · p. 4

D. Training and Fine-Tuning Limitations

LLMs remain difficult and costly to train and tune for specific tasks, including aligning outputs with human values and ethics. Black-box access constraints and accuracy losses under aggressive quantization add further limitations.

D. Training and Fine-Tuning Limitations

VI. OPEN PROBLEMS AND FUTURE RESEARCH GAPS

The section identifies unresolved challenges in multimodal integration, reliable self-correction, and adapting LLMs to rapidly changing social media. It frames these as future research needs.

VI. OPEN PROBLEMS AND FUTURE RESEARCH GAPS

  • Multimodal LLMs still face data explanation, cross-modal accuracy, and cross-domain reasoning challenges.The domains named are text, vision, and audio.Source: §VI · p. 4
  • LLM self-correction remains inconsistent and typically effective only on narrow tasks, leaving generalisable correction an open frontier.Source: §VI · p. 4
  • Rapidly changing social media poses adaptation challenges, with proposed research on content monitoring, AI-generated content detection, and ethical guidelines.Source: §VI · p. 4

VII. CONCLUSION

The study surveys LLM development and design factors, reports scaling-related gains in generalization and reasoning, and notes deployment, hardware, and societal concerns.

VII. CONCLUSION

  • The study traces LLMs from BERT and examines leading models including GPT, Gemini, and DeepSeek.Source: §VII · p. 5
  • The study reports that scaling LLMs improves generalization and reasoning, including emergent in-context learning, while training and deployment face rising data and computational demands.Source: §VII · p. 5
  • The conclusion calls for hardware-use optimization and future work on real-time multimodal reasoning and models consistent with human values.Source: §VII · p. 5
  • The study acknowledges LLM limitations including hallucinations and risks from unethical use.Source: §VII · p. 5

Related papers

Loading 2610.34281v1…