Source-linked AI summary
TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge
Ali Maatouk, Fadhel Ayed, Nicola Piovesan, Antonio De Domenico, Merouane Debbah, Zhi-Quan Luo
TL;DR
Telecommunications lacked a benchmark for evaluating LLM knowledge in the domain. The paper introduces TeleQnA, a 10,000-question dataset built with automated generation and human input, and finds that models perform better on general telecom questions than complex standards questions, while competing with active professionals.
Problem
The paper identifies a lack of benchmark datasets designed to evaluate LLMs’ proficiency in telecommunications.
Source: §I · p. 1Method
TeleQnA comprises 10,000 questions and answers from diverse telecom sources, generated through an automated framework involving two LLMs and human input at different stages.
Source: §I · p. 1Results
GPT-4 averaged 74% accuracy versus GPT-3.5’s 67%; models performed well on general telecom inquiries but struggled with complex standards questions, and context improved performance.
Source: §I · p. 1, B. Performance Benchmarks · p. 5Takeaways & Limitations
The findings emphasize the need for specialized telecom foundation models and report that LLMs can compete with active professionals in telecom knowledge.
Source: §I · p. 1, A. Conclusions · p. 6Question retention depends on human judgment, introducing subjectivity when questions have multiple valid options with varying degrees of correctness.
Source: B. Limitations · p. 6
Abstract
from arXiv · showhide
We introduce TeleQnA, the first benchmark dataset designed to evaluate the knowledge of Large Language Models (LLMs) in telecommunications. Comprising 10,000 questions and answers, this dataset draws from diverse sources, including standards and research articles. This paper outlines the automated question generation framework responsible for creating this dataset, along with how human input was integrated at various stages to ensure the quality of the questions. Afterwards, using the provided dataset, an evaluation is conducted to assess the capabilities of LLMs, including GPT-3.5 and GPT-4. The results highlight that these models struggle with complex standards related questions but exhibit proficiency in addressing general telecom-related inquiries. Additionally, our results showcase how incorporating telecom knowledge context significantly enhances their performance, thus shedding light on the need for a specialized telecom foundation model. Finally, the dataset is shared with active telecom professionals, whose performance is subsequently benchmarked against that of the LLMs. The findings illustrate that LLMs can rival the performance of active professionals in telecom knowledge, thanks to their capacity to process vast amounts of information, underscoring the potential of LLMs within this domain. The dataset has been made publicly accessible on GitHub.
I. INTRODUCTION
The paper addresses the lack of a telecom benchmark with TeleQnA, a dataset built from diverse telecom sources using automated question generation and human input. Its evaluation finds that LLMs handle general telecom inquiries better than complex standards questions, while also comparing them with active professionals.
I. INTRODUCTION
- TeleQnA fills the notable absence of a benchmark designed to evaluate LLMs’ proficiency in telecommunications.The paper introduces it as the first benchmark dataset tailored specifically to telecom knowledge.Source: §I · p. 1
- The dataset draws on standards and research articles, while its automated QnA generation uses two communicating LLMs and human input at different stages.The paper says human input helps ensure question quality and supports scalability across multidisciplinary topics.Source: §I · p. 1
- LLMs can compete with active telecom professionals in telecom knowledge, which the authors attribute to their ability to process extensive information.Source: §I · p. 1
II. DATASET SOURCES
- The source collection spans around 25,000 pages and approximately 6 million words across research, standards, and lexicon categories.Figure 1 presents the QnA dataset distribution among categories of collected source materials.Source: §II · p. 2, Figure 1 · p. 2
A. Standards Documents
The dataset draws on technical standards, diverse research materials, and a telecom terminology lexicon. Its source-selection choices aim to cover technical implementation, varied research, and related disciplines.
A. Standards Documents
- Technical specifications were uniformly sampled across standardization bodies, while standards overviews include summaries, reviews, and white papers.The paper describes standards documents as a way to capture technical facets and present-day implementations of telecom technologies.Source: A. Standards Documents · p. 2
B. Research Material
- Research sources span technical publications and reviews, with selection principles covering outlet diversity, citation relevance, bias minimization, and multidisciplinary topics.Sources include research articles, technical open-access books, reviews, and surveys; professionals also identified pivotal contributions.Source: B. Research Material · pp. 2–3
C. Telecom Lexicon
- The telecom lexicon was compiled as a dictionary of technical terminology found in the collected standards documents and research papers.The authors use this source to evaluate LLMs’ understanding of telecommunications terminology.Source: C. Telecom Lexicon · p. 3
III. DATASET CHARACTERISTICS
TeleQnA uses multiple-choice questions in a standardized JSON format, and the paper motivates automated generation as a way to address the demands of building comprehensive telecom coverage.
A. Question Type
- Multiple-choice questions are intended to assess accuracy, decision-making when multiple choices may be correct, and discrimination among plausible options.Source: A. Question Type · p. 3
B. Dataset Format
- Each question uses five JSON fields: question, options, answer, explanation, and category.The answer field identifies the correct option, and the category labels the source category.Source: B. Dataset Format · p. 3
IV. DATASET CREATION
- Because telecom expertise and challenging questions across many subdomains make human-only generation and crowdsourcing impractical, the authors use GPT-3.5’s API in an automated LLM framework.The paper presents Figure 3 as an overview of the generation process.Source: §IV · p. 3
A. Preprocessing
The pipeline prepares telecom documents, uses generator and validator LLM agents, and applies post-processing to remove questions that refer back to their source text and address other quality issues.
A. Preprocessing
- Preprocessing cleans and formats source documents and extracts acronyms with their corresponding definitions.Source: A. Preprocessing · p. 3
B. LLM Agents
- The generator creates self-contained multiple-choice questions from telecom-document segments and provides a rationale for the correct answer.It is prompted to support the answer by referencing sentences in the source document.Source: B. LLM Agents · p. 4
- A validator receives the question, options, and context without the answer or rationale, retaining a question only when it selects the generator’s designated correct option.The paper describes this as a way to mitigate potential misinformation or fabrication from the generator.Source: B. LLM Agents · p. 4
C. Post-Processing
- Post-processing filters questions that refer to source text with terms such as “figure” or “fig.”Source: C. Post-Processing · p. 4
- The authors shuffle answer options to counter GPT-3.5’s tendency to place the correct answer in option 1 and detect acronyms to map them to definitions.Source: C. Post-Processing · p. 4
D. Human Validation - First Stage
The paper combines expert review and clustering-based refinement to check question quality and reduce redundancy, then evaluates GPT-3.5’s accuracy across runs and batch sizes. Accuracy varies by about 1% across runs, while larger batches modestly reduce mean accuracy.
D. Human Validation - First Stage
- A telecom expert reviews each question, options, answer explanation, and source context to check correctness and ensure questions are self-contained.The review aims to discard improperly generated questions and questions tied to localized content, such as results specific to a proposed algorithm.Source: D. Human Validation - First Stage · p. 4
E. Refinement
- Ada v2 embeddings and K-Means cluster semantically similar questions before a second human review removes redundancy and performs a final quality check.The embeddings are computed from questions and their explanations.Source: E. Refinement · p. 4
V. PERFORMANCE EVALUATION
- Accuracy is the percentage of questions for which the respondent selects the option marked correct in the dataset.The evaluation assesses GPT-3.5 and GPT-4 and compares them with telecom professionals.Source: §V · p. 5
A. Batch Size and Variance Study
- About 1% standard deviation across runs supports a single run for assessing LLM proficiency, while larger batches modestly lower mean accuracy.The authors use batch size 5 in the remaining experiments to reduce the number of queries.Source: A. Batch Size and Variance Study · p. 5
B. Performance Benchmarks
GPT-4 outperforms GPT-3.5 across TeleQnA categories, though both models struggle with complex standards questions. On standards specifications, adding context yields a 22.5% relative accuracy gain for GPT-3.5, which the paper reports can then match GPT-4.
B. Performance Benchmarks
- GPT-4 averages 74% accuracy versus GPT-3.5’s 67%, and outperforms it by around 7% across categories.GPT-4 reaches approximately 87% in lexicon but only 64% in standards specifications.Source: B. Performance Benchmarks · p. 5
- LLMs and active telecom professionals perform comparably in general telecom knowledge and on intricate research and standards questions.Thirty professionals answered surveys with ten questions, two from each of the five categories, without search engines or external references.Source: B. Performance Benchmarks · p. 5
- The professional comparison is qualified by the breadth of telecom subdomains, which may include areas outside the professionals’ active work.The authors note that this breadth makes the questions challenging for professionals.Source: B. Performance Benchmarks · p. 6
C. Influence of Context
- 22.5% relative accuracy gain with context lets GPT-3.5 match GPT-4 on standards specifications.The experiment supplies batches of five question-option pairs with the three closest segments retrieved from standards pages.Source: C. Influence of Context · p. 6, Figure 5 · p. 6
A. Conclusions
- The authors conclude that TeleQnA evaluations show strengths on general telecom questions but challenges on complex inquiries, underscoring the need for specialized telecom foundation models.The paper introduces TeleQnA as an open-source benchmark tailored to telecommunications.Source: A. Conclusions · p. 6
B. Limitations
TeleQnA’s question-retention process still relies on human judgment, especially when multiple options are valid to differing degrees. The authors present the dataset as an initial step toward a broader industry collaboration on telecom-knowledge datasets.
B. Limitations
- Human judgment remains necessary to retain or discard generated questions, making judgments subjective when multiple options are valid but differ in correctness.The authors acknowledge that dataset generation is not flawless and continues to rely on human input.Source: B. Limitations · pp. 6–7
- For the HAP example, Option 2 is also an advantage, while Option 1 adds detail about the stratosphere’s low temperature and solar energy.The authors eliminated some such questions but retained others to test whether LLMs can distinguish options with differing degrees of correctness.Source: B. Limitations · p. 6
- Questions drawn from research papers can reflect authors’ subjective perspectives, and filtering them does not remove all human subjectivity.The authors say these factors mean dataset generation is not flawless despite efforts to optimize the process.Source: B. Limitations · pp. 6–7
C. Future Directions
- The authors frame TeleQnA as an initial step toward collaborative development of extensive telecom-knowledge datasets across the industry.They invite feedback on questions with nuanced correctness judgments and propose combining independently created datasets with TeleQnA.Source: C. Future Directions · p. 7
APPENDIX A SAMPLES OF TELEQNA
The appendix presents sample TeleQnA multiple-choice items spanning telecom terminology, general concepts, and standards-related knowledge. The examples include answers and explanations across these topics.
APPENDIX A SAMPLES OF TELEQNA
- Lexicon examples ask what EIRP stands for and what defines a heterogeneous network, with answers identifying the term and a multi-cell network, respectively.The heterogeneous-network answer describes a 3GPP access network with multiple cells having different characteristics.Source: Appendix A · p. 8
- General-knowledge samples cover free-space optical transmitters, software-defined radio flexibility, MIMO eigenmodes, and wireless-channel propagation parameters.The examples identify light diodes, software modification, min(nt, nr), and transmitter/receiver positions, carrier frequency, and scattering objects, respectively.Source: Appendix A · p. 8
- Standards-overview examples ask about Bluetooth’s frequency band and the wire pairs used in 10/100Base-T.The supplied answers are 2.4 GHz and pairs 2 and 3, respectively.Source: Appendix A · p. 8
- Standards-specification samples ask about a UE’s RRC buffer size and required inputs for the Nadrf MLModelManagement Delete operation.The supplied answers are 45 Kbytes and either a Storage Transaction Identifier or Unique ML Model identifier(s), respectively.Source: Appendix A · p. 9
APPENDIX B DETAILED BENCHMARKING RESULTS
The appendix reports GPT-3.5 and GPT-4 results from TeleQnA queries and includes a category-level accuracy table spanning the models and active professionals. The described results use randomly sampled five-question batches compared with correct answers.
APPENDIX B DETAILED BENCHMARKING RESULTS
- The appendix’s GPT-3.5 and GPT-4 results use randomly sampled batches of five TeleQnA questions.The sampled questions were provided to the model and compared with the actual correct answers.Source: Appendix B · p. 9
- Model responses were compared against the actual correct answers for the sampled questions.The passage describes the comparison procedure but supplies no numerical outcome in the provided excerpt.Source: Appendix B · p. 9
- Table II presents accuracy percentages for GPT-3.5, GPT-4, and active professionals across TeleQnA categories.The caption identifies the comparison scope but does not provide cell values.Source: Table II · p. 9