PaperlayerView exact revision on arXiv

Source-linked AI summary

ORAN-Bench-13K: An Open Source Benchmark for Assessing LLMs in Open Radio Access Networks

Pranshav Gajjar, Vijay K. Shah

arXiv:2407.06245v2cs.NIcs.AIcs.CLcs.LG

TL;DR

O-RAN lacks comprehensive tools for evaluating domain-specific LLM performance, and human assessment can be effortful and susceptible to error and bias. The paper builds a multi-stage question benchmark and an O-RAN RAG pipeline; ORANSight performs best across the three difficulty categories, while the tested LLMs remain short of proficiency in O-RAN.

  • Problem

    The paper identifies a lack of comprehensive O-RAN evaluation tools and notes that human review of generated outputs is effortful and susceptible to errors and bias.

    Source: §I · p. 1

  • Method

    The paper creates a 13,952-question benchmark through a three-LLM generation, validation, and categorization process, and proposes ORANSight, a RAG pipeline for O-RAN specifications.

    Source: §I · pp. 1–2, §III · p. 3

  • Results

    ORANSight was best across all three difficulty categories, with average performance increases of 21.55% for Macro and 22.59% for Weighted scores.

    Source: §VI · p. 4

  • Takeaways & Limitations

    Although the evaluated models scored better than a random classifier, the authors conclude that they are not proficient in O-RAN and that specialized models are needed.

    Source: §VI · p. 4, §VII · p. 4

Abstract

from arXiv · show

Large Language Models (LLMs) can revolutionize how we deploy and operate Open Radio Access Networks (O-RAN) by enhancing network analytics, anomaly detection, and code generation and significantly increasing the efficiency and reliability of a plethora of O-RAN tasks. In this paper, we present ORAN-Bench-13K, the first comprehensive benchmark designed to evaluate the performance of Large Language Models (LLMs) within the context of O-RAN. Our benchmark consists of 13,952 meticulously curated multiple-choice questions generated from 116 O-RAN specification documents. We leverage a novel three-stage LLM framework, and the questions are categorized into three distinct difficulties to cover a wide spectrum of ORAN-related knowledge. We thoroughly evaluate the performance of several state-of-the-art LLMs, including Gemini, Chat-GPT, and Mistral. Additionally, we propose ORANSight, a Retrieval-Augmented Generation (RAG)-based pipeline that demonstrates superior performance on ORAN-Bench-13K compared to other tested closed-source models. Our findings indicate that current popular LLM models are not proficient in O-RAN, highlighting the need for specialized models. We observed a noticeable performance improvement when incorporating the RAG-based ORANSight pipeline, with a Macro Accuracy of 0.784 and a Weighted Accuracy of 0.776, which was on average 21.55% and 22.59% better than the other tested LLMs.

I. INTRODUCTION

The introduction identifies a lack of O-RAN evaluation tools and the difficulty of assessing specialized LLMs through human review alone. It presents ORAN-Bench-13K and ORANSight as evaluation and domain-knowledge approaches, with specification assistance as an additional use case.

I. INTRODUCTION

  • The paper identifies the absence of O-RAN benchmarking tools and the effort and potential bias involved in human evaluation as key motivation for a comprehensive assessment strategy.The introduction also notes that hallucinations make domain-specific assessment difficult without skilled evaluators.Source: §I · p. 1
  • ORAN-Bench-13K is a benchmark for O-RAN specification knowledge, built with three LLM instances and containing 13,952 questions across three difficulty levels.The paper also evaluates Chat-GPT, Gemini-1.5, and Mistral-7B.Source: §I · pp. 1–2
  • The paper proposes ORANSight, a RAG-based pipeline using a FAISS database and O-RAN specifications to study domain-knowledge retrieval and the possibility of an O-RAN-centric LLM.The introduction reports state-of-the-art performance for ORANSight and says Chat-GPT and Gemini-1.5 are not sufficiently proficient in O-RAN knowledge.Source: §I · p. 2
  • The authors also explore ORANSight as a specification assistant, motivated by the volume and complexity of O-RAN documents facing engineers, researchers, and industry professionals.They describe incorporating RAG as a way to induce domain knowledge.Source: §I · p. 2

II. DATA SOURCES

The benchmark draws on 116 O-RAN specification documents comprising 2.53 million words.

II. DATA SOURCES

  • 116 O-RAN specification documents average 21,778 words each and contain 2.53 million words in total.The documents were sourced from.Source: §II · p. 2

III. BENCHMARK GENERATION

The benchmark pipeline chunks O-RAN specifications and uses generator, validator, and categorizer LLMs to create and sort multiple-choice questions. The resulting database contains questions spanning easy, intermediate, and difficult levels.

III. BENCHMARK GENERATION

  • Specifications are split into 1,536-character chunks with 256-character overlap to retain context while avoiding chunks that are too large or too small for valid question generation.The chunk size was selected by experimenting with different context sizes.Source: §III · pp. 2–3
  • The generator creates one O-RAN multiple-choice question per chunk; a validator checks its meaning and answer coherence against the source context using extractive Q&A.Questions proceed when the validator's answer matches the generated answer; otherwise, they are rejected.Source: §III · pp. 2–3
  • The categorizer separates valid questions into easy basic-fact items, intermediate comprehension or application items, and difficult items requiring deep standards knowledge or synthesis.Intermediate questions may also involve moderate calculations or reasoning.Source: §III · p. 3
  • 1,139 easy, 9,570 intermediate, and 3,243 difficult questions make up the final benchmark database.The benchmark is made from the O-RAN specification documents and is available online with supplementary files.Source: §III · p. 3

IV. ORANSIGHT

ORANSight combines a BGE-Small embedding generator, a FAISS database, and Mistral-7B in an open-source RAG pipeline. At inference, it retrieves relevant specification chunks and adds them to the query context.

IV. ORANSIGHT

  • ORANSight uses BGE-Small-1.5 embeddings, a FAISS database, and Mistral-7B; the authors choose Mistral to keep the system open source and adaptable.They select the small BGE variant for computational efficiency and an open-source implementation.Source: §IV · p. 3
  • FAISS supports rapid approximate-nearest-neighbor searches over indexed vectors, enabling retrieval of relevant vectors and their original document chunks.The paper describes FAISS as optimized for memory use and computational speed with large-scale, high-dimensional embeddings.Source: §IV · p. 3
  • For inference, ORANSight retrieves the top five relevant document chunks and appends them to the user query before Mistral-7B generates a prediction.For database creation, document chunks are split at 1,024 characters with 256-character overlap and embedded as 384-dimensional vectors.Source: §IV · p. 3

V. TESTED LLMS

The tested models include ChatGPT-3.5, Gemini-1.5, and Mistral-7B, all instruction-tuned. The section also describes Mistral’s efficiency-oriented architecture and identifies ORANSight in the accuracy figure as Mistral with RAG.

  • ChatGPT-3.5, Gemini-1.5, and Mistral-7B are the three instruction-tuned models described.Source: §V · p. 3
  • Mistral-7B uses Sliding Window Attention, which is described as faster to compute than the vanilla variant, and a Rolling Buffer Cache that enhances computational efficiency.Source: §V · pp. 3–4
  • Figure 4 presents primary accuracy scores and uses ORANSight to denote Mistral with RAG.Source: Figure 4 · p. 4

VI. RESULTS

ORANSight, the Mistral-plus-RAG pipeline, outperformed the other tested models on the reported aggregate scores and led across all three difficulty categories. The experiments used Langchain, with separate integrations for GPT, Gemini, BGE, and Mistral.

  • The experiments used Langchain; GPT used the OpenAI extension, Gemini the Google-GenAI extension, and BGE and Mistral-7B Hugging Face.Source: §VI · p. 4
  • ORANSight was the best-performing model across Easy, Intermediate, and Difficult, with reported absolute increases of 14.85%, 19.26%, and 18.17% across those categories.Source: §VI · p. 4
  • ORANSight averaged increases of 21.55% for Macro Accuracy and 22.59% for Weighted Accuracy over the other tested models.Table I presents cumulative performance scores for the tested models; Weighted Accuracy uses the number of generated questions as weights, while Macro Accuracy is an unweighted average.Source: Table I · p. 4, §VI · p. 4

VII. CONCLUSIONS AND FUTURE WORK

The paper concludes that its 13,952-question benchmark shows popular tested LLMs are not proficient in O-RAN, while ORANSight improves their scores. Future work includes an open-source fine-tuning dataset and a coding-based benchmark.

  • The benchmark contains 13,952 multiple-choice questions divided into Easy, Intermediate, and Difficult groups of 1,139, 9,570, and 3,243 questions, respectively.Source: §VII · p. 4
  • The tested ChatGPT, Gemini-1.5, and Mistral-7B models scored above a random classifier but were judged not proficient in O-RAN.Source: §VII · p. 4
  • ORANSight’s average increases over the other tested models were 21.55% for Macro Accuracy and 22.59% for Weighted Accuracy.Source: §VII · p. 4
  • Future work proposes an open-source fine-tuning dataset for O-RAN-proficient LLMs and a coding-based benchmark for O-RAN code understanding.Source: §VII · p. 4

A. Easy

The supplied sample questions test knowledge of O-RAN alarm management and interfaces, and include a question about near-real-time control.

A. Easy

  • One sample asks for the Alarm List’s purpose and gives storing IMS alarm-event history as the answer.Source: A. Easy · p. 4
  • Another sample asks which protocol communicates between the O-DU and O-RU, with FH-eCPRI as the answer.Source: A. Easy · p. 5
  • A further sample asks which O-RAN component controls the radio access network in near real time.Source: A. Easy · p. 5

B. Intermediate

The sample questions cover O-RAN management operations, an energy-saving policy object, and synchronization redundancy. Their supplied answers identify the listed management actions, O-RU energy-saving controls, and failover as the relevant functions.

B. Intermediate

  • The Service Management and Orchestration Framework can configure IP addressing, manage software updates, and add, remove, or modify network functions.The supplied answer selects “All of the above.”Source: B. Intermediate · p. 5
  • The NESPolicy Information Object Class provides policies for enabling or disabling O-RU energy-saving features.This is the supplied answer to the question about the object's purpose.Source: B. Intermediate · p. 5
  • Redundant sync flows from two different grandmasters in a ladder topology support failover and extended synchronization outages.The supplied answer selects failover and extended sync outages.Source: B. Intermediate · p. 5

C. Difficult

The difficult sample questions test synchronization status, a PSCell uplink counter, and test-tool requirements. Their supplied answers identify the current master clock, minimum UE counts at 100-millisecond intervals, and the RF analyzer as not required.

C. Difficult

  • “I USE” indicates that the device is synchronized to the current master clock.This is the supplied answer for the sync-source-status parameter in the ptp-status container.Source: C. Difficult · p. 5
  • The OR.CellUA.MinPscellUeUl counter tracks the minimum number of UEs connected to a PSCell in the uplink every 100 milliseconds.This is the supplied answer to the counter's purpose question.Source: C. Difficult · p. 5
  • A radio frequency analyzer is not required to test removal of an SN-terminated split bearer initiated by the MN (eNB).The supplied answer selects the RF analyzer as the unnecessary tool.Source: C. Difficult · p. 5

X. APPENDIC: B

The appendix includes generated ORANSight examples as a specification assistant, covering interface explanations, virtualization, and RIC API implementation guidance.

X. APPENDIC: B

  • ORANSight's sample text is presented as a possible use case for a specification assistant.The examples include generated explanations and implementation guidance.Source: §X · p. 5
  • The generated explanation describes fronthaul as connecting the O-RU and O-DU to carry user data and control information.It contrasts this with midhaul between the O-CU and O-DU.Source: §X · p. 5
  • The generated explanation describes midhaul as connecting the O-CU and O-DU, carrying user data and control information, and forming part of the WG5 Midhaul transport architecture.In D-RAN and RAN architectures without an HLS, these interfaces are typically considered part of the backhaul network.Source: §X · p. 5
  • The virtualization response describes running network functions such as O-RU and O-DU on software rather than dedicated hardware.It links resource sharing and easier function updates or replacement to flexibility, scalability, and cost-effectiveness.Source: §X · pp. 5–6
  • The RIC API guide covers architecture and specifications, Near-RT and Non-RT API implementation, integration, testing, and deployment.It describes Near-RT control via fine-grained data collection and actions over E2, and Non-RT functions including AI/ML workflows and policy-based guidance.Source: §X · p. 6
Loading 2407.06245v2…