PaperlayerView exact revision on arXiv

Source-linked AI summary

6G-Bench: An Open Benchmark for Semantic Communication and Network-Level Reasoning with Foundation Models in AI-Native 6G Networks

Mohamed Amine Ferrag, Abderrahmane Lakas, Merouane Debbah

arXiv:2602.08675v1cs.NIcs.AI

TL;DR

Existing telecommunications benchmarks provide limited evidence about foundation models' semantic, network-level reasoning under evolving 6G conditions. This paper introduces and validates 6G-Bench, then evaluates 22 diverse models, finding substantial variation in deterministic reasoning performance across tasks and capability groups.

  • Problem

    Existing telecommunications benchmarks do not jointly evaluate semantic network-level decision-making under evolving conditions, uncertainty, worst-case reasoning, and multi-turn regret.

  • Method

    6G-Bench defines 30 standardization-aligned tasks, constructs 10,000 very-hard MCQs from 113,475 scenarios, and retains 3,722 after automated and expert validation.

  • Results

    Deterministic pass@1 ranges from 0.228 to 0.829 across 22 evaluated foundation models, while leading intent and policy reasoning accuracy clusters between 0.87 and 0.89.

  • Takeaways & Limitations

    6G-Bench provides a standardized basis for assessing foundation models as semantic reasoning layers above standardized network functions in AI-native 6G systems.

Abstract

from arXiv · show

This paper introduces 6G-Bench, an open benchmark for evaluating semantic communication and network-level reasoning in AI-native 6G networks. 6G-Bench defines a taxonomy of 30 decision-making tasks (T1--T30) extracted from ongoing 6G and AI-agent standardization activities in 3GPP, IETF, ETSI, ITU-T, and the O-RAN Alliance, and organizes them into five standardization-aligned capability categories. Starting from 113,475 scenarios, we generate a balanced pool of 10,000 very-hard multiple-choice questions using task-conditioned prompts that enforce multi-step quantitative reasoning under uncertainty and worst-case regret minimization over multi-turn horizons. After automated filtering and expert human validation, 3,722 questions are retained as a high-confidence evaluation set, while the full pool is released to support training and fine-tuning of 6G-specialized models. Using 6G-Bench, we evaluate 22 foundation models spanning dense and mixture-of-experts architectures, short- and long-context designs (up to 1M tokens), and both open-weight and proprietary systems. Across models, deterministic single-shot accuracy (pass@1) spans a wide range from 0.22 to 0.82, highlighting substantial variation in semantic reasoning capability. Leading models achieve intent and policy reasoning accuracy in the range 0.87--0.89, while selective robustness analysis on reasoning-intensive tasks shows pass@5 values ranging from 0.20 to 0.91. To support open science and reproducibility, we release the 6G-Bench dataset on GitHub: https://github.com/maferrag/6G-Bench

I. INTRODUCTION

6G-Bench addresses the lack of standardized evaluation for foundation-model reasoning across semantic, network-level, and AI-native 6G tasks. It constructs and validates a broad benchmark aligned with ongoing standardization activities, then evaluates diverse foundation models and discusses deployment implications.

  • Motivation: Existing LLM studies use task-specific datasets, simulators, and metrics that hinder fair comparisons across communication, sensing, security, and control tasks.Related benchmarks remain narrow or domain-specific rather than generalizing across these areas.
  • Benchmark design: 6G-Bench defines 30 standardization-derived decision-making tasks across five capability categories for semantic communication and network-level reasoning.The categories cover intent and policy reasoning, resource management, trust and security, agentic control, and distributed intelligence.
  • Benchmark design: 113,475 scenarios produce 10,000 balanced, very-hard MCQs through prompts enforcing multi-step quantitative reasoning, uncertainty handling, and worst-case regret minimization.The construction pipeline uses heterogeneous reasoning models, deduplication, and anti-heuristic constraints.
  • Validation: 3,722 questions remain after automated structural and logical checks plus expert review, while the remainder supports training and fine-tuning.Filtering targets semantic correctness, quantitative soundness, and uniqueness under worst-case reasoning.
  • Evaluation: The study evaluates 22 foundation models and combines deterministic pass@1 with selective pass@k and task- and group-level aggregation.The model suite spans open-weight and proprietary, dense and mixture-of-experts, multimodal, code-specialized, and long-context systems.
  • Implications: The paper identifies which model classes may serve as semantic reasoning layers above standardized network functions and where further innovation or domain-specific training is needed.The stated deployment criteria include reliability, accountability, and safety expectations for future 6G networks.

C. Agentic Evaluation in Telecommunication Service Scenarios

The paper positions 6G-Bench against telecommunications benchmarks by emphasizing standardized, network-level, uncertainty-aware reasoning over multi-turn operational scenarios. Its pipeline reconstructs semantic states and action spaces, instantiates task-conditioned MCQs, validates them, and evaluates models across heterogeneous 6G conditions.

  • Related benchmarks: TeleAgentBench evaluates realistic telecommunications service workflows, while LDOT targets conceptual, mathematical, and network-optimization reasoning in wireless communications.These benchmarks broaden evaluation beyond factual telecom knowledge but address different task scopes.
  • Related benchmarks: CovertComBench evaluates constrained wireless covert-communication problems, with stronger conceptual and code-generation performance than mathematical-derivation performance.Its questions cover MCQs, optimization derivations, and code generation under detection-theoretic constraints.
  • Positioning: Existing benchmarks remain limited because they do not jointly model evolving network conditions, semantic intent and policy, worst-case reasoning, and multi-turn regret under uncertainty.Knowledge, table, and agentic benchmarks each abstract away at least part of this network-level decision setting.
  • Positioning: 6G-Bench grounds its 30 tasks in intents, policies, slices, trust relationships, and agent coordination extracted from 3GPP, IETF, ETSI, ITU-T, and O-RAN activities.This alignment is intended to reflect emerging architectural and operational requirements rather than ad hoc problem formulations.
  • Positioning: The benchmark uses realistic multi-turn episodes, regret-minimizing oracle decisions, and task-conditioned MCQs to assess deployment-relevant decisions in safety-, trust-, and SLA-critical contexts.This design complements isolated solvers and knowledge engines by evaluating foundation models as reasoning layers above network functions.
  • Pipeline: The construction pipeline extracts capabilities, reconstructs semantic states and action spaces, generates task-conditioned instances, validates them automatically and with experts, and computes task- and category-level metrics.The source material includes multi-turn UAVBench episodes and structured dialogue turns for semantic-state reconstruction.

2) Semantic Network State:

6G-Bench represents each turn with a semantic network state that combines the active slice, network metrics, and evolving operating conditions. Its scenarios predominantly target degraded, performance-critical regimes rather than nominal operation.

  • The semantic network state includes the active slice, latency, jitter, packet loss, throughput, and normalized edge-compute load, which evolve dynamically during each episode.
  • Throughput spans constrained regimes below 50 Mbps (16.9%) to high-capacity regimes above 400 Mbps (25.5%).
  • 24.3% of instances have edge-compute load above 0.8, exposing decisions to near-saturated compute conditions.

3) Semantic Communication Model:

6G-Bench models semantic communication as meaning-bearing state and action abstractions exchanged across multi-turn network interactions. Agents choose actions under uncertain dynamics and are evaluated by worst-case future regret rather than instantaneous metrics.

  • Semantic communication exchanges meaning-bearing abstractions rather than raw symbols, combining intent, network state, policies, and operational state.
  • The semantic state captures mission or network intent, 6G network conditions, policy and service constraints, and UAV state such as battery, speed, and position.
  • Semantic actions include slice selection, compute offloading, autonomy adjustment, sensing requests, and agent-to-agent coordination.
  • The environment transition incorporates uncertainty from network variability, sensing noise, and environmental disturbances.
  • Agents optimize future consequences over a finite horizon, with costs covering mission degradation, safety risk, and SLA violations.
  • 6G-Bench converts these episodes into multiple-choice problems where the correct action minimizes worst-case regret using semantic communication artifacts and network-aware reasoning.

B. Task Taxonomy and Standardization-Aligned Categories

6G-Bench organizes 30 decision-making tasks into standardization-aligned capability categories focused on meaning-bearing abstractions, including intents, policies, slices, trust relationships, and agent coordination. The taxonomy evaluates reasoning under changing network conditions, constraints, and future trade-offs.

  • Task taxonomy: 6G-Bench defines 30 tasks organized into five capability-oriented categories derived from 3GPP, IETF, ETSI, ITU-T, and O-RAN standardization themes.
  • Task taxonomy: The taxonomy emphasizes decisions over intents, policies, slices, trust relationships, and agent coordination rather than isolated network functions.
  • Intent and policy reasoning: Intent reasoning covers feasibility, conflict resolution, drift detection, conservative continuation, and consistency across replanning.
  • Network slicing and resource management: Resource-management tasks evaluate slice selection and switching, fairness versus safety, compute placement, graceful degradation, SLA prediction, and autonomy downgrades.
  • Network slicing and resource management: The benchmark favors reasoning about anticipated future conditions and controlled degradation instead of reacting only to instantaneous metrics.
  • Network slicing and resource management: Additional tasks assess multi-agent slice negotiation and scheduler reconfiguration after network or AI scheduling changes.

3) Trust, Security & SLA Awareness Performance:

6G-Bench evaluates trust, security, and SLA-aware reasoning through decisions that balance performance against authorization, policy, regulatory, and service constraints. Its tasks cover access control, resource exposure, agent identity, third-party exposure, security response, and agentic coordination.

  • Motivation: Trust- and security-aware reasoning treats authorization, policy, regulatory constraints, and SLAs as conditions on acceptable network decisions.Latency or throughput optimization alone may be insufficient when trust assumptions or contractual guarantees are violated.
  • Trust and Security Tasks: T9 tests whether models reject edge offloading requests when trust, policy, or authorization constraints outweigh performance benefits.
  • Trust and Security Tasks: T16, T18, and T26 assess compute-resource exposure, AI-agent authorization and identity, and third-party data, API, or compute exposure.These decisions incorporate pricing, allocation, authentication, identity mapping, regulatory constraints, operator policies, and user consent.
  • Trust and Security Tasks: T30 evaluates automated detection, isolation, and recovery for adversarial incidents and non-adversarial faults while respecting policy and SLA constraints.
  • Agentic Coordination: Agentic-control tasks extend reasoning to knowledge exposure, cross-domain interoperability, agent communication, task offloading, lifecycle management, and training-as-a-service decisions.These tasks consider latency, security, trust, resource availability, privacy, quality of service, roaming, and SLA constraints.

5) Distributed Intelligence & Emerging 6G Use Case Performance:

6G-Bench covers distributed intelligence and emerging 6G uses where communication, sensing, learning, and control interact across agents and heterogeneous data sources. Its evaluation protocol computes overall, task-level, group-level, and pass@k metrics for these decisions.

  • Capability Scope: Distributed-intelligence scenarios combine communication, perception, and reasoning in integrated sensing, digital-twin, and public-safety applications.These settings require coordination across agents and data sources under latency, privacy, and resource constraints.
  • Capability Scope: T22–T25 evaluate federated learning orchestration, digital-twin control, sensing-enhanced decisioning, and disaster or public-safety coordination.The tasks cover privacy-preserving training, telemetry and command channels, sensing-stream fusion, slice allocation, and escalation policies.
  • Capability Scope: T29 evaluates immersive and AR resource prioritization by balancing throughput, latency, and fairness across network slices and edge resources.
  • Benchmark Coverage: Figure 4 summarizes benchmark coverage across five capability groups and the individual tasks T1–T30.The caption identifies subfigure a as category coverage and subfigure b as per-task MCQ counts.
  • Evaluation Protocol: The unified evaluation protocol uses task-conditioned prompts and reports overall, per-task, group-level, and pass@k performance metrics.Evaluation instances are multiple-choice questions derived from α3-Bench episodes and annotated with taxonomy task identifiers.
  • Evaluation Protocol: Pass@k samples k stochastic predictions for reasoning-intensive instances and counts success when any sample matches the ground truth.Pass@k is non-decreasing in k, with pass@1 equal to deterministic single-shot accuracy.

2) Task-Conditioned Prompt Construction:

Each evaluation prompt combines task metadata, an episode summary, an MCQ question, and four answer options. The model is instructed to return a canonical answer object under reproducible low-temperature decoding.

  • Prompt Contents: Each prompt includes TASK_ID, TASK_NAME, and the formal definition of the target task Tk.
  • Prompt Contents: The prompt also provides an episode summary describing the mission, network state, and dialogue context.
  • Prompt Contents: The MCQ stem and four answer options A–D complete the standardized task-conditioned prompt.
  • Output and Reproducibility: The model is instructed to act as an expert 6G network AI agent evaluator and respond with a JSON object containing an answer choice.The accepted answer value is one of A, B, C, or D.
  • Output and Reproducibility: Temperature is set to zero or very low, with an optional seed derived from episode and task identifiers to improve reproducibility.

3) Answer Extraction and Robust Parsing:

6G-Bench robustly parses model outputs before computing accuracy at overall, task, group, and reasoning-intensive pass@k levels. Its group aggregation gives equal weight to tasks rather than sample counts.

  • Answer Extraction: Responses are first parsed as JSON, checking the fields answer, choice, label, or option before applying fallback letter matching.
  • Answer Extraction: Outputs from which no valid A–D option can be recovered are marked invalid and counted as incorrect.
  • Evaluation Records: Each evaluated question records its episode, task, question, ground truth, parsed prediction, and metadata for later analysis.Metadata includes difficulty, source turn, and raw model output.
  • Accuracy Metrics: The protocol computes overall MCQ accuracy and per-task accuracy using the ground-truth labels and model predictions.
  • Group-Level Accuracy: Group-level accuracy is the unweighted mean of per-task accuracies within each predefined capability group.This makes each task contribute equally despite task-specific differences in sample size.
  • Robustness Metrics: Pass@k evaluates reasoning-intensive tasks using k stochastic samples, with larger k measuring robustness and completeness under stochastic decoding.Pass@1 is equivalent to deterministic single-shot accuracy.

IV. PERFORMANCE EVALUATION

6G-Bench evaluates foundation models using a rigorously generated and validated question pool designed to test difficult semantic decisions under quantitative uncertainty. The evaluation separates a broad training pool from a high-confidence test core.

  • Evaluation Protocol: The benchmark evaluates diverse foundation models across task- and group-level accuracy, with the protocol distinguishing deterministic performance from selective robustness analysis.The evaluation covers intent and policy reasoning, resource management, and other capability groups.
  • Question Generation: 10,000 multiple-choice questions are generated from 113,475 AI-native 6G scenarios using task-conditioned prompting.The scenarios cover intent, policies, telemetry, trust boundaries, and multi-agent interactions.
  • Question Generation: Each question requires at least four reasoning steps involving numeric extraction, quantitative projection, uncertainty or worst-case bounds, and future-regret comparison.The design targets semantic reasoning rather than surface heuristics.
  • Question Generation: 10,000 questions are balanced across tasks and capability groups, deduplicated, semantically filtered, and constructed so the correct option is the least-bad decision under worst-case evolution.The full pool is also intended for training and fine-tuning specialized 6G models.
  • Validation: 3,722 questions form the final evaluation set after automated filtering and expert review for structure, logic, semantic well-posedness, uniqueness, and worst-case correctness.The curated set is used exclusively for reported pass@1 and selective pass@k results.

3) Evaluated Foundation Models for 6G-Bench:

The evaluated model suite spans diverse architectures, parameter scales, context lengths, and openness profiles relevant to AI-native 6G deployment. Results show that capability varies by task and does not follow model scale alone.

  • Evaluated Models: 22 foundation models span code-specialized, general-purpose, multimodal, compact, dense, mixture-of-experts, open-weight, and proprietary systems.The suite is organized by release date and deployment-relevant model characteristics.
  • Evaluated Models: Context lengths range from 16k tokens to 1,048k tokens, while MoE models expose substantially fewer active parameters per token than total parameters.Examples include Llama 4 Maverick with 400B total and 17B active parameters.
  • Intent & Policy Reasoning: Intent and policy reasoning leaders cluster between 0.87 and 0.89 in AccG1, with qwen/qwen3-coder-next highest at AccG1 = 0.886.The leading models are closely grouped rather than separated by scale alone.
  • Intent & Policy Reasoning: Intent drift detection reaches AccT 3 = 0.973, whereas intent conflict resolution peaks at AccT 2 = 0.867, revealing distinct task difficulty profiles.Detecting semantic inconsistency is comparatively easier than resolving normative conflicts.
  • Intent & Policy Reasoning: Conservative continuation under uncertainty peaks at 0.916, while replanning consistency reaches AccT 15 = 0.938.These tasks expose differences in risk-aware and temporally consistent reasoning.
  • Network Slicing & Resource Management: Network slicing and resource management has a wider spread, with top AccG2 values between 0.76 and 0.81 and meta-llama/llama-4-maverick highest at AccG2 = 0.805.Task-level strengths differ across slice selection, compute placement, and coordinated resource adaptation.

D. Trust, Security & SLA Awareness Performance

Trust, security, agentic control, and distributed-intelligence tasks reveal uneven capability profiles across models. Safety-sensitive and multi-agent decisions remain differentiated even when some perception- and control-oriented tasks achieve high accuracy.

  • Trust, Security & SLA Awareness: Trust, security, and SLA awareness reaches a highest group score of AccG3 = 0.838 for deepseek/deepseek-v3.2.The group reflects risk-sensitive decisions involving policy compliance and security constraints.
  • Trust, Security & SLA Awareness: SLA violation prediction reaches 0.913, while network security detection and response reaches AccT 30 = 0.890.Trust-aware offloading peaks at AccT 9 = 0.835, showing asymmetric difficulty across trust dimensions.
  • AI-Native Networking & Agentic Control: AI-native networking and agentic control includes strong preemptive autonomy downgrade performance, with qwen/qwen3-coder-next and gpt-5.2-codex both reaching AccT 11 = 0.958.RAG augmentation and agent identity and onboarding are more discriminative tasks.
  • AI-Native Networking & Agentic Control: Llama-4-maverick leads network-knowledge RAG augmentation at AccT 17 = 0.877 and agent identity and onboarding at AccT 18 = 0.885.Most competing models remain around 0.80–0.83 on T17 and 0.77–0.83 on T18.
  • Distributed Intelligence & Emerging 6G Use Cases: Distributed intelligence and emerging 6G use cases peak at AccG5 = 0.806 for meta-llama/llama-4-maverick, with wider variation reflecting coordination demands.The group combines communication, sensing, learning, and control across heterogeneous entities.
  • Distributed Intelligence & Emerging 6G Use Cases: Sensing-enhanced ISAC decisioning reaches AccT 24 = 0.913, while device–network offload arbitration peaks at 0.780.Perception-centric decisions are comparatively stronger than learning orchestration and resource admission.

G. Robustness Analysis via pass@k Evaluation

The benchmark’s robustness analysis shows that multiple reasoning attempts can substantially improve correctness, but gains vary by task and model scale. Deterministic pass@1 remains the primary metric for safety- and SLA-critical deployment, while pass@k provides complementary evidence about recoverability and ambiguity resolution.

  • Overall robustness: pass@3 reaches 0.888 for deepseek-v3.2-exp and pass@5 reaches 0.916 for qwen/qwen3-coder-next, despite different pass@1 rankings.The divergence indicates that stochastic exploration can change model rankings and expose recovery ability beyond deterministic accuracy.
  • Task-level behavior: pass@5 reaches 0.912 for deepseek-v3.2-exp and ministral-14b on intent conflict resolution, compared with pass@1 values around 0.83–0.86.Alternative reasoning paths frequently converge to correct semantic reconciliation on T2.
  • Task-level behavior: pass@5 reaches 0.946 for llama-4-maverick and deepseek-v3.2-exp on agent interoperability and federation, while T30 reaches 0.972 for deepseek-v3.2-exp.These tasks show especially strong robustness under multiple attempts, although the reported implication is task-specific.
  • Model dependence: 0.448 remains unchanged across all k for llama-3.2-1b, showing that pass@k gains are not universal when semantic capacity is insufficient.The benchmark therefore treats robustness as complementary to, rather than a substitute for, adequate deterministic capability.
  • Model scale: 0.829 is the highest reported pass@1 among medium-scale models, exceeding the large-model range of 0.789–0.796.The results do not show monotonic improvement with model size; medium-scale models achieve the strongest overall deterministic accuracy in the cited comparison.
  • Deployment implications: Single-GPU local inference is substantially slower than API-hosted deployment, often by more than an order of magnitude across the evaluated services.The deployment discussion connects this throughput gap to clustered GPU provisioning at the network edge or core for latency-sensitive semantic services.
  • Benchmark scope: 6G-Bench evaluates 22 foundation models using 3,722 validated questions from a generated pool of 10,000 questions based on 113,475 scenarios.The benchmark combines standardized task coverage with expert-validated evaluation data for comparing model reasoning.
Loading 2602.08675v1…