OTel: Building Domain-Specialized Telecom LLM Foundations for Intelligent Networks

• Dense, interdependent, continuously revised telecom standards require domain-specific training data, retrieval pipelines, and evaluations reflecting practitioners’ questions.The source identifies 3GPP specifications, O-RAN documents, GSMA reference documents, and IETF RFCs as examples of the complex corpora involved.Source: §1 · p. 1
• Existing telecom benchmarks cover public 5G datasets [6], broad standards understanding, mathematical reasoning [1], O-RAN [2], operational-log root cause analysis [8], and a shared leaderboard.The GSMA Open Telco AI Leaderboard brings these efforts together in a shared evaluation interface.Source: §1 · p. 1
• Without shared training infrastructure and reproducible baselines, practitioners must independently assemble data, build cleaning pipelines, and implement evaluations, slowing progress and weakening comparisons.The introduction identifies this as a gap between telecom evaluation and the resources needed to build benchmark-capable systems.Source: §1 · p. 1
• OTel releases four derived datasets and 30 full-parameter post-trained baselines across embedding, reranking, and language models, evaluated on held-out OTel partitions.The datasets draw on roughly 1.1 million raw training points, cleaned to 326,767 high-confidence examples; the sources include 3GPP, GSMA, O-RAN, RFCs, industry whitepapers, and telecom academic papers.Source: §1 · pp. 1–2
• OTel is a collaborative effort involving more than 100 domain experts from industry and academia, and its models had over 16 million downloads by May 2026.The project also received 157+ pieces of media coverage worldwide; the authors describe OTel as a reproducible starting point for community expansion.Source: §1 · pp. 1–2
• Embedding retrieval reaches 93.5% NDCG@10, reranking reaches 0.952 MRR@10, and language-model correctness reaches 88.2% with OTel fine-tuning.The paper reports that fine-tuning improves performance consistently across all three model families.Source: §1 · p. 2

Dataset collection and curation

• OTel’s corpus draws on six categories of public telecom documents, with over 100 experts contributing approximately 1.1 million raw training points.Contributors include Yale University, GSMA, NetoAI, Khalifa University, the University of Leeds, and UT Dallas.Source: §2 · p. 2
• A six-stage enrichment pipeline converts Yale QA-source triples into retrieval-ready supervision using document joins, passage generation, hard negatives, claim verification, context minimization, and contrastive formatting.Hard negatives are rescored by a reranker, and reference answers are decomposed into atomic claims checked against candidate context.Source: §2 · p. 2
• Shard evaluation exposed raw-corpus noise, with Acc@1 ranging from 71.1% to 3.9%; sequential filtering and deduplication yielded 326,767 retained training points.The Yale subset fell from 680K to 220,334 examples, while other contributors’ 420K examples fell to 106,433.Source: §2 · p. 2

Dataset formats

• OTel releases four Apache-2.0 task-specific datasets for contrastive retrieval, binary-labeled reranking, context-grounded instruction tuning, and abstention tuning.Raw source documents are not redistributed; asset-level documentation describes provenance, intended use, and source-specific constraints.Source: §2 · p. 2

Post-trained baselines

• The release includes 30 full-parameter post-trained baselines: 10 embedding models, 3 rerankers, and 17 language models spanning 22M to 32B parameters.Models start from public open-weight checkpoints and are trained on matching OTel datasets across AMD and NVIDIA GPU systems.Source: §2 · p. 2
• OTel fine-tuning raises embedding NDCG@10 by +9.6 to +60.2 percentage points, reranker MRR@10 by +0.535 to +0.598 absolute, and LLM correctness by +3.7 to +10.0 percentage points.All fine-tuned rerankers exceed 0.944 MRR@10, versus 0.35–0.42 for base models; OTel-LLM-27B-IT reaches 88.2% correctness.Source: §2 · p. 3

Evaluation protocol

• Evaluation uses held-out splits, task-specific metrics, and GPT-4o mini judging of context-grounded answers, reflecting OTel LLMs’ intended RAG use rather than unrestricted QA.Splits are 90/10 for LLMs and embeddings and 95/5 for rerankers, all with seed 42; results include bootstrap standard errors (n=10).Source: §2 · p. 3
• Collaborator-led retrieval stress tests found comparatively strong O-RAN retrieval but weaker academic-paper and GSMA PRD results, motivating targeted curation and harder negatives.These diagnostics covered multiple telecom subdomains but were not reported as formal benchmark tables because protocols were not standardized.Source: §2 · p. 3

Broader implications

• Telecom’s specialized, interdependent standards and severe consequences of deployment errors make it a consequential but difficult domain for AI research.Examples of risk include misconfigured networks, incorrect protocol interpretation, and unsafe automation.Source: §3 · p. 3

Democratizing telecom AI research

• By unifying aligned training data, post-trained baselines, and reproducible evaluation, OTel lowers infrastructure barriers and aims to provide telecom AI a shared foundation like MedQA, LegalBench [4], and FinBen [9].Researchers can start from OTel’s baselines and focus on specific improvements rather than independently assembling data access, cleaning, training, and evaluation infrastructure.Source: §3 · pp. 3–4

Industry uptake and real-world validation

• 16 million model downloads and 157+ media mentions within months of release provide early evidence of widespread interest in OTel.Coverage spans telecom trade press, general technology outlets, and regional business media; the authorship includes industry, academic, and infrastructure organizations.Source: §3 · p. 4

Responsible deployment and safety

• Context-grounded RAG, abstention-tuned variants, and OTel-Safety train models to answer from retrieved evidence or decline when context is insufficient or off-topic.The release also includes model and dataset cards, Croissant 1.1 metadata with Responsible AI fields, and provenance documentation for all 30 baselines and four datasets.Source: §3 · p. 4

Limitations and future directions

• OTel remains English-only and text-centric, relies on corpus-held-out rather than external evaluation, and aggregate scores obscure subdomain variation without first-class per-subdomain reporting.These limitations exclude multilingual users and document diagrams, tables, and structured data, and constrain assessment of cross-domain generalization.Source: §3 · p. 4
• Future releases should add multilingual and multimodal data, improve selected subsets and evaluation pools, standardize per-subdomain metrics, and validate on external benchmarks.The enrichment pipeline, training recipe, naming convention, and extensible held-out partitions are designed to support incremental community contributions.Source: §3 · p. 4