Source-linked AI summary
TeleMath: A Benchmark for Large Language Models in Telecom Mathematical Problem Solving
Vincenzo Colle, Mohamed Sana, Nicola Piovesan, Antonio De Domenico, Fadhel Ayed, Merouane Debbah
TL;DR
LLM performance on telecom mathematical problem solving has not been comprehensively assessed, despite progress on general mathematical reasoning and other telecom tasks. TeleMath provides a 500-QnA benchmark built from SME seeds using synthetic-generation pipelines. Evaluations report that reasoning-oriented models perform better than general-purpose counterparts.
Problem
A comprehensive assessment of LLM mathematical problem-solving capabilities in telecommunications remains lacking, despite evaluations of broad mathematics and other telecom tasks.
Source: §I · p. 1Method
TeleMath expands SME-generated seed problems through a synthetic QnA framework and releases a 500-QnA benchmark for telecom mathematical problem solving.
Source: §I · p. 1Results
Reasoning-oriented models designed to think step by step and explore multiple approaches perform significantly better on TeleMath than general-purpose models.
Source: §I · p. 1, §VI · p. 6Takeaways & Limitations
TeleMath provides a publicly available resource for benchmarking and understanding LLM capabilities on telecommunications-specific mathematical problems.
Source: §VI · p. 6The 50-problem seed does not fully represent telecommunications, and the category distribution skews toward traditional telecommunications engineering (30.6%).
Source: §V · p. 5
Abstract
from arXiv · showhide
The increasing adoption of artificial intelligence in telecommunications has raised interest in the capability of Large Language Models (LLMs) to address domain-specific, mathematically intensive tasks. Although recent advancements have improved the performance of LLMs in general mathematical reasoning, their effectiveness within specialized domains, such as signal processing, network optimization, and performance analysis, remains largely unexplored. To address this gap, we introduce TeleMath, the first benchmark dataset specifically designed to evaluate LLM performance in solving mathematical problems with numerical solutions in the telecommunications domain. Comprising 500 question-answer (QnA) pairs, TeleMath covers a wide spectrum of topics in the telecommunications field. This paper outlines the proposed QnAs generation pipeline, starting from a selected seed of problems crafted by Subject Matter Experts. The evaluation of a wide range of open-source LLMs reveals that best performance on TeleMath is achieved by recent models explicitly designed for mathematical or logical reasoning. In contrast, general-purpose models, even those with a large number of parameters, often struggle with these challenges. We have released the dataset and the evaluation code to ease result reproducibility and support future research.
I. INTRODUCTION
Telecom mathematical problem solving by LLMs remains insufficiently evaluated despite advances in general mathematical reasoning and telecom-related applications. The paper introduces TeleMath and a synthetic QnA framework, then turns to the dataset’s sources and characteristics.
I. INTRODUCTION
- The paper identifies a gap in assessing LLMs’ mathematical problem-solving abilities in telecom, where tasks require numerical precision and domain-specific knowledge.Prior work covered broad mathematical benchmarks and telecom tasks such as protocol summarization, document classification, and general telecom knowledge, but not telecom mathematical problem solving.Source: §I · p. 1
- TeleMath is introduced as the first benchmark dataset for evaluating LLMs on mathematical problems in telecommunications.The paper also proposes a framework for generating synthetic QnAs from SME-created seed problems and releases 500 QnAs for assessment.Source: §I · p. 1
II. DATASET SOURCES AND CHARACTERISTICS
- The benchmark’s QnAs are distributed across telecommunications categories, and the dataset section introduces their sources and characteristics.Figure 1 depicts the category distribution; the supplied passage does not report category-specific counts.Source: §II · p. 1, Figure 1 · p. 2
A. TeleMath Seed Dataset
TeleMath’s seed dataset was created with SME-designed problems and step-by-step solutions, while its QnAs use numerical answers and standardized metadata.
A. TeleMath Seed Dataset
- 10 SMEs created 50 seed problems across telecommunications topics and difficulty levels, each with a solution and detailed step-by-step explanation.The explanations are intended to support generation of the TeleMath dataset.Source: A. TeleMath Seed Dataset · p. 2
B. Dataset Characteristics
- TeleMath uses numerical answers to support straightforward evaluation and reflect calculations used in telecommunications engineering decisions.Numerical answers avoid false negatives that can arise when equivalent symbolic formulas differ syntactically, and reflect calculations such as signal-to-noise plus interference ratios used in engineering decisions.Source: B. Dataset Characteristics · p. 2
- Each QnA is standardized in JSON with a numerical question and answer, category, tags, and a Basic or Advanced difficulty label.The answer is a decimal or integer, and its unit is stated or implied by context to avoid unit conversion during validation.Source: B. Dataset Characteristics · p. 2
III. TELEMATH DATASET CREATION
The dataset creation framework expands a small SME-generated seed set through two complementary synthetic-data pipelines. Its four blocks are problem decomposition, blueprint generation, synthetic data generation, and post-processing.
III. TELEMATH DATASET CREATION
- Generating hundreds of SME-authored problems with step-by-step solutions and formula derivations would be prohibitively expensive and time-consuming.Source: §III · p. 2
- Two complementary pipelines expand the seed set: one generates blueprints for numerical-answer problems, while the other handles equation-based solutions.Together, the pipelines are intended to generate large volumes of high-quality data scalably.Source: §III · p. 2
- The dataset creation framework has four blocks: problem decomposition, blueprint generation, synthetic data generation, and post-processing.Source: §III · p. 2
A. Problem Decomposition
The pipeline decomposes SME problems into self-contained subproblems, then derives reusable blueprints that generate diverse QnAs. Blueprint generation uses code for numerical answers and symbolic expressions for equation-based solutions.
A. Problem Decomposition
- Problem decomposition uses an LLM to split SME problems into granular subproblems, using the original problem, solution, and step-by-step explanation as context.The decomposition process derives subproblem solutions and reviews whether each is well-posed, unambiguous, and self-contained without relying on extra information from the original problem.Source: A. Problem Decomposition · p. 2
B. Blueprint Generation
- Blueprints are reusable executable code or symbolic formulas derived from subproblems and instantiated to generate diverse QnAs.Source: B. Blueprint Generation · p. 3
- For numerical-answer subproblems, code-driven blueprint generation produces executable Python code and discards it if its output on original parameter values mismatches the expected result.Source: B. Blueprint Generation · p. 3
- For equation-based subproblems, the symbolic math-driven path standardizes solution expressions in LaTeX, parses them with SymPy, and converts them into structured algebraic forms with free parameters.Source: B. Blueprint Generation · p. 3, Figure 2 · p. 3
C. Synthetic Data Generation
The pipeline generates new QnAs by proposing contextually plausible inputs for derived subproblems and instantiating their code-driven or symbolic blueprints. The examples show how an original problem can be represented as a subproblem and blueprint, then used to create a QnA.
C. Synthetic Data Generation
- New QnAs are generated by supplying plausible input parameters and executing the associated Python or Sympy blueprint.The original problem and step-by-step solution guide input proposals, keeping them realistic for the domain.Source: C. Synthetic Data Generation · pp. 3–4
- A wireless link at 3.5 GHz over 0.2 km has a reported free-space path loss of 89.35 dB; the problem asks for minimum transmit power to achieve 15 dB SNR.The setup specifies 1 MHz bandwidth, 2 dBd transmit and receive antenna gains, 5 dB noise figure, and a 23 dBm EIRP limit.Source: C. Synthetic Data Generation · p. 3
- The example links an original problem to a derived subproblem and its associated blueprint.The figure caption identifies the three elements shown.Source: Figure 3 · p. 3
- A symbolic-solution example frames QnA generation as a question about Shannon channel capacity using bandwidth and linear SNR.The displayed question asks for capacity from B and SNRlinear.Source: Figure 4 · p. 4, C. Synthetic Data Generation · p. 4
D. Post-Processing
Post-processing filters implausible generated inputs, edits questions to insert generated parameters, and validates semantic fidelity. It also assigns metadata and labels difficulty using a model-answering and solution-length heuristic.
D. Post-Processing
- Filtering removes outputs outside domain-specific bounds, such as implausible distances, frequencies, or power values.The thresholds are intended to retain realistic and physically plausible instances.Source: D. Post-Processing · p. 4
- Question editing inserts generated parameter names and values into the original question while preserving its text structure.A validator then checks structural equivalence and dimensional consistency against the original.Source: D. Post-Processing · p. 4
- Categories, tags, and difficulty are assigned heuristically; difficulty uses answer correctness and average solution-token count to separate Basic from Advanced problems.The token-count distribution is described as bimodal, with shorter solutions labeled Basic and longer solutions Advanced.Source: D. Post-Processing · p. 4
IV. PERFORMANCE EVALUATION
The evaluation samples 16 responses per model-question pair under shared decoding settings, with shorter generation limits for two Qwen math models. Performance is measured using single-attempt accuracy and majority-vote accuracy.
IV. PERFORMANCE EVALUATION
- Each model-question pair receives 16 independently generated responses at temperature 0.6 and top-p 0.90.The maximum generation length is 16,384 tokens except for the two Qwen2.5-Math-Instruct models, which use 4,096.Source: §IV · p. 4
- pass@1 measures whether a correct answer is generated in a single attempt, averaged over sampled answers.Source: §IV · p. 4
- cons@16 counts a problem as solved when the most frequent answer among 16 generations is correct.Source: §IV · p. 4
A. Models Performance Analysis
Qwen3-32B leads the TeleMath evaluation, while reasoning-focused models generally outperform non-reasoning models, including cases where a smaller reasoning model surpasses larger general-purpose models. Performance also varies by model scale and domain.
A. Models Performance Analysis
- Qwen3-4B surpasses substantially larger non-reasoning models, including models about 17 times its size on Information Theory.DeepSeek-R1-Distill-Llama-70B scores 53.21% pass@1 and 60.80% cons@16; Phi-4-reasoning+ scores 53.56% and 58.40%, respectively.Source: A. Models Performance Analysis · pp. 4–5
- 69.51% pass@1 and 76% cons@16 are Qwen3-32B’s average scores, the highest among evaluated models.Table I compares pass@1 and cons@16 accuracy across TeleMath categories.Source: A. Models Performance Analysis · p. 4, Table I · p. 5
- Among non-reasoning models, Qwen2.5-Math-72B-Instruct leads with 39.99% pass@1 and 46.48% cons@16.Llama-3.3-70B-Instruct follows at 36.23% pass@1 and 40.20% cons@16.Source: A. Models Performance Analysis · p. 5
- Larger models tend to score better within the Qwen3, Qwen2.5-Math, and Llama-3.x families, while Computer Networking and Telecommunications Engineering are challenging for many models.Electrical Engineering is a stronger domain; Llama-3.3-70B-Instruct scores 63.19% pass@1 and 66.67% cons@16 there, more than 25 percentage points above its overall average.Source: A. Models Performance Analysis · p. 5
V. LIMITATIONS
TeleMath’s coverage is limited by a 50-problem seed that does not represent the full telecommunications domain, and its use of Qwen models may bias scores in their favor.
V. LIMITATIONS
- The 50-problem seed does not fully represent telecommunications, and traditional telecommunications engineering accounts for 30.6% of the category distribution.The authors note that other important categories may consequently be under-represented.Source: §V · p. 5
- Using Qwen models for problem decomposition, blueprint generation, and review may favor those models in evaluation.Semantic validation is intended to limit this potential bias by checking generated QnAs against the original problems.Source: §V · p. 6
VI. CONCLUSIONS
TeleMath is a 500-pair benchmark for telecommunications mathematics, built from expert knowledge with synthetic generation. Evaluations found that models designed for mathematical or logical reasoning outperformed even larger general-purpose models.
VI. CONCLUSIONS
- Models designed for mathematical or logical reasoning outperform even larger general-purpose counterparts on TeleMath.The conclusion presents this as the key finding from evaluating diverse open-source LLMs.Source: §VI · p. 6
- TeleMath comprises 500 curated question-answer pairs based on expert knowledge and synthetic generation pipelines.Source: §VI · p. 6
- The publicly available dataset is intended to support benchmarking and research on LLM mathematical capabilities in telecommunications and similar technical fields.Source: §VI · p. 6