PaperlayerView exact revision on arXiv

Source-linked AI summary

SignalReasoner: Assessing the Upper Bound of 3B Models for Signal Mathematical Reasoning

Guozheng Sun

arXiv:2608.17301v1cs.AI

TL;DR

Signal-processing problems remain difficult for small general-purpose models, motivating domain-specific post-training for mathematical reasoning. This report compares direct RL with domain-aware CoT SFT followed by RL across GRPO, GSPO, and GMPO, achieving 39.12% accuracy versus 12.37% for the untrained base model.

  • Problem

    Specialized signal-processing problems remain difficult for sub-10B models because they require domain knowledge, symbolic reasoning, and interpretation of physical system equations.

  • Method

    The report compares direct verifiable-reward RL with wireless-domain CoT SFT followed by RL, benchmarking GRPO, GSPO, and GMPO.

  • Results

    39.12% overall accuracy is achieved by B-GMPO, compared with 12.37% for the untrained Base model.

  • Takeaways & Limitations

    SFT provides a useful domain-specific starting point, while GSPO and GMPO show the clearest gains over GRPO after SFT, with benefits depending on the RL objective.

  • Takeaways & Limitations

    Answer-only rewards can improve accuracy and efficiency while reducing the transparency and completeness of intermediate reasoning traces.

Abstract

from arXiv · show

Post-training with supervised chain-of-thought fine-tuning and reinforcement learning from verifiable rewards has substantially improved the mathematical reasoning capabilities of large language models (LLMs). However, their application to signal processing problems remains relatively under-explored. This report investigates reinforcement fine-tuning strategies for adapting Qwen2.5-3B-Base to graduate-level signal mathematical problems from WirelessMATHBench-XL, a comprehensive benchmark for mathematical reasoning in this domain. We examine two training paradigms: (i) direct reinforcement learning (RL) on WirelessMATHBench-XL with verifiable rewards; and (ii) supervised fine-tuning (SFT) on a distilled wireless-domain chain-of-thought corpus, followed by the same domain-specific RL stage. Across both paradigms, we benchmark Group Relative Policy Optimization (GRPO), Group Sequence Policy Optimization (GSPO), and Geometric-Mean Policy Optimization (GMPO). We aim to assess whether domain-aware CoT SFT serves as an effective initialization for subsequent RL, and whether GSPO or GMPO offer advantages in stability or accuracy over GRPO for signal reasoning tasks. Our best model achieves an overall accuracy of 39.12\%, representing a more than threefold improvement over the untrained Base model (12.37\%).

1 Introduction

The report studies reinforcement fine-tuning of Qwen2.5-3B for graduate-level signal-processing mathematics, comparing direct RL with domain-aware CoT SFT followed by RL. It evaluates GRPO, GSPO, and GMPO, reporting substantial accuracy gains and identifying GMPO after SFT as the strongest route.

  • Motivation: Signal-processing problems remain challenging because they combine symbolic equations, physical assumptions, and lengthy scientific contexts involving technologies such as MIMO, beamforming, RIS, and ISAC.The report motivates specialized post-training because general-purpose models struggle with specialized engineering problems.
  • Training paradigms: The study compares direct RL with Qwen2.5-3B-Base against Wireless-CoT-Mix SFT followed by RL on WirelessMATHBench-XL.Wireless-CoT-Mix contains 3,542 examples combining distilled wireless-domain CoT trajectories and NuminaMath-CoT samples.
  • RL algorithms: GRPO, GSPO, and GMPO are benchmarked across both fine-tuning routes for signal-processing mathematical reasoning.GRPO estimates advantages from multiple responses, GSPO optimizes full sequences, and GMPO stabilizes token-level updates through geometric-mean aggregation.
  • Contributions: 39% overall accuracy is achieved after reinforcement learning, compared with 12% for the untrained base model on WirelessMATHBench-XL.The report frames this as a more than threefold improvement in overall accuracy.
  • Contributions: GSPO and GMPO outperform GRPO in accuracy, convergence speed, and output token efficiency, with GMPO after SFT achieving the best overall result.The comparison is conducted across both training routes on WirelessMATHBench-XL.

2 Related Work

Related work progresses from preference-based alignment and chain-of-thought prompting toward verifiable-reward reinforcement learning and domain-specific mathematical post-training. WirelessMATHBench-XL provides the signal-processing benchmark context, while wireless-specific CoT distillation aligns structured reasoning with the target domain.

  • Preference and verifiable-reward alignment: RLHF uses human-preference reward models with PPO, whereas DPO removes the reward model but remains tied to pairwise preference data and excludes verifiable rewards.Verifiable rewards are deterministically computed from outputs, such as mathematical correctness or code execution results.
  • Mathematical reasoning post-training: STaR, GRPO, DeepSeek-R1, and DAPO advanced mathematical reasoning through self-training, rule-based rewards, and techniques for stabilizing long-chain-of-thought reinforcement learning.STaR bootstraps reasoning from correct rationales; DeepSeekMath introduced GRPO; DeepSeek-R1 demonstrated large-scale rule-based-reward RL; DAPO introduced four stabilization techniques.
  • Chain-of-thought reasoning: CoT prompting elicits step-by-step reasoning, while self-consistency and exemplar-free prompting further support structured reasoning in multi-step tasks.Kojima et al. showed that “Let’s think step by step” can trigger the effect without exemplars.
  • Domain-specific CoT post-training: Large CoT datasets enable SFT to instill structured reasoning, and this report distills wireless-specific trajectories from DeepSeek-V3 to align that prior with signal processing.The cited datasets include MetaMathQA, OpenMathInstruct-1, and NuminaMath-CoT.
  • Wireless-domain applications: Signal-processing and wireless tasks require mathematical precision in areas such as channel estimation, interference management, and beamforming, motivating domain-focused language-model research.Earlier wireless LLM work emphasized domain-specific knowledge extraction, technical-standard recall, and higher-level wireless tasks.
  • Signal-domain benchmarks: WirelessMATHBench introduced 587 expert-level signal-processing math problems, while WirelessMATHBench-XL expanded the benchmark to 4,027 problems for evaluating compact-model adaptation.The successor also showed that compact models fine-tuned with GRPO can approach much larger general-purpose models.

3 Method

The method compares two training routes and three verifiable-reward policy-optimization algorithms for signal reasoning. Route B uses domain-aware SFT initialization, while GRPO, GSPO, and GMPO differ in how they estimate advantages, aggregate importance ratios, and apply clipping.

  • Training routes: Route B applies SFT on Wireless-CoT-Mix to initialize signal-domain reasoning and general mathematical chain-of-thought behavior.The initialization targets step decomposition, symbolic manipulation, and final-answer formatting before signal-domain reinforcement learning.
  • GRPO: GRPO removes the critic and computes baseline-free advantages by normalizing rewards across multiple sampled responses.Its clipped token-level surrogate includes a KL penalty toward a frozen reference policy, while all response tokens share the same advantage.
  • GSPO: GSPO shifts importance sampling and clipping from individual tokens to complete sequences, matching the optimization unit to the response-level reward.Its sequence-level, length-normalized ratio is designed to mitigate training collapse caused by token-level importance-sampling noise.
  • GMPO: GMPO replaces GRPO’s arithmetic mean with a geometric mean, narrowing the objective range and reducing sensitivity to extreme token-level ratios.The geometric mean dampens individual-token outliers and requires more uniform sequence-wide consensus for a large update.
  • GMPO: GMPO clips token-level ratios in log-space before geometric aggregation, preserving gradients from unclipped tokens instead of suppressing an entire response.This differs from GSPO’s sequence-level clipping and is evaluated as a stability-oriented alternative to GRPO.

4 Task and Training Routes

Section 4 defines the WirelessMATHBench-XL task and contrasts two training routes for Qwen2.5-3B-Base: direct signal-domain RL versus domain-aware CoT SFT followed by RL. Both routes compare GRPO, GSPO, and GMPO, while Route B initializes wireless reasoning before verifiable-reward specialization.

  • Task: 4,027 graduate-level signal-processing problems from 970 technical papers form WirelessMATHBench-XL, spanning topics including MIMO, RIS, ISAC, beamforming, NOMA, and satellite communications.The benchmark includes multiple-choice and fill-in-the-blank questions, with 3,227 training examples and 800 test examples.
  • Task: 3,542 examples comprise Wireless-CoT-Mix, combining 1,546 wireless-domain question–reasoning pairs with 1,996 general-domain examples from NuminaMath-CoT.The wireless portion is distilled from DeepSeek-V3 using WirelessMATHBench-XL and WirelessMathBench.
  • Route A: Route A directly applies RL to WirelessMATHBench-XL from the raw Qwen2.5-3B-Base checkpoint, testing whether signal-domain verifiable rewards induce reasoning in a compact base model.This route compares GRPO, GSPO, and GMPO without SFT initialization.
  • Route B: Route B first fine-tunes Qwen2.5-3B-Base on Wireless-CoT-Mix and then applies RL to WirelessMATHBench-XL to assess domain-aware CoT initialization.The route asks whether domain-aware CoT SFT provides a better initialization for subsequent signal-domain RL.
  • Route B: The Route B SFT stage teaches wireless reasoning patterns, mathematical decomposition, and final-answer discipline while preserving general reasoning through NuminaMath-CoT; RL then specializes signal-processing performance.GRPO, GSPO, and GMPO are again compared at the RL stage.

5 Experimental Setup

The experiments compare Base and Base-SFT against six RL variants to test SFT initialization and RL algorithm choice. Training uses VeRL with specified one-epoch SFT and 1,000-step RL configurations, while rewards compare extracted boxed expressions with ground truth.

  • Compared methods: Eight methods comprise Base, Base-SFT, and six RL variants: A-GRPO, A-GSPO, A-GMPO, B-GRPO, B-GSPO, and B-GMPO.The design tests whether SFT improves the RL starting point and which RL algorithm is most effective under each starting point.
  • Training configuration: SFT trains Qwen2.5-3B-Base on Wireless-CoT-Mix for 1 epoch at 3 × 10−6 learning rate and batch size 36.The maximum response length is 768 tokens.
  • Training configuration: RL training uses group size 4, learning rate 1 × 10−6, batch size 18, and 1,000 steps on three NVIDIA RTX 4090 GPUs.RL generation uses a maximum response length of 768 tokens.
  • Reward function: The reward extracts all boxed expressions from each response and compares them element-wise with the ground-truth answer.The setup defines M as the number of ground-truth boxed expressions and m as the number of extracted expressions matching exactly.

6 Experimental Results

Reinforcement fine-tuning raises overall accuracy substantially over the Base model, while SFT alone provides a strong signal-specific initialization. GSPO and GMPO shorten reasoning traces dramatically, trading detailed auditability for inference efficiency.

  • Main results: Overall accuracy rises from 12.37% for the Base model to 34–39% after reinforcement fine-tuning.SFT alone reaches 28.75% overall accuracy on Wireless-CoT-Mix.
  • Main results: SFT reduces average output length from 907.41 to 604.45 tokens while injecting useful signal-specific reasoning behavior.The distilled wireless-domain CoT corpus improves performance before RL.
  • Output length: A-GSPO and A-GMPO produce 279.25 and 395.84 tokens, while B-GSPO and B-GMPO produce 243.59 and 258.87 tokens on average.These methods retain concise derivations but generate much shorter reasoning traces than GRPO variants.
  • Output length: GSPO and GMPO achieve competitive or superior accuracy with dramatically fewer tokens, improving inference efficiency but reducing auditability and error diagnosis.The shortening effect results from implicit penalties on long sequences and sacrifices detailed step-by-step derivations.
  • SFT optimization: SFT training loss decreases from roughly 0.75 to about 0.36, while validation loss falls from 1.01 to approximately 0.684 before plateauing and slightly increasing to around 0.71.The curves indicate stable optimization followed by a validation-loss plateau.
  • Route and algorithm comparisons: For GSPO and GMPO, SFT-initialized variants reach higher reward levels more rapidly than Base counterparts, whereas GRPO routes converge to similar final levels.SFT first establishes a wireless-aware reasoning prior that RL subsequently refines.

7 Conclusion · Appendix

The report evaluates reinforcement fine-tuning routes and policy-optimization algorithms for Qwen2.5-3B-Base on WirelessMATHBench-XL. SFT followed by RL delivers the strongest reported accuracy, while GSPO and GMPO improve early reward growth and output efficiency but do not establish more complete reasoning.

  • 7 Conclusion: 39.12% overall accuracy is achieved by B-GMPO after SFT followed by RL, versus 12.37% for the untrained Base model.SFT alone reaches 28.75% overall accuracy before subsequent RL.
  • 7 Conclusion: 37.75% and 39.12% are achieved by B-GSPO and B-GMPO, respectively, in the SFT-then-RL setting.The report compares direct RL with SFT followed by RL under GRPO, GSPO, and GMPO.
  • 7 Conclusion: Both SFT and subsequent RL contribute to performance improvements on WirelessMATHBench-XL.Overall accuracy rises from 12.37% for Base to 28.75% with SFT alone and 39.12% after subsequent RL with B-GMPO.
  • 7 Conclusion: GSPO and GMPO show the clearest gains over GRPO in the SFT-then-RL setting, with faster early reward growth.This comparison spans the three evaluated algorithms.
  • 7 Conclusion: 243.59 and 258.87 output tokens are averaged by B-GSPO and B-GMPO, compared with 600.10 for B-GRPO.The shorter responses improve output efficiency.
  • 7 Conclusion: Shorter responses should not be interpreted as evidence that the model generates more complete reasoning.The passage explicitly distinguishes output efficiency from reasoning completeness.

A Data and Model Preprocessing … A.3 Model Configuration

The preprocessing pipeline constructs a verified wireless-domain SFT corpus, supplements it with general mathematical reasoning data, and prepares RL data and model configurations for reliable training. It also addresses answer-format and termination-token mismatches that otherwise disrupt verification or generation.

  • A.1 SFT Data: 3,542 examples form Wireless-CoT-Mix, combining distilled wireless-domain and general-domain mathematical reasoning data for SFT.The corpus draws from WirelessMATHBench-XL, WirelessMathBench, and NuminaMath-CoT.
  • A.1 SFT Data: 1,613 WirelessMATHBench-XL training instances are sampled approximately halfway for SFT construction from the 3,227-example training split.WirelessMATHBench-XL contains 3,227 training and 800 test instances.
  • A.1 SFT Data: Each distilled candidate includes domain background, a question, a masked equation, and a ground-truth answer, with reasoning ending in a boxed final result.DeepSeek-V3 generates concise step-by-step reasoning designed to reach the provided answer.
  • A.1 SFT Data: 512 tokens limit generated reasoning, temperature is 1.0, and failed generations are retried up to five times before being discarded.These constraints apply after requiring every boxed expression to exactly match its corresponding ground-truth answer.
  • A.1 SFT Data: 1,546 verified question–reasoning pairs remain in the wireless SFT subset after answer verification and length filtering.Adding 1,996 NuminaMath-CoT examples yields the final 3,542-example corpus, formatted as two-turn conversations.
  • A.2 RL Data: Bare-letter MCQ ground-truth answers in both RL splits are wrapped in boxed format to prevent verifiable-reward matching failures.The original dataset stores MCQ answers as raw letters such as A, B, C, or D.
  • A.3 Model Configuration: Replacing Base-model generation and tokenizer configurations with Qwen2.5-3B-Instruct configurations resolves the end-of-sequence mismatch that causes repetitive generation during SFT.The mismatch is between <|endoftext|> and the SFT chat template’s <|im_end|> delimiter.

B Training Hyperparameters

All reinforcement-learning variants use a shared base training configuration and differ only in algorithm-specific parameters. These parameters specify distinct KL penalties and clipping ranges for GRPO, GSPO, and GMPO.

  • Shared configuration: All RL variants share group size, learning rate, batch size, and training steps, differing only in algorithm-specific parameters.These settings constitute the common base configuration across experiments.
  • GRPO: GRPO uses a KL loss coefficient of 0.01 and a clipping ratio of 0.2.These are GRPO-specific hyperparameters.
  • GSPO: GSPO uses clip_ratio_low = 3 × 10−4 and clip_ratio_high = 4 × 10−4, with no KL penalty.Its clipping range is described as tight and is paired with the absence of a KL penalty.
  • GMPO: GMPO uses a KL loss coefficient of 0.001 and a clipping ratio of 0.4.These are GMPO-specific hyperparameters.

C Extended Benchmark Comparison

The extended benchmark comparison places 3B SignalReasoner variants below the strongest external models but competitive with several smaller baselines. Among reinforcement-learning variants, B-GMPO achieves the highest accuracy, while B-GSPO uses the fewest output tokens.

  • External-model comparison: 57.87% overall accuracy makes GPT-5 the strongest listed external model, followed by DeepSeek-R1 at 57.37% and DeepSeek-V3.1 at 56.87%.These external models substantially outperform the 3B SignalReasoner variants.
  • Open-source baseline comparison: 39.12% overall accuracy makes B-GMPO the strongest 3B model, exceeding Qwen2.5-7B-Instruct (25.75%), Gemma 3 27B (31.50%), and all listed math-specialized baselines except Qwen2.5-Math-72B-Instruct (42.13%).B-GMPO also obtains the strongest SignalReasoner Fill-in score at 38.03%.
  • Efficiency trade-off: 243.59 tokens is B-GSPO’s shortest average response, alongside 37.75% overall accuracy; B-GMPO uses 258.87 tokens for the best overall accuracy.A-GRPO and B-GRPO generate substantially longer responses at 668.33 and 600.10 tokens, respectively.
Loading 2608.17301v1…