Source-linked AI summary
Learning to Reason in 13 Parameters
John X. Morris, Niloofar Mireshghallah, Mark Ibrahim, Saeed Mahloujifar
TL;DR
Existing low-rank adaptation methods leave open whether reasoning can be learned with updates below rank-1 adapter sizes. TinyLoRA scales low-rank updates to a single parameter and, with RL, achieves strong math-reasoning performance with byte-scale updates, though the findings are limited to math datasets.
Problem
Existing LoRA methods do not establish whether reasoning can be learned with updates below rank-1 adapter sizes.
Method
TinyLoRA uses shared trainable vectors to scale low-rank updates down to a single parameter.
Results
Across difficult math benchmarks, TinyLoRA with RL recovers strong performance with orders-of-magnitude smaller updates, including 91% GSM8K accuracy with 13 parameters and 87% of absolute improvement with 196 parameters.
Takeaways & Limitations
The results suggest that large models trained with RL can learn some math-reasoning tasks from extremely small, often sub-1KB updates, whereas SFT is less effective in this regime.
Takeaways & Limitations
The findings are limited to math datasets and may not generalize to science, creative writing, or other domains.
Abstract
from arXiv · showhide
Recent research has shown that language models can learn to \textit{reason}, often via reinforcement learning. Some work even trains low-rank parameterizations for reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91\% accuracy on GSM8K with only 13 trained parameters in bf16 (26 total bytes). We find this trend holds in general: we are able to recover 90\% of performance improvements while training $1000x$ fewer parameters across a suite of more difficult learning-to-reason benchmarks such as AIME, AMC, and MATH500. Notably, we are only able to achieve such strong performance with RL: models trained using SFT require $100-1000x$ larger updates to reach the same performance.
1 Introduction
TinyLoRA addresses the gap between standard LoRA's million-parameter updates and the much smaller updates that may suffice for reinforcement-learning-based reasoning. Across math reasoning tasks, RL supports strong performance with extremely few parameters, whereas SFT needs substantially larger updates.
- Standard LoRA reduces full-model updates to millions of adapter parameters, but rank-1 LoRA still requires at least 3M parameters for Llama3-8B.
- Fewer trained parameters reduce memory and communication costs, allow more adapters in memory, and can help retain information in the base model.
- RL can achieve comparable performance with much smaller updates than supervised learning, motivating an RL-based alternative to standard LoRA SFT.
- 95% accuracy on GSM8K is reached from a 76% baseline while training 10,000 parameters with Qwen-2.5-7B-Instruct.
- 91% GSM8K accuracy is achieved with TinyLoRA and GRPO using only 13 trained parameters, totaling 26 bytes.
- RL updates remain effective at tiny capacities: GRPO reaches 90% accuracy with fewer than 100 parameters, while similarly sized SFT updates barely outperform the base model.
- Across six difficult math benchmarks, 196 trained parameters retain 87% of Qwen-2.5-7B-Instruct's absolute performance improvement.
2 Related Work
Related work places TinyLoRA within efforts to reduce low-rank adaptation costs and study learning under data or parameter constraints. The cited work motivates comparing SFT and RL and examining how model scale affects the update size needed for performance.
- Parameter-efficient adaptation has expanded through methods including VeRA, VB-LoRA, LoRA-XS, UniLoRA, AdaLoRA, LoRA Drop, NoRA, WeightLoRA, and ShareLoRA.
- TinyLoRA targets update sizes below 10K parameters, extending beyond the 10K–10M scale common in other low-rank adaptation work.
- Prior studies report that SFT can use a thousand examples and, under certain settings, RL can learn from one example.
- As model size grows, fewer absolute parameters are needed to reach 95% of full-finetuning performance across the Qwen2.5-Instruct family.
- Related work compares SFT reasoning-token training with reinforcement-learning methods and studies LoRA scaling, learning capacity, and update sparsity.
- Concurrent work showed LoRA can match full finetuning at rank 1 but did not explore parameterizations below 3M parameters.
3 Update Capacity of SFT and RL
This section contrasts supervised finetuning and reinforcement learning as training signals with different information and capacity requirements. SFT exposes full demonstrations without reward annotations, whereas RL uses sampled continuations whose useful signal is concentrated in rewards.
- Supervised finetuning: SFT trains with next-token prediction on demonstrations sampled from a high-reward distribution.The model maps word-problem inputs x to solution outputs y.
- Reinforcement learning: Policy-gradient RL trains on fresh batches of continuations and their rewards, regenerated after each epoch.For each prompt, the observed data contains sampled outputs and reward values.
- Information content: RL exposes more sampled data than SFT, but binary rewards bound the relevant information to k bits per prompt.Most continuation content is noise without reward annotations.
- Signal separation: Reward annotations separate task-relevant features from irrelevant variation, while resampling accumulates correlated signal and cancels uncorrelated variation.This is the section’s proposed mechanism for RL’s efficient learning under limited capacity.
- Signal separation: SFT must treat all demonstration tokens as informative because demonstrations lack reward annotations distinguishing useful structure from irrelevant details.The hypothesis is that this requires absorbing more information than low-capacity RL.
4 Parameter-Efficient Finetuning with TinyLoRA
TinyLoRA extends parameter-efficient adaptation below conventional LoRA’s model-width-dependent scale. It combines low-rank spectral directions with a low-dimensional trainable projection and weight sharing, reaching parameter counts as low as one.
- TinyLoRA: TinyLoRA builds on LoRA and LoRA-XS to scale parameter-efficient updates down to a single trained parameter.The method targets the low-capacity regime that prior LoRA variants do not reach.
- LoRA and LoRA-XS: LoRA trains matrices A and B while freezing W, requiring O(dr) parameters per adapted module and typically millions overall.Applying LoRA across m modules and n layers yields O(nmdr) trainable parameters.
- LoRA and LoRA-XS: LoRA-XS reduces per-module trainable parameters from O(dr) to O(r^2) by learning a matrix that recombines dominant singular directions of W.The singular directions come from a truncated SVD and outperform randomly initialized LoRA in practice.
- Parameter sharing: Table 1 compares per-layer parameter usage using adapted-module count m, model width d, rank r, and TinyLoRA projection dimension u.Its stated comparison includes the possibility of reducing TinyLoRA to a single trainable parameter.
- Reducing trainable parameters: TinyLoRA replaces LoRA-XS’s r × r trainable matrix with a low-dimensional vector projected through a fixed random tensor.Each module trains only u parameters.
- Parameter sharing: Full weight tying shares one trainable vector across all adapted modules, reducing total parameters to u and potentially one.Without full tying, the count scales as O(nmu/ntie).
5 Experiments
The experiments evaluate parameter-efficient reasoning updates with SFT and GRPO across Qwen-2.5 and Llama-3 models and math benchmarks. They compare full finetuning, LoRA, LoRA-XS, and TinyLoRA across update sizes, while addressing inference constraints in vLLM.
- Experimental setup: The experiments target improved math reasoning while minimizing trained parameters on GSM8K and the harder MATH training set.GSM8K contains 7,500 math word problems.
- Experimental setup: The study compares supervised finetuning with GRPO reinforcement learning on instruction-tuned Llama-3 and Qwen-2.5 models.GRPO is the specified reinforcement-learning method.
- Experimental setup: Four baselines are evaluated—full finetuning, LoRA, LoRA-XS, and TinyLoRA—with ranks 1, 8, 64, and 256.TinyLoRA additionally varies the number of shared layers, and RL uses exact-match reward.
- Experimental setup: Learning rates are swept separately for each update size and selected using performance averaged over three random seeds.This controls for the known relationship between update size and effective learning rate.
- Evaluation: Evaluation includes GSM8K validation and seven additional math benchmarks for MATH experiments, including MATH500, AIME 24, and AMC23.The listed suite also includes Minerva, GAOKAO, OlympiadBench, and CollegeMath.
- Implementation: Because vLLM supports neither LoRA variants nor ranks below 4, inference uses merged weights while the final forward pass uses the true LoRA model.Truncated importance sampling mitigates the resulting numerical mismatch.
6 Results
TinyLoRA recovers most GSM8K gains with extremely small updates, and RL is substantially more parameter-efficient than SFT. Performance also depends on model size, backbone, sharing strategy, and update precision.
- 95% of the net GSM8K improvement is recovered with only 120 trained parameters, while one trained parameter yields a 4% performance increase.
- SFT vs. RL: At 13 parameters, RL reaches 91% GSM8K accuracy from a 76% baseline, whereas SFT reaches 83%; at 120 parameters, RL reaches 95% and SFT 84%.
- Training dynamics: Updates as small as 16 parameters receive reward during training, while larger parameter counts generally achieve higher rewards and longer responses.
- Scaling across backbones: Qwen reaches about 82% GSM8K accuracy with one parameter and 94.7% with 13, while LLaMA barely improves with fewer than five parameters.
- Scaling across backbones: Across model sizes, low-parameter adaptation works better for larger models, which reach 95% of full-finetuning performance with fewer absolute parameters.
- Parameter sharing: With all-layer sharing and float16 precision, Qwen reaches 70% GSM8K accuracy, an absolute improvement of over 10%.
7 Ablations
The ablations identify preferred TinyLoRA configurations: a small frozen rank and greater per-module expressivity are favored over extensive parameter sharing.
- The ablation sweeps frozen rank r, trainable projection dimension u, and weight tying factor ntie across Qwen and LLaMA 3B models.
- Frozen rank: Increasing frozen rank from 1 to 2 yields modest gains, but larger ranks degrade performance; the main experiments therefore use r = 2.
- u and ntie: For a fixed budget, performance generally improves with larger u and smaller ntie, favoring expressive per-module updates with less sharing.
8 Discussion
The discussion links TinyLoRA's promise to scaling trends and model-specific responsiveness, while emphasizing that the evidence is currently limited to math-style reasoning.
- Scaling trends: As model size grows, LoRA becomes more effective at smaller parameter counts, allowing larger models to be controlled with fewer parameters.
- The authors suggest that 13-parameter reasoning updates may mainly change generation style because task knowledge may already reside in the base model.
- Qwen and LLaMA: Qwen models generally require around 10x fewer updated parameters than LLaMA models for equivalent performance, possibly reflecting architectural or pretraining differences.
- Limitations: The findings are limited to math-style reasoning and may or may not generalize to science or creative writing.
9 Conclusion
TinyLoRA shows that effective task adaptation can require far fewer trainable parameters than previously thought, with updates often approaching full-finetuning performance in files smaller than 1KB. The results further indicate that this extreme compression depends on reinforcement learning, large models, and the task setting.
- TinyLoRA enables effective model tuning with far fewer parameters than previously thought necessary.
- Updates from TinyLoRA often approach full-finetuning performance while occupying less than 1KB.
- The results suggest that some models can learn certain tasks while absorbing very few bytes during reinforcement learning with verification.
- Strong performance with TinyLoRA is reported only for reinforcement learning on large models, not as effectively without RL or with smaller models.
A.1 Additional ablations
The ablations examine how to allocate a fixed trainable-parameter budget across tied layers and ranks. At matched update size, fewer unique parameters perform better, and Qwen2.5 substantially outperforms Llama at 100 parameters.
- The ablation varies the number of tied layers and rank r for Qwen2.5-3B-Instruct.
- Lower r_trainable yields higher performance at matched parameter counts, indicating that parameters are better spent on fewer unique parameters.
- At 100 parameters, Llama achieves 60% accuracy while Qwen averages 74% across three random seeds.