PaperlayerView exact revision on arXiv

Source-linked AI summary

Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective

Junnan Liu, Hongwei Liu, Linchen Xiao, Shudong Liu, Taolin Zhang, Zihan Ma, Songyang Zhang, Kai Chen

arXiv:2505.19815v1cs.CLcs.AI

TL;DR

The paper asks how reasoning trajectories contribute to LLM reasoning and proposes RaML, which models them as pseudo-gradient updates within a meta-learning training framework. Its analyses report correspondence between LLM reasoning and meta-learning, suggesting meta-learning principles as a perspective for understanding and improving LLMs.

  • Problem

    The role of reasoning trajectories in LLM reasoning remains insufficiently understood, despite their use in complex problem solving.

    Source: §1 · p. 1

  • Method

    RaML treats trajectories as pseudo-gradient parameter updates and models each question as a meta-learning task with trajectory adaptation in the inner loop.

    Source: §1 · p. 1, §2.3 · p. 5

  • Results

    The paper reports empirical evidence of correspondence between LLM reasoning and meta-learning, including analyses of training techniques, trajectories, tokens, and generalization.

    Source: §1 · p. 2, §6 · p. 14

  • Takeaways & Limitations

    The framework offers a way to interpret LLM reasoning and practical insights for improving models using established meta-learning techniques.

    Source: §6 · p. 14

    The experiments mainly focus on mathematical reasoning, while other reasoning tasks are included for generalization study.

    Source: §3.1 · p. 6

Abstract

from arXiv · show

We propose a novel framework for comprehending the reasoning capabilities of large language models (LLMs) through the perspective of meta-learning. By conceptualizing reasoning trajectories as pseudo-gradient descent updates to the LLM's parameters, we identify parallels between LLM reasoning and various meta-learning paradigms. We formalize the training process for reasoning tasks as a meta-learning setup, with each question treated as an individual task, and reasoning trajectories serving as the inner loop optimization for adapting model parameters. Once trained on a diverse set of questions, the LLM develops fundamental reasoning capabilities that can generalize to previously unseen questions. Extensive empirical evaluations substantiate the strong connection between LLM reasoning and meta-learning, exploring several issues of significant interest from a meta-learning standpoint. Our work not only enhances the understanding of LLM reasoning but also provides practical insights for improving these models through established meta-learning techniques.

1. Introduction

The paper addresses the need to understand how reasoning trajectories relate to LLM reasoning and proposes RaML, a meta-learning perspective that treats trajectories as pseudo-gradient updates. It uses this perspective to analyze reasoning training and reports empirical correspondence between LLM reasoning and meta-learning principles.

1. Introduction

  • RaML treats reasoning trajectories as pseudo-gradient descent updates to model parameters, framing LLM reasoning through meta-learning.The paper presents this as a methodology for understanding and optimizing LLM reasoning.Source: §1 · pp. 1–2
  • Each question is a distinct meta-learning task, with reasoning trajectories as inner-loop parameter adaptation and answers as the query set.Training aims to identify an effective meta-initialization for efficient adaptation and accurate responses.Source: §1 · pp. 1–2
  • The paper reports empirical evidence of strong correspondence between LLM reasoning and meta-learning principles, alongside analysis of SFT, RL, trajectory counts, tokens, and generalization.It also explores potential improvement pathways and preliminary confirmatory experiments.Source: §1 · p. 2

2. RaML: Interpreting LLM Reasoning as Meta-Learning

RaML interprets reasoning as parameter adaptation through trajectories, then models reasoning-task training as meta-learning. It connects training techniques to this framework by considering their trajectory-generation and optimization processes.

2.2. Reasoning Trajectories as Parameter Update

  • Reasoning trajectories are modeled as multi-step gradient descent on LLM parameters, rather than as algorithms that merely solve explicit numerical optimization problems.The paper describes each question as an optimization task and the trajectory as a parameter-update mechanism before the answer.Source: §2.2 · p. 3
  • The analysis assumes a simplified transformer with one self-attention layer and a two-layer feed-forward network, omitting normalization and other components.The pseudo-gradient formulation is developed for this model setting.Source: §2.2 · p. 3

2.3. A Meta-Learning Perspective on LLM Reasoning

  • RaML casts each question as an independent task, with trajectory-conditioned parameter updates in the inner loop and answer optimization in the outer loop.The trajectory acts as a latent support set and the answer as the query set; inner-loop updates are implicit rather than direct weight updates.Source: §2.3 · pp. 4–5
  • RaML resembles MAML, while its LLM-generated reasoning trajectories also make the inner loop resemble L2O, with the model generating a question-specific optimization path.The final answer is optimized in the outer loop using a second-order gradient.Source: §2.3 · pp. 4–5

2.4. Instantiation of Training Techniques within RaML

  • RaML distinguishes training techniques by inner-loop trajectory optimization: off-policy methods use manually collected or synthesized trajectories, while on-policy methods generate them from the model distribution.Off-policy methods follow trajectories from an optimal meta-optimizer; on-policy RL explores its own optimization path, with greater freedom and potentially greater optimization outcomes.Source: §2.4 · pp. 5–6

3. Empirical Analysis on LLM Reasoning from Meta-Learning Perspective

The experiments examine LLM reasoning through meta-learning, comparing training strategies, trajectory properties, and generalization across reasoning domains. They report gains from trajectory-based training and connect trajectory length and token roles to inner-loop optimization.

3.1. Experiment Setup

  • The study focuses mainly on mathematical reasoning, using Qwen2.5-7B-Base trained from scratch with SFT or (Zero-)GRPO and evaluating on math benchmarks plus GPQA and LiveCodeBench.Training data came from Open Reasoner Zero and were filtered from approximately 57k questions to 39k.Source: §3.1 · p. 6

3.2. Inner Loop Optimization v.s.Reasoning Trajectory Source

  • GRPO-trained models did not consistently outperform SFT-trained models on the same base LLM; DeepSeek-R1-Zero scored 71.0 versus DeepSeek-V3's 39.8 on AIME24.The paper interprets SFT trajectories from human annotations or advanced models as oracle guidance that supports stable inner-loop optimization, and reports substantial enhancements when RL follows SFT.Source: §3.2 · pp. 7–8
  • The paper frames SFT as stable inner-loop optimization from oracle-like trajectories and RL as a means of exploring alternative optimization paths.It proposes SFT initialization followed by RL exploration as a way to combine these roles.Source: §3.2 · pp. 7–8

3.3. Inner Loop Optimization Steps v.s.Reasoning Trajectory Tokens

  • Models trained with longer reasoning trajectories consistently outperformed those trained with shorter trajectories, consistent with longer trajectories representing more inner-loop update steps.The paper relates this pattern to findings that extended inner-loop updates improve performance and to the strong performance of long-CoT models such as DeepSeek-R1 [27].Source: §3.3 · pp. 8–9
  • Reflection tokens such as “Wait” and “Alternatively” mark significant objective changes, while end-of-thinking delimiters are associated with quicker convergence that may not reach the optimum.The authors propose that reflection tokens help escape saddle points and that delimiters accelerate convergence; they note QwQ does not fully adapt to no-thinking mode and cite additional Qwen3 experiments.Source: §3.3 · pp. 8–9

3.4. Task Generalization v.s.Reasoning Generalization

  • SFT and GRPO models improved on AIME24 and LiveMathBench-Hard, while all trained models outperformed the base model on GPQA and LiveCodeBench.The evaluation treats the first pair as within-domain and the latter pair as cross-domain generalization.Source: §3.4 · p. 9

3.5. More Discussions about LLM Reasoning

  • RaML is also used to interpret recent developments in LLM reasoning as improvements that can be understood through a meta-learning lens.The paper points to further discussion of these developments in Appendix D.Source: §3.5 · p. 10

4. Advancing LLM Reasoning Drawing Inspiration from Meta-Learning

The section tests meta-learning-inspired changes to reasoning training and decoding, finding benefits from multiple trajectories per question and shorter reasoning paths. It also outlines open questions and training directions for applying meta-learning ideas to LLM reasoning.

4.1. Manipulating Training Reasoning Trajectories per Question

  • Increasing synthetic reasoning trajectories from 1 to 32 improves Qwen2.5-7B-Base performance and reasoning stability across benchmarks.Training frequency per question was held equal across trajectory counts.Source: §4.1 · p. 10
  • Multiple trajectories per question improve GRPO performance and stability, while larger trajectory counts accelerate convergence.The authors link this pattern to GRPO's use of multiple trajectories per question, in contrast with PPO and naive SFT.Source: §4.1 · p. 11, Figure 9 · p. 11
  • Training on multiple trajectories per question corresponds to enlarging the meta-learning support set, which improves inner-loop optimization and training performance.Source: §4.1 · p. 11

4.2. Incentivizing Reasoning Efficiency by Optimization Lens

  • Summarized trajectories yield Qwen3-32B performance comparable to thinking mode, especially on Pass@16, with fewer reasoning tokens, and outperform no-thinking mode.The comparison covers AIME24, MATH500-L5, and LiveMathBench-Hard; Figure 11 presents the experiments.Source: §4.2 · p. 12, Figure 11 · p. 12
  • The authors propose finding decoding methods that reproduce shorter optimal reasoning trajectories while maintaining comparable reasoning performance.Their summarized-trajectory method is an approximation; more advanced approaches are left for future work.Source: §4.2 · p. 12

4.3. Future Directions

  • Future work asks how self-generated trajectories and token contributions shape reasoning, and proposes adaptive trajectory sampling, token filtering, and question ratios for cross-domain generalization.The paper also raises how meta-features developed through trajectory optimization contribute to generalization.Source: §4.3 · pp. 12–13

5. Related Work

Related work spans LLM reasoning methods, theoretical analyses of transformer capabilities, and meta-learning approaches. The paper distinguishes its focus on trajectory-aided reasoning and internal parameter updates from work on demonstrations or models describing learning algorithms.

  • Prior work studies LLM reasoning through in-context learning, chain-of-thought prompting, supervised fine-tuning, reinforcement learning, and long-chain-of-thought models.Source: §5 · p. 13
  • Research on transformer capabilities includes universal approximation, computational power, chain-of-thought necessity, and learning algorithms implemented within trajectories.Other studies examine the role of demonstration examples in in-context learning.Source: §5 · p. 13
  • Unlike ICL work centered on demonstration examples or studies of learning algorithms described in natural language, this paper examines token-level trajectory updates to internal parameters across SFT and RL.The authors contrast explicit parameter optimization with ICL's limited demonstrations and frame their analysis as applicable to a broader range of problems.Source: §5 · pp. 13–14

6. Conclusion

The paper frames LLM reasoning trajectories as pseudo-gradient updates within a meta-learning perspective. The conclusion reports theoretical and empirical support for this framing and points to meta-learning principles as a source of practical insights.

6. Conclusion

  • The paper conceptualizes reasoning trajectories as pseudo-gradient updates and reports theoretical analysis and empirical evidence connecting LLM reasoning with meta-learning.Source: §6 · p. 14

A. Notations

  • The notation appendix lists symbols for answers, trajectories, token embeddings, model-parameter variation, network parameters, and question loss.Table 4 is identified as an illustration of the paper's notations.Source: Table 4 · p. 24, A. Notations · p. 24

B. Proof of Proposition 2.1

The proof constructs parameter transformations to establish the stated proposition. It uses linear maps and, where needed, an approximate solution for a matrix in the construction.

  • The proof seeks parameter transformations that preserve the stated equality, beginning with a linear transformation of the attention-related matrices.It invokes the basis-mapping theorem to construct the transformation.Source: B. Proof of Proposition 2.1 · pp. 24–25
  • A linear map can be constructed to send each basis vector to a specified target vector, providing the proof's mapping step.Source: B. Proof of Proposition 2.1 · p. 25
  • The proof chooses W′q so that Q=Q′, then simplifies Equation (11).Source: B. Proof of Proposition 2.1 · p. 25
  • When E_t,0 is outside the span of E_l,: row vectors, the proof allows an approximate C solution using methods such as least squares.Source: B. Proof of Proposition 2.1 · pp. 25–26
  • The construction uses the Moore–Penrose pseudoinverse of E_l,: and establishes a parameter set that may not be unique.A feed-forward network may also address differences in attention computations, according to the proof.Source: B. Proof of Proposition 2.1 · p. 26

C.1. Implementation Details of Visualization of Pseudo Gradient Update

The appendix visualizes QwQ reasoning trajectories and parameter-loss landscapes using four AIME2024 questions, then projects pseudo-gradient updates onto those landscapes.

C.1. Implementation Details of Visualization of Pseudo Gradient Update

  • Four AIME2024 questions provide the examples used to visualize pseudo-gradient updates and parameter landscapes.The questions include problems in walking-rate arithmetic, logarithms, complex numbers, and geometry.Source: §C.1 · p. 26, Question · p. 27

Question

  • QwQ-32B generates trajectories for each question, and the process tracks the negative log-probability of the final answer at each position.Source: Question · p. 27
  • The landscape is formed by perturbing non-embedding parameters along two randomly selected directions and evaluating negative log-probability from the instruction and question.Source: Question · p. 27
  • To project an update trajectory onto the landscape, the method fixes the time dimension and chooses the closest contour to the corresponding loss value.Source: Question · p. 27

C.2. Training Details

Training uses a filtered question set, synthesized supervised trajectories, and GRPO rewards that distinguish answer correctness from format correctness.

  • Filtering removes incomplete, unverifiable, image-dependent, or externally informed questions, leaving approximately 39k questions with answers.Source: §C.2 · p. 28
  • SFT synthesizes 640k trajectories from 10k sampled questions, retaining approximately 470k for Qwen2.5-Math-72B-Instruct and 550k for DeepSeek-R1-Distill-Qwen-14B.Source: §C.2 · p. 28
  • The SFT response format separates reasoning in <think> tags from a conclusion and boxed final answer inside <conclusion> tags.Source: §C.2 · p. 29
  • GRPO assigns rewards of 1, 0.0, −0.5, and −1 according to whether answer and format are each correct or incorrect.Math-Verify checks equivalence between the ground truth and generated answer.Source: §C.2 · p. 29
  • Experiments use veRL, PyTorch 2.6.0, Transformers 4.51.3, and vLLM 0.8.4 on NVIDIA A800 GPU clusters.The clusters also use Intel Xeon Platinum 8336C CPUs.Source: §C.2 · p. 29

C.3. Evaluation Details

The evaluation section specifies mathematical and generalization benchmarks and metrics, then interprets recent reasoning-training techniques through a meta-learning lens.

C.3. Evaluation Details

  • Pass@k measures performance and mG-Pass@k measures stability across generated responses.Source: §C.3 · p. 30

D. More Discussions on Recent LLM Reasoning Progress

E. More Experimental Results

Additional visualizations examine pseudo-gradient updates across QwQ and Qwen3, including thinking-mode contrasts and incorrect trajectories.

Loading 2505.19815v1…