TemplateRL: Structured Template-Guided Reinforcement Learning for LLM Reasoning

Motivation

• Unstructured self-sampling limits consistent generation of high-quality trajectories, transferable strategy learning, and interpretable reasoning.The paper identifies low hit rates and instability on weak models, difficulty transferring high-level patterns such as divide-and-conquer, and limited support for diagnosis or expert intervention.Source: §1 · pp. 1–2

Approach

• TemplateRL constructs templates with MCTS on a small seed set, retrieves relevant templates to guide rollouts, and can add patterns abstracted from successful rollouts.The templates encode high-level problem-solving patterns and steer generation toward validated strategies.Source: §1 · p. 2

Contributions

• Template-guided trajectories form interpretable decision chains that support error diagnosis and expert refinement.The paper also reports that explicit templates support cross-domain generalization and flexible knowledge updates.Source: §1 · p. 2

Results

• 99% on AIME and 41% on AMC: TemplateRL outperforms GRPO on a 7B backbone, with stable training on weak models and cross-domain gains.The reported out-of-distribution benchmarks include BALROG, GPQA-D, and MMLU-Pro, with improvements across text and vision modalities.Source: §1 · p. 2

LLMs as Markov Decision Processes

• LLM generation is modeled as a token-level Markov Decision Process, with the question and generated prefix forming the state.The policy samples the next token from the vocabulary action space.Source: §2 · p. 2

LLM Optimization

• The entropy-regularized objective includes a KL term that prevents the policy from deviating excessively from a reference policy.The return is defined over the trajectory, and the reward model supplies the reward.Source: §2 · p. 2

Template Action

• A template action is a prompt that elicits a specific type of reasoning from a partial solution, such as a sub-question or inference step.Source: §2 · p. 2

3 Template-Guided RL

• The framework has three stages: template construction, template-guided training, and optional template updates.The templates encode high-level strategic guidance inspired by reusable approaches to similar problems.Source: §3 · p. 2, Figure 2 · p. 3, §3.1 · p. 3

3.1 Template Construction

• MCTS explores solution trees from a small seed set, with action edges and model-generated partial solutions at child nodes.A root-to-leaf path forms a solution trajectory, and the search assigns each trace a final reward.Source: §3.1 · p. 3
• Rejection sampling selects a path that balances solution quality against path complexity, with a parameter controlling that trade-off.The selected action sequences are aggregated into a library of generalized strategies, with each template storing its average problem-condition complexity.Source: §3.1 · pp. 3–4

3.2 Template-Guided Training

• For each training question, the method retrieves the top-k templates with the closest problem-condition complexity and rolls out iteratively under their actions.The final guided rollout is the history after the retrieved template’s actions have been applied.Source: §3.2 · p. 4
• Multi-guidance training independently samples rollouts for each template and uses the resulting rollouts to update the policy.The optimization can be expressed through template-specific objectives and combined contributions from the guidance groups.Source: §3.2 · p. 4
• Guidance groups increase the probability of obtaining at least one positive-advantage trajectory and reduce gradient variance compared with unguided training.The stated guarantee assumes independent groups of trajectories with per-template positive-advantage probabilities.Source: §3.2 · p. 4
• Under the proposition’s stated condition on query similarity, retrieving a high-quality template from a related question yields a mini-group with higher positive-advantage probability than the policy alone.The result is conditional on the proposition’s assumptions about the advantage function and the new query.Source: §3.2 · p. 5

3.3 Optional Template Update

• Optional updates extract action-sequence patterns from correct rollouts and add them to the library for continued expansion.Source: §3.3 · p. 5

4.1 Experimental Setup

• Experiments use 5.5K MATH level 35 problems (500 for the library, 5K for training), evaluate math and OOD benchmarks, and compare with Standard GRPO and Oat-Zero.Source: §4.1 · p. 5

4.2 Improved Performance

• TemplateRL maintains improvements across 1.5B–8B models, Qwen and Llama architectures, base and instruction-tuned models, and other RL algorithms.A 32B analysis reports that structured guidance enhances rather than restricts exploration.Source: Figure 4 · p. 6, §4.2 · p. 6
• 55.8 average score: TemplateRL beats Oat-Zero by 7.1 points and GRPO by 12.0; AIME24 reaches 33.3% versus GRPO’s 16.7% (+99.4% relative gain).Gains rise with task difficulty: MATH500 shows +9.4%, AMC +40.9%, and AIME24 +99.4%.Source: §4.2 · p. 6

4.3 Enhanced Training Stability

• On Llama-3.2-3B-Base, GRPO rewards fall from 0.3 to near 0.0 after 100 steps, while TemplateRL keeps rewards above 0.25.TemplateRL also achieves higher training rewards overall on Qwen2.5-Math-7B-Base and Llama-3.2-3B-Base; the authors attribute its stability mainly to template guidance during sampling.Source: §4.3 · p. 6

4.4 Broad Generalization

• OOD evaluation beats GRPO across tasks (+6.1% BALROG), while multimodal evaluation improves across five benchmarks (+8.4% average).The OOD tasks include agentic reasoning, graduate-level science, and general knowledge; multimodal benchmarks cover math, general understanding, and visual perception.Source: Figure 6 · p. 7, §4.4 · p. 7, Table 2 · p. 7

4.5 Dynamic Extensibility

• Training-time library expansion from successful rollouts improves all benchmarks, including AIME24 (+10.2%); test-time expansion also improves results using majority-voted pseudo-supervision.At test time, the library is enriched sequentially from voted outputs.Source: Table 3 · p. 7, §4.5 · p. 7, Table 4 · p. 8

4.6 Ablation Study and Analysis

• On a geometry problem, TemplateRL identifies a strategy such as divide-and-conquer before solving subproblems, producing clearer, more interpretable reasoning than GRPO.Source: Figure 7 · p. 8, §4.6 · p. 8
• With fixed total rollouts, more diverse template guidance often yields higher rewards; the authors choose |g| = 2 by default.Source: §4.6 · pp. 7–8, Figure 8 · p. 8

• RL research has focused on algorithmic refinements, data-level optimizations, and exploration enhancement, but remains constrained by unstructured self-sampling; TemplateRL integrates explicit reasoning templates into policy optimization.Source: §5 · p. 8
• Template guidance has appeared mainly in inference-time retrieval, decomposition, and abstract thought patterns; TemplateRL brings structured templates into RL training to learn both concrete steps and underlying strategic logic.Source: §5 · p. 8

• TemplateRL is a framework for LLM reasoning that constructs structured templates and integrates them during policy optimization.Source: §6 · p. 9
• Experiments demonstrate improvements across models, datasets, and modalities, with cross-domain generalization and dynamic extensibility.Source: §6 · p. 9
• The framework produces interpretable decision chains.Source: §6 · p. 9

Limitations

• The evaluated scope is primarily text and multimodal reasoning; robotic control, scientific discovery, and embodied AI agents are future extensions.Source: Limitations · p. 9

Appendix

• The appendix elaborates the MCTS implementation, template-library construction, and adaptive retrieval and instantiation mechanisms.Source: Appendix · p. 13
• Supplementary experiments and case studies are also included in the appendix.Source: Appendix · p. 13

A TL;DR: Main Contributions and Takeaways

• Existing RL methods rely on unstructured self-sampling and lack systematic structured guidance for improving trajectory quality, strategy transferability, and interpretability.The passage frames these as limitations of prevailing methods.Source: A TL;DR: Main Contributions and… · pp. 13–14
• TemplateRL combines an explicit reasoning-template library, built via offline MCTS on seed data, with template guidance during online RL training.The framework steers policy optimization toward validated strategic patterns.Source: A TL;DR: Main Contributions and… · p. 14
• TemplateRL is reported to outperform GRPO by +99% on AIME, +41% on AMC, and +17% on Minerva Math on a 7B backbone.It also consistently improves across 1.5B–8B scales and Qwen and Llama architectures.Source: A TL;DR: Main Contributions and… · p. 14
• The library is interpretable and editable, and supports continuous updates during training and inference.Structured trajectories expose strategic patterns for error diagnosis and expert intervention.Source: A TL;DR: Main Contributions and… · p. 14

B.1 Notation and Assumptions

• The grouped-sampling setup assumes trajectories within a group may be dependent while different groups are approximately independent.For a single query, G trajectories are divided into m groups of size gs = G/m.Source: §B.1 · p. 14

B.4 Why template transfer increases success probability

• For nearby queries, a retrieved MCTS template with positive advantage on its seed problem can retain positive advantage when the advantage function is Lipschitz and the distance is sufficiently small.The passage links this transfer condition to higher per-group template success probability than policy rollouts.Source: §B.4 · pp. 15–16

B.2 Proof of Proposition 3.1

• The probability of at least one positive trajectory in a batch increases with the number of independent groups and approaches 1 when per-group success probability is positive.The leading-order variance of the averaged gradient estimator scales as O(1/m), even when samples within groups are dependent.Source: §B.2 · pp. 14–15
• Increasing the number of independent groups, rather than only adding correlated samples within one group, is identified as the effective route to variance reduction.The effective sample size is the number of independent clusters.Source: §B.2 · p. 15

B.3 Proof of Proposition 3.2

• Template transfer gives at least probability rh of a positive trajectory in group h, and strictly improves per-group success when rh exceeds policy-rollout probability.Under independent groups, the batch-level probability of a positive sample exceeds the policy-rollout probability when the transfer probabilities are higher.Source: §B.3 · p. 15

C.1 Monte Carlo Tree Search (MCTS)

• The search uses five predefined reasoning actions: Divide and Conquer, Self-Reflection, System Analysis, One-Step Thought, and Chain-of-Thought.MCTS builds a tree whose edges denote actions and whose child nodes contain partial solutions generated by the policy.Source: §C.1 · p. 16
• MCTS explores solution trees through selection, expansion, simulation, and backpropagation, using UCT for selection and self-consistency for early simulation termination.Terminal-node reward uses self-consistency majority-vote confidence.Source: §C.1 · pp. 16–17, Figure 9 · p. 17

C.2 Thought Template Library Construction

• For each of 500 seed problems, MCTS explores solution paths, and the selected trajectory maximizes a score balancing reward and action-count complexity with b = 0.95.The selected path is abstracted into its high-level action sequence.Source: §C.2 · pp. 17–18

C.3 Adaptive Template Retrieval

• Retrieval computes a question’s PCC, ranks templates by distance, and selects the top-k similar patterns to guide rollout generation.The templates balance exploiting strategies effective on similarly complex problems with model-internal exploration.Source: §C.3 · pp. 18–19
• In the PCC = 2 example, retrieval selects T1 and T2 while filtering out overly complex T3; T1 is streamlined, whereas T2 adds verification.The passage says T2 may be preferred in high-stakes scenarios where correctness verification is critical.Source: Figure 11 · p. 19, §C.3 · p. 19
• Correct rollouts can optionally be mined for action sequences and incorporated into the library during RL training.The passage reports that Tables 3–4 empirically validate dynamic updates during training and inference.Source: §C.3 · p. 20

C.2 Thought Template Library Construction

• Templates aggregate identical action sequences and store their average PCC, producing 103 unique templates from 500 seed samples.Each template pairs a generalized strategy with typical problem complexity.Source: §C.2 · p. 18, §C.4 · p. 20

C.4 Discussion on Additional Method Details

• Templates are discovered as sequences of predefined actions, and action-level abstraction unifies different surface phrasings that share an action label.This design is described as providing recognizable reasoning concepts while allowing effective combinations to emerge through MCTS.Source: §C.4 · p. 20

D Experimental Details; E.1 Out-of-Domain Results

• The experiments compare five baselines on Qwen2.5-Math-7B, with GRPO using the same 5.5k training samples; training uses binary final-answer accuracy as its reward.Appendix E.1 provides detailed results on three out-of-distribution benchmarks for Qwen2.5-Math-7B-Base.Source: D Experimental Details · p. 20, §E.1 · p. 21, Table 5 · p. 21

E.3 Extension to Multimodal Domains

• TemplateRL beats CoT and GRPO across multimodal benchmarks, with a +8.4% average improvement over GRPO.The evaluation spans mathematical reasoning, general understanding, and visual perception.Source: §E.3 · p. 21

E.2 Extension to More Models; E.4 Extension to Other RL Algorithms

• Reported results include AMC gains of +17.5 points (+35.0% relative) and AIME24 gains of +3.3 points (+11.0% relative); TemplateRL averages +3.2% and +5.6% on two additional models.The additional-model averages are for Qwen2.5-Math-1.5B-Instruct and Llama3.2-3B-Instruct, respectively.Source: §E.2 · p. 21, §E.4 · p. 22

E.5 Analysis on Highly Capable Models

• Template-guided sampling scores 83.3 on AIME25 and 82.8 on GPQA-D before RL training, and TemplateRL further improves post-training performance.The evaluation uses pass@16 on Qwen3-32B; the authors say future work will investigate scaling behavior across model sizes.Source: §E.5 · p. 23, Table 8 · p. 23

E.6 Discussion on Template Construction Strategy

• 55.4 vs. 42.9 average: MCTS-based templates outperform end-to-end generation, which itself beats GRPO at 42.9 vs. 35.9.On AIME, MCTS scores 33.3 vs. 23.3 for end-to-end generation (+43.0%); MCTS provides a further +29.1% improvement over end-to-end.Source: §E.6 · p. 24, Table 9 · p. 24

E.7 Computational Overhead Analysis.

• TemplateRL uses 20% more training time than GRPO but yields a +27.4% relative performance gain and 3.1 vs. 2.9 accuracy points per GPU-hour.Training takes approximately 18 vs. 15 GPU-hours; optional dynamic updates add approximately 2% training time and 0.3 seconds per test question.Source: §E.7 · pp. 24–25

E.8 Ablation Study on the Action Space

• 64.7 average: the full five-action set performs best, while the single-action {a5} baseline achieves 55.9 average accuracy.Expanding the action set consistently improves performance; the authors also report that reduced sets can suit resource-constrained settings.Source: Table 10 · p. 24, §E.8 · p. 25

E.9 Sensitivity to Training Data Size

• 64.7% (5K) to 66.1% (10K) and 67.1% (15K): performance rises with training-data size.The reported AIME24 improvement from 5K to 10K is +3.4 percentage points.Source: §E.9 · p. 25, Table 11 · p. 25

E.10 Sensitivity to Pattern-Matching Metrics

• 66.0 average: unified ranking performs best, compared with 64.7 for PCC-only and 60.2 for PS-only matching.PCC-only has lower computational overhead than unified ranking; semantic matching is reported as the weakest approach.Source: §E.10 · p. 26

E.11 Sensitivity to Seed Data Size.

• 38.7% (100 samples) to 51.4% (1,000 samples): average accuracy increases with seed data size; 500 samples is the default.The authors identify 500 samples as a balance between performance enhancement and data efficiency.Source: §E.11 · p. 26

E.12 Statistics on Reasoning Actions

• DC and OST are the most frequent library actions at 31.4% and 30.3%, followed by CoT at 25.5%.SR and SA account for 7.2% and 5.7%, respectively; the authors associate the more frequent actions with successful solution generation during MCTS exploration.Source: §E.12 · p. 26, Table 14 · p. 27

E.13 Statistical Analysis

• p = 0.0156: the Wilcoxon signed-rank test rejects no difference between TemplateRL and GRPO across benchmarks at α = 0.05.The comparison uses Qwen2.5-Math-7B-Base.Source: §E.13 · p. 27

F Case Study

• In the arithmetic-mean case, TemplateRL gives a more structured solution; on the complex-number case, it stays coherent while GRPO reasoning deteriorates.The examples contrast stepwise reasoning with repetitive notation in the first case and coherent use of Vieta’s formulas with incoherent fragments in the second.Source: F Case Study · p. 27

G.1 Justification for Choosing MCTS.; G.2 Justification of Key Design Choices

• MCTS incrementally explores promising trajectories; DFS can become trapped after early errors, while BFS grows exponentially in candidate states.The design choices include a five-action set and PCC matching, which distinguishes problems by reasoning requirements rather than surface similarity.Source: §G.1 · p. 28, §G.2 · p. 28

G.3 Why A Small Set of Generic Patterns Can Generalize?; G.4 Are Templates Optimized for Test Sets?

• Math-derived templates improve results on BALROG (+6.1%), GPQA-D (+6.6%), and MMLU-Pro (+7.4%).The authors describe the transferred templates as strategic patterns rather than domain-specific solutions.Source: §G.4 · pp. 28–29, §G.3 · p. 28

G.5 How does performance scale with template set size?

• Performance improves substantially below 100 templates, with gains slowing beyond that library size.The authors suggest future work on dynamically adjusting library size by domain or difficulty.Source: §G.5 · p. 29

G.6 Robustness to Seed Set and Cross-Domain Generalizability

• Math-seed templates improve BALROG by +6.1% and GPQA-D by +6.6%, and the reported results support cross-domain transfer.The authors describe the seed-derived patterns as task-agnostic strategic logic rather than surface-level heuristics.Source: §G.6 · p. 29

G.7 Distinction between TemplateRL and Related Works

• Unlike ReMA’s implicit meta-strategy discovery through agent interactions, TemplateRL builds explicit MCTS-derived libraries that people can inspect and edit.The passage also identifies cross-domain transfer and avoiding multi-agent training overhead as properties of this approach.Source: §G.7 · p. 29

G.8 Future Work

• Future work includes richer templates for agentic scenarios, heterogeneous model orchestration, broader domains, multi-agent coordination, and deployment efficiency.The authors also propose investigating template diversity and robustness to unknown adversarial conditions.Source: §G.8 · p. 29
• Figures 13 and 14 compare GRPO and TemplateRL on simple and difficult algorithm problems from MATH.The captions identify the problem types and dataset but do not state comparative outcomes.Source: Figure 13 · p. 30, Figure 14 · p. 31