Rewarding Structural Conformance of Reasoning using Process Mining

1 Introduction

• Correct solutions received higher TACReward alignment than incorrect ones across four models, with statistically significant separation for every model (U-test, p < 10−10).The distributions were computed against DeepSeek-R1 (671B) as the reference on DeepMath-103k.Source: Figure 1 · p. 2, §1 · p. 2
• Binary outcome rewards can provide little differentiation when responses to a problem have similar correctness, while step-level labels are costly and challenging to integrate into sparse-reward frameworks.Human annotation is costly, and dense rewards may require architectural modifications that risk sparse-reward stability.Source: §1 · p. 2
• Reasoning quality can be characterized by the sequence and composition of cognitive activities rather than the specific content of individual steps.Different solution approaches may share a structural skeleton, such as formulating a strategy, applying a formula, simplifying, and verifying.Source: §1 · p. 2
• TACReward treats reasoning as a structured activity sequence and uses process mining to align policy reasoning with a more mature teacher, without additional human annotation cost.Its scalar reward ranges from 0 to 1 and is designed to integrate with sparse-reward policy-gradient methods.Source: §1 · p. 2

2.1 Event Log & Process Mining

• An event log is a multiset of traces, each a finite sequence of events from one case; process mining discovers process models, checks conformance, and enhances or extends models.Figure 2 maps activities to boxes, flow to arrows, and process states to circles.Source: §2.1 · p. 3, Figure 2 · p. 3

2.2 Notations

• An autoregressive LRM is denoted by policy πθ, a stochastic policy over sequences parameterized by θ; its response y is represented as a token sequence.The sequence is written y1, y2, …, y|y|.Source: §2.2 · p. 3

2.3 Sparse Reward Policy Gradient Methods for LRMs

• Sparse-reward policy gradients maximize expected response reward; a response-independent baseline reduces estimator variance, and the advantage is R(x, y) − b(x).The reward function supplies a scalar signal based on response quality for query x.Source: §2.3 · p. 3

2.4 Group-based Policy Gradient Methods for LRMs

• Group-based methods sample multiple responses for each query and assign typically sparse outcome rewards, such as binary correctness, with a common baseline choice.The responses are sampled from the policy model.Source: §2.4 · p. 4

3.1 Trace: Formalize Reasoning

• For each query, TACReward samples G policy responses, segments reasoning into discrete steps, maps them to 20 activities, and formalizes a teacher reference trace.A general-purpose model extracts the traces, and the teacher generates one response per problem.Source: §3.1 · pp. 4–5

3.2 Align: Examine the Alignment

• Each policy trace becomes an Inductive Miner process model, then a minimum-total-cost alignment with the teacher reference trace captures synchronous and mismatched moves.A mismatch can represent a reference-only activity or a model-only activity.Source: §3.2 · p. 5

3.3 Check: Evaluate the Conformance

• Fitness measures how well the model explains reference behavior, precision measures extra behavior against the trace, and their F1 score yields a 0–1 reward that can be added to task reward.Fitness uses a finite worst-case deviation cost for stable normalization.Source: §3.3 · p. 6

4.1 Foundational Policy Gradients Methods with TACReward

• GSPO + TAC raised Avg. * from 17.2 to 32.5 (+89.2%) over GSPO alone in the 240-step comparison.The authors attribute this synergy to sequence-level GSPO updates matching TACReward’s sequence-level conformance score.Source: §4.1 · p. 6, Table 2 · p. 7
• TACReward improved Avg. * by +6.1% with RLOO and +12.7% with GRPO, but yielded limited gains and slightly reduced the overall average with PPO.The authors attribute this asymmetry to PPO’s sensitivity to reward variance and credit assignment under sparse rewards, versus variance reduction in RLOO and GRPO.Source: §4.1 · p. 7

4.2 Comparison with State-of-the-Art Models

• With the 7B backbone, GSPO + TAC outperformed all baselines, particularly on Olympiad and KSAT 2025, and remained competitive (often better) at 1.5B.Table 3 compares the methods after 3,000 training steps.Source: Table 3 · p. 7, §4.2 · pp. 7–8
• During training, GSPO + TAC showed stable, high accuracy-reward trajectories, while TACReward increased alongside accuracy for both model sizes.Figure 5 compares accuracy reward among GRPO, GSPO, and GSPO + TAC at 1.5B, and plots TACReward and accuracy reward for 7B and 1.5B.Source: Figure 5 · p. 8, §4.2 · p. 8

4.3 Analysis

• Policy performance degraded only marginally on most benchmarks even when teacher accuracy on DeepMath-103k fell by up to 28%.The authors report that structural patterns in teacher traces can remain informative when the teacher’s final answer is wrong.Source: §4.3 · p. 8, Table 4 · p. 8
• DeepSeek-V3.2, Gemini-2.5-Flash, and GPT-5.4-nano yielded nearly identical policy performance when the extractor could follow the 20-activity taxonomy.GPT-5.4-nano matched or slightly exceeded the default extractor on every benchmark.Source: Table 5 · p. 8, §4.3 · p. 9
• Fitness alone produced longer reasoning chains, precision alone produced shorter ones, and using both yielded lowest entropy and moderate length; either alone degraded benchmark performance.Figure 6 tracks entropy, mean length, and TACReward over 240 steps.Source: §4.3 · p. 9, Figure 6 · p. 9
• The 20-activity taxonomy is tailored to mathematics; smaller teacher and extractor models may be usable, but their exact performance trade-offs and optimal configurations remain unexplored.The authors note that other reasoning domains may require a new taxonomy.Source: §4.3 · p. 9

5 Conclusion

• Structural similarity improves sparse-reward performance, while fitness and precision show their intended optimization effects.TACReward compares policy reasoning with a more logically mature teacher without additional annotations or architectural changes.Source: §5 · p. 9
• TACReward may extend to code generation and other reasoning domains where the underlying process is well-defined.The paper identifies code generation as a potential broader application.Source: §5 · p. 9
• In well-defined domains, domain knowledge can specify the reference trace without a teacher, as in loan assessment’s expected analyst steps.TACReward could act as a structural guardrail for policy reasoning in such settings.Source: §5 · p. 10

A Integrating TACReward with Sparse Reward Policy Gradients Methods

• TACReward is described as an add-on for PPO, RLOO, GRPO, and GSPO sparse-reward policy-gradient methods.The appendix frames integration as adding its reward to the existing response reward.Source: A Integrating TACReward with Sparse… · p. 13, §A.3 · p. 14

A.1 Proximal Policy Optimization (PPO) with TACReward

• PPO samples one response per query and incorporates TACReward into that response’s reward.The method also uses a fixed reference policy, with β controlling KL regularization strength.Source: §A.1 · p. 13

A.2 REINFORCE with Leave-One-Out (RLOO) with TACReward

• RLOO adds TACReward to each response’s reward and uses a leave-one-out baseline computed from the other group samples.RLOO generates multiple responses for each input query.Source: §A.2 · p. 13

A.3 Group-based Policy Optimization with TACReward

• GRPO and GSPO use group-relative advantages, and TACReward is added to each response’s reward.The appendix states that GRPO assigns the same advantage to all tokens in a response.Source: §A.3 · p. 14

B Details of Experiments

• Policy training used the TRL framework with DeepSpeed ZeRO-3 optimization.Source: §B.1 · p. 14
• Table 6 summarizes key hyperparameters and distributed training configuration for the experiments.Source: §B.1 · p. 14
• Parameters not otherwise specified followed the TRL GRPOTrainer defaults.Source: §B.1 · p. 14

B.2 Experimental Settings

• Experiments use Qwen2.5-1.5B/7B-Instruct and DeepSeek-R1-Distill-Qwen-1.5B/7B as non-reasoning and reasoning-enhanced base models, respectively.Source: §B.2 · p. 14
• MATH-500 and AIME 2024 results are reported for reference because of potential contamination; main quantitative conclusions use the remaining benchmarks.Training uses DeepMath-103k, and the evaluation set includes seven mathematical reasoning benchmarks.Source: §B.2 · p. 14
• Evaluation uses Pass@1 with temperature 0.6, top_p 0.95, and max_tokens 16384.Answers are extracted using the DeepMath Evaluation Library.Source: §B.2 · p. 14
• Training used TRL on four NVIDIA H200 GPUs, AdamW at a constant learning rate of 1 × 10^-6, and a zero KL coefficient.The global batch size was 128; micro-batch sizes were 8 for 1.5B models and 4 for 7B models.Source: §B.2 · p. 14
• Table 6 is identified as the hyperparameter and training-configuration table for TACReward with GSPO.Source: Table 6 · p. 15

C.1 Computational Complexity of the optimal alignment.

• With a fixed activity set, bounded intermediate states, and no branching beyond local model moves, alignment costs O(n log n), dominating the overall O(n) score computation.The general A* search is exponential in trace length; summing move costs takes O(n + m), and precision costs O(n · |Σ|).Source: §C.1 · p. 15

C.2 TAC as a Proxy for Step-level Reward

• TACReward returns one scalar from fine-grained process alignment, whose local move costs implicitly assign credit at reasoning transitions; fitness penalizes missing steps and precision penalizes excessive or unjustified behavior.Their harmonic mean captures completeness and selectivity.Source: §C.2 · pp. 15–16

C.3 Why Single-Trace Can Be Effective; C.5 On Imperfect Reference Traces.

• A single-trace process model captures ordering constraints, optional branches, and loops, while the reference trace need only be more mature than the policy—not necessarily correct.The paper says this abstraction enables conformance checking of structurally invalid reasoning behaviors.Source: §C.3 · p. 16, §C.5 · p. 16

C.4 Why Response-Level Comparison Is Insufficient.

• Raw-text edit distance measures insertions, deletions, and substitutions but cannot reliably distinguish structural reasoning errors from benign reordering.Trace formalization separates reasoning structure from linguistic realization for structural comparison.Source: §C.4 · p. 16

C.6 Role of Reward Components.

• The reward-component ablation compares training with fitness alone, precision alone, or both components; Table 7 summarizes the results.The supplied text gives no component-level outcome values.Source: §C.6 · p. 16, Table 7 · p. 16

C.7 Role of Mathematical Taxonomy.

• Removing the Mathematical Reasoning Taxonomy degrades performance, with larger drops on multistep benchmarks such as AIME 2025 and KSAT 2025.The text attributes this to rewards becoming less sensitive to structural differences and more dominated by surface variation without an explicit taxonomy.Source: §C.7 · pp. 16–17, Table 8 · p. 17

D.1 Prompt for Generation; D.2 Prompt for Formalizing Trace

• The generation prompt requires step-by-step reasoning before one boxed final answer, and trace formalization outputs one CSV row per non-empty line using exactly one allowed activity label.The formalization instructions prohibit inventing activity labels.Source: §D.1 · p. 17, §D.2 · p. 18

E Reasoning Process Visualization with and without TACReward

• The visualization material includes a LiveMathBench No.95 game problem and a list of reasoning activity labels.The supplied passages identify the example and labels but do not state a comparison outcome.Source: E Reasoning Process Visualization with… · p. 19

• Step 700 records Wrong F1: 47, while step 1000 records Correct F1: 0.55.Source: F Reasoning Process Visualization… · p. 20

• A Minerva-Math example asks for the frequency, wave number, and wavelength conversions for red light emitted by excited lithium atoms.Source: G Reasoning Process Visualization… · p. 21
• The listed activity sets include steps such as Apply Known Formula, Check Validity, and Conclude Final Result, with some lists also including Interpret Result and Verify With Example.Source: G Reasoning Process Visualization… · pp. 21–22
• A LiveMathBench example asks which of Huey, Dewey, or Louie is closest to Donald using their specified geometric positions.Source: G Reasoning Process Visualization… · p. 22