PACT: From Credit Assignment to Critic Alignment

• Trajectory-level rewards must be attributed to token-level updates, but a token-level Markov representation alone does not characterize that attribution or how reward information changes during generation.Source: §1 · p. 2
• Completeness, Prefix Consistency, and Neutrality uniquely determine token-level credit, providing a unified perspective on phenomena across LLM reinforcement-learning algorithms.Source: §1 · p. 2
• With an ideal teacher, OPD yields an expected policy gradient proportional to credit-induced gradients, while response-level RLOO yields the same expected contribution despite coarser granularity.Source: §1 · p. 2
• Bounded outcome rewards yield approximate credit sparsity, and intermediate critic errors can help explain the empirical difficulty of GAE-based actor-critic methods in long-horizon settings.Source: §1 · p. 2
• PACT updates the actor before the critic to enable importance-sampling correction during critic training and better align the critic with the updated policy.Source: §1 · p. 2
• 72.87% average accuracy across four mathematical benchmarks exceeds GRPO and PPO by 8.80 and 13.16 percentage points; SWE-bench Verified pass rate is 67.4%, above PPO, GRPO, and SAO by 2.4, 2.0, and 3.8 points.Source: §1 · p. 2

2.1 Notation: Reward Information Flow

• Trajectories interleave generated tokens and observations (empty observations recover ordinary generation); Vi is expected terminal reward given information through step i, while reward depends on the full trajectory.Source: §2.1 · p. 3

2.2 Regularity Conditions of Credit Assignment

• Credit attributes outcomes to generated tokens, while Completeness accounts for reward deviation, Prefix Consistency fixes prefix credit, and Neutrality requires zero expected next-token credit.Source: §2.2 · p. 3

2.3 Unique Representation Theorem

• For integrable rewards measurable at the stopping time, the three conditions uniquely determine credit as Ci = Vi − Vi−1; omitting any condition permits alternatives.Source: §2.3 · p. 4, It admits the representation · p. 4

It admits the representation

• For consecutive-token segments, segment-level credit aggregates token-level credit, recovering coarse-grained schemes such as turn-level attribution.Source: It admits the representation · p. 4

3.1 The Teacher in On-Policy Distillation is an Implicit Critic

• Under the ideal-teacher assumption, OPD matches the unique token-credit policy gradient up to scaling, with the teacher defined via KL-regularized reward improvement.Source: §3.1 · pp. 4–5

3.2 Group-Level Baselines as Gradient-Equivalent Credit Surrogates

• RLOO's response-level leave-one-out baseline yields a token-level policy-gradient estimator equivalent in expectation to unique credit, but not necessarily equally statistically efficient.Source: §3.2 · pp. 5–6

3.3 Approximate Credit Sparsity and Error Sensitivity of GAE

• With bounded outcome rewards, Theorem 4 bounds the expected number of credit increments above any fixed magnitude independently of maximum response length.Source: §3.3 · p. 6
• For outcome-only rewards, λ < 1 retains intermediate critic errors that can become comparable to or dominate true credit, while λ = 1 leaves only prefix value error.Source: §3.3 · p. 6

4.1 Motivation

• PPO-style training uses critic values from the previous policy iteration, leaving the critic one policy update behind the actor.The lag matters especially when token-level credit is computed by differencing consecutive conditional values.Source: §4.1 · p. 7

4.2 Probabilistic Value Estimation

• For rewards normalized to [0, 1], BCE and MSE have the same optimal value prediction, while BCE critics converge substantially faster in the reported pretraining ablation.The paper therefore uses a sigmoid-parameterized critic trained with BCE rather than MSE.Source: §4.2 · p. 7

4.3 Off-Policy Critic Synchronization

• PACT completes actor updates first, then uses an additional pass over the rollout batch to form importance ratios and train the critic for the updated actor.This Actor-then-Critic order is intended to synchronize critic values with the policy being optimized.Source: §4.3 · p. 8
• Because exact continuation ratios can have high variance on long responses, PACT uses detached current-token ratios and masks critic losses outside [ρmin, ρmax].The surrogate needs one post-actor forward pass and no additional rollout generation.Source: §4.3 · p. 8

5.2 Main Results

• 67.4% pass rate on SWE-bench Verified exceeds GRPO, PPO (λ = 1.0), and SAO by 2.0, 2.4, and 3.8 percentage points.PACT records the highest pass rate among the compared methods.Source: Table 2 · p. 9, §5.2 · p. 9

5.1 Experimental Setup

• The experiments train Qwen3.5-4B for mathematical reasoning and Qwen3.6-35B-A3B for agentic coding, evaluating on four math benchmarks and SWE-bench Verified.Each rollout round produces 512 trajectories; the mathematical evaluation reports Avg@16 accuracy.Source: §5.1 · p. 9

5.2 Main Results

• 72.87% Avg@16 accuracy across four math benchmarks exceeds GRPO, PPO (λ = 1.0), and SAO by 8.80, 13.16, and 21.73 percentage points.PACT records the highest accuracy on all four benchmarks.Source: §5.2 · p. 9

5.3 Ablation Studies

• With rollout policy, initialization, and on-policy data fixed, BCE critics achieve lower BCE and MSE losses and greater value separation than MSE critics across both model scales.Value separation is the mean predicted value on positive samples minus that on negative samples.Source: §5.3 · p. 9
• 72.87% versus 67.74% average accuracy, a 5.13-point gain across all four benchmarks, is reported when PACT includes critic-target importance sampling correction.The ablation also reports more stable training with the correction.Source: §5.3 · p. 9

6 Conclusion

• Completeness, Prefix Consistency, and Neutrality uniquely determine token-level credit, providing a common basis for analyzing existing algorithms.The conclusion connects this characterization to OPD, RLOO, approximate credit sparsity, and GAE critic errors.Source: §6 · pp. 9–10
• PACT applies importance sampling correction after actor updates to align the critic with the updated policy, and uses BCE to improve value estimation.It reports 72.87% average accuracy across four math benchmarks and a 67.4% pass rate on SWE-bench Verified.Source: §6 · p. 10

B.1 Reinforcement Learning for Large Language Models

• Prior LLM reinforcement-learning work includes critic-free baselines and value-based methods for long, heterogeneous trajectories; this paper instead characterizes token-level credit and its implications.The cited work also includes asynchronous training systems and refinements to critic-free optimization.Source: §B.1 · p. 14

B.2 Credit Assignment in Reinforcement Learning

• Earlier credit-assignment approaches address delayed rewards through return redistribution, temporal transport, future conditioning, or action-influence estimates, using different targets and procedures.The paper studies credit under three regularity conditions for outcome-supervised LLM reinforcement learning.Source: §B.2 · pp. 14–15

C.1 Reward Normalization and Variance Bound

• A bounded reward can be positively affinely normalized without changing which policy maximizes expected reward.Source: §C.1 · p. 15

C.2.1 Prefix Consistency Implies Adaptedness

• With finite token sequences over a finite vocabulary, Prefix Consistency makes accumulated credit depend only on the observed prefix, so the process is adapted to the prefix filtration.The proof represents accumulated credit as a measurable function of that prefix.Source: §C.2.1 · pp. 15–16

C.2.2 Proof of Unique Representation Theorem

• For integrable terminal reward R, the unique credit at step i is the increment E[R | Fi] − E[R | Fi−1], up to almost-sure equality.The conditional expectations are defined through the information available in each prefix.Source: §C.2.2 · p. 16, It admits the representation · p. 16

It admits the representation

• The proof accumulates credit into a prefix-measurable process whose terminal value is fixed by Completeness; Neutrality makes it a martingale, determining its successive increments uniquely.The argument concludes uniqueness up to almost-sure equality.Source: It admits the representation · p. 17

C.2.3 Necessity of the Regularity Conditions

• Completeness, Prefix Consistency, and Neutrality are logically independent; removing any one permits assignments that differ from the conditional-reward increments.Completeness prevents ignoring reward deviation, Prefix Consistency prevents future information revising earlier credit, and Neutrality prevents predictable zero-sum transfers.Source: §C.2.3 · pp. 18–19
• Without Completeness, an assignment may explain none of the realized reward deviation; without Prefix Consistency, future outcomes can revise credit assigned to an earlier prefix.Source: §C.2.3 · p. 18
• Without Neutrality, predictable compensating terms can move credit between token positions while preserving prefix measurability and the terminal sum.The second-token credit in the construction is already completely predictable before that token is generated.Source: §C.2.3 · p. 19

C.2.4 Aggregation of the Unique Token-Level Representation

• Summing unique token-level increments over any partition into consecutive segments gives the corresponding segment-level credit.Source: §C.2.4 · p. 19

C.3 Teacher Credit Equivalence Theorem

• Under an ideal teacher, OPD's expected update is equivalent up to scaling to the policy gradient induced by unique token-level credit.The teacher acts as an implicit critic.Source: §C.3 · pp. 19–20

C.4 Gradient Unbiasedness of the RLOO Estimator

• For RLOO, an independent response-level baseline yields zero expected score contribution when replacing the terminal reward with token-level credit.The response-level signal is not itself token-level credit, yet it preserves the expected policy-gradient contribution.Source: §C.4 · p. 21

C.5 Approximate Credit Sparsity Theorem

• Under bounded rewards, the theorem establishes approximate sparsity of token-level credit over a bounded trajectory horizon.The proof extends credit by zero after termination and uses its representation as adjacent conditional-value differences.Source: §C.5 · pp. 21–22

C.6 Gradient of BCE with Importance-Weighted Targets

• For on-policy and exact importance-weighted targets, approaching minimum expected BCE implies convergence to the target conditional mean in mean square.When expected BCE approaches its minimum at adjacent prefixes, the estimated credit approaches the true credit in mean square.Source: §C.6 · p. 23

C.7 Current-Token Importance Weighting as a One-Step Policy-Tracking Surrogate

• The practical current-token importance target is a one-step truncation of the full policy correction, tracking a hybrid policy that uses the updated policy for the current token and the rollout policy thereafter.This avoids the multiplicative continuation weight while moving the critic target toward the updated policy.Source: §C.7 · pp. 24–25

D Policy Aligned Critic Training Procedure

• PACT completes actor updates before critic training, using token-wise importance ratios and critic targets.Algorithm 1 summarizes the procedure; the critic targets are formed from importance-weighted rewards.Source: D Policy Aligned Critic Training… · p. 25

E.1 Prompt of Agentic Reasoning for Math

• The math benchmark prompt preserves each problem and ground-truth answer, excludes the answer from policy input, and encourages Python use for calculation, enumeration, and verification.The prompt also requires the final answer in a boxed expression on the last line.Source: §E.1 · pp. 25–26

E.2 Further Empirical Evidence for Theorem 4

• During PACT training, responses become progressively longer while the mean adjacent-value difference decreases.The reported trend is consistent with the intuition behind Theorem 4.Source: §E.2 · p. 26, Figure 4 · p. 26

E.3 Training Dynamics

• Figures 5 and 6 show the evolution of mean trajectory rewards during training on mathematical reasoning and agentic coding.The supplied passage identifies the plotted training outcomes but does not state their comparative values.Source: §E.3 · p. 26

E.4 Key Training Hyperparameters

• Table 3 is identified as summarizing key training hyperparameters for mathematical reasoning and coding.The supplied passage does not state individual hyperparameter values.Source: §E.4 · p. 26
• PACT applies TIS with a ratio range of [0, 2] for actor updates, while PPO, GRPO, and SAO use rollout log-probabilities.PACT without critic importance sampling retains actor-side corrections, the BCE critic objective, and the Actor-then-Critic order.Source: §E.4 · p. 26
• Figure 5 plots training reward on agentic mathematical reasoning with Qwen3.5-4B, showing raw mean trajectory rewards and their exponential moving averages.The moving-average coefficient is α = 0.3, and each rollout round corresponds to four training steps.Source: Figure 5 · p. 27

• Uniqueness holds only under Completeness, Prefix Consistency, and Neutrality; different requirements may yield different credit representations.Source: F Limitations · p. 27
• The defined credit is statistical, based on conditional expectations, and does not characterize the counterfactual effect of replacing an individual token or action.Source: F Limitations · p. 27
• The paper characterizes credit under its proposed conditions but does not solve its exact estimation in practical LLM reinforcement learning.Source: F Limitations · p. 27