Source-linked AI summary
Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning
Oriol Corcoll, Qiyao Wei, Ciarán Mark Gilligan-Lee
TL;DR
Long-horizon RL must assign credit to actions whose consequences may emerge hundreds or thousands of interactions later, while standard discounting privileges temporal proximity. EWM explains realized outcomes using earlier events and recursively redistributes reward through a conservative operator. Across controlled RL benchmarks and recommendation settings, it outperforms or improves on the reported baselines, with gains on delayed-reward tasks.
Problem
Standard RL assigns more credit to temporally nearby actions, although outcomes may depend on decisions made hundreds or thousands of interactions earlier.
Source: §1 · p. 1, §3.2 · p. 4Method
EWM uses explanations to identify earlier events contributing to realized outcomes and recursively redistributes reward through a conservative operator.
Source: §1 · p. 2Results
Takeaways & Limitations
Explanations can provide structured credit when temporal propagation becomes inefficient.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Generative Recommenders increasingly use Reinforcement Learning (RL) to optimize outcomes that may appear hundreds or thousands of interactions after a decision. Standard RL propagates reward according to temporal proximity rather than explicit relevance, attenuating the signal assigned to distant but important actions and creating a source of sample inefficiency. We introduce Explanatory World Models (EWM), an attribution framework that uses explanations to identify the earlier events that plausibly contributed to a realized outcome. We instantiate EWM with a frontier language model and use its explanations as learning signals using a conservative hierarchical operator that recursively redistributes reward while preserving the undiscounted episode return. To investigate what constitutes a useful explanation, we operationalize three complementary variants inspired by Halpern's actual causation, Lombrozo's explanatory simplicity, and Deutsch's hard-to-vary explanations. We evaluate EWM on two controlled long-horizon RL benchmarks, a public sequential-recommendation benchmark, and an industry-scale generative recommender. On Crafter and WordCraft, EWM substantially outperforms PPO and LaRe, with its largest gains when rewards are most delayed. On KuaiRand, EWM outperforms SFT and LaRe offline and A2C online. Finally, in an offline evaluation of an industry generative recommender, EWM considerably improves Recall and increases behavioral proxies. These results suggest that explanations can provide structured and reusable credit for learning from delayed outcomes.
1 INTRODUCTION
Long-horizon rewards can arrive hundreds or thousands of decisions after the actions that contributed to them, while standard RL assigns credit by temporal proximity. EWM uses explanations to attribute outcomes to earlier events and redistribute reward to them.
1 INTRODUCTION
- Standard RL discounts distant actions, so identifying which decisions deserve credit for outcomes delayed by hundreds or thousands of interactions can require extensive trial and error.Source: §1 · p. 1
- The paper explores explanation criteria inspired by actual causation, explanatory simplicity, and hard-to-vary explanations.Source: §1 · p. 2
- EWM attributes a realized event or reward to earlier events that explain it, then redistributes reward to those events while preserving episode return.Source: §1 · p. 2
- The conservative hierarchical operator recursively follows explanatory relations to redistribute value, and explanations can be reused across reward functions.Source: §1 · p. 2
- Across Crafter and WordCraft, EWM outperforms PPO and LaRe; it also outperforms SFT and LaRe offline on KuaiRand and improves recall and engagement measures in an industry recommender.The online KuaiRand comparison is against PPO, with performance competitive with an oracle method.Source: §1 · p. 2
2 RELATED WORK
EWM relates to prior work on credit assignment, language-model rewards, world models, and long-term recommendation, but derives attribution from explanations of observed trajectories.
Credit assignment and reward redistribution
- Unlike methods based on learned memory links, return decomposition, or counterfactual attribution, EWM derives relevance from a prior-informed explanation of the realized trajectory and redistributes observed reward.Source: §2 · p. 3
Explanations and language-model rewards
- Unlike LaRe’s latent-reward representation and return decoder, EWM asks which observed events explain an outcome and reallocates an externally specified reward.Source: §2 · p. 3
World models and long-term recommendation
- Unlike forward world models and successor features, EWM is backward-looking and represents reward-independent explanatory relations among observed events.Source: §2 · p. 3
3 EXPLANATORY WORLD MODELS
EWM turns explanations of observed outcomes into weighted event attributions, then uses a conservative hierarchical operator to redistribute rewards before standard RL updates. The framework can use frontier language models and supports online and offline learning.
3.1 WHAT IS AN EXPLANATION?
- EWM treats an explanation as a weighted hypothesis linking preceding events to an observed phenomenon, and evaluates six prompt variants with different explanatory criteria.The variants include HP, inspired by actual causality; D, by hard-to-vary explanations; and LB and LS, by explanatory virtues.Source: §3.1 · p. 4
3.2 CONSERVATIVE HIERARCHICAL REWARD REDISTRIBUTION
- Discounted returns favor temporally nearby actions regardless of their relevance, weakening signals for distant decisions that contributed to an outcome.Source: §3.2 · p. 4
- CHO recursively reallocates reward across eligible predecessors using normalized explanation weights, stopping at depth, eligibility, or threshold boundaries while preserving total reward.Source: §3.2 · p. 5
3.3 FRONTIER MODELS AS EWM FOR RL
- A frontier language model explains realized outcomes from trajectories, and CHO uses the resulting graph to redistribute rewards recursively without additional model calls.Source: §3.3 · p. 6
- EWM inserts an explanation graph and reward redistribution between a trajectory and the RL update; online it processes realized outcomes, while offline it can cache graphs and rewards from logged trajectories.Source: §3.3 · p. 6
4 LONG-HORIZON RL EXPERIMENTS
The long-horizon experiments compare explanation-based redistribution with PPO and LaRe in Crafter and WordCraft, finding its clearest gains when rewards depend on longer or deeper dependencies.
4.1 CRAFTER
- Crafter aggregates technology-achievement rewards into a terminal reward and compares recurrent PPO, LaRe, and depth-1 EWM under shared architecture and training settings.The controlled setup tests learning as rewards become delayed over longer horizons.Source: §4.1 · p. 6
- At L=1000, EWM reaches about 2× LaRe on Crafter, and their gap widens as episode-length caps increase.At L=150, the two credit methods remain close; vanilla PPO stays near the random-policy floor.Source: §4.1 · p. 6, Figure 3 · p. 7
4.2 WORDCRAFT
- WordCraft tests sparse terminal rewards across compositional recipe dependencies, comparing PPO, LaRe, EWM-LLM, and an oracle explainer on two difficulty levels.Difficulty varies distractor count and target-chain depth, with episodes capped at 50 mixing attempts.Source: §4.2 · pp. 6–7
- On WordCraft’s harder 3-mix split, only EWM solves the task, reaching almost oracle score while LaRe barely learns and PPO remains near random.All methods reach maximum return on the easier 2-mix split.Source: Figure 4 · p. 7, §4.2 · p. 7
5 REAL-WORLD EXPERIMENTS
The real-world experiments assess explanation-based recommendation learning offline and online on KuaiRand, and evaluate EWM-reach in an industry recommender using offline replay.
5 REAL-WORLD EXPERIMENTS
- The KuaiRand study covers reward-weighted fine-tuning, online RL, and an industry-scale recommender, with online EWM substantially improving cumulative return over vanilla A2C.The online experiment uses an EWM-inferred disengagement rule; the study also evaluates offline recommendation learning and an industry recommender.Source: §5 · p. 7
5.1 KUAIRAND-OFFLINE
- LB yields the largest mean AUC improvement (1.99 percentage points) and log-loss reduction (5.01 × 10−3), while LaRe has the largest mean differences on four ranking metrics.LB’s mean NDCG@10 difference is close to zero and its mean MRR difference is slightly negative.Source: §5.1 · p. 8, Figure 5 · p. 8
5.2 KUAIRAND-ONLINE
- Mean evaluation return rises from 3.773 for Vanilla A2C to 6.736 for EWM (w5) and 13.249 for EWM (unbounded), versus 15.653 for the Oracle gate.The 5-step gate increases mean session length from approximately 8.1 to 12.3 recommendations and mean reward per step from 0.467 to 0.546; unbounded EWM reaches 30 steps but averages 0.442 reward per step.Source: §5.2 · pp. 8–9, Figure 6 · p. 9
5.3 INDUSTRY GENERATIVE RECOMMENDER
- The industry evaluation derives Reach from recommendation-rooted explanatory graphs and assesses EWM-reach RLFT by offline replay against held-out logged contexts.Table 2 reports fold-changes against SFT; Recall measures reproducing the logged recommendation, while converted and broad reach require a further stream or two.Source: §5.3 · p. 9, Table 2 · p. 9
6 CONCLUSION
The conclusion presents EWM as an explanation-based approach to long-horizon credit assignment, reporting gains across controlled RL, KuaiRand, and an industry-scale recommender.
6 CONCLUSION
- Across Crafter and WordCraft, EWM outperforms PPO and LaRe under sparse rewards.The conclusion characterizes the gains as occurring across the controlled long-horizon benchmarks.Source: §6 · p. 9
- EWM improves over SFT and LaRe in offline KuaiRand experiments and over PPO online.Source: §6 · p. 9
- In an industry-scale generative recommender, EWM increases recall and downstream engagement measures.The paper concludes that explanations can provide structured credit when temporal propagation becomes inefficient.Source: §6 · p. 9
A EWM REDISTRIBUTION ANALYSIS
Figure 7 illustrates how EWM recursively reallocates a conserved episode return, while the KuaiRand offline experiment uses explanation-derived training weights for engagement examples. The section describes the dataset, recommendation inputs, annotation prompts, and weighting pipeline.
A EWM REDISTRIBUTION ANALYSIS
- Across three stone-reaching episodes, depth-1 moves terminal +4.00 onto achievements and depth-2 credits enabling actions, with wood receiving about +1.6 to +1.8.The episode return remains +4.00 after redistribution; depth-1 alone would leave place table and wood pickaxe without credit.Source: Figure 7 · p. 13, A EWM REDISTRIBUTION ANALYSIS · p. 13
B.1.1 DATASET AND PREPROCESSING
- KuaiRand uses 1,000 chronological interaction sessions, split by stored order into 700 training sessions and 300 evaluation sessions.Evaluation targets come from randomly exposed interactions.Source: §B.1.1 · p. 14
B.1.2 RECOMMENDATION MODEL AND INPUT REPRESENTATION
- The recommendation model receives a user’s interaction history and a candidate video, while recorded engagement labels remain unchanged by credit assignment.The target is like or skip, and supervised loss applies to answer tokens rather than history and candidate-description tokens.Source: §B.1.2 · p. 14
B.1.3 CREDIT-ASSIGNMENT MECHANISMS
- Eight conditions include Uniform, LaRe, and six EWM prompts that vary explanation criteria, including Halpern-style counterfactual reasoning and Lombrozo-style breadth or simplicity.Prompts identify up to five history steps plausibly contributing to retention, with importance scores and short explanations.Source: §B.1.3 · p. 14
- Positive importance scores redistribute session outcomes across selected steps; if none is valid, reward stays at the final step, preserving the undiscounted outcome.Discounted returns become exponential training weights, normalized across examples for weighted answer-token cross-entropy in an offline supervised objective.Source: §B.1.3 · pp. 14–15
B.1.4 TRAINING BUDGET
The section specifies offline training and evaluation procedures, then describes the simulated KuaiRand environment, A2C policy, action-gating conditions, and training budget.
B.1.4 TRAINING BUDGET
- Each offline condition trains for one epoch with batch size 4 and AdamW at an initial learning rate of 10^-5, using seeds 1, 2, and 3.All final evaluation snapshots occur at optimization step 5,423.Source: §B.1.4 · p. 15
B.1.5 EVALUATION PROTOCOL
- Offline evaluation reports AUC and log loss on 300 held-out targets, plus HR@10, HR@20, NDCG@10, and mean reciprocal rank for 213 positive-target users.Ranking compares each positive target against 99 sampled unseen videos; bars summarize paired method-minus-Uniform differences across three seeds, with one-standard-error bars.Source: §B.1.5 · p. 15
B.2.1 DATASET AND SEQUENTIAL RECOMMENDATION ENVIRONMENT
- The simulated KuaiRand environment uses is_click feedback, DeepFM-generated training feedback, and an MF-based evaluation reward matrix; episodes end at disengagement or 30 recommendations.These scores are model-based surrogates rather than directly observed engagement from online deployment.Source: §B.2.1 · pp. 15–16
B.2.2 A2C POLICY
- The A2C policy uses a two-layer 64-by-64 network with actor and value heads, separate Adam optimizers, and discount factor 0.9.Training samples actions from the categorical policy; evaluation uses deterministic action selection.Source: §B.2.2 · p. 16
B.2.3 ACTION-GATING CONDITIONS
- The four simulated conditions are Vanilla A2C, EWM gates over the five most recent items or the whole episode, and an oracle gate using the environment’s disengagement window.The discovered rule is fixed before training and applied as an action mask during both rollout collection and evaluation; it changes actions, not action rewards.Source: §B.2.3 · p. 16
B.2.4 TRAINING BUDGET
- The four gating conditions use seeds 2023–2025 and train for 100 epochs targeting 10,000 environment steps per epoch.All conditions have complete 100-epoch records for all three seeds.Source: §B.2.4 · p. 16
B.2.5 EVALUATION PROTOCOL
The reported protocol uses free-running episodes and summarizes each seed over late training epochs. It also reports episode length and reward per step alongside returns.
B.2.5 EVALUATION PROTOCOL
- Primary evaluation is free-running, terminating at the environment’s disengagement rule or the 30-step horizon, with 100 episodes per recorded epoch.Alternative evaluation variants are excluded from the reported results.Source: §B.2.5 · pp. 16–17
- Each seed’s performance is summarized by mean evaluation return over epochs 76–100, with mean and sample standard deviation reported across seed summaries.Mean episode length and reward per step are also reported over the same window.Source: §B.2.5 · p. 17
- Learning curves show unsmoothed three-seed mean returns, with shaded bands marking the minimum-to-maximum seed range rather than confidence intervals.Because each evaluation has the same number of episodes, reward per step equals total evaluation reward divided by total recommendation steps in the window.Source: §B.2.5 · p. 17
B.2.6 RESULTS
In simulated KuaiRand, EWM gates achieve higher cumulative returns than Vanilla, with longer sessions contributing to the gains. The section also describes a MovieLens-1M experiment and its data representation.
B.2.6 RESULTS
- EWM (w5), EWM (unbounded), and the Oracle gate achieve approximately 1.79×, 3.51×, and 4.15× Vanilla’s cumulative return, respectively.Mean returns are 6.736 ± 0.269, 13.249 ± 0.184, and 15.653 ± 0.252, compared with Vanilla’s 3.773 ± 0.143.Source: §B.2.6 · p. 17
- The unbounded gate and Oracle reach the 30-step horizon throughout the final evaluation window, but the unbounded gate likely excludes some allowed high-reward items.Mean seed-level reward per step is approximately 0.442 for EWM (unbounded) and 0.522 for the Oracle gate; the reported gains concern gated policies, not reward redistribution alone.Source: §B.2.6 · p. 17
B.3 MOVIELENS-1M
- The separate MovieLens-1M experiment compares four reward-allocation methods using a global random 90/10 data split rather than chronological or per-user splitting.User and movie features are categorical embeddings, yielding 32-dimensional user and 56-dimensional movie representations.Source: §B.3 · p. 17, §B.3.1 · pp. 17–18
B.3.2 TRAINING AND EVALUATION REWARD MODELS
The appendix specifies the MovieLens reward simulators, sequential environment, PPO setup, and four credit-assignment conditions. It also details how EWM allocates episode rewards and how training and evaluation are conducted.
B.3.2 TRAINING AND EVALUATION REWARD MODELS
- DeepFM supplies dense training rewards, while a six-factor MF model supplies evaluation rewards from the same interaction split, not independent ground truth.DeepFM uses user and movie features; MF uses user and movie identities, so optimizing DeepFM rewards need not maximize MF evaluation return.Source: §B.3.2 · p. 18
B.3.3 SEQUENTIAL RECOMMENDATION ENVIRONMENT
- Episodes sample a user uniformly from 6,040 users, recommend one movie per action, and stop after at most 30 actions.The observation contains the user index and most recent recommendation; the initial movie, when randomized, is not a policy action.Source: §B.3.3 · p. 18
B.3.4 ACTOR INPUT AND STATE REPRESENTATION
- The actor receives a 57-dimensional state formed from the average movie representation and average recent feedback.Pretrained movie embeddings come from the first DeepFM ensemble member; initial feedback is zero.Source: §B.3.4 · p. 19
B.3.5 PPO POLICY
- All algorithms use the same PPO actor–critic, with a shared two-layer 64-unit MLP and separate masked-action and value heads.Training samples actions; evaluation takes the deterministic argmax, with minibatch-normalized advantages and gradient clipping.Source: §B.3.5 · p. 19
B.3.6 TEMPORAL CREDIT-ASSIGNMENT CONDITIONS
- The four learning-reward conditions are dense per-action rewards, terminal lump reward, uniform redistribution, and EWM-inferred step rewards.Lump sums the dense rewards at the terminal step; uniform divides that terminal reward across prior actions.Source: §B.3.6 · p. 19
B.3.7 EWM CREDIT REDISTRIBUTION
- EWM receives a training-split user profile, movie descriptions, and aggregate episode outcome, but not individual step rewards.Invalid judgments retain the lump allocation; successful judgments are cached by prompt version, user, and ordered movie sequence for reuse across sequential training runs.Source: §B.3.7 · p. 19
B.3.8 TRAINING BUDGET
- Training uses seeds 2023, 2024, and 2025, with 100 epochs and 100 episodes per epoch for each condition.PPO uses batch size 1,024 and one optimization pass per minibatch.Source: §B.3.8 · p. 19
B.3.9 EVALUATION PROTOCOL
- Evaluation uses deterministic actions in the MF-based environment, with 100 episodes before training and after each epoch.The recommended main-table report is epoch-100 return per seed and the mean and sample standard deviation across seed-level means.Source: §B.3.9 · p. 20
B.3.10 RESULTS
In MovieLens-1M, EWM outperforms the compared reward-allocation baselines. In anonymized KuaiRand, EWM performs worse than dense baselines when item information is limited.
B.3.10 RESULTS
- EWM outperforms dense feedback, uniform redistribution, lump terminal reward, and anonymized-prompt EWM in MovieLens-1M.Figure 8(a) reports mean evaluation return across seeds, with shading showing the seed minimum–maximum range rather than confidence intervals.Source: Figure 8 · p. 20
- For one MovieLens session, EWM predictions align especially well with MF predictions where MF and DeepFM disagree.The figure describes this as an illustrative example, not an aggregate result.Source: Figure 8 · p. 20
- In anonymized KuaiRand, EWM performs worse than dense baselines when recommendation items lack semantic information.The prompt represents videos primarily through anonymous item and tag identifiers.Source: Figure 9 · p. 20