PaperlayerView exact revision on arXiv

Source-linked AI summary

Hindsight Memory-PRM: Supervising Memory Management with Auditable Hindsight Credit

Haoxuan Jia, Yang Liu, Yingguang Yang, Yancheng Chen, Chongyang Zhang, Hao Zheng, Qian Li, Yulin Huang, Jianshen Zhang, Yongzhi Qi, Shang Luo, Kefu Xu, Hao Peng, Junyu Lu, Du Cheng, Philip S. Yu, Bin Chong

arXiv:2608.29605v1cs.CL

TL;DR

Long-horizon memory operations are difficult to supervise because their value is delayed and episode-level outcomes provide coarse credit. Hindsight Memory-PRM uses logged retrieval, citation, and controlled-deletion evidence to train a utility critic and assign online intervention-calibrated credit, reaching 77.5% on held-out LoCoMo and 79.0% on LongMemEval. The paper reports that intervention-based calibration and multi-version organization underpin gains beyond tested baselines, while noting domain-dependent schemas and other scope limits.

  • Problem

    Long-horizon agents must manage memory operations whose delayed value is not observable when taken, while episode-level outcomes provide coarse supervision across many decisions.

  • Method

    The method uses retrieval, citation, and controlled-deletion records to train an operation-conditioned utility critic offline and assign intervention-calibrated entry credit online within GRPO.

  • Results

    77.5% on LoCoMo and 79.0% on LongMemEval are reported for the resulting local 8B manager, with LoCoMo performance exceeding the API teacher and reproduced baselines.

  • Takeaways & Limitations

    Intervention-based credit improves on denser observational attribution, and the learned manager develops a multi-version memory organization not recovered by tested open-loop controls.

  • Takeaways & Limitations

    The schema is domain-dependent; forgetting is under-stressed, probe generation binds to one closed-source model, and Eq. (6) is conservative rather than an estimator of D_i.

Abstract

from arXiv · show

Memory operations of long-horizon LLM agents are hard to supervise: an operation's value is unobservable when it is taken. But they are special -- they leave machine-readable evidence in the trajectory: retrieval hits and answer-time citations. Hindsight Memory-PRM exploits this audit trail twice: offline to train an operation-conditioned memory-utility critic, and online, where retrievals, citations, and one controlled deletion-and-reanswer per probe settle an intervention-calibrated entry-level presence credit, propagated along version chains as an action-level proxy reward -- no per-operation human labels, no Monte-Carlo replay of continuations. On held-out LoCoMo a local 8B policy reaches 77.5% under a fixed shared reader, surpassing its API teacher (65.1%) and all reproduced external systems, at one eighth the context of Mem0's official operating point; on LongMemEval, 79.0%. Ablations attribute the gain to causal calibration rather than signal density, and the policy converges to a multi-version memory organization whose gains no tested open-loop baseline reproduces.

1 INTRODUCTION

Hindsight Memory-PRM addresses sparse, delayed supervision for memory operations by exploiting retrieval, citation, and intervention evidence in trajectories. It combines offline utility modeling with online calibrated credit and reports gains across long-term memory benchmarks.

  • Motivation: Long-horizon memory operations affect downstream success, but episode-level outcomes assign the same coarse signal to many decisions.Nearly half of rollout groups receive identical exam scores, leaving no within-group gradient for individual operations.
  • Approach: Interventional self-supervision uses retrievals, citations, and controlled deletion tests as machine-readable evidence of delayed memory utility.Autogenerated, source-anchored probes increase evidence density without per-operation human labels or replaying every continuation.
  • Approach: Hindsight Memory-PRM trains an operation-conditioned memory-utility critic offline and assigns entry-level online credit from retrievals, citations, and one deletion-and-reanswer.The measured credit is combined with coverage and annealed critic-shaping terms in the GRPO advantage.
  • Results: 77.5% on LoCoMo is achieved by a local 8B policy, exceeding its API teacher at 65.1%, budget-matched Mem0 at 69.7%, and official Mem0 at 74.7%.The official Mem0 comparison uses 8× as many context tokens.
  • Results: The reward-design ladder rises from 63.4 to 70.2 to 77.5, while open-loop and rule-based controls do not recover the full benefit.The result also holds on LongMemEval at 79.0%, and the policy develops a multi-version organization.
  • Contributions: The paper contributes hindsight relabeling, intervention-calibrated GRPO credit, controlled comparisons, and analysis of multi-version organization and transfer.The contribution scope covers two long-term memory benchmarks and examines schema sensitivity across benchmarks.

2 PRELIMINARIES AND PROBLEM SETUP

The setup models an agent that makes turn-level memory-management decisions over a deterministic external bank while receiving only an episode-level QA reward. Sparse outcome supervision creates a credit-assignment burden that grows with the number of operations.

  • Setup: At turn t, the agent observes o_t and applies a memory action a_t ∈ A_mem to bank M_t, with deterministic transitions and end-of-episode QA accuracy R(τ) ∈ [0, 1].Actions include WRITE, MERGE, and NOOP; READ is executed deterministically by the retriever.
  • Setup: The memory bank contains event, profile, and automatic verbatim layers, and each turn triggers one write gate per managed layer.MERGE appends to an entry’s retained version chain, while the automatic verbatim layer preserves every turn.
  • Credit assignment: With K memory operations and additive outcome contributions, a baselined policy gradient weights every operation by the same episode-level term R(τ) − b.The formulation assumes individual contribution variance σ² and pairwise uncorrelated contributions.
  • Credit assignment: Sample size for fixed precision scales linearly with K under the stated additivity and zero-covariance assumptions.The measured mean is approximately K ≈ 37 gate decisions in this setting.
  • Motivation: The method is designed to reduce sparse-outcome credit burden rather than recover the ideal per-action difference reward.The setup motivates finer attribution for memory-RL methods trained only with episode-level outcomes.

3 HINDSIGHT MEMORY-PRM

Hindsight Memory-PRM turns auditable retrieval and answer-support traces into ordinal memory-utility supervision, then uses verified intervention credit and critic shaping for online optimization. Its design combines zero detection, targeted probes, tiered relabeling, and conservative temporal credit.

  • Overview: The training loop rolls out memory management, relabels logged operations into utility targets, trains critic PRMM, and combines measured credit with task reward in GRPO.The critic is trained offline on teacher trajectories, then frozen for LoCoMo or periodically refreshed for LongMemEval.
  • Auditable evidence: Difference-reward replay is avoided because retrieval logs record recalls and citations record support for answers already produced.These traces make parts of counterfactual attribution cheap without replaying every continuation.
  • Zero detection: Under the retrieval invariant, an entry absent from every logged read-path candidate set has token-identical contexts before and after deletion.On LoCoMo, 74.6% of entries satisfy this condition, certifying zero contribution without replay.
  • Probe construction: Source-anchored probes are gated by prior unanswerability, anchor support, and lack of support from remaining turns.Removing the anchor turn makes 98.7% of gated probes incorrect on development re-tests.
  • Hindsight relabeling: A weighted sum of retrieval recall, outcome contribution, and counterfactual gain performs poorly, with development Spearman never exceeding 0.10.The method instead uses citation calibration, content-cluster labels, and monotone ordinal tiers.
  • Utility critic: The critic ranks operations in context using decision-time context, bank-state rows, and candidate operations; a hinge ranking term improves development rank correlation sixfold.Separate write, merge, and NOOP adapters share a frozen backbone with oracle routing.
  • Verified credit: Online optimization uses session episodes, shared rollout exams, one final-bank writeback, and verified deletion-and-reanswer tests to assign signed ordinal credit.A correct-to-wrong flip marks necessity, while a wrong-to-correct flip marks misleading evidence; non-flips receive an intermediate cap.
  • Composite reward: The composite step reward combines task outcome, miss penalties, measured credit, and annealed critic shaping, with suffix-min pessimism for memory actions.NOOP receives its expert score directly, while isolated high critic scores cannot inflate memory credit through the suffix minimum.

4 EXPERIMENTS

Experiments show that audit-based, intervention-calibrated memory supervision improves long-horizon management across benchmarks, budgets, ablations, transfer, and learned organization. The gains depend on both critic coverage and measured attribution, while domain-specific schemas and deletion remain important boundaries.

  • 4.3 CONSTRUCTION OF THE CREDIT SIGNAL: Intervention-calibrated branches add 7.3 points beyond observational attribution, while observational signals add 6.8 points beyond sparse outcome feedback.The comparison isolates intervention-based information within the shared scaffold.
  • 4.4 VALIDITY OF THE CRITIC: Deleting top-scored entries costs 9.6 points versus 2.1 for random and 0.8 for bottom-scored entries, supporting critic validity beyond ranking correlation.Six pre-specified development checks also pass, including miss AUC 0.951 and mis-attach AUC 0.985.
  • 4.5 ABLATIONS: The critic and audited terms have approximately additive effects, costing 6.5 and 8.1 points respectively when removed from the full method.Their simple effects relative to removing both are 6.0 and 7.6 points, with an interaction of +0.5.
  • 4.6 SECOND BENCHMARK, CROSS-BENCHMARK TRANSFER, AND SAMPLE EFFICIENCY: 79.0% on LongMemEval extends the pattern, with gains of 18.0 points over the untrained backbone and 13.0 over MemBuilder.The ladder reaches 66.0/72.5/79.0 on 400 independent units.
  • 4.6 SECOND BENCHMARK, CROSS-BENCHMARK TRANSFER, AND SAMPLE EFFICIENCY: Direct transfer falls from 77.5 to 58.4, but rebuilding a domain-appropriate schema without policy retraining closes 62% of the gap to native LongMemEval training.The audited policy retains 90% of native accuracy under the adapted schema, versus 67% for outcome-only training.
  • 4.6 SECOND BENCHMARK, CROSS-BENCHMARK TRANSFER, AND SAMPLE EFFICIENCY: The policy reaches its plateau after roughly 120 rollouts and finishes at 1,712, whereas outcome-only training uses 28,000 rollouts to reach 6% of that gain.Both comparisons use the same SFT initialization.

5 RELATED WORK

Related memory systems differ in construction and retrieval, but existing managers generally lack operation-level training from later retrieval and answer support. Trained managers instead use outcome scalars or observationally attributed rewards.

  • 5 RELATED WORK: Closest prompt-based managers do not train each operation using later retrieval and answer support.The related systems include Mem0, A-Mem, and MemBuilder.
  • 5 RELATED WORK: Trained managers use outcome scalars or observationally attributed rewards rather than the paper’s intervention-based attribution.The passage contrasts these approaches with the proposed direction.

6 CONCLUSION

Hindsight Memory-PRM uses auditable retrieval, citation, and controlled-deletion evidence to supervise memory management. Across two benchmarks, intervention-based credit improves attribution and yields multi-version memory organization.

  • Retrieval, citation, and controlled-deletion records train a memory-utility critic offline and assign intervention-calibrated presence credit online.
  • Across two long-term memory benchmarks, intervention-based credit improves over denser observational attribution.
  • The learned manager develops a multi-version memory organization.
  • Extending auditable hindsight supervision to delayed actions such as caching or planning remains future work.

A PROOF OF PROPOSITION 1

The proposition bounds how critic shaping can separate the shaped objective from the design objective. Annealing contracts this worst-case gap, while optimization error enters additively.

  • Setup: The design objective is J(θ) := Eτ∼πθ[R(τ)], while the shaped objective adds critic-based terms to the non-learned reward signals.
  • Uniform bound: Bounded critic outputs and at most Ntot shaped action steps imply a uniform trajectory shaping bound of B Ntot.
  • Objective gap: A maximizer of the shaped objective is 2βk-optimal under the design objective.
  • Annealing: Annealing makes the bound contract monotonically to 2λ∞BNtot, and it is zero when λ∞ = 0.
  • Practical bound: With λ∞ = 0.2, the worst-case bound is 17.6 reward units, while late-training measured shaping share falls to 1.8% of total absolute reward mass.
  • Approximate maximization: If training reaches only an ϵopt-approximate shaped maximizer, optimization error enters the design-objective gap additively.

B PROOF OF PROPOSITION 2

The suffix-minimum shaping rule is pessimistic and robust to bounded critic perturbations, rewarding improvements to bottleneck operations rather than isolated score inflation. Its formal properties distinguish it from the sum-credit alternative used in ablation.

  • Definition: For memory-action steps, suffix-minimum credit is defined as ˜st = mins∈I, s≥t ps, while the NOOP branch bypasses this rule.
  • Pessimism: The suffix minimum is pessimistic: credit never exceeds the current step’s critic score, and later high scores cannot lift earlier credit.
  • Stability: The suffix minimum is 1-Lipschitz, so bounded critic perturbations cause no larger credit perturbation.
  • Anti-inflation property: Non-minimizing operations have zero local sensitivity within their score gap, so isolated non-bottleneck inflation leaves total shaping unchanged.
  • Anti-inflation property: Raising total credit requires improving an operation that attains a suffix minimum, closing the compensate-one-low-step-with-one-high-step channel.
  • Ablation: In the sum-credit arm, any single-point inflation increases total credit linearly, and the end-to-end difference is 11.4 points.
  • Training pipeline: The implementation trains an operation-conditioned critic offline, then uses measured rollout credit and critic shaping in GRPO policy optimization.

E EXTENDED EXPERIMENTAL SETUP

The experiments use held-out, dialogue-level benchmark protocols with fixed readers and controlled component comparisons. They also test robustness, reward design, and training horizon under explicit cost and reproducibility constraints.

  • Robustness and capacity: 67.2 versus 67.5 shows that pinning high-dependency memories does not improve the unpinned full method.Retrieval misses account for 13 of 89 errors.
  • Benchmark protocols: 38.0 to 52.0 is the retrieval-only range on LongMemEval, where dense retrieval with k=50 is the strongest listed reader.The 400 questions are independent units under MemBuilder’s protocol.
  • Benchmark protocols: 975 LoCoMo test questions span D6–D10, while LongMemEval uses 400 test questions after separate training and development splits.Development sets support checkpoint selection and design checks; test sets are reserved for final evaluation.
  • Evaluation controls: The evaluation fixes construction, answering, judging, and service endpoints across prompt-based systems, while local 8B rows use the evaluated policy as constructor and a shared API reader.The table also reports actual answering-context tokens because entry lengths differ.
  • Reward-design controls: 70.6 ± 0.8 versus 70.2 shows that matching observational reward scale does not explain the full method’s gain.A shuffled-flip placebo reaches 70.5 ± 0.9 while preserving tier frequencies.
  • Training cost and horizon: Approximately 30–50% wall time is added per step by online counterfactuals, and the method uses approximately 1.6× as many API calls per rollout as outcome-only training.LongMemEval uses refresh mode after 2,400 steps, while LoCoMo uses frozen mode for 214 steps.

F THE THREE GATES FOR SYNTHETIC PROBES

Synthetic probes turn source-anchored memory questions into controlled deletion tests through three behavioral gates. The section specifies their operational interpretation, calibration, acceptance checks, and construction limitations.

  • Gate construction: Three gates require that the answer is not available from priors, supported by the anchor turn, and unsupported by remaining turns.Passing all gates provides model-relative operational evidence for the deletion label uCF.
  • Yield and limitations: 51–59% of generated questions pass the gates overall, with aggregation probes passing at 33%.Pass rates are 97% adversarial, 72% profile, and 53% factual by type.
  • Gate validation: 98.7% of gated probes become incorrect after anchor removal on development re-tests.The reported rate comes from majority outcomes over three independent re-answers.
  • Deletion protocol: Source-turn mapping supports one-to-many and many-to-one mappings while deleting exactly the entry anchored by each synthetic probe.Auxiliary cited entries are retained, and multiple citations do not trigger joint deletion.
  • Ablations: 73.9 ± 0.9 results when source-turn mapping is removed, because misses cannot be localized and anchored deletion cannot be executed.This ablation identifies source anchoring as a contributor to usable supervision density.
  • Critic acceptance: 0.55 Spearman is the development ranking threshold, alongside AUC, cross-expert scale, style-adversary, and deletion-causality checks.The checks are fixed before critic training, and D5 is excluded from critic training.

H ANTI-EROSION: IMPLEMENTATION AND RECORDS

The anti-erosion design limits proxy-reward drift through bounded shaping, refresh checks, rollback, and attack evaluations. Calibration experiments also separate citation, retrievability, clustering, ranking, and schema adaptation effects.

  • Reward monitoring: 9.6% to 1.8% is the shaping share reduction across the three reported training phases.Shaping is the only learned proxy among the four reward components.
  • Refresh and rollback: Every 200 steps, the critic refreshes from replay samples and rolls back if development metrics fall below floors or historical tolerances.The update is anchored toward the offline snapshot.
  • Refresh and rollback: 0.62 to 0.66 is the rise in LongMemEval development ranking correlation across 12 refresh points, with one rollback triggered.The audit metric detects degradation relative to development data but cannot rule out shared critic-distribution drift.
  • Attack surface: 3.1–9.4 points below control is the score range for five audit-signal manipulations.Query copying and oversized entries increase retrieval but reduce answer correctness; citation baiting and prompt injection do not reach the judge.
  • Target calibration: Development Spearman never exceeds 0.10 across weighted label variants and a 94× capacity range.Retrievability and counterfactual gain correlate at ρ = 0.05, while manual importance and counterfactual gain correlate at ρ = −0.07.
  • Target calibration: 100% citation recall and 65% reverse precision make citation a negative filter rather than a positive utility criterion.Citation restricts expensive counterfactual measurement to candidate entries.
  • Target calibration: 0.083 to 0.533 is the development Spearman increase from pure regression to regression-plus-ranking on identical labels.Clustering reduces high-similarity, high-label-gap pairs from 145 to 18.
  • Schema adaptation: The LongMemEval schema adapts layer content and write rules while keeping credit construction, critic architecture, and optimization fixed.Adaptations address speaker asymmetry, mutable attributes, broader subjects, and procedural content.

K PER-DIALOGUE RESULTS AND BLOCK-LEVEL STATISTICS

Block-level evaluation treats the five LoCoMo dialogues as the independent units and reports paired comparisons, uncertainty, and online-cost accounting. The results extend to independent LongMemEval questions with positive gains across all five types.

  • Statistical unit: Table 14 reports 3-seed mean accuracy for each of the five LoCoMo test dialogues.The 975 questions are nested within shared dialogues, so dialogue blocks—not questions—are the independent units.
  • LoCoMo block results: 77.5% is higher than Mem0’s official k=200 result on all five LoCoMo dialogue blocks.The one-sided paired sign test gives p = 0.031; the 95% bootstrap interval for the mean is [76.1, 78.6].
  • LoCoMo block results: Every paired LoCoMo slope rises toward the method across all five dialogues, matching the 5/5 sign-test wins.Figure 3 visualizes the per-dialogue paired slopes from three-seed means.
  • Cost accounting: 14 points higher final accuracy accompanies approximately 0.55M calls for the method versus 1.34M for outcome-only training.The approximately 1:50 ratio applies only to online calls at a shared threshold, not total experimental cost.
  • LongMemEval results: 15.0 points is the paired LongMemEval difference versus Mem0 across 400 independent questions, with paired bootstrap 95% CI [11.2, 18.8] and McNemar p < 10^-6.Gains are positive in all five question types and concentrate in multi-session reasoning (+19) and knowledge update (+21).

L CONTROL MATRICES FOR CHAIN CREDIT AND RETRIEVAL SCAFFOLD

The controls test whether chain-credit broadcasting and retrieval scaffolds explain the reported gains. Results indicate that useful consolidation depends on closed-loop feedback and that credit sharing is not the sole source of multi-version hubs.

  • Chain-credit controls: 84% of answer flips in 200 multi-version entries came from cited versions, concentrating hub value in cited rather than uniformly distributed versions.Multi-version organization persisted under controls, while normalization retained similar structure and accuracy.
  • Chain-credit controls: Disabling credit sharing lowered both merge rate and accuracy, suggesting chain sharing encourages useful consolidation despite overestimation bias.The controls make credit broadcasting unlikely to be the sole cause of multi-version hubs.
  • Action dynamics: The merge share rose fivefold from 3.0% to 14.5% during LoCoMo training, overtaking write by approximately step 50 as writing halved.NOOP settled at the approximately 30/37 level reported in §4.5, showing reallocation toward consolidation rather than increased writing.
  • Retrieval-scaffold controls: The verbatim layer and per-version retrieval interface each contributed approximately 2–3 points, insufficient to explain the 7.8-point gap to the strongest external system.Rule-based merging reached a similar version count of 5.7 but only 66.5% accuracy, while whole-chain versus hit-version-only return differed by about 1.9 points.
  • Attribution validation: Entry-level presence credit correlated with replay marginals at Spearman 0.72 across 300 operations, with correlations of 0.78 for WRITE and 0.61 for MERGE.The replay replaces each selected action with NOOP under the same seed and decoding configuration.
  • Reader robustness: With GLM-5-Chat as answerer and judge, accuracy was 74.8 ± 0.4 for the method, versus 67.9 ± 0.5 for Mem0 and 61.2 ± 0.4 for outcome-only training.The banks were unchanged while retrieval, answering, and judging were rerun over three re-evaluation seeds.
Loading 2608.29605v1…