Source-linked AI summary
MemRiskBench: Trace-Aware Risk-Preserving Evaluation for Long-Horizon LLM Agents
Jianhua Jiang, Dongbo Yuan, Weihua Li
TL;DR
Long-horizon agents accumulate memory across sessions, but aggregate scores can obscure memory-specific risks and per-risk failure rates. MemRiskBench evaluates those risks with deterministic trace checks and a risk-preserving subset selector. At a 20% subset size, the selector preserves ranking, risk coverage, and high-risk detection while reducing compute 5×.
Problem
Aggregate evaluation scores provide no explicit per-model, per-risk pass rates, while memory-specific failures can be sparse and high impact.
Source: §1 · p. 2Method
Results
At a 20% subset size, ranking, risk coverage (1.00), and high-risk detection (1.00) are preserved while compute is reduced 5×.
Source: §1 · pp. 2–3Takeaways & Limitations
The benchmark supports model iteration under a fixed evaluation budget and auditing existing models via deterministic traces.
Source: §9 · p. 13Summary contamination (R6) is excluded from main scoring because its contamination surface area is too broad for a first benchmark.
Source: §8.8 · p. 13
Abstract
from arXiv · showhide
Long-horizon LLM agents accumulate memory across sessions, creating sparse but high-impact risks: stale facts, conflicting updates, cross-user leakage, revoked-memory reuse, and constraint decay. Standard aggregate scores hide per-risk failure rates--a model achieving 78% average accuracy may still leak data in 4% of episodes--and benchmark compression preferentially discards the rare high-severity events that distinguish a mostly-working model from one that occasionally causes harm. We present MemRiskBench. The primary contribution is a five-category risk taxonomy (plus one documented, unscored category) operationalized by deterministic trace grounded checks, instantiated as a 120-episode scripted benchmark with full trace logging and no LLM-as-judge on the pass/fail path, evaluated on five locally run quantized instruction-tuned models. Second, a risk-preserving subset selector: a coverage-constrained greedy selector on deterministic trace-derived features that retains full ranking (Spearman rho = 0.975, deterministic; CI collapses to a point estimate with zero bootstrap variance), risk coverage (1.0), and high-risk model detection (1.0) at a 20% subset size, reducing compute 5x. Unlike ranking-only subset selectors, this selector additionally preserves risk-type coverage and high-risk detection using trace-grounded deterministic features that do not require an LLM judge. All episodes, traces, the scoring implementation, and the selector are released to support reproducible evaluation and risk assessment of deployed LLM agents
1 Introduction
MemRiskBench targets memory-specific harms that aggregate task scores and conventional benchmark compression can obscure. It pairs a trace-scored, five-risk benchmark with a subset selector designed to preserve rankings and risk signals at lower evaluation cost.
1 Introduction
- Aggregate task success can conceal sparse high-impact failures, including cross-scope leakage and constraint decay, while ordinary subsampling may drop rare severe events.The motivating example contrasts 78% average task success with 4% leakage and 25% constraint decay.Source: §1 · p. 2
- Prior memory benchmarks measure remembering through QA or extraction, while compression work targets ranking fidelity rather than preserving high-severity failure modes.The paper distinguishes its memory-violation focus from output-level harm taxonomies.Source: §1 · p. 2
- At a 20% subset size, the selector preserves ranking, risk coverage (1.00), and high-risk detection (1.00), reducing compute 5×.The benchmark comparison reports the subset retains the full benchmark’s ranking, coverage, and high-risk detection.Source: §1 · pp. 2–3
- The benchmark combines five risk categories, 120 scripted episodes, trace-grounded deterministic scoring, and evaluation on five locally run quantized models.The pass/fail path uses no LLM-as-judge; the contribution also reports a retrieval-filtering baseline performing worse than no filtering.Source: §1 · pp. 2–3
- The introduction previews a pipeline of episode execution, deterministic scoring, and risk-preserving selection across 120 episodes and five local models.This yields 600 episode-model pairs with full traces.Source: §1 · pp. 2–3
2 Related Work
Related work measures memory capability, compresses benchmarks to preserve rankings, or catalogs output-level harms. MemRiskBench distinguishes itself by evaluating memory violations and preserving risk signals alongside rankings.
2.1 Memory-capability benchmarks.
- Memory-capability benchmarks such as LongMemEval (Wu et al., 2025), LoCoMo, and MemBench focus on retrieval recall or QA accuracy, not multi-session risk.Other cited examples include MemoryAgentBench (Hu et al., 2025) and BEAM (Tavakoli et al., 2025).Source: §2.1 · p. 3
2.2 Benchmark compression and subset selection.
- Benchmark compression methods target ranking fidelity, while MemRiskBench adds risk coverage and high-risk detection; risk-stratified sampling scored Spearman 0.698 and high-risk detection 0.345.Ndzomga (2026) propose a 30%–70% historical-pass-rate filter, achieving Spearman ρ ≈0.94 while reducing task counts by 44%–70%.Source: §2.2 · pp. 3–4
2.3 Harm taxonomies and safety evaluation.
- Weidinger et al. (2024) catalog output-level sociotechnical risks, whereas MemRiskBench evaluates memory violations as the unit of evaluation.The paper characterizes its approach as adding a process-level view of when and how memory causes harmful failures.Source: §2.3 · p. 4
3 Risk Taxonomy
The taxonomy defines five deterministically scored memory risks and a sixth documented but unscored risk. The risks cover outdated, conflicting, out-of-scope, revoked, and constraint-violating memory use, plus contamination of persistent summaries.
3 Risk Taxonomy
- R1–R5 are deterministically scored, using text, tool, memory, and constraint checks; R6 summary contamination is documented but unscored.The checks include string containment or regex, call verification or argument equality, memory usage/status/scope, and constraint activity or violation monitoring.Source: §3 · p. 4
3.1 R1: Stale memory reliance.
- R1 occurs when an agent uses a stored value despite a newer valid update superseding it.Source: §3.1 · p. 4
3.2 R2: Conflicting memory misresolution.
- R2 occurs when incompatible facts coexist and the agent violates the episode’s resolution rule or blends the facts.Resolution rules may use timestamp, trusted source, explicit correction, or scope.Source: §3.2 · p. 4
3.3 R3: Cross-scope leakage.
- R3 is unauthorized cross-user, cross-scope, or cross-memory information appearing in an unauthorized context.Source: §3.3 · p. 4
3.4 R4: Revoked memory reuse.
- R4 occurs when an agent reuses memory after deletion, revocation, or invalidation.Source: §3.4 · p. 5
3.5 R5: Constraint decay.
- R5 occurs when an agent violates a safety, policy, or task constraint after unrelated turns.Source: §3.5 · p. 5
3.6 R6: Summary contamination (documented, not scored).
- R6 covers noisy, malicious, or incorrect content compressed into summaries that later influence behavior, but it is not scored in the main benchmark.Integrating it into main scoring requires a hybrid deterministic–LLM evaluation pipeline.Source: §3.6 · p. 5
4 Benchmark
MemRiskBench evaluates scripted memory-risk episodes with event-level traces and deterministic scoring. It also reports scoring revisions, a safety-aware refusal variant, and the released evaluation artifacts.
4.1 Episode structure.
- An episode is a directed acyclic graph of events, with memory, tool-call, constraint, and message state logged at each decision point.At each decision point, the agent produces a tool call, final answer, or clarification question.Source: §4.1 · p. 5
4.2 Determinism and scoring.
- Each episode receives a boolean success flag, a score from 0 to 1, violation and per-risk evidence, and per-check results from deterministic scoring.The pass/fail path uses no LLM-as-judge; each episode has 1–3 required checks, and its score is the fraction passed.Source: §4.2 · pp. 5–6
4.3 Strict-matching revision (control analysis).
- Revised expected values changed scores for 39 of 600 trace-episode pairs, while preserving within-risk model ordering and the headline size–risk interaction.The authors re-scored existing traces, not model reruns, and released both original and revised values with a per-model, per-risk audit.Source: §4.3 · p. 6
4.4 Safety-aware refusal scoring (protocol).
- A safety-aware variant counts refusal without a tool call as a safety-pass for R3 and R5, alongside the default scoring convention.The release includes per-model, per-risk pass rates under both conventions.Source: §4.4 · p. 6
4.5 Release contents.
- The release includes episode definitions, the scorer, trace schema, subset selector, bootstrap audit script, and raw traces for five models across 120 episodes.The raw release contains 600 traces.Source: §4.5 · p. 6
5 Risk-Preserving Subset Selection
The selector chooses a fixed-size episode subset to preserve ranking, risk-type coverage, and high-risk model detection. It uses trace-derived features and greedy forward selection, with performance tied to the reference-model set.
5.1 Problem statement.
- A risk-preserving subset of k episodes maximizes a combined objective for ranking fidelity, risk-type coverage, and high-risk model detection.The objective weights all three terms equally; high-risk models are those whose pass rate ranks in the bottom two of five.Source: §5.1 · p. 6
5.2 Episode feature vector.
- Each episode has a 12-dimensional trace-derived feature vector covering difficulty, discrimination, failure density, and complexity.The discrimination feature is cross-model pass-rate spread; the other features capture tier and trajectory length, model failures, risk types, tool calls, and constraints.Source: §5.2 · p. 7
5.3 Discrimination feature and cross-validation.
- Leave-one-model-out cross-validation computes discrimination features from four reference models before evaluating the held-out model’s rank.The resulting rankings are consistent with features computed using all five models.Source: §5.3 · p. 7
5.4 Selector algorithm.
- Greedy forward selection adds the episode that most improves the objective; six baselines include random, risk-stratified, difficulty, disagreement, and clustering selectors.Stochastic baselines use 1,000 bootstrap resamples, while deterministic baselines return identical subsets and point-estimate intervals.Source: §5.4 · p. 7
5.5 Reference-set dependence.
- The selector relies on the v3 reference ranking, and performance is not claimed to transfer equivalently to a new model family without a reference set.Practitioners can rebuild the ranking on their own models, at the cost of recomputation; offline discrimination estimation is left to future work.Source: §5.5 · p. 7
6 Experiments
Across five local quantized models, aggregate task success coexists with sharply different per-risk outcomes. At 20% subset size, the risk-preserving selector retains ranking, coverage, and high-risk detection; its ranking signal comes from cross-model pass-rate spread.
6.1 Models. / 6.2 Protocol.
- The protocol evaluates five locally run quantized instruction models and reports 95% percentile bootstrap intervals over 1,000 resamples.Qwen2.5-7B uses Q3 quantization because the 16 GB GPU cannot hold a Q4 model alongside the prompt cache.Source: §6.1 · pp. 7–8, §6.2 · p. 8
6.3 RQ1: Aggregate scores hide per-risk failure rates.
- Average task success spans 0.775 to 0.885 across five models, while Qwen2.5-1.5B passes only 1 of 24 cross-scope leakage episodes.The leakage comparison does not survive Holm correction; constraint decay shows an inverse pattern, with 1.5B obeying all 24 explicit constraints while larger models and Phi violate some.Source: §6.3 · p. 8
6.4 RQ2–RQ3: Risk-preserving compression.
- At 20% subset size, the selector achieves Spearman 0.975, risk coverage 1.0, and high-risk detection 1.0; stratified sampling reaches Spearman 0.698 and high-risk detection 0.345.The selector’s deterministic confidence interval collapses to a point estimate; stratified sampling misses at least one high-risk model in half of resamples.Source: §6.4 · pp. 8–9, §6.3 · p. 8
6.5 RQ4: Feature ablation.
- Removing cross-model pass-rate spread drops Spearman to 0.308, while removing any other feature leaves it at 0.975.Coverage remains stable because the v3 reference set meets risk-coverage objectives at k ≥ 24.Source: §6.5 · p. 9
6.6 RQ5: Held-out-model generalization (concept verification).
A held-out-model check selects a 24-episode subset from four reference models, while the paper also reports refusal-scoring sensitivity and a retrieval-level baseline. The held-out comparison is descriptive rather than a statistical generalization claim.
6.6 RQ5: Held-out-model generalization (concept verification).
- With Qwen2.5-1.5B held out, the selected 24-episode subset preserves reference-model ranking at Spearman ρ = 1.000 and covers all five risk types.The subset yields a 0.812 held-out average pass rate versus 0.775 on the full benchmark, but the authors treat this composition-sensitive comparison as descriptive only.Source: §6.6 · p. 10
6.7 Sensitivity analysis: safety-aware refusal scoring.
- Safety-aware refusal scoring changes per-model, per-risk pass rates only for refused R3 and R5 episodes; both default and safety-aware numbers are released.A full quantitative sensitivity table is left to the online repository.Source: §6.7 · p. 10
6.8 Memory baseline.
- RAG and recency filtering perform worse than no filtering on MemRiskBench.The authors conclude that memory risk requires dedicated evaluation rather than off-the-shelf retrieval.Source: §6.8 · p. 10
7 Analysis
Per-risk results show notable inversions across model size and family, while difficulty effects vary by model. The selector targets risk coverage and high-risk detection in addition to ranking fidelity.
7.1 Per-risk failure patterns.
- Qwen2.5-1.5B obeys all 24 explicit constraints while larger models violate 2–3; Phi-3.5-mini leads Qwen models on leakage and revoke, with slightly worse constraint decay.The 3B model passes 19/24 revoke episodes versus 18/24 for 7B, but quantization effects cannot be fully excluded.Source: §7.1 · pp. 10–11
7.2 Difficulty scaling.
- For 3B and larger models, pass rates drop monotonically from easy to hard, while Qwen2.5-1.5B shows no further drop at hard.The hard tier therefore does not expose a gap that size would close for Qwen2.5-1.5B.Source: §7.2 · p. 11
7.3 Selector positioning relative to concurrent work.
- Unlike the Mid-Range Difficulty Filter, the selector targets risk-type coverage and high-risk model detection alongside ranking, using trace-derived features and a 20% subset.The approaches are described as complementary; comparison under a common protocol is left to future work.Source: §7.3 · p. 11
8 Limitations
The evaluation is constrained by simulated tools, narrow model coverage, and scoring and selector design choices. Several comparisons and generalization claims therefore remain bounded by the benchmark setup.
8.1 Simulated tool environment.
- Episodes use deterministic scripted stubs rather than real browsers, email clients, or office suites; a real-environment follow-up could extend the taxonomy to adversarial user behavior.The proposed follow-up concerns settings beyond scripted stubs.Source: §8.1 · pp. 11–12
8.2 Model family and size coverage.
- The five evaluated models are local and under 8B parameters, with three from Qwen; no frontier-class API models were evaluated.The size range reflects a 16 GB consumer-GPU constraint; Qwen2.5-7B at Q4 K M and 13B+ models exceed the paper’s scope.Source: §8.2 · p. 12
8.3 Quantization confound.
- Qwen2.5-7B uses Q3 K M while the other models use Q4 K M, confounding stale and revoke rankings; a Q4-only rerun may alter them.The authors interpret the constraint-decay inversion as real but cannot fully exclude quantization effects.Source: §8.3 · p. 12
8.4 Strict string matching and label-prefix revision.
- Revising label-prefixed expected values changed 39 of 600 scores, while the release preserves both versions and an audit showing unchanged within-risk model ordering.The label prefix was stripped on 19 of 20 affected episodes, without rerunning models.Source: §8.4 · p. 12
8.5 Safety-aware refusal scoring.
- Default scoring treats refusals as failures, although refusal is safety-correct in R3 and R5; a safety-aware variant counts those refusals as passes.The release ships both sets of numbers.Source: §8.5 · p. 12
8.6 RQ5 generalization.
- Held-out-model evaluation used one smallest model and a subset skewed toward conflict episodes, so RQ5 is concept verification rather than statistical generalization.A multi-model, balanced held-out study is planned.Source: §8.6 · p. 12
8.7 Reference-set dependence.
- The selector depends on the v3 reference ranking, so a new model family may require recomputation; no automated reference-set drift detector is provided.The feature matrix and selector are released to support recomputation.Source: §8.7 · p. 12
8.8 R6 not scored.
- R6 summary contamination is documented but excluded from main scoring because its surface area was considered too broad for this first benchmark.R6 integration is planned for a future version.Source: §8.8 · p. 13
9 Conclusion
MemRiskBench makes long-horizon agent memory risks measurable under controlled, scripted conditions. Its selector preserves ranking, risk coverage, and high-risk model detection at reduced compute, while several extensions remain open.
9 Conclusion
- At a 20% subset size, the selector retains ranking (Spearman 0.975, deterministic; zero bootstrap variance), risk-type coverage, and high-risk model detection while reducing compute 5×.The paper identifies larger and more diverse benchmarks, real-environment extensions, and paired defense studies as open directions.Source: §9 · p. 13
Broader Impact and Ethical Considerations
The benchmark is intended for diagnostic use under controlled, synthetic conditions, not as an evaluation of deployed systems. The paper also identifies potential dual-use and broader ethical-scope boundaries.
Broader Impact and Ethical Considerations
- The benchmark uses synthetic episodes and involves no real users, data, or deployed systems; its stated purpose is diagnostic use by developers and auditors.The paper does not address broader societal implications such as deployed-assistant privacy, consent, or data retention.Source: Broader Impact and Ethical… · p. 13, §9.2 · p. 14
9.1 Dual-use risk.
- Released episode definitions could probe commercial agents; mitigations include intended-use documentation, withholding adversarial traces, and recommending deployment-monitored traces alongside scripted scores.The authors caution practitioners against relying solely on scripted evaluation.Source: §9.1 · p. 14
9.2 Limitations of ethical scope.
- The paper does not address broader societal implications of long-horizon memory systems, including deployed-assistant privacy, consent, and data retention.Source: §9.2 · p. 14