Source-linked AI summary
STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?
Hanxiang Chao, Yihan Bai, Rui Sheng, Tianle Li, Yushi Sun
TL;DR
Long-term assistant memory lacks systematic evaluation of implicit conflicts, where later observations invalidate earlier beliefs without explicit contradiction. This paper introduces STALE to test state resolution, premise resistance, and policy adaptation, finding that recognizing stale memories does not reliably yield application, with the strongest model reaching 55.2% overall accuracy.
Problem
Long-term assistant memory lacks systematic evaluation of implicit conflicts, where later observations invalidate earlier beliefs without explicit contradiction.
Method
STALE models long-term memory as latent user-state tracking and evaluates 400 expert-validated conflict scenarios through 1,200 queries across three probing dimensions.
Results
Across systems, recognizing outdated memories did not reliably yield application, with propagated conflicts and stale presuppositions remaining challenging; Gemini-3.1-pro achieved 55.2% overall accuracy.
Takeaways & Limitations
The findings support explicit write-side state adjudication as a promising direction for robust agentic memory.
Takeaways & Limitations
STALE evaluates one-shot implicit state transitions with single conflict pairs, rather than repeated updates, coupled propagation, or gradual state drift in open-ended interactions.
Abstract
from arXiv · showhide
Large Language Model (LLM) agents are increasingly expected to maintain coherent, long-term personalized memory, yet current benchmarks primarily measure static fact retrieval, overlooking the ability to revise stored beliefs when new evidence emerges. We identify a critical and underexplored failure mode, Implicit Conflict: a later observation invalidates an earlier memory without explicit negation, requiring contextual inference and commonsense reasoning to detect. To rigorously evaluate this capability, we introduce STALE, a benchmark of 400 expert-validated conflict scenarios (1,200 evaluation queries across three probing dimensions) spanning over 100 everyday topics with contexts up to 150K tokens. We propose a three-dimensional probing framework that tests State Resolution (detecting that a prior belief is outdated), Premise Resistance (rejecting queries that falsely presuppose a stale state), and Implicit Policy Adaptation (proactively applying updated states in downstream behavior). A systematic evaluation of frontier LLMs and specialized memory frameworks reveals a pervasive gap between retrieving updated evidence and acting on it, with even the best evaluated model achieving only 55.2% overall accuracy. Models often accept outdated assumptions embedded in a user's query, and they struggle to recognize when a change in one aspect of the user's state should invalidate related memories. To establish an initial baseline for state-aware memory, we further present CUPMem, a prototype that strengthens write-time revision through structured state consolidation and propagation-aware search, suggesting that explicit state adjudication is a promising direction for robust agentic memory.
1 Introduction
Long-term LLM assistants must track users’ evolving latent states, but existing memory evaluations largely test static retrieval rather than implicit conflict resolution. STALE addresses this gap with expert-validated scenarios and three probes targeting outdated states, stale premises, and downstream adaptation.
- Motivation: Long-term assistants need memory that maintains continuity and adapts to changing personal circumstances, making memory updating a first-class concern.In these settings, memory supports coherent and responsible assistance rather than serving only as a convenience feature.
- Problem: Implicit Conflict occurs when later observations invalidate earlier memories without explicit negation, requiring contextual inference, structural reasoning, and commonsense knowledge.The failure can arise through co-referential updates or cascading invalidation across structurally related attributes.
- Evaluation Gap: Existing benchmarks predominantly measure static fact retrieval, while update- and conflict-sensitive conversational memory remains limited.Static retrieval asks whether a model can recover specific information from prior interactions, rather than whether it can revise beliefs when circumstances change.
- STALE Benchmark: STALE contains 400 expert-validated conflict scenarios and 1,200 evaluation queries across over 100 everyday topics, with contexts up to 150K tokens.The benchmark is designed for long-term memory under implicit conflict in user-assistant dialogues.
- Probing Framework: STALE probes State Resolution, Premise Resistance, and Implicit Policy Adaptation to isolate distinct failures in update-sensitive memory.These dimensions test recognizing outdated information, rejecting queries that falsely presuppose an old state, and applying updated states in downstream behavior.
2 Related Work
Prior long-term memory research has evaluated static observation recovery and increasingly implicit reasoning, while parallel work develops explicit memory mechanisms for selecting, compressing, and extracting information. STALE instead focuses on detecting when later observations revise latent user states and invalidate assumptions tied to earlier memories.
- Long-Term Memory Benchmarks for LLM Agents: Early benchmarks such as LoCoMo and LongMemEval primarily evaluated static observation recovery over extended interaction histories.Later benchmarks broadened evaluation to implicit reasoning and autobiographical person understanding.
- Long-Term Memory Benchmarks for LLM Agents: STALE complements prior benchmarks by testing whether later observations revise latent user states and invalidate assumptions licensed by earlier, unlinked memories.Its setting differs from choosing between competing factual answers or inferring a missing fact.
- Long-Term Memory Frameworks: Parallel long-term memory frameworks address deliberate information selection, compression, and extraction despite substantially larger context windows.Examples named in the passage include Mem0, Zep, and LiCoMemory.
3 STALE
STALE defines implicit conflict as a new observation that invalidates a prior belief through contextual or commonsense inference without explicit correction. It operationalizes these conflicts through formal axioms, two structural types, expert-reviewed generation, and multidimensional evaluation.
- Formal Definition: An implicit conflict requires a new observation to make a prior belief invalid while no later utterance explicitly negates, corrects, or marks it obsolete.The definition captures belief revision that remains invisible at the dialogue’s surface level.
- Conflict Types: STALE distinguishes co-referential conflicts, where the same attribute receives incompatible implied values, from cascading conflicts caused by changes in related attributes.Type II conflicts require inferring that an upstream state change logically precludes a prior belief about a downstream attribute.
- Benchmark Construction: The generation pipeline anchors an earlier state, creates an implicit invalidating observation, applies type-specific attacks, and filters candidates through plausibility, conflict, and implicitness checks.Failed cases are regenerated with evaluator feedback before multi-turn dialogue packaging and haystack construction.
- Evaluation Framework: Evaluation probes State Resolution, Premise Resistance, and Implicit Policy Adaptation to test recognition of stale beliefs, rejection of misleading assumptions, and downstream state-aware behavior.The framework is designed to measure more than standard retrieval accuracy.
- Evaluation Framework: 95.8% evaluation agreement with human judgments was achieved by an LLM judge scoring responses against foundational state logic rather than synthetic reference strings.This approach reduces reference bias in evaluating implicit-conflict resolution.
4 Experiments
Experiments evaluate LLMs, memory frameworks, and CUPMEM on STALE using fixed constructed memories and a Gemini-3.1-flash-lite judge. Results show broad difficulty with implicit-conflict resolution and a gap between retrieving updated evidence and using it to govern answers.
- Evaluation setup: The evaluation spans closed- and open-source LLMs, five memory frameworks, and the proposed CUPMEM prototype.Plain LLMs receive the full dialogue history as a chronological long-context input.
- Main results: The three probing dimensions indicate that implicit-conflict resolution is multifaceted rather than a single retrieval problem.The main results evaluate each system across the benchmark’s three probing dimensions.
- Evaluation setup: Responses are probed separately against the same constructed memory, which remains fixed during evaluation.Gemini-3.1-flash-lite serves as the LLM judge for conflict and updated-state awareness.
- Main results: 55.2% overall accuracy is achieved by the strongest model, Gemini-3.1-pro, while Qwen3.5-27B reaches 31.3% and Gemini-3.1-flash-lite reaches 22.4%.Most memory frameworks fall below 10%, demonstrating substantial difficulty with implicit-conflict resolution.
- Diagnostic analysis: 77.5% of SR/PR cases and 67.8% of IPA cases contain updated evidence in LightMem’s top-20 retrieval results, yet its failure rate remains 78.6%.This comparison separates retrieval visibility from final answer correctness.
- Diagnostic analysis: The current-state adjudication gap reflects retrieved evidence failing to become a stable judgment that guides downstream planning and generation.The findings characterize the limitation as conversion of retrieved evidence into downstream behavior, not merely recall.
5 Bridging the Gap: From Retrieval to State Adjudication (CUPMEM)
CUPMEM addresses the gap between retrieving updated evidence and using it in downstream reasoning by framing memory management as explicit state tracking with write-side adjudication.
- 5 Bridging the Gap: From Retrieval to State Adjudication (CUPMEM): CUPMEM is proposed to address the current-state adjudication gap revealed by diagnostics in Section 4.4.The gap is that retrieving updated evidence does not guarantee that it governs downstream reasoning.
- 5 Bridging the Gap: From Retrieval to State Adjudication (CUPMEM): CUPMEM reframes memory management as explicit state tracking with write-side adjudication.It is described as a prototype for current-state updating and propagation-aware memory.
- 5 Bridging the Gap: From Retrieval to State Adjudication (CUPMEM): Unlike generic construction-time updates, CUPMEM targets conflict-specific revision of stored states.The passage distinguishes updating entries during construction from conflict-targeted state revision.
6 Conclusion
STALE reframes long-term assistant memory as latent user-state tracking and systematically evaluates implicit conflict resolution. Its findings show that recognizing outdated memories does not ensure updated beliefs are applied, while models remain vulnerable to stale presuppositions.
- 6 Conclusion: STALE introduces a benchmark of 400 expert-validated conflict scenarios and 1,200 evaluation queries for systematic implicit conflict-resolution evaluation.The benchmark uses a three-dimensional probing framework.
- 6 Conclusion: Recognizing that a memory is outdated does not imply applying the updated belief.
- 6 Conclusion: Models are highly susceptible to queries that presuppose stale information.
A Limitations and Future Work · B Dialogue, Information, and State
The paper frames dialogue as sparse, partial evidence of an evolving latent user state, while positioning STALE and CUPMem as targeted approaches with important scope, data, evaluation, and architectural limitations.
- A Limitations and Future Work: STALE measures latent belief revision under one-shot implicit conflict, not repeated updates, coupled attribute propagation, or gradual state drift.Each instance contains a single conflict pair, so the benchmark evaluates a specific capability rather than all long-term memory challenges.
- A Limitations and Future Work: LLM-generated scenarios validated by experts enable broad coverage and scale but may not represent organic user-assistant interaction distributions.The benchmark mitigates this concern by grounding instances in everyday scenarios and applying strict quality control.
- A Limitations and Future Work: 95.8% human agreement supports the LLM-as-judge protocol, yet judges may miss nuanced correct answers, especially for open-ended IPA queries.STALE performance also entangles implicit-conflict resolution with instruction following and long-context retrieval.
- A Limitations and Future Work: CUPMem is a targeted prototype that depends on a predefined state schema, which enables stale-state adjudication but restricts attribute domains.Inferring evolving user states from sparse dialogue without this scaffolding remains unresolved.
- B Dialogue, Information, and State: The user state is an unobservable, evolving configuration inferred from incomplete and noisy utterance evidence, making complete temporal recovery fundamentally difficult.Future states are unpredictable, while past states are hard to reconstruct because dialogues are fragmented and selectively informative.
- B Dialogue, Information, and State: User-assistant interactions are temporally sparse and localized, with sessions separated by potentially long intervals and driven by immediate needs or intentions.Dialogues therefore provide incomplete coverage of the user’s life and cognition.
- B Dialogue, Information, and State: Natural-language messages express the same underlying preferences, beliefs, or facts in surface-divergent ways shaped by momentary intent and linguistic choice.Observable messages are linguistically mediated rather than direct representations of user information.
- B Dialogue, Information, and State: A user’s goals, emotions, attention, and prior experiences influence message formulation, constituting a latent, time-dependent state.Information extracted from a single message is only a partial observation of that underlying state.
C Cost Analysis and Model Usage … D.2 Construction Prompts
The paper documents a multi-model construction and evaluation pipeline, then specifies how STALE scenarios are seeded, validated, probed, packaged, and temporally audited. Its prompts enforce realistic, implicit conflicts and ensure that updated states govern later queries and downstream tasks.
- C Cost Analysis and Model Usage: C Cost Analysis and Model Usage: Different models handled observation generation, conflict quality control, probe generation, session packaging, distractor filtering, timestamp construction, and evaluation judging.Qwen3.5-Plus generated m_o; GPT-5.2 generated m_n and performed conflict-quality control; Gemini-3.1-pro generated probes; GPT-5.2 and GPT-5.1-Chat packaged sessions; Gemini-3.1-flash-lite filtered distractors, constructed timestamps, and judged evaluation.
- D.1 Seed Ontology: D.1 Seed Ontology: The manually constructed seed ontology covers everyday user attributes likely to produce implicit conflicts after state changes.It contains 10 high-level categories and 104 fine-grained attributes, providing a broad but non-exhaustive seed space.
- D.2 Construction Prompts: D.2 Construction Prompts: Old-state prompts ground each attribute in a realistic, stable, substantial, state-dependent user scenario while avoiding other attributes and fleeting states.The generated output includes a person description, context scenario, old information statement, and attribute value.
- D.2 Construction Prompts: D.2 Construction Prompts: Type I conflicts require co-referential statements about the same attribute with incompatible values and no explicit linguistic negation.Generation and verification require independent plausibility, state conflict, and implicit-constraint compliance, including avoidance of the underlying attribute name and old scenario details.
- D.2 Construction Prompts: D.2 Construction Prompts: Type II conflicts update attribute A and indirectly invalidate attribute B through a commonsense dependency A -> B.The new statement must not mention B, the dependency chain, or the old scenario; verification checks plausible propagation and indirect emergence of the conflict.
- D.2 Construction Prompts: D.2 Construction Prompts: Probe-generation prompts create three queries testing direct state validity, resistance to false old-state premises, and downstream adaptation to the new state.The adversarial query must exclude new-state leakage, while the downstream task must omit both old and new details and require silently incorporating the updated state.
- D STALE Construction Details: D STALE Construction Details: Target facts are packaged as simulated user–assistant sessions and inserted chronologically among deduplicated auxiliary dialogues filtered for interference.Facts may appear in the opening turn or a later turn; sessions between the old and new facts cannot contradict or directly elaborate on the old fact, while later sessions face stricter filtering.
- D.2 Construction Prompts: D.2 Construction Prompts: Timestamp prompts preserve Session_o ≺ Session_n ≺ Q and audit whether the new state still constrains both validation and downstream queries.They generate later old–new dates, estimate the latest plausible query time, and revise samples when either queried condition no longer holds.
D.3 Manual Revision Standards in Dataset Construction · D.4 Context Length and Session Statistics · D.5 Attribute Distribution
STALE’s construction process combined expert manual review, explicit conflict-validity checks, and a second-pass audit. Its 400 contexts used 50-session histories averaging 151.8K formatted tokens and covered all 10 high-level categories and 103 fine-grained attribute types.
- D.3 Manual Revision Standards in Dataset Construction: Each finalized instance was reviewed by at least one LLM-evaluation expert, with ambiguous cases revised and a second-pass random audit checking the completed dataset.This quality-control process supplemented candidate-level review before finalization.
- D.3 Manual Revision Standards in Dataset Construction: Reviewers evaluated unwrapped evidence, explanations, and three queries against the intended conflict type and probing objective using four labels: ACCEPT, WEAK REJECT, WRONG TYPE, and REJECT.Local edits were made directly in the annotation interface when possible.
- D.3 Manual Revision Standards in Dataset Construction: Evidence review required mo to encode a persistent belief and mn to provide the actual implicit invalidation, including incompatible same-attribute values for Type I without explicit correction language.Statements describing momentary events, unrelated attributes, or underspecified beliefs were revised or rejected.
- D.3 Manual Revision Standards in Dataset Construction: The SR, PR, and IPA queries were revised to test outdated-belief resolution, preserve stale premises without mn cues, and require updated-state-dependent downstream behavior.Reviewers also avoided queries that made the target memory unnecessary or revealed the update directly.
- D.4 Context Length and Session Statistics: 151.8K tokens was the average and median formatted context length across 400 instances, with a 164.9K-token maximum and closely matched Type I and Type II averages.Type I averaged 151.7K formatted tokens, while Type II averaged 151.9K; contexts comprised 50 temporally ordered sessions followed by a probing query.
- D.5 Attribute Distribution: The final dataset covered all 10 high-level seed ontology categories and 103 fine-grained attribute types after generation, verification, filtering, and manual editing.The seed ontology guided generation rather than imposing strict balancing.
- D.5 Attribute Distribution: The 400 accepted examples were broadly distributed across ten everyday attribute domains, but the distribution was not exactly uniform because candidates were rejected or revised during validation and construction.Domains included preferences and values, spatiotemporal context, routines, health, identity, social networks, focus, possessions, capabilities, and digital footprint.
E Experimental Details · E.1 Evaluation Prompts
The evaluation prompts separate response generation from automatic scoring across three probing dimensions. A joint judge compares model responses with old and updated states, hidden invalidation logic, and strict Boolean rubrics.
- E.1 Evaluation Prompts: Response-generation prompts provide serialized conversation history directly to the assistant, using explicit questions for SR and PR but the latest user request for IPA.SR and PR ask the model to answer after reviewing history, whereas IPA asks it to respond directly to the latest query.
- E.1 Evaluation Prompts: Automatic scoring gives a judge the old state, updated state, hidden invalidation logic, and three probing responses, producing Boolean decisions with brief reasoning.The judge evaluates whether each response correctly deduces and respects the hidden updated state.
- E.1 Evaluation Prompts: Dimension 1 passes only when the response recognizes that the old information may no longer be valid, rather than assuming it remains true or claiming ignorance.This rubric tests explicit state-resolution behavior.
- E.1 Evaluation Prompts: Dimension 2 passes only when the response detects a false premise based on the old state or acknowledges the new state, rather than blindly agreeing.The adversarial question is designed to expose reliance on outdated assumptions.
- E.1 Evaluation Prompts: Dimension 3 passes only when the final action, plan, or recommendation safely follows the new state and is not generic or vague.Responses fail when their actions comply with outdated constraints.
- E.1 Evaluation Prompts: The judge evaluates underlying logic rather than surface-level word matching and returns separate reasoning and pass fields for all three dimensions in JSON.Each dimension includes brief reasoning and a Boolean pass value.
- E.1 Evaluation Prompts: The user-side judge input organizes ground truth and model outputs into separate sections for Dimension 1, Dimension 2, and Dimension 3.The template includes M_old, M_new, hidden logic, each question, and its corresponding target-model response.
E.2 Effect of Real-world LLM Calls
Repeated calls to the same models and instances reveal moderate dimension-level stochasticity but stable overall conclusions. Across runs, PR remains near zero, Type II trails Type I, and all models remain below 60% overall, while larger evaluation scale should attenuate variance.
- Repeated-call protocol: Five repeated calls on fixed 20-instance subsets measured response variation without changing datasets, prompts, or judge inputs.Mean accuracy and standard deviation were computed across the five repeated runs for each conflict type.
- Results: 1.7–4.7% overall standard deviations indicate stable aggregate accuracy across repeated runs.Individual dimensions show moderate variance because each instance contributes 5 percentage points on the small subset.
- Results: PR remains near zero, Type II performance remains below Type I, and every model stays below 60% overall in every run.These repeated-run results preserve the paper’s core findings despite stochastic variation in individual dimensions.
- Interpretation: At the 200-instance scale used in the main evaluation, stochastic variation should be substantially attenuated.The observed per-dimension variance is attributed to LLM generation rather than instability in the benchmark itself.
E.3 Human Agreement Analysis for Automatic Evaluation
Human validation indicates that the automatic LLM judge closely matches expert assessment across models, conflict types, and probing dimensions. Disagreements are limited and skew toward conservative false negatives, especially for open-ended IPA tasks.
- Validation Design: 240 model responses were manually validated across Gemini-3.1-pro and GPT-5.4, Type I and Type II conflicts, and all three probing dimensions.The stratified study annotated 20 examples for each model–type combination and evaluated three probing responses per example.
- Overall Agreement: 95.83% overall agreement with human labels and Cohen’s κ = 0.9152 show strong automatic-judge alignment beyond chance.The judge also achieved 98.02% precision and 95.19% F1.
- Error Profile: 1.50% false positive rate indicates limited overestimation of model performance, while 7.48% false negatives account for most disagreements.There were 2 false positives among 133 human-incorrect responses, compared with 8 human-correct responses rejected by the judge.
- Agreement by Dimension: 98.75% agreement on SR and 97.50% on PR exceeded 91.25% on IPA, where downstream planning and recommendation responses are more open-ended.The corresponding Cohen’s κ values were 0.9728 for SR, 0.9134 for PR, and 0.8261 for IPA.
- Agreement by Model: 96.67% agreement for Gemini-3.1-pro and 95.00% for GPT-5.4 support scalable LLM-as-judge evaluation, with automatic scores appearing mildly conservative.Cohen’s κ was 0.9241 for Gemini-3.1-pro and 0.8389 for GPT-5.4.
E.4 Attention analysis details
The attention analysis measures query-conditioned routing among old-session, new-session, and neighboring-session spans. Query-to-evidence routing exceeds positional baselines, while weak cross-session attention and Type II patterns indicate limited reconciliation of conflicting memories; the analysis remains diagnostic rather than causal.
- Setup: Inputs are annotated with old-session, new-session, final-query, and neighboring-session spans used as positional noise baselines.The neighboring sessions replace target evidence spans when computing corresponding noise baselines.
- Attention measures: The analysis computes average attention mass for Sessionn →Sessiono, Q→Sessiono, and Q→Sessionn.The score averages post-softmax attention from tokens in a query span to the entire key span.
- Query-to-evidence routing: Q→Sessiono and Q→Sessionn consistently exceed neighboring-session baselines, whereas Sessionn →Sessiono remains weak and near its noise baseline.This pattern suggests query-conditioned routing to task-relevant evidence but limited evidence for explicit cross-session reconciliation before answering.
- Type I/Type II comparison: Type II shows weaker query-to-new attention and weaker cross-session attention than Type I.The reported attention patterns are consistent with the performance gap between the two conflict types.
- Relative attention ratio: The diagnostic analysis uses updated-to-outdated attention ratios, with ratios above one indicating stronger relative attention to updated evidence.Ratios are computed within correctness splits and as per-example means grouped by whether each dimension is answered correctly.
- Limitations: The analysis is diagnostic rather than causal because attention does not fully determine predictions and sampled groups are limited in size.Despite these limitations, separation from noise baselines and correctness-conditioned ratio differences support query-conditioned routing as a key factor in implicit-conflict resolution.
E.5 Diagnostic Case Studies of LightMem on STALE … H Broader Impacts
The case studies show that LightMem can retrieve updated evidence yet fail to adjudicate which state should govern behavior. CUPMEM addresses this gap through typed temporal memory, write-time invalidation, constrained readout, and explicit attention to privacy and misuse risks.
- E.5 Diagnostic Case Studies of LightMem on STALE: In the coastal-to-desert case, LightMem stored desert evidence but did not consolidate the implied environmental transition into a revised current state.The old coastal-dampness memory remained retrievable, and the old window-seal memory ranked first while the best desert evidence ranked fourth.
- E.5 Diagnostic Case Studies of LightMem on STALE: The friend-scheduling case shows that updated evidence can support natural decisions while remaining vulnerable to premise-laden requests.The old two-night memory ranked first and the new one-evening evidence second on PR, yet IPA remained vulnerable despite acknowledging the night-shift constraint.
- E.5 Diagnostic Case Studies of LightMem on STALE: LightMem’s traces show stale memories dominating premise-laden queries despite updated evidence being stored and sometimes retrieved.In the Seattle→Austin case, the old Seattle memory ranked first while the best Austin evidence ranked tenth; analogous brittleness appeared in other cases.
- F CUPMEM Design Details: CUPMEM is motivated by the need to stabilize transitions from later evidence to a revised current-state basis rather than merely preserve retrievable traces.Its design makes write-side revision conflict-targeted for latent user-state updating.
- F.1 Memory Representation: CUPMEM represents memory as typed state domains and local slots, with traceable attributes, fixed schema construction, temporal status, provenance, and supporting evidence.Stale states are archived rather than deleted, and uncertain replacements can receive an UNKNOWN_CURRENT marker.
- F.2 Write-Time State Consolidation and Invalidation: CUPMEM separates same-slot updates from latent invalidation, using affected state regions and common-sense extrapolation to identify propagated conflicts beyond directly touched domains.An LLM-based adjudicator can archive stale states, mark old defaults unsafe when replacements are underdetermined, or decline revision when evidence is insufficient.
- F.3 Constrained Readout at Query Time: At query time, CUPMEM performs premise-centered, status-aware readout and verifies whether queried premises are supported, outdated, or unresolved without redefining state on the fly.The design treats reliable write-time consolidation and stale-state retirement as primary, with query behavior derived from those decisions.
- H Broader Impacts: The work seeks safer personalized assistance but recognizes risks involving privacy, consent, persistent profiling, sensitive-attribute inference, dependence, and incorrect state updates.The benchmark uses synthetic and manually reviewed scenarios as a mitigation described in the broader-impacts discussion.