Source-linked AI summary
Auditing Future-Memory Use in Point-in-Time Language Agents
Rongtian Ye
TL;DR
PIT agents may retain post-cutoff facts even when external evidence respects a historical cutoff, raising the question of whether that access predicts evidence gathering. The paper uses independent memory probes and paired identity conditions to test the prediction, finding no prespecified positive allocation shift in either model and an inverse pattern in DeepSeek.
Problem
It is unclear whether access to future facts predicts which company a PIT agent investigates when external evidence remains cutoff-valid.
Source: §1 · p. 1Method
Results
Takeaways & Limitations
PIT evaluation should pair external-evidence cutoff checks with separate assessments of memory accessibility, search allocation, and final outputs.
Source: §7 · p. 8The logs record observable investigation actions and final outputs, but not the internal use of remembered information.
Source: §8 · p. 8
Abstract
from arXiv · showhide
Point-in-time (PIT) agents can recall post-cutoff facts even when their external evidence respects a historical cutoff. We examine whether this access predicts evidence gathering in 48 historical company-comparison tasks. Independent closed-book revenue probes identify the better-remembered company for each model on tasks with unequal recall, fixing the predicted direction before agent runs. We then compare paired runs with real or anonymized company identities while keeping external evidence PIT-valid. Both Qwen and DeepSeek recalled future facts, but neither supported the prespecified shift toward the better-remembered company. Qwen's four allocation estimates on 28 unequal-recall tasks were negative, with descriptive intervals spanning zero. A prospective DeepSeek replication showed an inverse pattern across all four metrics on 20 such tasks, with intervals below zero. These estimates remained negative after each company, pair, and future-fact omission, and all four post-hoc max-T-adjusted sign-flip p-values were below .05. In a prospectively specified follow-up, explicit future facts produced small estimated shifts away from the supplied-fact company, with intervals spanning zero, while final choices shifted toward it. The mechanism behind the natural inverse pattern remains unresolved. These findings support separate assessments of memory accessibility, search allocation, and final outputs alongside cutoff checks on external evidence.
1 Introduction
PIT agents can retain access to post-cutoff facts even when their external evidence respects a historical cutoff. This study tests whether independently measured future-fact access predicts observable evidence gathering and finds no prespecified positive shift in either model.
1 Introduction
- The study asks whether future facts accessible to a model predict what it investigates when all external evidence is PIT-valid.Final answers and citation dates alone cannot distinguish recalled outcomes from changed search behavior, motivating analysis of tool calls and evidence selections.Source: §1 · p. 1
- Temporal Policy Contamination (TPC) is a systematic shift in observable information seeking toward the direction predicted by independently measured access to post-cutoff facts.The audit distinguishes data integrity, policy integrity, and output integrity.Source: §1 · p. 1
- Across 48 historical public-company comparisons, independent closed-book revenue probes measured each model’s recall before paired runs with real or anonymized identities.Four metrics measured allocation during early search, all actions and decision batches, and opened sources.Source: §1 · pp. 1–2
- Both models recalled post-cutoff facts, but neither supported the prespecified shift toward the better-remembered company.Qwen’s four allocation estimates were negative with descriptive intervals spanning zero; DeepSeek showed a robust inverse pattern across all four metrics.Source: §1 · p. 2
- The contributions include a directional audit, paired PIT evaluation, semantic re-identification audit, and prospective follow-up on explicit future facts.The audit fixes recall-based predictions before agent runs and tests them against observed information seeking.Source: §1 · p. 2
2 Related Work
Related work spans temporal contamination, knowledge-aware retrieval, and historical financial-agent evaluation. This paper connects those strands by testing whether independently measured future-fact recall predicts search under PIT-valid evidence.
2 Related Work
- Temporal-contamination research includes retrieval-based and behavioral diagnostics, temporal enforcement, and methods relating future knowledge to answers or decisions (Deng et al., 2024; Zhang and Stadie, 2026a; Liu et al., 2026; Zhang et al., 2026).Prompted-cutoff tests report that earlier-date instructions do not reliably suppress causally related future knowledge.Source: §2 · p. 2
- Knowledge-aware retrieval work studies how model knowledge can inform retrieval decisions, including adaptive policies such as KbPO and AdaSearch (Lin et al., 2025).Related approaches use confidence-based retrieval, tool/no-tool rollouts, and segment-level credit assignment (Chen et al., 2026; Kumar et al., 2026).Source: §2 · p. 2
- Financial-agent work evaluates PIT research, look-ahead bias, contamination, and identity masking (Xiao et al., 2026; Benhenda, 2026; Li et al., 2025; Zhu et al., 2026b).This study pairs real and anonymized identities and measures future-revenue recall before testing a prediction about evidence allocation.Source: §2 · p. 2
3 Problem and Audit Framework
The framework defines a directional test of whether accessible post-cutoff knowledge is associated with observable search behavior while external evidence remains PIT-valid. It separates that policy question from evidence-cutoff and output checks.
3 Problem and Audit Framework
- Each task asks which of two companies will have higher future revenue growth using evidence available by a historical cutoff, while the post-cutoff label is withheld from the PIT environment.The SEC implementation requires filing availability to strictly precede the cutoff date.Source: §3 · pp. 2–3
- Normal and Anonymized conditions present paired versions of the same task, changing identity presentation while preserving evidence content and PIT validity.A separate audit assesses whether identities can be recovered from anonymized text.Source: §3 · p. 3
- Future-memory accessibility is measured by closed-book recovery of post-cutoff revenue within 5% relative error, and the better-remembered company sets each task’s predicted direction.Directions are fixed before trajectories are collected; the original Qwen audit used two probes per fact and the prospective DeepSeek replication used five.Source: §3 · p. 3
- TPC predicts a positive memory-aligned Normal–Anonymized allocation contrast: revealing identities should increase relative evidence allocation to the better-remembered company.The contrast pairs trajectories and orients task differences toward the better-remembered company; positive and negative values represent shifts toward and away from that company, respectively.Source: §3 · p. 3
- The audit distinguishes data integrity of external evidence, policy integrity of observable actions, and output integrity of final claims and citations.It enforces the evidence cutoff, records final outputs, and tests the directional allocation hypothesis.Source: §3 · p. 3
4 Experimental Setup
The experiments use a fixed historical company-comparison task pool, paired identity conditions, model-specific memory probes, and logged agent trajectories. Allocation and other outcomes are analyzed with paired task comparisons and descriptive uncertainty intervals.
4.1 Tasks and paired environments
- The fixed policy set contains 48 tasks selected from a 194-task SEC filing pool, covering 19 companies, 32 pairs, and 77 future revenue facts.The corpus contains 453 historical filings divided into 49,854 chunks; Normal and Anonymized preserve cutoffs, dates, labels, structure, and numerical content, with a centralized cutoff gate.Source: §4.1 · pp. 3–4, Figure 1 · p. 4
4.2 Model-specific memory and policy audits
- Qwen’s two-probe directions were frozen before 48 paired Normal and Anonymized trajectories, while DeepSeek’s five-probe directions and 20-task asymmetric subset were frozen before policy runs.The DeepSeek prospective replication used a structured adapter after initial format failures, then reran the 96-run schedule under policy-v2.Source: §4.2 · p. 4
4.3 Measurements, diagnostics, and uncertainty
- Allocation metrics track early entity-directed batches, full-investigation actions and batches, and opened evidence; uncertainty uses paired task-level bootstrap intervals from 10,000 resamples.The intervals summarize variability within the evaluated task set, where tasks share companies, pairs, and future facts.Source: §4.3 · pp. 4–5
- Additional analyses include Qwen identity-cue trajectories, semantic re-identification probes, and evaluator-side sensitivity grids for memory directions.The identity-cue study scheduled Normal, Anonymized, and Cue conditions; the separate re-identification audit used frozen anonymized chunks.Source: §4.3 · p. 5
- A prospective follow-up compares otherwise identical Anonymized tasks supplying either company’s true target-year revenue, while retrieved evidence remains cutoff-restricted.Its primary contrast is positive when allocation shifts away from the company whose fact is supplied.Source: §4.3 · p. 5
5 Results
Both models recalled post-cutoff revenue, but neither showed the prespecified positive allocation shift toward the better-remembered company. DeepSeek showed a robust inverse pattern, while the results also document differences in investigation behavior and measurement diagnostics.
5.1 Memory accessibility and primary allocation results
- Qwen recovered future revenue within 5% relative error in 124 of 252 probes (49.2%), while DeepSeek mean accessibility was 0.8413 over 630 probes.DeepSeek’s measurement covered the same 126 facts.Source: §5.1 · p. 5
5.2 DeepSeek’s inverse allocation pattern is robust
- DeepSeek’s four allocation estimates remained negative after every prespecified omission; post-hoc max-T-adjusted sign-flip p-values ranged from .0036 to .0385.Across the full task set, signed-memory correlations were negative, while directional memory variation was concentrated in the asymmetric subset, whose task composition differed from the tied subset.Source: §5.2 · p. 5
- Exploratory absolute-allocation contrasts showed greater withdrawal on the better-remembered side: −0.750 for early allocation, −4.650 for calls, −1.700 for batches, and −3.250 for opened sources.All four descriptive intervals were below zero; Qwen did not show the same consistent absolute pattern.Source: §5.2 · pp. 5–6
5.1 Memory accessibility and primary allocation results
- Across 20 asymmetric tasks, DeepSeek’s four allocation estimates were negative with intervals below zero, while Qwen’s four estimates on 28 tasks were negative with intervals spanning zero.The prespecified positive shift was unsupported in both models.Source: §5.1 · p. 5, Figure 2 · p. 6
5.3 Identity cues and output behavior
- Both models used fewer batches and calls in Normal than Anonymized; accuracy was higher for Qwen in Normal and unchanged for DeepSeek.For Qwen, accuracy was 36/48 versus 31/48; for DeepSeek, it was 29/48 in both conditions.Source: §5.3 · p. 6
- On asymmetric tasks, the better-remembered company was the ground-truth winner in 15/28 Qwen tasks and 9/20 DeepSeek tasks; accuracy and final-answer alignment had weak, inconsistent memory associations.Source: §5.3 · p. 6
5.4 Measurement and mechanism diagnostics
- Qwen’s five-probe expansion closely tracked its original memory measurement: 26 of 28 originally asymmetric tasks retained direction, one became tied, and one changed sign.Batch allocation was more sensitive than K5 to sparse evaluator-side injections; at 30% sparsity, intervals excluded zero in 85.0% of batch transformations versus 33.5% for K5.Source: §5.4 · p. 6
- In Qwen’s Identity Cue study, five of six broad outcome contrasts shared the Normal–Anonymized sign, but all four memory-aligned allocation intervals spanned zero; the natural inverse pattern’s mechanism remained unresolved.Anonymized identities remained recoverable, and re-identification difficulty had inconsistent associations with allocation; in the explicit-fact follow-up, allocation intervals spanned zero while final choices shifted toward the supplied-fact company.Source: §5.3 · p. 6, Figure 3 · p. 7, §5.4 · p. 7
6 Discussion
The natural inverse allocation pattern is robust for DeepSeek, but its mechanism remains unresolved. The discussion distinguishes search behavior from final decisions and recommends evaluating temporal integrity at multiple stages.
6 Discussion
- Search substitution is a plausible explanation for DeepSeek’s inverse pattern, but the mechanism remains unresolved.The exploratory absolute-count decomposition and explicit-fact follow-up were consistent with withdrawal, while the follow-up’s allocation intervals spanned zero.Source: §6 · p. 7
- Investigations were shorter in Normal for both models; accuracy rose for Qwen but was unchanged for DeepSeek, while explicit future facts shifted final choices toward the supplied-fact company despite weak allocation responses.In Qwen’s Cue study, five of six broad outcome contrasts shared the Normal–Anonymized sign, while all four memory-aligned allocation intervals spanned zero.Source: §6 · p. 7
- The authors recommend combining source-availability checks, independent memory probes, trajectory analysis, and assessment of final claims and citations in PIT evaluations.For anonymization-based historical controls, semantic re-identification tests complement identifier checks by measuring identity information retained in the content.Source: §6 · p. 7
7 Conclusion
The paper presents a trajectory-based audit combining independent future-memory probes with paired PIT-valid agent runs. Its results support separate assessments of memory accessibility, search allocation, and final outputs alongside external-evidence cutoff checks.
7 Conclusion
- The audit pairs independent future-memory probes with agent runs under PIT-valid external evidence to test whether recalled post-cutoff facts relate to search allocation.Source: §7 · p. 8
- Both models recalled post-cutoff facts, but neither supported the prespecified shift toward the better-remembered company; DeepSeek showed a robust inverse pattern, while Qwen’s negative estimates had intervals spanning zero.Source: §7 · p. 8
8 Limitations
The experiments are limited by their scope and task dependence, residual identity recovery, and unobserved internal memory use; protocol differences and unresolved mechanisms further constrain interpretation.
8 Limitations
- The evaluation covers two fp8 model endpoints and model-specific subsets; repeated companies, pairs, and future facts make task-level dependence relevant.Bootstrap intervals summarize the evaluated task set, while sign-flip inference relies on task-level sign symmetry.Source: §8 · p. 8
- Anonymized identities remain recoverable, the directional strata are small, and residual identity recovery may confound identity-associated contrasts.The logs capture investigation actions and final outputs, not the internal use of remembered information.Source: §8 · p. 8
- Different finalization protocols limit cross-model comparisons of stopping behavior, and the role of parametric future memory in DeepSeek’s inverse pattern remains unresolved.The explicit-fact follow-up produced small withdrawal estimates with intervals spanning zero.Source: §8 · p. 8
A Study Design Summary
- Task selection used Qwen’s original memory scores and historical-prior agreement, while cutoff rules required pre-cutoff filings and excluded unresolved-provenance candidates.The selection procedure also imposed quotas and feasibility checks before corpus construction and policy runs.Source: A Study Design Summary · p. 10, §B.1 · pp. 10–11
B.2 Worked task: AMD and Amazon, FY2018
- The worked example preserves dates and numerical evidence when anonymizing identities, while target-year outcomes and labels remain evaluator-only.The revenue probes test closed-book recall of future revenue rather than forecasting.Source: §B.2 · pp. 11–12, §B.3 · p. 12
C Observable Policy Metric Definitions
- Retrieval applies availability checks to visible content and metric sources; allocation metrics classify entity-directed calls and omit undefined paired contrasts without imputation.Opened documents and chunks count as unique evidence units, while bilateral batches expose both entities.Source: §B.4 · p. 13, C Observable Policy Metric Definitions · pp. 13–14
D Interaction and Canonicalization Protocol
- The protocol allows up to 18 investigation batches, excludes evidence-free finalization from allocation metrics, and excludes earlier Qwen debugging records from reported policy results.A 24-call Qwen configuration prematurely terminated parallel investigations.Source: D Interaction and Canonicalization… · p. 14
E Familiarity Diagnostics
- Familiarity measures did not consistently align with memory: more-familiar and better-remembered entities differed on 10 identity-familiarity and 14 historical-familiarity tasks.All four reported familiarity-effect intervals spanned zero.Source: E Familiarity Diagnostics · p. 14
F Additional Identity-Cue Results
The Identity Cue condition’s memory-aligned estimates were negative across all four metrics, despite positive raw shifts for entity-A-oriented action and batch asymmetry. Qwen’s K5 estimate also remained negative across dependency and finalization checks.
F Additional Identity-Cue Results
- The study yielded 143 valid decisions from 144 scheduled trajectories; C–A used 48 valid pairs, while N–A and C–N used 47.The N/A runs were separate from the original Qwen policy audit.Source: F Additional Identity-Cue Results · pp. 14–15
- C–A differences were −0.521 batches, −0.854 calls, +0.0074 parallel rate, −0.0346 bilateral rate, −0.0417 forced-finalization rate, and +0.1042 accuracy.Task-level distance from Normal was smaller for Identity Cue on batches and bilateral rate, but larger on calls and parallel rate.Source: F Additional Identity-Cue Results · p. 15
- All four memory-aligned C–A means were negative on 28 asymmetric tasks, with 95% descriptive intervals spanning zero.Raw entity-A-oriented action and batch asymmetry shifts were +0.0933 and +0.0885, with intervals above zero.Source: F Additional Identity-Cue Results · p. 15
G Qwen Dependency and Finalization Checks
- Qwen’s K5 estimate was −0.0463 across 28 valid asymmetric pairs and stayed negative under each company, pair, and future-fact omission.It was −0.0482 for 21 pairs where both conditions reached K5, −0.0473 for 14 matched-finalization pairs, and −0.0349 for five pairs without forced finalization.Source: G Qwen Dependency and Finalization… · p. 15
H Qwen Five-Probe Memory Robustness
A post-hoc five-probe check largely preserved Qwen’s task-level memory directions. Separate output diagnostics found little association between memory measures and accuracy differences, while final-choice alignment with the better-remembered entity was similar across identity conditions.
H Qwen Five-Probe Memory Robustness
- Reusing original policy trajectories as a post-hoc measurement check, two- and five-probe scores across 126 facts correlate at 0.9077 by Pearson and 0.9061 by Spearman.Source: H Qwen Five-Probe Memory Robustness · p. 15
- Five-probe reclassification retained the original direction on 26 of 28 asymmetric tasks; one became tied, one flipped, and 10 original ties became asymmetric.The reclassification table reports defined paired contrasts and 95% descriptive task-bootstrap intervals.Source: H Qwen Five-Probe Memory Robustness · p. 15, Table A4 · p. 15
I Output and Accuracy Diagnostics
- Memory measures had weak correlations with task-level accuracy differences, and final choices matched the better-remembered entity on 15/28 Normal versus 14/28 Anonymized Qwen tasks.DeepSeek counts were 10/20 and 9/20, respectively; the better-remembered entity was the ground-truth winner on 15/28 Qwen and 9/20 DeepSeek tasks.Source: I Output and Accuracy Diagnostics · p. 15
J Synthetic Sensitivity Analysis
The sensitivity analysis altered evaluator-side batch labels in fixed Qwen trajectories to test how transformation intensity affected allocation estimates. At α = .10, P1’s interval-exclusion rate was highest under homogeneous transformation and low under sparse transformations.
J Synthetic Sensitivity Analysis
- The transformation probabilistically replaced entity-directed A-only, B-only, or bilateral batch labels with the better-remembered entity’s unilateral label.Neutral labels, sequence lengths, and batch positions remained unchanged.Source: J Synthetic Sensitivity Analysis · p. 15
- Sparse replicates transformed 8 of 28 tasks at 30% sparsity or 14 of 28 at 50%, using 1,000 seeded transformations and 1,000 task bootstraps per cell.Intervals used the 2.5th and 97.5th percentiles of paired transformed-minus-original task mean differences.Source: J Synthetic Sensitivity Analysis · p. 16
- At α = .10, P1 interval-exclusion rates were 29.8% under homogeneous transformation, 0.6% at 30% sparsity, and 3.6% at 50% sparsity.The table summarizes synthetic sensitivity on 28 fixed sequences; Excl. is the percentage with a 95% descriptive interval entirely above zero.Source: J Synthetic Sensitivity Analysis · p. 16, Table A5 · p. 16
K DeepSeek Inverse-Pattern Robustness
DeepSeek’s inverse allocation pattern remained negative under dependency omissions, while the explicit-fact follow-up’s allocation estimates pointed away from the supplied-fact entity with intervals spanning zero. Final choices nevertheless shifted toward that entity, leaving the natural inverse pattern’s mechanism unresolved.
K DeepSeek Inverse-Pattern Robustness
- All four DeepSeek allocation estimates remained negative after each company, pair, or future-fact omission.They stayed negative in 16/16 company, 17/17 pair, and 36/36 fact omissions.Source: K DeepSeek Inverse-Pattern Robustness · p. 16, Table A6 · p. 17
- The post-hoc global sign-flip test gave p = .0036, but its reference distribution assumes task-level sign symmetry and shared task dependencies limit interpretation.The test used shared sign flips across the four metrics within each task.Source: K DeepSeek Inverse-Pattern Robustness · p. 16
- For calls, the better-remembered side’s N–A change was −6.900 versus −2.250 on the worse-remembered side, yielding S = −4.650.Negative S indicates greater N–A withdrawal on the better-remembered side; this decomposition is exploratory.Source: K DeepSeek Inverse-Pattern Robustness · p. 16
L DeepSeek Runtime and Finalization Provenance
- The policy-v2 rerun produced 48 valid trajectories per condition after a structured A/B final-answer adapter was fixed; finalization is excluded from investigation metrics.Qwen and DeepSeek finalization rates are not compared across models because their finalization procedures differ.Source: L DeepSeek Runtime and Finalization… · p. 17
M Semantic Re-identification Audit
- The DeepSeek semantic re-identification audit’s small strata, especially in closed-set probes, limit its directional test; residual identity recovery may still affect the N–A comparison.K5 and batch did not maintain the predicted direction across modes and correlation measures.Source: M Semantic Re-identification Audit · p. 17
N Explicit Future-Fact Mechanism Follow-Up
- All four explicit-fact withdrawal estimates were positive, with 95% descriptive intervals spanning zero; final choices moved toward the supplied-fact entity on 17 tasks.Choices moved away on four tasks and remained unchanged on 27; the natural inverse mechanism remains unresolved.Source: N Explicit Future-Fact Mechanism… · pp. 17–18
O Model and Endpoint Metadata
- Both endpoints were recorded as fp8, and requests used temperature 0 with reasoning and provider fallback disabled.Investigation requests used the fixed tool schema and automatic tool choice.Source: O Model and Endpoint Metadata · p. 18