A Historical Corpus Is Not a Historical System: Auditing Hindsight Leakage in Stateful Data Discovery

1 Introduction

• Freezing the candidate corpus and index does not prevent interaction memory from containing post-anchor observations that change rankings.Aggregate recall cannot show whether a gain came from historical evidence or later experience.Source: §1 · p. 1
• PIT Discovery binds corpus snapshot D_t to memory M_<i, then compares PIT and Future views while holding retrieval conditions and memory capacity fixed.Its behavior-only traces are audited with Temporal Violation Rate, and immutable manifests and row-level memory positions make the boundary auditable.Source: §1 · pp. 1–2
• Future raises Asset Recall@100 over PIT by 2.62–5.24 points, with all 12 paired query-cluster intervals excluding zero.On five timestamped FreshStack topics, Future exceeded PIT by 2.72 points [1.75, 3.71].Source: §1 · p. 2
• Behavior-only trace memory was harmful under PIT, and Future masked 32.7–48.4% of that harm; positive-feedback memory yielded S < P < F in all 12 settings.For positive-feedback memory, Future added another 4.11–9.58 points beyond PIT.Source: §1 · p. 2
• The paper formulates historical system state, provides a paired protocol and temporal audit, and measures hindsight inflation across harmful and useful memory mechanisms.Its conclusion is that a score cannot represent historical capability unless every state influencing it was historically available.Source: §1 · p. 2

2 Related Work

• Multi-asset benchmarks span multimodal retrieval [6], source-aware multi-hop search [5], and coherent table–passage products [27], while their standard protocols use stateless retrievers.HybridQA, TAT-QA, and ConvFinQA provide the table–text evidence structures behind DPDisc [3] [4] [28].Source: §2 · pp. 2–3
• Conversational and long-term memory benchmarks model history or cross-session reasoning [1] [18] [23] [24] [26], but do not pair PIT and Future states at one anchor with entry-level timestamp certification.PIT Discovery adds the paired replay and entry-level audit.Source: §2 · p. 3
• Temporal retrieval studies vary corpus, model, query, or distribution time [14] [16] [17] [22] [25], whereas this counterfactual changes only interaction-memory eligibility.FreshStack [22] is used for natural-order validation.Source: §2 · p. 3
• Recommendation audits vary training interactions, histories, item availability, or split policy [7] [8], while PIT Discovery fixes corpus, index, anchor, ranking, reranker, and budgets.It changes memory-time eligibility and reports Temporal Violation Rate beside retrieval quality, without claiming priority for temporal leakage.Source: §2 · p. 3

3.1 A corpus snapshot is only part of the state

• A valid historical state combines the frozen corpus and retrieval parameters with memory induced only by events observed before the anchor.Freezing the corpus alone can combine yesterday’s assets with tomorrow’s experience.Source: §3.1 · p. 3

3.2 Three clocks define historical validity

• Historical validity requires eligibility checks on asset-entry and interaction-observation times; a cache’s materialization timestamp alone cannot certify its contents.The protocol audits selected entries at the event level.Source: §3.2 · pp. 3–4

3.3 Memory views

• PIT uses only earlier events, while paired PIT and Future views have equal lookup budgets and differ only in eligible event timestamps.Every memory view excludes the anchor event; full-stream Naive serves as a leak control expected to match Future.Source: §3.3 · p. 4

3.4 The paired replay identifies memory-time bias

• Paired replay compares Future and PIT for the same request and base ranking, attributing their score difference to memory-time eligibility under fixed budgets and deterministic reranking.The estimate excludes long-run effects from branch-dependent user behavior.Source: §3.4 · p. 4

3.5 Measurement and validity

• Hindsight Gain is the Future-minus-PIT quality difference; Temporal Violation Rate must be identically zero for PIT to be valid.Future-h provides a dose–response curve, while the estimand excludes training-data contamination, corpus drift, user satisfaction, and unobserved online-policy effects.Source: §3.5 · p. 4
• The protocol reports signed PIT memory effects and Hindsight Gain when both memory conditions may underperform Stateless, rather than framing recovered harm as retained utility.The masked-harm fraction is used to describe this regime.Source: §3.5 · p. 4

4 The PIT Discovery Protocol

• PIT Discovery uses six replay stages and enforces immutable inputs, label-free event construction, paired memory views, and machine-checkable temporal validity.The protocol converts a fixed multi-asset benchmark into an auditable replay.Source: §4 · p. 4

4.1 Immutable assets and typed products

• Requests map to products of typed table and text assets, supporting asset-level recall and product-level metrics without answer generation.Cryptographic hashes record corpus, serialized assets, type vectors, queries, embeddings, and retrieval outputs; they expose accidental mismatches, not adversarial attacks.Source: §4.1 · pp. 4–5

4.2 Deterministic query streams

• Deterministic random and semantic-burst streams model diffuse access and short runs of nearby queries, with each query appearing once.Headline stream construction forbids relevance labels because gold-asset grouping could manufacture ideal trace-reuse conditions.Source: §4.2 · p. 5

4.3 Online trace construction

• The append-only trace log is built online from retrieval behavior, preventing anchor gold products or future-derived representations from entering past events.The revenue-request example keeps the same product across conditions; a following-week similar request enters only Future.Source: §4.3 · p. 5

4.4 Paired replay at an anchor

• At each anchor, PIT and Future select at most 32 entries from the same base top-200 list and use the same deterministic reranker.Each output row records provenance, budgets, quality metrics, and Temporal Violation Rate for inspection of paired differences.Source: §4.4 · p. 5

4.5 Audit checks

• Audit checks require pre-anchor PIT entries, exclude the anchor’s own trace, and verify Naive matches Future; any failed contract invalidates the paired row.A negative PIT memory effect can remain a valid result, while a positive score with nonzero TVR fails the historical claim; the protocol can accommodate other memory internals when availability times are observable and views share capacity.Source: §4.5 · p. 5

5 Experimental Setup

• Each PIT–Future pair fixes the corpus, index, anchor, base ranking, reranker, and budgets, changing only memory availability.The shared setup is designed as a paired counterfactual.Source: §5 · p. 5

5.1 Benchmark and corpus construction

• The benchmark uses ConvFinQA and HybridQA as core domains, with TAT-QA reserved for robustness; query labels evaluate assets but never order streams or build memory.Corpus objects become immutable typed assets, and snapshot manifests record source, asset, type-array, and query-file hashes.Source: §5.1 · p. 5

5.2 Retrievers and trace memory

• BM25 and hybrid retrieval provide identical top-200 base lists across conditions; hybrid retrieval fuses the top-200 BM25 and dense rankings with Reciprocal Rank Fusion.The dense component uses all-mpnet-base-v2 embeddings and normalized cosine similarity.Source: §5.2 · p. 6
• Behavior-only trace memory stores each query’s top 50 assets and raw scores, while a separate positive-feedback cache uses relevant assets as simulated clicks.The trace memory excludes relevance labels and answers; the positive-feedback cache is a label-based mechanism stress test.Source: §5.2 · p. 6

5.3 Streams, anchors, and conditions

• Label-free streams comprise seeded random permutations and semantic topical bursts of 4–16 requests; a gold-overlap ordering is excluded from reported results.Each stream has 400 anchors across four positions, and each headline cell contains 2,000 paired anchor observations.Source: §5.3 · p. 6
• The FreshStack study sorts five timestamped topics by query time and evaluates 592 anchors with at least eight past and eight future events.It uses BM25 retrieval and query-local TF–IDF for trace selection, without relevance labels in stream ordering or memory construction.Source: §5.3 · p. 6

5.4 Metrics and uncertainty

• Asset Recall@100 is the headline metric; paired query-cluster contrasts are bootstrapped with 5,000 resamples, and unseen deployment streams require a new paired replay.The reported diagnostics include Hindsight Gain, PIT memory effect, masked-harm fraction, and Temporal Violation Rate.Source: §5.4 · p. 6

5.5 Implementation and reproducibility

• Deterministic manifests record seeds, retrieval and memory hashes, budgets, selected memory positions, and output hashes; PIT runs abort on nonzero TVR.Fifteen unit tests cover schema validity, natural ordering, budget parity, self-trace exclusion, gold-label isolation, and temporal invariants.Source: §5.5 · p. 6

6.2 Leakage masks a harmful memory mechanism

• PIT trace memory reduces recall by 6.57–14.37 points in all 12 cells; Future masks 32.7–48.4% of that harm.Even with Future memory, scores remain below Stateless in every cell, so leakage makes the harmful mechanism look less harmful without reversing its effect.Source: §6.2 · p. 7

6.4 Leakage begins locally in semantic streams

• Ten future queries add 0.65–1.36 recall points in semantic streams, while no random-stream horizon-10 interval excludes zero.At horizon 500, semantic streams recover 56%, 42%, and 96% of full HG for ConvFinQA, HybridQA, and TATQA, respectively. Across primary cells, HG falls from 7.26 points at the 20% anchor to 1.43 at 80%, despite full slot fill in both views.Source: §6.4 · p. 7

6.5 The effect survives design changes

• Switching BM25 and hybrid retrieval changes HG by at most 0.44 points within each domain and stream; held-out TATQA gains of 4.69–5.13 points fall within the core-domain range.On 592 timestamped FreshStack anchors, Future exceeds PIT by 2.72 points [1.75, 3.71], extending the comparison beyond generated stream orders.Source: §6.5 · p. 7
• Future improves Asset Recall at every cutoff: HG is 0.32–2.02 points at k=10, 0.66–3.71 at k=20, and 1.68–4.73 at k=50.After removing anchors with a selected neighbor of cosine similarity at least 0.99, 99.9–100.0% of observations remain and all 12 intervals still exclude zero; complete-product success is directionally positive but not uniformly nonzero.Source: §6.5 · pp. 7–8

6.1 Future systematically inflates historical recall

• Future raises Asset Recall@100 over PIT by 2.62–5.24 percentage points across all 12 combinations, with every paired interval excluding zero.The paired comparison holds the anchor query, corpus snapshot, base ranking, reranking rule, and budgets fixed; only memory eligibility changes.Source: §6.1 · p. 7, Table 5 · p. 8

6.3 Useful memory remains vulnerable to hindsight

• The positive-feedback cache yields PIT utility of 4.65–18.96 points over Stateless and another 4.11–9.58 points of HG, with all paired intervals excluding zero.This proxy assumes perfect clicks and tests whether hindsight persists when prior feedback improves PIT retrieval.Source: §6.3 · p. 7, Table 6 · p. 8

6.6 The audit detects the exact invalid state

• PIT has maximum TVR zero, while Future selects post-anchor entries in 50.1–54.0% of its memory slots; Future and Naive match exactly on every query and metric.The results pair retrieval quality with temporal validity rather than treating a higher score as proof of a historical replay.Source: §6.6 · p. 8, Table 7 · p. 9

• A retrieval score describes a historical system only when corpus, model, and interaction state were available at the anchor.For derived state, timestamp it by its latest possible ancestor; tied or coarse timestamps require proof that an event preceded the anchor.Source: §7 · p. 9
• Model and memory updates need separate versions, availability intervals, and dependency provenance linking every contributing event to the anchor.A checkpoint timestamp alone omits the event lineage that shaped learned parameters; black-box systems without that lineage cannot certify hidden state as historical.Source: §7 · p. 9
• TVR gates historical validity, ΔPIT = P−S measures PIT utility, and HG = F−P measures contamination magnitude.Nonzero TVR can invalidate a run even when HG is zero, because future entries may not change Recall@100.Source: §7 · p. 9
• A release gate should freeze corpus and model hashes, stream order, memory recipe, and budgets, then rebuild PIT and Future from one event log if checks fail.The audit artifact should record selected entry identifiers, observation times, dependency timestamps, and capacity so reviewers can recompute TVR without rerunning retrieval.Source: §7 · p. 9

8 Limitations

• Answer quality, user satisfaction, and downstream task success are outside the evaluation, which measures Asset Recall as evidence discovery.The results cover three DPDisc table–text domains, two retrievers, two transparent memory mechanisms, and five FreshStack topics.Source: §8 · p. 9
• Fixed-log replay isolates direct memory-time eligibility but excludes long-run branch-dependent behavior.Query-cluster intervals cover repeated requests only in the tested workloads.Source: §8 · p. 9
• Broader deployment claims require a new paired replay with auditable event lineage; systems without timestamps or dependencies can undergo only a partial audit.The evaluation reports uncertainty and scope conditions for its tested workloads.Source: §8 · p. 9

9 Conclusion

• PIT Discovery pairs replay branches that hold the anchor, corpus, retriever, and budgets fixed while changing memory-time eligibility.This comparison isolates memory-time eligibility.Source: §9 · pp. 9–10
• 2.62–5.24 points: Future raises Asset Recall@100 over PIT in every primary cell across three DPDisc domains.FreshStack shows the same direction under recorded order.Source: §9 · p. 10
• Historical evaluation must bind D_t to M_<i, version θ_t and derived state, and preserve observation and dependency timestamps in an auditable manifest.The state contract makes historical scores independently verifiable.Source: §9 · p. 10

Ethical Considerations

• The work uses public research datasets and does not collect new personal data or involve human subjects.Memory traces contain benchmark query and retrieved-asset identifiers rather than user profiles.Source: Ethical Considerations · p. 11
• Post-anchor memory can overstate historical capability and encourage deployment of unreliable components.The authors report the negative PIT result, scope conditions, uncertainty, and validity checks without presenting leaked utility as progress.Source: Ethical Considerations · p. 11
• For real logs, practitioners should minimize retained query content, enforce access controls, and treat event-time provenance as sensitive operational metadata.Released manifests include hashes and configuration metadata but exclude credentials and private production logs.Source: Ethical Considerations · p. 11