PaperlayerView exact revision on arXiv

Source-linked AI summary

Freshness and the Limits of Heuristic Trend Detection in Temporal RAG

Matthew Grofsky

arXiv:2509.19376v2cs.LGcs.AI

TL;DR

Atemporal RAG ranking can miss freshness, historical validity, and change over time. The paper adds a lightweight temporal layer and separates recency-based freshness from heuristic topic evolution; freshness improves on a diverse CVE test, while the tracker’s low score is chiefly tied to its labeling rule.

  • Problem

    Semantic-proximity ranking struggles with questions whose answers depend on recency, historical validity, or change over time.

    Source: §1 · p. 1

  • Method

    A model-agnostic temporal layer combines per-slice topic tracking with query-time semantic and half-life recency scoring, plus date filtering for as-of queries.

    Source: §1 · p. 2

  • Results

    On NVD CVEs, Latest@10 is 0.60 versus 0.20 for semantic-then-newest; tracker macro-F1 rises from 0.08 to 0.49 by changing the labeling rule, while HDBSCAN yields 0.10.

    Source: §1 · p. 3, §7 · p. 15

  • Takeaways & Limitations

    The results separate freshness as a tractable but partial capability from topic evolution, whose low heuristic score is driven more by labeling than clustering.

    Source: §1 · p. 3, §7 · p. 15

    Trend detection still relies on synthetic ground truth, and real-data freshness remains partial and parameter-sensitive, with optimal α and h depending on corpus density and timespan.

    Source: Threats to Validity · p. 14

Abstract

from arXiv · show

We present a lightweight, model-agnostic temporal layer for RAG and use cybersecurity data to separate two problems that are usually conflated. For freshness, a half-life recency prior surfaces the newest relevant item where a cosine-only baseline scores 0.00; on a hard NVD CVE test, where the freshest item is not the most similar, it reaches Latest@10 of 0.60 versus 0.20 for a semantic-then-newest baseline, but stays partial and parameter-sensitive. For topic evolution, a heuristic tracker's low 0.08 macro-F1 is driven by the labeling rule, not the clusterer (HDBSCAN: 0.10; fixing the rule alone reaches 0.49, and 0.96 without clustering noise). We contribute a reproducible decoupling of the two, with honest real-data scope and a reference implementation.

1 Introduction

Atemporal semantic ranking struggles when answers depend on recency, historical validity, or change over time. The paper proposes a lightweight temporal layer for RAG and separates freshness retrieval from heuristic topic evolution.

1 Introduction

  • Atemporal semantic ranking struggles with latest-update, historical-validity, and change-over-time questions, risking stale evidence and incorrect as-of answers.The paper motivates adding time because embeddings capture semantic similarity but ignore temporal dynamics.Source: §1 · pp. 1–2
  • The proposed model-agnostic layer preserves timestamps and combines per-slice topic tracking with query-time recency reranking and time-bounded as-of retrieval.Its four-stage pipeline normalizes and embeds inputs, tracks topics, then performs time-aware retrieval.Source: §1 · p. 2
  • A half-life prior elevates recent, on-topic evidence without discarding older canonical sources, while topic labels expose growth, decay, and drift.The fused score is a tunable end-user relevance control; labels provide operator interpretability.Source: §1 · p. 3
  • The authors position the layer as a compact, reproducible approach for settings where retraining is impractical and temporal correctness matters.The method is demonstrated with cybersecurity streams but described as domain-agnostic.Source: §1 · p. 3
  • 0.49 macro-F1 follows replacing the per-week-delta labeling rule, versus 0.10 when HDBSCAN replaces K-means with the rule fixed; the initial tracker scores 0.08.The authors localize the dominant controllable factor to labeling, not clustering.Source: §1 · p. 3

2 Method

The method uses a four-stage, model-agnostic pipeline: normalize and embed records, track topics across time slices, then retrieve with temporal signals. Its topic tracker links adjacent slices, while retrieval handles latest-intent and as-of queries.

2.1 Stage 1: Data Normalization

  • Incoming streams are standardized with UTC timestamps, a compact text representation, and optional entity information for downstream processing.Timestamps are coerced to UTC, and text_repr concatenates salient fields.Source: §2.1 · p. 4, §2 · p. 4

2.2 Stage 2: Embedding

  • Normalized text_repr is encoded as 384-dimensional, L2-normalized Sentence-BERT vectors stored as float16.This lightweight embedding choice balances semantic structure with storage efficiency.Source: §2.2 · p. 4

2.3 Stage 3: Weekly Topic Tracking

  • Weekly K-means clusters are matched one-to-one across adjacent weeks, then labeled emergence, growth, decay, drift, or stable using operational rules.Drift is defined as 1 − cos(ct−1, ct); cluster counts use an elbow-method choice of K.Source: §2.3 · p. 5
  • The topic tracker’s low trend-label score is driven by comparing clusters with their immediately preceding-week matches, rather than by the clustering algorithm.The method uses weekly slices and greedy one-to-one centroid-cosine matching.Source: §2.3 · p. 5

2.4 Stage 4: Time-Aware Retrieval

  • As-of queries first filter documents to dates at or before the cutoff; recency queries rerank candidates with a semantic-plus-decay score.The score uses tunable α and half-life h; freshness evaluation uses the full candidate pool because a narrow cosine cutoff can exclude the newest relevant item.Source: §2.4 · pp. 5–6

3 Implementation

The implementation follows the four-stage pipeline through sequential scripts and generated artifacts, with an interactive dashboard for exploration. Operational parameters and time-slicing granularity can be adjusted for different domains and use cases.

3 Implementation

  • The implementation comprises Python scripts that produce Parquet and CSV artifacts, plus a Streamlit application for interactive result exploration.The stages run sequentially, each building on the previous stage.Source: §3 · p. 6, §3.1 · p. 6

3.1 Pipeline Stages and Artifacts

  • The sequential pipeline ingests and normalizes JSONL or CSV, creates embeddings, clusters and labels weekly trends, then exposes results in an operator dashboard.Artifacts include Parquet event data and CSV cluster/trend outputs; the dashboard includes an as-of week selector.Source: §3.1 · p. 6

3.2 Parameters and Operational Guidance

  • The default half-life is 14 days and the drift threshold is 0.2, but these heuristic parameters may need tuning for other domains or embedding models.Daily slices may suit fast incident response, while weekly slices are proposed for strategic analysis; half-life can be shortened for breaking news or lengthened for stable corpora.Source: §3.2 · p. 7

3.3 Beyond Security: Sports and News

  • Sports and news examples use weekly topic clusters and recency weighting to surface changes, with α and h trading recency against stability.Examples include lineup or injury changes in sports and topic drift as news stories reframe.Source: §3.3 · p. 7

4 Related Work

The related work places temporal RAG between established temporal retrieval and more deeply integrated systems. The paper combines time-aware retrieval and topic tracking as a reproducible engineering pattern, while positioning richer models as options rather than demonstrated necessities.

4.1 Temporal Information Retrieval (TIR)

  • Classical TIR uses temporal filtering or priors, while Dynamic Topic Models model smooth topic evolution; the paper evaluates a simpler discrete-slice heuristic.Its experiments attribute the heuristic tracker’s low score to labeling rather than clustering, so richer models are motivated directions, not demonstrated necessities.Source: §4.1 · pp. 7–8

4.2 Temporal Knowledge and Time-Sensitive QA

  • Temporal Knowledge Graphs add validity intervals, and TimeQA evaluates as-of correctness, motivating time-scoped context for LLMs without retraining.These efforts highlight a failure mode in atemporal systems.Source: §4.2 · p. 8

4.3 Time-Aware RAG and LLM Systems

  • Existing time-aware RAG approaches include TKG integration and query-time freshness boosting; this work combines such ideas into one reproducible layer.The passage describes the contribution as a unified, end-to-end engineering pattern.Source: §4.3 · p. 8

4.4 Positioning

  • The layer is a model-agnostic post-processing reranker, unlike TempRALM [8] and TA-RAG [9], which integrate scoring or redesign the pipeline more deeply.It needs a broad or time-aware candidate set because a narrow top-K cosine cutoff may remove the newest relevant item before reranking.Source: §4.4 · p. 8

5 Evaluation

The evaluation tests trend detection and time-sensitive retrieval across synthetic cybersecurity logs, real logon data, and NVD CVE descriptions. The setup uses corpus-specific recency sweeps for real-data freshness tests.

5 Evaluation

  • The evaluation covers scripted trend detection and time-sensitive retrieval across three corpora: synthetic logs, CERT logons, and NVD CVE descriptions.Source: §5 · p. 9

5.1 Experimental Setup

  • Real-corpus freshness tests sweep α and h and report the best observed setting in small query samples, against cosine-only and semantic-then-newest baselines.The semantic-then-newest baseline sorts the top 50 cosine candidates by timestamp and evaluates the top 10.Source: §5.1 · p. 9

5.2.1 Synthetic Log Stream

  • The 12-week synthetic stream scripts Okta growth, AWS S3-to-Snowflake drift, and vulnerability-scanning decay to provide trend ground truth.Source: §5.2.1 · p. 9

5.2.2 Real-World Logon Data

  • CERT contains 849,579 logon and logoff events over 71 weeks, but near-constant product=unknown text makes it degenerate for trend detection; it tests freshness at scale instead.Source: §5.2.2 · p. 9

5.2.3 NVD CVE Descriptions

  • The NVD test uses 4,500 timestamped CVEs with genuine semantic spread, so the freshest relevant item need not be the most similar; relevance uses query-specific aliases.Source: §5.2.3 · p. 9, Table 4 · p. 10

5.3 Metrics

The metrics assess trend-label classification, as-of filtering, and retrieval of the newest relevant document. Each metric is evaluated over a small, specified set of labeled weeks or queries.

5.3 Metrics

  • Trend Label F1 is macro-F1 over 16 labeled topic-weeks for growth, drift, and decay, giving equal weight to each class.Source: §5.3 · p. 10
  • As-of Correctness checks whether returned documents are dated at or before the cutoff after filtering, over three queries.Source: §5.3 · p. 10
  • Latest@10 counts queries where the newest relevant document appears in the top 10; Latest-Set@10 accepts any document sharing the newest timestamp.The measures are evaluated over 3 synthetic, 2 CERT, and 5 NVD-CVE topic queries.Source: §5.3 · p. 10

5.4 Results on Synthetic Data

On synthetic data, the recency prior succeeds on a controlled freshness test, while trend-label performance is constrained chiefly by the labeling rule. The results also characterize the prior’s sensitivity to α.

5.4 Results on Synthetic Data

  • The per-week-delta rule misses persistent multi-week states because it fires only at transitions, and its growth threshold also misses the scripted 20-to-30 increase.Source: §5.4 · pp. 10–11
  • 0.49 macro-F1 follows replacing the per-week-delta rule with a baseline-relative rule, versus 0.08 initially; HDBSCAN instead yields 0.10 with the rule fixed.Applying the baseline-relative rule to true topic volumes without clustering reaches 0.96.Source: Table 5 · p. 11, §5.4 · p. 11
  • 1.00 Latest@10 for the fused score versus 0.00 for cosine-only retrieval on synthetic data is a controlled sanity check, not evidence of a hard freshness solution.Relevant embeddings are nearly identical; cosine spread among 301 Okta-relevant events is 0.002.Source: §5.4 · p. 10, Table 6 · p. 11
  • Latest@10 remains perfect for α ≤ 0.95 and degrades only as α approaches 1; at α = 1.0, pure cosine returns the 0.00 baseline.At α = 1.0 the newest-relevant items have cosine ranks #49, #64, and #124.Source: Table 7 · p. 11, §5.4 · p. 11

5.5 Results on Real-World Data

On real data, the recency-fused score improves discovery of the newest relevant CVEs, but gains are partial and tuning-sensitive. Top-10 relevance remains comparable to the baselines, with a precision trade-off as recency weight rises.

5.5 Results on Real-World Data

  • Latest-Set@10 reaches 1.00 on CERT at α=0.3, h=3, but falls to 0.00 at the synthetic-tuned α=0.7, h=14 setting.The real-data sweeps are small (CERT N=2; NVD CVE N=5), and the synthetic-optimal parameters do not transfer to CERT.Source: §5.5 · p. 12, Table 10 · p. 13
  • nDCG@10 stays in 0.90–0.98 and Precision@10 in 0.68–0.84, while P@10 falls from 0.84 to 0.68 as recency weight rises.No single α dominates: α=0.7 gives Latest@10 0.40 at P@10 0.76, while α=0.3 gives Latest@10 0.60 at P@10 0.68.Source: §5.5 · p. 12, Table 8 · p. 12
  • Latest@10 reaches 0.60 with fused α=0.3, versus 0.20 for semantic-then-newest and 0.00 for cosine; the newest CVE is outside the cosine top-50 in four of five queries.The fused score is evaluated over the full pool; Table 11 shows that it can lift a moderately similar newest item but not very poor semantic matches.Source: §5.5 · p. 12, Table 11 · p. 13

6 Limitations and Future Work

The temporal layer relies on heuristics whose parameters and scope constrain its current evidence. The discussion identifies improvements for topic tracking and large-corpus retrieval, while real-data trend ground truth remains open.

6 Limitations and Future Work

  • Fixed weekly slices and the heuristic half-life prior could be replaced by adaptive slicing and learned-to-rank models with richer temporal features.Suggested features include seasonality and burstiness.Source: §6 · p. 14
  • Macro-F1 rises from 0.08 to 0.49 when the per-week-delta rule is replaced by a baseline-relative rule, and to 0.96 with clustering noise removed.HDBSCAN yields 0.10 with the original rule; the experiments do not isolate K-means limitations as the cause of low performance.Source: §6 · p. 14
  • The current re-ranking approach post-processes ANN candidates, and its performance on very large corpora could improve through time-aware indexing.Time-partitioned vector indices are one proposed direction.Source: §6 · p. 14

Threats to Validity

  • Trend detection still lacks real-data ground truth, and recency performance is parameter-sensitive, with optimal α and h depending on corpus density and timespan.The best observed NVD CVE Latest@10 is 0.60, not 1.00.Source: Threats to Validity · pp. 14–15

Ethical and Privacy Considerations

7 Conclusion

The study separates temporal RAG into freshness and topic evolution: a recency prior offers partial, parameter-sensitive freshness gains, while heuristic topic labeling remains an open challenge.

7 Conclusion

  • Latest@10 reached 0.60 on diverse CVE data versus 0.20 for semantic-then-newest, with nDCG@10 ≥0.90, but freshness remained partial and parameter-sensitive.The freshest CVE item was not the most similar; the recency prior improved freshness at a modest precision cost.Source: §7 · p. 15
  • 0.08 macro-F1 was driven by the topic tracker's labeling rule: fixing it raised the score to 0.49, or 0.96 without clustering noise, while swapping clusterers yielded 0.10.The results point toward change-aware labeling rather than a more powerful clusterer.Source: §7 · p. 15
  • Basic freshness can be partially addressed with a lightweight temporal prior, but deeper temporal understanding for AI remains an open research challenge.The study frames freshness and topic evolution as distinct temporal mechanisms.Source: §7 · p. 15

8 Reproducibility Note

The supplementary repository provides evaluation scripts and pins library versions, noting that the α-sensitivity result depends on the embedding build.

8 Reproducibility Note

  • Evaluation scripts for synthetic and logon data, plus v2 CERT and CVE freshness analyses, are included in the supplementary repository.The v2 analyses can be run with python v2/run_all.py.Source: §8 · p. 15
  • Library versions are pinned in requirements-v2.txt because the α-sensitivity result depends on the embedding build.Source: §8 · p. 15
  • The v2 runner includes eval_cert_freshness.py and eval_cve_freshness.py.Both scripts are within the v2 analysis scripts.Source: §8 · p. 15
Loading 2509.19376v2…