Source-linked AI summary
TimelyRAG: Semantic-Temporal Hybrid Retrieval for Time-Critical Question Answering in Overlapping-Evolving Documents
Youngeun Nam, Joeun Kim, Hwanjun Song, Susik Yoon, Jae-Gil Lee, Byung Suk Lee
TL;DR
Time-critical QA over amended documents is difficult because versions can share most of their content while differing in clause validity. TimelyRAG adds temporal compatibility to semantic ranking, and experiments report consistent retrieval gains and improved downstream generation quality. The paper concludes that reliable QA over evolving documents requires version-valid, clause-level temporal retrieval.
Problem
Amendments preserve much document content while changing clauses, and existing time-sensitive retrieval mainly addresses independent snapshots rather than matching similar versions to clause-level validity.
Source: §1 · pp. 1–2Method
TimelyRAG reranks candidates using semantic relevance and temporal compatibility, falling back to insertion-time signals and reducing reliance on uncertain temporal evidence when event-time signals are missing or ambiguous.
Source: §1 · p. 2, §3.3 · p. 4Results
Takeaways & Limitations
Reliable QA over evolving documents requires retrieving version-valid, clause-level evidence beyond document-level relevance.
Source: §6 · p. 9TimelyRAG depends on a first-stage retriever that includes relevant versions in its candidate pool, and TimelyQABench is synthetic, with real-world versioned corpora left for future work.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Although large language models (LLMs) and retrieval-augmented generation (RAG) have advanced open-domain question answering (QA), they remain unreliable when documents evolve through amendments. Existing time-sensitive retrieval methods address only the disjoint-evolving environment, where each update is an independent snapshot. However, laws, policies, and regulations often operate in overlapping-evolving environments, where amendments override earlier clauses while preserving most content, creating strong semantic overlap across versions. We propose TimelyRAG, a retriever-agnostic framework that incorporates temporal distance into ranking to align queries with version-appropriate documents. We also introduce TimelyQABench, the first benchmark for regulation-heavy domains with overlapping-evolving challenges. Experiments show consistent gains, up to +28.6% in nDCG@10, highlighting the importance of temporal reasoning for reliable QA over evolving documents. All resources are available at https://github.com/kaist-dmlab/TimelyRAG.
1 Introduction
Time-critical QA over amended documents must distinguish semantically similar versions by clause validity, a gap in methods focused on independent snapshots or document-level dates. TimelyRAG adds temporal validity to ranking, and TimelyQABench evaluates retrieval in regulation-focused overlapping-evolving settings.
1 Introduction
- Amendments in laws, policies, and regulations preserve most content while revising key clauses, making topical similarity insufficient for identifying evidence valid at query time.Source: §1 · pp. 1–2
- Existing time-sensitive benchmarks mainly reflect independent snapshots, while overlapping amendments require matching query event times to clause-level validity rather than document dates.Source: §1 · pp. 1–2
- TimelyRAG combines semantic similarity with temporal distance to rank versions by whether the needed clause was valid at the query or referenced event time.Source: §1 · p. 2
- TimelyQABench evaluates regulation-focused overlapping-evolving corpora whose successive amendments retain substantial semantic similarity.Source: §1 · p. 2
- Experiments report retrieval improvements of up to +28.6%.Source: §1 · p. 2
2 Related Works
Prior temporal retrieval and QA work addresses timestamped or evolving information, but much of it assumes independent snapshots or structured temporal facts. The related-work discussion distinguishes these settings from retrieval over overlapping textual amendments.
2.1 Time-Aware RAG Methods
- Time-aware RAG methods use timestamp filtering, recency scoring, or temporal supervision, but mainly target independent snapshots rather than scoring similar versions by effective dates or scope.Source: §2.1 · p. 2
2.2 Temporal KG Reasoning
- Temporal KGQA models constraints over structured facts, whereas TimelyRAG retrieves clause-valid versions from semantically overlapping textual amendments.Source: §2.2 · pp. 2–3
2.3 Time-Sensitive QA Benchmarks
- Existing time-sensitive QA benchmarks use general-purpose corpora and provide limited coverage of incremental document evolution, primarily reflecting independent snapshots.Source: §2.3 · p. 3
3 TimelyRAG Framework
TimelyRAG ranks evolving-document candidates by combining semantic relevance with temporal compatibility, targeting clauses valid in the query’s temporal context. It first retrieves candidates, then reranks them using available temporal signals and query-adaptive weighting.
3 TimelyRAG Framework
- A time-critical query requires evidence valid in its temporal context, so retrieval combines semantic relevance with temporal compatibility.Source: §3 · p. 3
3.1 Semantic-Temporal Hybrid Retrieval
- A lexical or semantic retriever forms the candidate pool, then temporal reranking combines semantic relevance and validity; it cannot recover a gold document missing from that pool.Source: §3.1 · p. 3
3.2 Stage 1: Semantic Retrieval
- Stage 1 retains top-N candidates by textual relevance; experiments use a pool larger than the final evaluation cutoff and report retrieval metrics at k=10.Source: §3.2 · p. 4
3.3 Stage 2: Temporal Re-Ranking
- Stage 2 combines semantic relevance and temporal compatibility, obtaining document event times from metadata or normalized text and falling back to insertion times when event times are missing or ambiguous.Source: §3.3 · p. 4
3.3.1 Temporal Signal Handling
- Event times represent referenced periods and take priority; insertion times mark availability and substitute only for affected terms when event times are missing, ambiguous, or low-confidence, subject to DIT ≤ QIT.Source: §3.3.1 · p. 4
3.3.2 Temporal Compatibility
- Clause event-time distance is zero within its validity interval and increases toward the nearest boundary; point timestamps use absolute distance, and available alignments combine into overall distance.Source: §3.3.2 · p. 4
3.3.3 Balancing Semantic/Temporal Signals
- Explicit event times or fine-grained temporal expressions call for stronger temporal discrimination, while weak or absent temporal cues shift reliance toward semantic relevance.Source: §3.3.3 · p. 5
4 TimelyQABench Construction
TimelyQABench is a controlled benchmark for regulatory questions whose answers depend on which amended clause was valid in the query’s temporal context. Its construction combines domain-specific questions, evolving document families, and multi-model validation.
4.1 Domain-Specific QA Creation
- The benchmark covers four regulation-intensive domains and six temporal-query categories, including exemptions, validity periods, event conditions, overrides, and scope restrictions.The categories include cases where an older clause remains valid for a specific population and rules valid only during a limited period.Source: §4.1 · p. 5
- Using Llama-3.3-70B, the authors generate 3,000 Korean QA instances per domain, each linked to the minimal gold documents needed under its temporal context.The benchmark contains 12,000 instances in total.Source: §4.1 · p. 5
4.2 Corpus Generation
- Evolving document families retain most content while revising selected clauses, making semantically similar but temporally invalid versions hard negatives.Gold documents are the minimal versions required to derive the answer; other versions in the family are temporally challenging non-gold documents.Source: §4.2 · pp. 5–6
4.4 TimelyQABench Analysis
- TimelyQABench provides comprehensive clause-level temporal annotations and regulation-focused domain coverage, whereas TimeQA and CRAG only partially annotate temporal signals.The benchmark is presented as practical for evaluating retrieval of the clause valid at query time.Source: Table 2 · p. 6, §4.4 · pp. 6–7
4.3 Multi-Dimensional Validation
- Three independent LLM validators assess relevance, answer derivability, time sensitivity, and domain specificity; instances are retained when validators agree on support and temporal reasoning.The filtering is intended to preserve examples requiring clause-level temporal alignment.Source: §4.3 · p. 6
4.4 TimelyQABench Analysis
- Candidate documents in TimelyQABench have substantially higher semantic overlap than those in existing time-sensitive QA benchmarks.The authors conclude that semantic matching alone is insufficient to identify the temporally valid version.Source: Table 3 · p. 6, §4.4 · p. 7
5 Experimental Results
Across overlapping- and disjoint-evolving benchmarks, TimelyRAG improves retrieval, and generation evaluations report higher win rates. It preserves performance on static retrieval while adding limited reranking overhead.
5.1 Experimental Setup
- With BM25, BGE-M3, NV-Embed-V2, and BGE-Gemma2 as first-stage retrievers, TimelyRAG is evaluated on overlapping-, disjoint-, and non-evolving benchmarks.Retrieval uses nDCG, MAP, Recall, and Hit metrics; downstream answer comparisons use the same generator and GPT-5 as judge.Source: §5.1 · p. 7
5.2.1 Overlapping-Evolving Benchmark
- Table 4 reports retrieval gains across sparse and dense backbones, including up to 24.8% nDCG@10 improvement over BM25 on University and 28.6% over BGE-Gemma2 on Terms of Service.On TimelyQABench, TimelyRAG also consistently outperforms deployable time-aware baselines with both sparse and dense retrieval.Source: §5.2.1 · pp. 7–8, Table 5 · p. 8
5.2.2 Disjoint-Evolving Benchmark
- On TS-Retriever, TimelyRAG consistently improves baselines across metrics, complementing lexical and semantic retrieval in disjoint-evolving settings.The reported gains favor temporally valid evidence over merely recent or lexically similar passages.Source: Table 6 · p. 8, §5.2.2 · p. 8
5.2.3 General RAG Benchmark
- On BEIR-FiQA, time-aware scoring becomes inactive without temporal context, preserving the original retriever performance.The authors report this result as maintaining baseline effectiveness on static corpora.Source: Table 7 · p. 8, §5.2.3 · p. 8, Figure 4 · p. 8
5.3 Generation Evaluation
- TimelyRAG-augmented generation achieves higher win rates across all domains and retrievers.The evaluation compares answers for correctness and temporal validity.Source: §5.1 · p. 7, Figure 4 · p. 8, §5.3 · p. 8
5.4 Temporal Sensitivity Analysis
- On University, optimal α(Q) values cluster near 1.0 for both sparse and dense retrievers, at 84.6% and 85.0%, respectively.The authors interpret this distribution as indicating a need for clause-level temporal validity rather than semantic similarity alone.Source: §5.4 · p. 8, Figure 5 · p. 9
5.5 Ablation Study
- Ablations indicate that query-document temporal alignment, event-time activation, and temporal granularity are effective for fine-grained time-critical retrieval.The variants use uniform distance-term weighting, remove the event-time indicator, or omit the granularity term.Source: §5.5 · pp. 8–9, Table 8 · p. 9
5.6 Computational Overhead
- TimelyRAG adds 2.15 ms/query to BM25 and 2.64 ms/query to BGE-M3 while improving nDCG@5, with further candidate-pool gains tapering after 50 candidates.The temporal module uses approximately 5.9 × 10^3 CPU-side operations per query; increasing the pool from 10 to 50 candidates substantially improves retrieval, with only marginal gains thereafter.Source: Table 9 · p. 9, §5.6 · p. 9, Table 10 · p. 9
6 Conclusion
TimelyRAG combines semantic relevance with clause-level temporal compatibility to distinguish similar versions with different validity. The conclusion presents TimelyQABench as a controlled benchmark for regulation-heavy documents with successive amendments.
6 Conclusion
- TimelyRAG is a retriever-agnostic framework for time-critical QA over overlapping-evolving documents.Its clause-level temporal compatibility is combined with semantic relevance.Source: §6 · p. 9
- TimelyQABench provides a controlled benchmark for regulation-heavy domains with successive amendments.The conclusion reports consistent retrieval gains across retrievers and improved downstream generation quality.Source: §6 · p. 9
- The authors conclude that reliable QA over evolving documents requires version-valid, clause-level temporal retrieval beyond document-level relevance.TimelyRAG distinguishes semantically similar versions with different temporal validity.Source: §6 · p. 9
Limitations
TimelyRAG’s temporal-distance function and the study’s data and retrieval setup impose several scope limits on the conclusions it can support.
Limitations
- The temporal-distance function does not yet model complex temporal logic, causal relations, or multi-event dependencies.Extending it to these cases remains future work.Source: Limitations · p. 10
- As a reranking framework, TimelyRAG depends on the first-stage retriever including relevant document versions in its candidate pool.Relevant versions absent from that pool are outside the reranker’s stated reach.Source: Limitations · p. 10
- Temporal signals may be harder to obtain without explicit amendment or effective-date conventions, and TimelyQABench is synthetically constructed.The paper identifies extension to real-world versioned corpora as future work.Source: Limitations · p. 10
Ethics Statement
The paper describes safeguards and intended use for its synthetic benchmark, then details how query and document time granularity inform temporal scoring.
Ethics Statement
- The synthetic benchmark may reflect biases or artifacts from its underlying models and prompts, despite multi-model validation and a 50-instance human check.The authors state that TimelyRAG is not a substitute for professional legal, policy, or regulatory advice.Source: Ethics Statement · p. 10
A Temporal Granularity
- Finer temporal granularity imposes a stronger temporal constraint, while coarser granularity represents broader or more general time references.Granularity balances semantic relevance and temporal compatibility through the gating function.Source: A Temporal Granularity · p. 12
A.1 Final Granularity
- The final granularity uses the finer query or document resolution, smoothing large gaps by moving one level toward the finer side when the gap is at least two.A finer final granularity increases the temporal-compatibility weight relative to semantic similarity.Source: §A.1 · p. 12
- When query event time is missing, granularity defaults to month; missing document times fall back to insertion time subject to DIT < QIT.With multiple query-event or document-event times, the pair minimizing Eq. (4) is selected.Source: §A.1 · p. 12
A.2 Query-Side Estimation
- Query-side granularity is estimated from explicit time expressions, selecting the finest matched level and defaulting to month when none appears.The levels range from hour to year and are identified from corresponding timestamp patterns.Source: §A.2 · p. 12
A.3 Document-Side Estimation
- Document-side granularity is the finest resolution present in at least 70% of candidate documents, based on hour, day, and month timestamp proportions.This estimate is combined with the query-side estimate to obtain final granularity.Source: §A.3 · p. 12
B Multi-Dimensional Validation Results
Across the evaluated backbones, TimelyRAG consistently improves retrieval on a benchmark built around regulation-focused, overlapping versions; validation also reports high relevance and consistency between a pilot human review and LLM judgments.
B Multi-Dimensional Validation Results
- TimelyQABench has an average relevance score of 4.93 and requires 3.10 passages on average to derive a correct answer.The relevance score is on a five-point scale; the passage count reflects the need to consult multiple revised versions.Source: B Multi-Dimensional Validation Results · p. 13
C.1 Backbone Retrievers
- The evaluated backbones include BM25, BGE-M3, Sentence-BERT, DPR, Contriever, and ColBERT, each tested both alone and with TimelyRAG reranking.The set includes lexical, dense, and late-interaction retrieval approaches.Source: §C.1 · p. 13
C.2 Datasets
- TimelyQABench covers legal, university, and company regulations, plus terms of service; comparisons also include TS-Retriever and FiQA.TS-Retriever is based on Wikipedia revisions and timestamped questions; FiQA covers financial-domain questions.Source: §C.2 · p. 13
C.3 Results and Analysis
- Every retriever improves consistently across all metrics with TimelyRAG, while baseline scores on TimelyQABench are strikingly low.The benchmark’s versions share substantial semantic overlap and differ in clause-level validity periods, effective dates, or scope restrictions.Source: Table 12 · p. 13, §C.3 · p. 14
C.4 Human Validation
- On 50 randomly sampled queries, two independent annotators found a trend consistent with full-dataset LLM validation and no substantial discrepancy.Both evaluations indicated high relevance and domain specificity.Source: §C.4 · p. 14, Table 13 · p. 14
C.5 Category-Wise Analysis
Category results show mostly preserved or improved performance with one retrospective-query degradation, while expanded ablations report gains across domains and retriever types.
C.5 Category-Wise Analysis
- On the Company dataset, TimelyRAG improves or preserves nDCG@10 in most categories, but degrades clearly for Historical, Retrospective Fact.The authors note that retrospective queries may require more precise handling of past validity intervals.Source: §C.5 · p. 14, Table 14 · p. 14
C.6 Expanded Ablation
- TimelyRAG outperforms Naive-Dist and variants without event-time activation or temporal-granularity modeling in all eight settings.The settings span four domains and sparse and dense retrievers.Source: §C.6 · p. 14, Table 15 · p. 14
D.1 Benchmark Dataset Generation Prompt
- The benchmark-generation prompt specifies query timestamps, semantically overlapping but temporally distinct documents, and gold document labels.It is designed to create realistic time-sensitive Korean QA pairs for regulation-intensive domains.Source: §D.1 · pp. 14–15
D.2 Benchmark Validation Prompt
The benchmark validation prompt standardizes assessment of QA relevance, required documents, temporal characteristics, and domain-specific language. It is presented alongside the benchmark-generation prompt.
D.2 Benchmark Validation Prompt
- The validation prompt asks annotators to assess query relevance to its category and identify documents needed to derive the correct answer.Source: §D.2 · p. 15
- Annotators identify the temporal types involved and describe how frequently an answer may change over time.Source: §D.2 · p. 15
- The prompt also evaluates domain-specific language in queries and documents, aiming to standardize validation across the dataset.Source: §D.2 · p. 15
- Table 16 is the prompt for generating the time-critical Korean QA and corpus benchmark.Source: Table 16 · p. 16
- Table 17 is the validation prompt for annotating time-critical QA samples across four dimensions.Source: Table 17 · p. 17