PaperlayerView exact revision on arXiv

Source-linked AI summary

Intent-Driven Dynamic Chunking: Segmenting Documents to Reflect Predicted Information Needs

Christos Koutsiaris

arXiv:2602.14784v1cs.IRcs.AIcs.CLcs.LG

TL;DR

Traditional document chunking can misalign segments with users’ information needs, fragmenting answers or adding irrelevant text. IDC predicts likely user queries and optimizes chunk boundaries with dynamic programming; across six QA datasets, it outperformed traditional methods on five and matched the best baseline on the sixth.

  • Problem

    Fixed-length and coherence-based segmentation is query-agnostic, risking fragmented answers or irrelevant text when document segments do not match users’ information needs.

  • Method

    IDC predicts likely user queries with a generative model and uses dynamic programming to optimize document boundaries around those predicted intents.

  • Results

    Across six QA datasets, IDC outperformed traditional chunking on five and matched the best baseline on the sixth, improving R@1 by 5%–67%.

  • Takeaways & Limitations

    IDC produced 40–60% fewer chunks while achieving 93–100% answer coverage, supporting efficient indexing for long, heterogeneous documents.

  • Takeaways & Limitations

    IDC depends on LLM-generated intents, and some datasets had small sample sizes (n=15), limiting statistical power.

Abstract

from arXiv · show

Breaking long documents into smaller segments is a fundamental challenge in information retrieval. Whether for search engines, question-answering systems, or retrieval-augmented generation (RAG), effective segmentation determines how well systems can locate and return relevant information. However, traditional methods, such as fixed-length or coherence-based segmentation, ignore user intent, leading to chunks that split answers or contain irrelevant noise. We introduce Intent-Driven Dynamic Chunking (IDC), a novel approach that uses predicted user queries to guide document segmentation. IDC leverages a Large Language Model to generate likely user intents for a document and then employs a dynamic programming algorithm to find the globally optimal chunk boundaries. This represents a novel application of DP to intent-aware segmentation that avoids greedy pitfalls. We evaluated IDC on six diverse question-answering datasets, including news articles, Wikipedia, academic papers, and technical documentation. IDC outperformed traditional chunking strategies on five datasets, improving top-1 retrieval accuracy by 5% to 67%, and matched the best baseline on the sixth. Additionally, IDC produced 40-60% fewer chunks than baseline methods while achieving 93-100% answer coverage. These results demonstrate that aligning document structure with anticipated information needs significantly boosts retrieval performance, particularly for long and heterogeneous documents.

I. INTRODUCTION

Intent-Driven Dynamic Chunking (IDC) aligns document segmentation with predicted user queries through dynamic programming, addressing the query-agnostic limitations of fixed-length and coherence-based chunking. Across six QA benchmarks, IDC improved retrieval on five datasets, reduced chunk counts, maintained high answer coverage, and added minimal offline-indexing cost.

  • Motivation: Fixed-length chunking can fragment answers or dilute relevance, while coherence-based methods optimize document structure rather than users’ information needs.These limitations motivate segmentation that accounts for anticipated questions instead of relying only on uniform windows or discourse boundaries.
  • Method: IDC predicts likely user queries and uses dynamic programming to create answer-sized chunks that optimally answer those predicted questions.The method is designed to produce focused chunks containing complete, relevant information without excess content.
  • Evaluation: 5% to 67% gains in Recall@1 occurred on five of six QA datasets, while IDC tied the best baseline on the sixth.The evaluation covered six benchmarks across four domains.
  • Efficiency: 40–60% fewer chunks than baselines were produced alongside 93–100% answer coverage, improving indexing efficiency.IDC is intended to retain complete answers while using fewer indexed segments.
  • Efficiency: <$0.01 per long document was the reported overhead for IDC, supporting its suitability for offline indexing.The work analyzes IDC’s efficiency and cost in addition to retrieval performance.

II. RELATED WORK · A. Document Segmentation Methods

Document segmentation methods evolved from simple fixed-length windows to coherence-based and supervised approaches. However, coherence-based methods remain query-agnostic, motivating intent-driven segmentation that considers users’ potential questions.

  • A. Document Segmentation Methods: Fixed-length chunking remains common because it is simple.Document segmentation research spans several decades.
  • A. Document Segmentation Methods: Callan found that fixed windows often divide answers between chunks.
  • A. Document Segmentation Methods: Retrieval performance breaks down quickly when segment length deviates from optimal values.This result was reported by Wartena.
  • A. Document Segmentation Methods: TextTiling detects topic shifts through lexical-cohesion valleys of low similarity.
  • A. Document Segmentation Methods: C99 clusters sentences by semantic similarity to identify topic boundaries.
  • A. Document Segmentation Methods: Coherence-based methods create internally consistent segments but optimize document structure without considering user queries.This query-agnostic limitation motivates the intent-driven approach.

B. Query-Aware Document Expansion · III. METHODOLOGY · A. Overview of IDC

Prior query-aware retrieval methods predict queries to expand document content but leave segmentation unchanged, whereas IDC uses predicted user intents to optimize document chunk boundaries. IDC comprises Intent Simulation and Boundary Optimization, aligning chunks with likely information needs.

  • B. Query-Aware Document Expansion: doc2query predicts likely answerable questions and appends them before indexing to bridge vocabulary gaps.docT5query extends this approach with T5-generated questions that are more diverse and fluent, improving retrieval gains.
  • B. Query-Aware Document Expansion: docT5query uses T5 to generate more diverse, fluent questions than doc2query.The passage reports improved retrieval gains for docT5query.
  • B. Query-Aware Document Expansion: InPars uses GPT-3 to create synthetic query-document training pairs, while Promptagator generates useful query variations with minimal examples.Both methods extend query-aware document expansion through large language model prompting or generation.
  • B. Query-Aware Document Expansion: Expansion methods leave underlying document splitting unchanged, so appended questions cannot repair information fragmented across suboptimal chunks.The generated queries become part of each document’s index entry rather than changing its segmentation.
  • B. Query-Aware Document Expansion: IDC extends query prediction from content expansion to document structure by using predicted queries to drive segmentation.This addresses fragmentation caused when important information is split across chunks.
  • A. Overview of IDC: IDC realigns document segmentation with user information needs through Intent Simulation and Boundary Optimization.Intent Simulation predicts likely user queries, while Boundary Optimization segments the document to maximize alignment between chunks and predicted intents.

B. Intent Simulation · C. Sentence Embedding and Scoring · D. Boundary Optimization

IDC simulates likely user intents, scores candidate sentence chunks in a shared embedding space, and uses dynamic programming to select globally optimized boundaries. Length and boundary penalties, bounded chunk lengths, and light post-processing regulate chunk size and segmentation complexity.

  • B. Intent Simulation: IDC uses Gemini 2.5 Flash to generate hypothetical questions covering a document’s main topics and key details, with section-wise generation and top-k sampling for longer documents.Section-wise generation supports topic coverage, while stochastic decoding promotes intent diversity.
  • B. Intent Simulation: 10–15 questions are generated for documents under 100 sentences, versus 35–40 questions for documents over 400 sentences; redundant questions above 0.85 cosine similarity are removed.The adaptive counts aim to provide coverage without over-segmentation.
  • C. Sentence Embedding and Scoring: 1536-dimensional transformer embeddings encode sentences and predicted intents in a shared vector space, while each candidate chunk embedding averages its sentence embeddings.This representation enables intent-aware scoring of candidate chunks.
  • C. Sentence Embedding and Scoring: Intent relevance R(Ci,j) quantifies how well a candidate chunk could answer at least one predicted question.The score connects chunk content to the simulated information needs.
  • D. Boundary Optimization: The utility function maximizes segmentation quality while λ penalizes overly long chunks and β penalizes over-segmentation.Because |Cm|^2 grows quickly with chunk size, λ is typically small, such as 0.0005 after tuning.
  • D. Boundary Optimization: Dynamic programming defines f(j) as the maximum utility for segmenting sentences 1 through j, using f(0) = 0 and considering chunks up to 10–15 sentences.The maximum-length constraint reduces complexity to O(N × L), essentially linear in document length.
  • D. Boundary Optimization: The hybrid architecture assigns semantic reasoning to the LLM and structural boundary optimization to dynamic programming, avoiding the lack of global optimality from left-to-right marker insertion.After optimization, very short adjacent chunks may be merged and overly long chunks split at natural paragraph boundaries.

IV. EXPERIMENTAL SETUP · A. Datasets · B. Baselines

The evaluation covered six question-answering datasets across four domains and compared IDC with four segmentation baselines under standardized retrieval preprocessing. The datasets varied substantially in document length and structure, while baselines represented fixed-length, overlapping-window, coherence-based, and paragraph-based segmentation.

  • A. Datasets: Six question-answering datasets spanned news articles, Wikipedia, academic papers, and technical documentation, with documents ranging from 12–495 sentences.Table I reports the dataset characteristics.
  • B. Baselines: IDC was compared against four baseline segmentation strategies.The baselines covered fixed-length, sliding-window, coherence-based, and paragraph-based approaches.
  • B. Baselines: Fixed-Length used non-overlapping 6-sentence chunks.
  • B. Baselines: Sliding Window used 6-sentence chunks with 50% overlap.
  • B. Baselines: Coherence-Based applied TextTiling-like topic boundary detection.
  • B. Baselines: Paragraph-Based used natural paragraph breaks as segmentation boundaries.
  • B. Baselines: All methods used identical sentence tokenization, embedding models, and hybrid retrieval combining 60% dense retrieval with 40% BM25.

C. Evaluation Metrics · V. RESULTS · A. Retrieval Performance

The evaluation used retrieval accuracy, ranking quality, chunk counts, and answer coverage, with IDC achieving the best or tied-best top-1 retrieval on all six datasets. Its largest gains occurred on long, heterogeneous documents, while Qasper favored paragraph-based segmentation on some metrics.

  • C. Evaluation Metrics: R@1, R@5, and MRR measured retrieval performance, alongside chunk counts and the percentage of answers fully contained within single chunks.R@1 is the fraction of queries whose top-ranked chunk contains the answer.
  • A. Retrieval Performance: IDC achieved the highest R@1 on five of six datasets and tied on the sixth, Qasper.These results are reported in Table II for Recall@1, Recall@5, and MRR.
  • A. Retrieval Performance: 0.667 R@1 on the 495-sentence arXiv paper exceeded baselines’ 0.400, a 67% relative improvement.The largest improvements occurred on long, heterogeneous documents.
  • A. Retrieval Performance: 0.533 R@1 on Fiori technical documentation exceeded 0.333, representing a +60% improvement.Fiori was among the long-document settings where IDC’s gains were most pronounced.
  • A. Retrieval Performance: On SQuAD 2-doc, IDC’s improvement was statistically significant across 293 queries, with p < 0.05 and Cohen’s d ≈0.41.The dataset-level result was reported for a large SQuAD 2-doc set.
  • A. Retrieval Performance: On Qasper, IDC tied Paragraph on R@1 at 0.250 but had lower R@5 (0.500 vs 0.600) and MRR (0.333 vs 0.367).The passage attributes this exception to structured papers whose section boundaries naturally align with distinct questions.

B. Segmentation Efficiency · C. Efficiency Analysis

IDC reduced chunk counts while improving answer coverage, keeping complete answers intact. Its preprocessing is offline, with fast dynamic-programming segmentation, low per-document costs, and retrieval latency matching baselines.

  • B. Segmentation Efficiency: 39 versus 83 chunks on arXiv yielded a 53% reduction, while 177 versus 304 on Fiori yielded a 42% reduction.Fewer chunks reduce index sizes and accelerate retrieval.
  • B. Segmentation Efficiency: 93.3% versus 80% answer coverage on arXiv and 100% versus 86.7% on Fiori show that IDC preserves complete answers.The comparisons are against Fixed on arXiv and baselines on Fiori.
  • C. Efficiency Analysis: 1–2 seconds for short documents and 10–15 seconds for very long documents (>400 sentences) characterize IDC’s offline preprocessing time.Intent generation dominates at ∼1s via Gemini 2.5 Flash API, while DP segmentation takes <200ms.
  • C. Efficiency Analysis: ∼500ms online retrieval latency is identical for IDC and baselines because query embedding and index lookup dominate.IDC’s preprocessing is entirely offline.
  • C. Efficiency Analysis: ∼$0.0002–0.0005 per document is the Gemini 2.5 Flash preprocessing cost for short documents (<100 sentences).The cost analysis varies by document length.
  • C. Efficiency Analysis: ∼$0.002–0.005 per document applies to long documents (400+ sentences, ∼15k tokens).For 1,000 documents, total preprocessing costs range from $0.20 for short documents to $5.00 for long documents.
  • C. Efficiency Analysis: $0.20 versus $5.00 is the estimated total preprocessing cost for 1,000 short versus long documents.These estimates assume parallelization is feasible; API rate limits may bottleneck large-scale processing.

VI. DISCUSSION

IDC improves retrieval by aligning chunk boundaries with predicted information needs, creating focused, answer-sized segments through dynamic sizing. Its benefits are greatest for long, heterogeneous documents, but diminish when existing structure already matches likely query boundaries.

  • Why IDC Works: IDC creates answer-sized segments by predicting likely user questions, unlike fixed-length and coherence-based methods that overlook query relevance.Its boundaries contain complete, focused content rather than arbitrarily fragmented or merely topically consistent information.
  • When IDC Excels: Fiori +60%, arXiv +67%, and SQuAD 2-doc +5% are the largest gains, occurring on long, heterogeneous documents and multi-document collections.These settings are especially challenging for static segmentation.
  • When IDC Excels: IDC generates 40–60% fewer chunks than baselines while achieving 93–100% answer coverage.Baselines achieve 80–87% answer coverage.
  • Why IDC Works: Dynamic chunk sizing outperforms uniform approaches by using larger segments for broad explanations and smaller segments for specific facts.This adapts chunk size to the breadth of the anticipated information need.
  • When IDC Ties Baselines: On well-structured Qasper papers, paragraph-based segmentation achieves comparable results because existing boundaries already align with distinct topics and questions.IDC provides no advantage when document structure reflects likely query boundaries.
  • Limitations: IDC depends on accurate LLM-generated intents, faces limited statistical power with datasets as small as n=15, and adds offline indexing time.The offline processing cost is acceptable for most applications.

VII. CONCLUSION

The conclusion presents Intent-Driven Dynamic Chunking (IDC), which uses LLM-generated likely questions and dynamic programming to align document segments with predicted information needs. Across six QA datasets, IDC improved retrieval over traditional chunking on five datasets while reducing chunk counts, and it is especially suited to long, heterogeneous documents with minimal offline-indexing overhead.

  • Core contribution: IDC segments documents using LLM-generated likely questions and dynamic programming to optimize boundaries around predicted user intents.This produces segments aligned with actual information needs.
  • Evaluation: 5 of 6 QA datasets showed IDC outperforming traditional chunking methods, with R@1 improvements ranging from 5% to 67%.The evaluation covered six diverse question-answering datasets.
  • Efficiency: 40–60% fewer chunks were produced while maintaining high retrieval effectiveness across the evaluated datasets.The supplied conclusion passage reports fewer chunks but truncates the accompanying effectiveness value.
  • Practical advantages: IDC is particularly effective for long, heterogeneous documents where static segmentation fails to isolate relevant content.It adds minimal computational overhead for offline indexing and does not affect query-time latency.
  • Future work: Future work will address multi-hop query synthesis, real-user-feedback-driven re-segmentation, and domain-specialized intent generation for technical corpora.These directions extend IDC beyond single-chunk retrieval and toward adaptive, domain-aware segmentation.
Loading 2602.14784v1…