PaperlayerView exact revision on arXiv

Source-linked AI summary

ChunkRank: Model-Aware Text Chunking and Abstention-Aware Answer Selection for LLM Pipelines

Amit Nautiyal, Ayush Bhatt, Gaurav Nautiyal

arXiv:2609.29828v1cs.CL

TL;DR

Long documents can exceed fixed model context windows, while existing splitters may require manual sizing that mismatches a model’s tokenizer. ChunkRank derives model-aware chunks and selects among per-chunk answers; across tested tasks, content-based rankers do not reliably beat first-non-empty selection, with reader abstention explaining the result.

  • Problem

    Long documents exceed fixed model context windows, and existing splitters require user-supplied sizes without consulting the target model’s tokenizer.

    Source: §1 · p. 1

  • Method

    ChunkRank derives chunk boundaries from model token budgets and selects among non-empty answers produced independently for each chunk.

    Source: §1 · p. 1, §4.3 · p. 5, §4.4 · p. 5

  • Results

    Across NaturalQuestions, TriviaQA, and HotpotQA, no content-based ranker reliably beats first-non-empty selection; reader abstention, not answer position, explains the pattern.

    Source: §7.5 · p. 9, §8 · p. 10

  • Takeaways & Limitations

    Chunking is not needed when a document fits a large-window model, delimiting where the resource applies.

    Source: §8 · p. 10

    The character-ratio fallback is calibrated for Latin-script English and can under-count Chinese or Japanese tokens by up to 71%.

    Source: §9 · p. 11

Abstract

from arXiv · show

We present ChunkRank, an open-source Python library that derives chunk boundaries from a target model's tokenizer and context window, and selects an answer among candidates produced independently per chunk. It ships a validated registry of 90 models across 15 providers and six answer-selection methods, and needs only three core dependencies. For chunking, ChunkRank avoids context-window overflow automatically from the model name, whereas character-based splitters overflow or waste the budget, and a fidelity study across 11 languages shows why token-exact budgets matter beyond English. For answer selection we report a negative result: on NaturalQuestions, TriviaQA and HotpotQA, with extractive and generative readers, no content-based ranker reliably beats taking the first non-empty answer. The reason is reader abstention on chunks that lack the answer, not answer position. A long-context baseline shows that chunking matches single-call reading on single-hop questions, so ChunkRank targets small-window and beyond-window settings. Code, registry and evaluation harness are released.

1 Introduction

ChunkRank addresses the need to split documents for fixed model context windows and the lack of standalone, model-aware tooling for chunking and post-chunk answer selection. Its evaluation finds that reader abstention makes first-non-empty selection difficult for content-based rankers to beat.

1 Introduction

  • Existing splitters require user-supplied sizes and ignore the target model’s tokenizer, so the same character window can represent 100–400 tokens across encoders.Documents routinely exceed model context windows, making reliable splitting a recurring need.Source: §1 · p. 1
  • ChunkRank packages model-aware chunking and answer ranking as reusable, standalone components rather than utilities embedded in larger retrieval frameworks.Its contributions include a versioned registry of 90 models across 15 providers.Source: §1 · pp. 1–2
  • Across NaturalQuestions, TriviaQA, and HotpotQA, no content-based ranker reliably beats first-non-empty selection because reader abstention dominates chunk-level answer selection.The study includes extractive and generative readers and paired bootstrap tests.Source: §1 · p. 2
  • ChunkRank makes overflow and answer selection explicit in agent loops, offline document workflows, and RAG systems without requiring a pipeline rewrite.The motivation includes answer multiplicity: independently queried chunks can yield different answers, while applications often return the first chunk’s answer.Source: §1 · pp. 1–2

2 Related Work

Prior work offers general NLP toolkits, chunking utilities, and retrieval rankers, but does not package automatic model-to-tokenizer-and-budget resolution with standalone post-chunk answer selection. ChunkRank positions itself as lightweight middleware rather than a retrieval system.

2.1 NLP Toolkits and Resource Papers

  • General NLP toolkits provide stable, versioned APIs for common tasks, while ChunkRank targets model-aware segmentation and answer selection as a standalone component.The paper situates ChunkRank in the resource tradition of NLTK, spaCy, Stanza, Flair, and Transformers.Source: §2.1 · p. 2

2.2 Text Chunking in LLM Pipelines

  • Existing splitters still require manual chunk-size or tokenizer configuration; ChunkRank’s distinguishing feature is automatic resolution from a named model.The comparison defines model awareness as deriving chunk size from the target tokenizer and context window without manual configuration, and auto registry as shipping named-model parameters.Source: §2.2 · pp. 2–3, Table 1 · p. 3

2.3 Answer Ranking and Passage Re-ranking

  • Retrieval rankers select relevant passages from a corpus, whereas ChunkRank selects among answers from chunks of one document without an index or retrieval stack.The related work describes the latter as a structurally simpler post-chunk setting.Source: §2.3 · p. 3

2.4 Retrieval-Augmented Generation

  • ChunkRank is not a RAG system, but can be used within one; its middleware role remains relevant as context budgets vary across models.The paper notes that long contexts do not eliminate chunking because cost scales with length, positional attention can degrade, and many self-hosted or edge models retain small budgets.Source: §2.4 · p. 3

3 Problem Definition

The paper formalizes model-aware chunking as a token-budget-constrained partition and post-chunk selection as scoring non-empty answers. These definitions distinguish hard budget compliance from the softer goal of preserving coherence.

3 Problem Definition

  • The effective chunk budget is the model’s maximum context window minus tokens reserved for the prompt template and generated output.Source: §3 · p. 3

3.1 Model-Aware Chunking

  • Chunks must satisfy the model tokenizer’s hard token limit, while coherence is a soft objective and configurable overlap trades storage for continuity.Source: §3.1 · p. 3

3.2 Post-Chunk Answer Selection

  • Selection chooses among non-empty per-chunk answers by scoring candidate content, document position, or reader confidence; it selects rather than generates a new answer.The candidate set contains answers produced for each chunk, excluding empty outputs.Source: §3.2 · p. 3

4 System Design

ChunkRank uses replaceable components for model configuration, tokenization, chunking, and answer ranking. Its design supports configurable token budgets and multiple chunking and selection strategies.

4 System Design

  • The four sequential components expose replaceable interfaces, allowing incremental adoption rather than an all-or-nothing framework.Source: §4 · p. 3

4.1 Model Registry

  • 90 preconfigured models across 15 providers are stored in JSON; validation caught and corrected two stale context windows.The registry is the source of model-specific parameters, and its validation harness is released for rerunning checks as the registry grows.Source: §4.1 · p. 4

4.2 Tokenizer Adapter

  • The tokenizer adapter unifies token counting and encoding; its character-ratio fallback is dependency-free but less accurate for non-Latin scripts.The fallback approximates token count as ⌊|s|/4⌋; tokenizer backends can be switched through registry updates without application-code changes.Source: §4.2 · p. 4

4.3 Chunker

  • The default sliding-window chunker binary-searches character ranges to meet token budgets in O(log B) tokenizer calls per chunk, without sentence-boundary detection.The optional semantic strategy groups sentences by cosine similarity before enforcing the budget, at the cost of a one-time embedding pass.Source: §4.3 · p. 5

4.4 Ranker

  • The ranker drops empty answers first and defaults to first-non-empty, breaking score ties by document order.The six methods include reader confidence, BM25, TF-IDF, dense embeddings, and cross-encoders; first-non-empty matches or beats tested content-based methods on the reported datasets.Source: §4.4 · p. 5
  • Lexical and semantic rankers trade scoring capability for implementation cost: BM25 and TF-IDF are deterministic, while embeddings and cross-encoders require added dependencies or slower inference.Cross-encoders score each candidate with a separate forward pass; dense embeddings are sensitive to encoder choice.Source: §4.4 · pp. 5–6

5 Implementation

ChunkRank keeps deployment lightweight while offering multiple integration levels and support for asynchronous execution, caching, and Python 3.10–3.13.

5.1 Dependency Philosophy

  • The core system needs only three runtime dependencies; neural backends are optional, and PyTorch is imported only for embedding or cross-encoder selection.The core dependencies are numpy, scikit-learn, and rank-bm25.Source: §5.1 · p. 6

5.2 API Design

  • Three API levels range from a one-shot function to a stateful pipeline and direct chunker and ranker components.The stateful Pipeline API caches chunker configuration across repeated queries on one document.Source: §5.2 · p. 6

5.3 Async, Caching, and Compatibility

  • Async pipelines run CPU-bound work in threads and LLM calls as coroutines; both support streaming, while a standard-library disk cache reuses chunks by SHA-256 key.The cache key covers text, model, strategy, and overlap.Source: §5.3 · p. 6
  • ChunkRank targets Python 3.10–3.13, and its test suite passes on all four versions.Registry access uses importlib.resources.files().Source: §5.3 · p. 6

6 Resource Release and Reproducibility

The release includes the library, maintained artifacts, and evaluation materials intended to support reuse, reproducibility, and extension.

Resource release

  • The release provides versioned artifacts and documentation describing their availability and maintenance.The released artifacts are intended to support reproduction and extension of the work.Source: §6 · p. 6

Distribution, licensing, and versioning

  • ChunkRank is distributed on PyPI and GitHub under Apache 2.0, with semantic versioning and a separately diffable model registry.The paper's results use release 2.0.0.Source: §6 · p. 6

Test coverage and evaluation artifacts

  • The public test suite covers the registry, tokenizers, chunking strategies, six selection methods, caching, and sync and async pipelines on Python 3.10–3.13.The released evaluation harness includes dataset filtering, configuration, and per-method result tables for independent checking and extension.Source: §6 · p. 6

Computational reproducibility

  • Reported results use pinned software dependencies and exact model revisions, and the neural rankings were verified bit-identical on CPU and Apple MPS.The evaluation harness calls the installed package's Ranker directly for each selection method.Source: §6 · pp. 6–7

Extensibility

  • Three extension points allow community additions through registry entries, chunking strategies, and ranking methods without forking the library.Extensions use the same Chunker and Ranker interfaces as internal components.Source: §6 · p. 7

7 Experiments

ChunkRank is evaluated for long-document QA, model-aware chunking, and answer selection. Chunking avoids overflow and preserves token budgets across languages, but tested content-based selectors do not reliably improve on first-non-empty; a long-context comparison identifies settings where chunking adds no benefit or can split multi-hop evidence.

7.1 Task and Motivation

  • The evaluation targets single-hop QA over long documents, where a small, predefined set of per-chunk answers must be selected rather than retrieved from a large corpus.The authors position the released harness as a reusable evaluation resource for this setting.Source: §7.1 · p. 7

7.2 Experimental Setup

  • NQ and TriviaQA each contribute 500 examples with answer contexts exceeding 8,000 tokens; extractive answers are produced per chunk, and selectors are compared over non-empty candidates using EM and token-level F1.The extractive reader can abstain on chunks; Table 2 defines empty-chunk and no-answer-example rates, while the evaluation uses 95% bootstrap intervals and paired tests against first-non-empty.Source: §7.2 · p. 7, Table 2 · p. 7

7.3 Chunking Correctness

  • For 500 TriviaQA documents, ChunkRank automatically achieves zero overflow from the model name, while token-aware alternatives require manual tokenizer configuration.ChunkRank's mean utilisation is 75.9% versus 84–86% for token-aware alternatives, a deliberate safety margin; character-based splitting overflows or requires wasteful shrinking.Source: Table 3 · p. 8, §7.3 · p. 8

7.4 Multilingual Tokenizer Fidelity

  • Across 11 languages, the character-ratio fallback undercounts some logographic and syllabic scripts by up to 71%, whereas tokenizer-derived budgets are safe across the tested scripts.A 6,000-English-token character window can hold about 20,000 Japanese tokens, or 3.4 times the budget.Source: §7.4 · p. 8

7.5 Answer Selection Results

  • Across the extractive-reader evaluation, no content-based ranker significantly beats first-non-empty on NaturalQuestions or TriviaQA, while reader abstention helps explain the result.BM25 and TF-IDF scores tie on 79–90% of multi-candidate examples, and real oracle headroom remains uncaptured by tested selectors.Source: §7.5 · p. 8, Table 4 · p. 9

7.6 Validation with a Generative LLM

  • With Claude Haiku 4.5, first-non-empty still is not beaten by embedding on NQ or TriviaQA, despite the generative reader leaving fewer examples with no answer.The run used 100 examples per dataset; the oracle remained above every method, including 68.0 EM on TriviaQA.Source: §7.6 · p. 9

7.7 Latency Analysis

  • On CPU, chunking up to 11 chunks takes under 15 ms, and the whole pipeline adds under 50 ms.Selection ranges from effectively free for first, confidence, and BM25 to about 30 ms for neural methods at 5–20 candidates.Source: §7.7 · p. 9

7.8 When Not to Chunk: A Long-Context Baseline

  • When documents fit Claude Haiku 4.5's 200k-token window, single-call and chunked reading are statistically indistinguishable on single-hop NQ and TriviaQA, while long-context reading wins on the HotpotQA multi-hop subset.For the single-hop tasks, input-token costs are essentially equal; the authors attribute the HotpotQA result to splitting evidence across chunks and limit the comparison's scope.Source: §7.8 · pp. 9–10, Table 5 · p. 10

8 Conclusion

ChunkRank is a lightweight, modular resource for model-aware chunking and post-chunk answer ranking. Its evaluations validate the chunking contribution but find no tested content-based selector reliably beats first-non-empty.

8 Conclusion

  • ChunkRank packages model-aware chunking and post-chunk ranking as standalone, lightweight, modular middleware, with a 90-model registry, three-tier API, PyTorch-free core, and three dependencies.The resource is extensible without forking, and evaluation code, configuration, and results are released for replication.Source: §8 · p. 10
  • Across three datasets and two reader types, no tested content-based ranker reliably beats first-non-empty, while chunking avoids overflow automatically from the model name.The long-context baseline also delimits when chunking is unnecessary: it is not needed when a document fits a large-window model.Source: §8 · p. 10
  • The authors identify future work in multilingual calibration data, learned chunk boundaries, multi-document fusion, and CPU-only embedding backends.The conclusion frames the contribution as a reproducible, community-maintainable resource rather than a stronger re-ranker.Source: §8 · p. 10

9 Limitations

ChunkRank is currently limited to single-document, single-hop use and selects among pre-generated answers rather than synthesizing evidence across chunks. Its tokenizer fallbacks and evaluation scope also impose boundaries on the overflow guarantee and on how broadly the answer-selection findings apply.

9 Limitations

  • Post-chunk selection cannot synthesize evidence across chunks, placing multi-document and multi-hop question answering outside the current scope.The HotpotQA study concatenates distractor paragraphs into one document and tests positional effects, not multi-hop reasoning.Source: §9 · p. 10
  • Because the ranker selects among pre-generated answers without merging complementary information, answers spanning multiple chunks may be incomplete.The evaluation focuses on factoid QA where one chunk contains the complete answer; cross-chunk synthesis needs different criteria and metrics.Source: §9 · pp. 10–11
  • Both chunking strategies use fixed rules; the paper does not explore learned approaches that optimize boundaries for downstream tasks.The authors identify learned chunk boundaries as future work.Source: §9 · p. 10
  • The len/4 character-ratio fallback is inaccurate for Chinese, Japanese, and Korean; users processing such text should install tiktoken or the target model’s Hugging Face tokenizer.The paper describes multilingual calibration data as a future community contribution.Source: §9 · p. 11
  • The semantic-similarity strategy requires sentence-transformers, reintroducing a PyTorch dependency that conflicts with the lightweight design goal.Lightweight CPU-only embedding backends are planned for a future release.Source: §9 · p. 11
  • The answer-selection evaluation focuses on factoid QA; summarization, multi-hop reasoning, and open-ended generation require reconsidered ranking criteria and metrics.The implemented post-chunk ranking is not suited to queries whose answers require synthesizing information across chunks.Source: §9 · p. 11
  • The evaluation uses Wikipedia-derived datasets; generative validation covers one provider model at n=100 per dataset, and untested learned or reader-aware selectors could still beat first-non-empty.The released harness can be extended to new domains, generators, and selectors without re-implementation.Source: §9 · p. 11
  • The zero-overflow guarantee is exact for 49 models with native tokenizers but approximate for 41 proxy-mapped models; the character fallback can under-count Chinese and Japanese tokens by up to 71%.At a 128k-token budget, an 11.3% proxy under-count is about 14k tokens, far beyond the stated 512–1,024-token reserve; exact tokenizers are recommended for proxy-mapped models or very large budgets.Source: §9 · p. 11

Ethics Statement

The ethics statement describes ChunkRank as local infrastructure without telemetry or independent external-service calls. The accompanying material outlines its pipeline, APIs, token-budget chunking algorithm, and multilingual tokenizer-fidelity results.

Ethics Statement

  • ChunkRank maintains no server, telemetry, or persistent state beyond an optional user-controlled local disk cache, and it does not independently call external services.The only network access in a typical deployment is the LLM API call made by the user’s application, outside ChunkRank’s control.Source: Ethics Statement · p. 11

A Architecture Diagram

  • Figure 1 distinguishes primary data flow with solid arrows from the optional disk-cache path with dashed arrows.It defines WM as the context window, TM as the tokenizer, and BM = WM − RM as the effective chunk budget.Source: Figure 1 · p. 14

B Three-Tier API

  • The examples present three API levels: a one-shot function, a stateful pipeline, and components that let users call their own LLM between chunking and ranking.The component example splits text, obtains an answer per chunk, and ranks those answers.Source: B Three-Tier API · p. 14

C Token-Budget Sliding Window

  • The default sliding-window chunker binary-searches a character range to meet the token budget, using O(log B) tokenizer calls per chunk.The pseudocode appends the selected text range and applies overlap when configured.Source: C Token-Budget Sliding Window · p. 14

D Multilingual Tokenizer Fidelity

  • The appendix identifies Table 6 as the source of the full per-language results for the multilingual tokenizer-fidelity evaluation.The supplied passage does not state the table’s values or comparisons.Source: D Multilingual Tokenizer Fidelity · p. 14

E Proxy-Tokenizer Error

The o200k base proxy under-counts tokens for measured non-OpenAI models, creating overflow risk that the default per-model reserve absorbs; proprietary-tokenizer models cannot be measured directly.

E Proxy-Tokenizer Error

  • The o200k base proxy under-counts by up to 11.3% for Mistral, and the default per-model reserve absorbs the resulting overflow risk.The comparison uses an 8,000-character English document; proprietary tokenizers from Claude, Gemini, Grok, and Cohere cannot be measured directly.Source: E Proxy-Tokenizer Error · p. 14, Table 7 · p. 15
  • The registry maps 41 non-OpenAI models to o200k base as an approximate token counter.Source: E Proxy-Tokenizer Error · p. 14
  • Logographic and abugida scripts under-count by up to 71% with the len/4 estimate, while Latin scripts are safe.Table 6 reports characters per o200k base token and len/4 error on UDHR Article 1; negative error indicates overflow risk.Source: Table 6 · p. 15

H Ranking Without Positional Bias (HotpotQA)

In a HotpotQA distractor diagnostic where answers were not front-loaded, no content-based method significantly beat first-non-empty selection. The result supports reader abstention, rather than answer position, as the explanation for that baseline’s strength.

H Ranking Without Positional Bias (HotpotQA)

  • The diagnostic uses 400 examples with verbatim non-yes/no answers, concatenated distractor paragraphs, and a 256-token budget across about eight chunks per example.The answer-bearing paragraph sits at mean index 4.6; the experiment tests whether first-non-empty’s strength is merely positional.Source: H Ranking Without Positional Bias… · p. 15
  • The authors attribute first-non-empty’s strength to reader abstention, and note that post-chunk selection cannot perform multi-hop reasoning.Source: H Ranking Without Positional Bias… · p. 15
  • No method significantly beats first-non-empty on HotpotQA distractor examples, even though only 10% of answer-bearing paragraphs are at position 0.Embedding, cross-encoder, and reader-confidence methods edge it by 0.4–1.0 EM, but paired bootstrap p > 0.4.Source: H Ranking Without Positional Bias… · p. 15, Table 10 · p. 16

Related papers

Loading 2609.29828v1…