PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Compression Trees: Multi-Resolution Knowledge Retrieval via Hierarchical Semantic Residuals

Junaid Farooq

arXiv:2608.21610v1cs.AI

TL;DR

Flat RAG uses fixed-size similarity retrieval that discards document hierarchy and may mismatch query specificity. This paper introduces Semantic Compression Trees, which store hierarchical semantic residuals and retrieve by progressive descent. Results support residual storage and its cost model, but not progressive top-down routing, especially when document selection is required.

  • Problem

    Flat RAG retrieves fixed-size chunks by similarity while discarding document hierarchy and treating queries with different specificity identically.

  • Method

    Semantic Compression Trees hierarchically store each node’s semantic residual beyond its parent and retrieve through bounded progressive descent.

  • Results

    Results are mixed: residual storage matches dense retrieval with fewer tokens when the document is given, while progressive descent underperforms for document selection.

  • Takeaways & Limitations

    Residual representation is worth keeping, but top-down routing is not supported by the evaluation.

  • Takeaways & Limitations

    Reported retrieval latency excludes cached query-embedding round trips and therefore is not end-to-end serving latency.

Abstract

from arXiv · show

Retrieval-augmented generation relies mostly on flat, fixed-granularity indexes: documents are cut into uniform chunks and retrieved by similarity, discarding the hierarchical structure of the source. We introduce Semantic Compression Trees (SCT), a hierarchical index in which each node stores only its semantic residual -- the information it adds beyond its parent -- and retrieval proceeds by progressive descent from the root, so that per-query cost is governed by tree depth rather than collection size. We evaluate on QASPER (50 papers, 173 questions) under two protocols differing only in whether the benchmark supplies the relevant document, with bootstrap confidence intervals and paired significance tests throughout. The results are mixed and we report them as such. When the document is given, SCT with a zero-LLM extractive compressor matches dense retrieval on answer quality (0.274 vs. 0.277 F1, $p = 0.37$) using 30% fewer context tokens and no LLM calls to build the index, and residual storage beats storing full summaries at each node (0.274 vs. 0.205, $p < 0.001$). Increasing the collection fifty-fold multiplies flat retrieval's per-query scoring work by 48.9x and SCT's by 6.4x. Progressive descent itself is not supported. Retrieving the same residuals without the tree performs identically when the document is given ($p = 0.27$), and descent is substantially worse when the system must select the document (0.122 vs. 0.165, $p < 0.001$). Routing accuracy localises the cause: descent selects the correct paper 20.2% of the time against 39.3% for flat retrieval, because that choice is made from the root residual, the most compressed node in the tree. We conclude that the residual representation is worth keeping and top-down routing is not.

1 Introduction

SCT addresses three limitations of flat RAG—similarity can diverge from relevance, fixed granularity cannot match query needs, and document hierarchy is discarded—by organizing knowledge hierarchically with semantic residuals. The paper positions this representation as source-agnostic and complementary to existing retrieval methods.

  • Motivation: Dense similarity can diverge from informational relevance, so the nearest passage may not best answer a query.The paper illustrates this with Arctic climate and policy-response passages.
  • Motivation: Fixed-size chunks treat high-level and detail-seeking queries identically, making fixed granularity suboptimal.Adaptive and hierarchical chunking address this partly but remain within a flat retrieval paradigm.
  • Motivation: Traditional chunking discards document hierarchy, including the semantic relationships and scoping encoded by sections and subsections.The paper argues that preserving hierarchy should extend from classification to retrieval.
  • Approach: Semantic Compression Trees store only each node’s semantic residual beyond its parent and retrieve through progressive descent across resolution levels.The framework is intended to make retrieval cost proportional to query specificity rather than corpus size.
  • Positioning: SCT is presented as source-agnostic, formally non-redundant, and complementary to retrievers that select child nodes during descent.The related-work comparison distinguishes SCT’s residual formulation and source flexibility from prior hierarchical approaches.

3 Semantic Compression Trees

An SCT represents a source as a hierarchy of compressed nodes whose residuals encode information absent from compressed parents. Concatenating residuals along a root-to-node path reconstructs the node’s content at the corresponding resolution.

  • Representation: Each SCT node represents a source region, has a compression level, and stores a semantic residual.The root is level 0 and is the most abstract representation.
  • Residual definition: The root residual is the compressed source, while each non-root residual is the child content after subtracting information captured by its compressed parent.Semantic subtraction retains only information contributed beyond the parent.
  • Accumulation: Accumulating residuals along a root-to-node path reconstructs semantic content at the node’s resolution level.Context is maximally compressed at the root and contains full detail at leaves.
  • Compression protocol: The compression protocol separately compresses text to a resolution level and extracts a residual from child content relative to the parent summary.This separates the abstract formulation from concrete compression implementations.
  • Compression strategies: LLM compression produces residuals using one call per node, whereas extractive compression retains sentences with novel terms and requires zero LLM calls.The extractive method computes novelty against the parent summary and runs in linear time.

4 Tree Construction

Tree construction separates source-specific hierarchy parsing from universal residual computation. The resulting framework supports multiple source types, and sibling computations can be parallelized.

  • Construction pipeline: SCT construction first parses a source into an intermediate hierarchy, decoupling structure detection from residual computation.The construction and retrieval algorithms can therefore remain universal across source types.
  • Source support: Markdown, PDFs, SQL databases, and plain text use different parsers to define hierarchy boundaries and represented regions.Adding a source type requires implementing only the parser interface.
  • Complexity: O(N) compression calls are required for N nodes, while independent sibling residuals enable parallel construction.Parallelization can reduce wall-clock time by a factor related to the branching factor.

5 Progressive Retrieval

Progressive retrieval descends a semantic-residual tree with a bounded beam and an optional similarity floor, aiming to make work depend on depth and branching rather than indexed-document count. Fixed-depth and exhaustive modes provide query-independent references for what the index exposes.

  • Progressive descent: Progressive retrieval starts from an SCT and descends through child residuals for a query.The algorithm accepts a maximum depth, beam width, and optional similarity floor.
  • Beam search: At each level, the system scores the union of frontier children by query–residual cosine similarity and retains the top k candidates.Selecting globally over the union keeps the frontier fixed-width instead of growing exponentially.
  • Resolution control: The optional floor halts descent when no candidate is sufficiently similar, allowing broad queries to stop at coarser resolutions.The default configuration disables the floor, which is evaluated separately.
  • Reference modes: Fixed-depth and exhaustive modes retrieve children without consulting the query, bounding what the index makes available rather than performing retrieval.Exhaustive mode returns the whole subtree to the depth bound.
  • Complexity: O(D·k·b) descent work and at most D·k accumulated residuals are independent of indexed-document count, while corpus routing adds a document-root scan.Here D is depth, k is beam width, and b is branching factor.

6 Experiments

The experiments compare SCT with flat retrieval under single-document and corpus protocols, measuring answer quality, retrieval quality, cost, and component effects. SCT matches dense retrieval when the document is given and scales better in scored nodes, but top-down descent performs poorly when document routing is required.

  • Setup: 50 papers and 173 answerable questions are evaluated under protocols where the document is either supplied or must be selected from the corpus.The primary protocol isolates index quality; the corpus protocol adds document routing.
  • Single-document results: 0.274 Answer F1 matches dense retrieval’s 0.277 when the document is given, while SCT uses 30% fewer context tokens and no LLM calls for index construction.The paired bootstrap comparison gives p = 0.366.
  • Corpus results: 0.122 versus 0.165 Answer F1 shows SCT’s largest gap when the system must find the document, with correct-paper routing at 20.2% versus 38.7–39.3% for baselines.The routing result identifies document selection as the main failure point in the corpus protocol.
  • Ablations: 0.069 F1 favors residual storage over full summaries at each node, although the comparison also uses different context volumes.The residual configuration scores 0.274 versus 0.205, p < 0.001; the full-summary variant retains 437 tokens versus 2,178.
  • Ablations: Retrieving SCT residuals without the tree has no measurable answer-quality cost when the document is given, with a paired difference of +0.0017 and p = 0.420.The confidence interval [−0.014, +0.017] is centered near zero, while depth changes answer F1 only from 0.2715 to 0.2741.
  • Scaling: 48.9× versus 6.4× is the collection-growth multiplier for flat retrieval versus SCT in index nodes scored per query.The comparison grows the indexed collection from one document to fifty and reports nodes rather than implementation-dependent latency.

7 Analysis

The analysis separates the value of residual storage from the failure of strictly top-down routing. Residuals preserve answer quality and evidence better than full summaries, while early document selection from compressed roots causes unrecoverable errors.

  • Routing and hierarchy: 0.122 vs. 0.147 Answer F1 shows that the tree underperforms SCT-Flat when residual content is held constant.Routing accuracy also rises from 0.202 to 0.347 for SCT-Flat, isolating retrieval strategy as the differing factor.
  • Routing and hierarchy: 0.202 routing accuracy remains unchanged across depth settings under P2, so later descent cannot recover from an initial wrong-document choice.Under P1, greater depth adds only 0.0026 F1 while increasing context by 25%.
  • Residual storage: 0.233 vs. 0.274 Answer F1 shows that SCT-LLM underperforms SCT-Extractive under P1.LLM compression reduces verbatim evidence recall to 0.009 despite retaining 0.590 token recall, indicating lossy paraphrase.
  • Protocol dependence: 0.358 vs. 0.202 routing accuracy reverses the compressor ranking under the corpus protocol, where SCT-LLM beats SCT-Extractive by 0.0154.The LLM-written root better describes a paper, which matters when document selection is the binding constraint.
  • Residual storage: 34% relative F1 gain shows that residuals outperform full summaries at each node.Residuals achieve 0.720 verbatim evidence recall and 0.895 token recall, versus 0.152 and 0.584 for full summaries.
  • Design implication: The practical recommendation is to retain residual storage but replace strictly top-down routing with flat selection followed by within-document resolution control.The analysis states that residuals match dense retrieval at 70% of the context budget with no LLM indexing cost, whereas top-down accuracy loss exceeds its cost saving.

8 Discussion

The discussion identifies compression-before-selection as the central trade-off: compressed representations reduce scoring work but can make document routing unrecoverable. This limitation may not transfer to exact structured hierarchies, motivating hybrid routing and broader evaluation.

  • Compression before selection: Selecting from compressed representations makes the earliest document decision from the least informative representation, while flat retrieval scores full passages first.A wrong top-down choice cannot be recovered at later levels, whereas flat retrieval pays collection-growing per-query work.
  • Compression before selection: 7.6× fewer scored nodes comes with a routing-accuracy loss that exceeds the cost saving on QASPER.The trade-off is framed as applying to RAPTOR, table-of-contents indexes, and other routers operating on document-level abstractions.
  • Scope boundary: Exact schema hierarchies may avoid the paper-routing failure because their node boundaries are genuine and table names are stronger routing keys than one-sentence summaries.Whether descent is competitive for structured sources remains empirical.
  • Future directions: Hybrid retrieval, stronger routing keys, and deeper corpora are proposed as direct follow-ups to the observed routing limitation.The proposed hybrid selects passages flat, then descends within the selected document while retaining residual representations.

9 Conclusion

SCT’s residual representation performs well when the relevant document is supplied, but progressive descent is not supported as a retrieval strategy. The paper therefore recommends retaining residual storage while abandoning top-down routing.

  • 0.274 vs. 0.277 F1: SCT with zero-LLM extractive compression matches dense retrieval when the relevant document is supplied.It uses 30% fewer context tokens and requires no LLM calls to build the index.
  • 34% relative: residual storage outperforms storing full summaries at each node.The comparison is statistically significant (p < 0.001).
  • 48.9× vs. 6.4×: fifty-fold collection growth multiplies flat retrieval’s scoring work more than SCT’s.The result supports the claim that descent decouples query cost from collection size.
  • Progressive descent performs identically to flat retrieval of the same residuals when the document is given (p = 0.27).
  • 0.122 vs. 0.165: progressive descent performs substantially worse when the system must find the document.Routing selects the correct paper 20.2% of the time for descent versus 39.3% for flat retrieval because descent starts from the most compressed root residual.
  • The paper’s recommended direction is to keep the residual representation rather than the progressive descent strategy.

Limitations

The evaluation is limited by its single dataset, dependence on well-structured documents, approximate residual subtraction, routing constraints, judge design, and non-end-to-end latency measurement.

  • 50 QASPER papers and 173 questions leave several comparisons unresolved because confidence intervals are wide.Other datasets would test different assumptions, especially whether single-path descent can assemble multi-hop evidence from several branches.
  • 28 of 50 papers have no subsections, limiting how much multi-resolution retrieval can be demonstrated.QASPER papers have explicit, consistently nested sections, while unstructured sources requiring inferred boundaries were not evaluated.
  • Semantic subtraction is approximate, so residuals may retain parent restatements or discard novel low-salience detail.A formal measure of residual quality remains open.
  • Top-down document routing is bounded by the root residual, which compresses an entire paper into one or two sentences.This creates a routing constraint that flat retrieval does not share because flat retrieval compares the query against every passage.
  • A single judge model from the answer-generation model family risks self-preference.The paper therefore reports judge scores alongside token-level and evidence-level metrics.
  • Reported retrieval latency excludes the cached query-embedding round trip and is therefore not end-to-end serving latency.

Reproducibility

The experiments are reproducible through recorded run metadata, scripted table generation, cached API responses, and deterministic retrieval components, while hosted-model generation is not bitwise reproducible from scratch.

  • Every experiment records its code revision, deployment names, tokeniser, bootstrap seed, per-question system records, and measured API usage.
  • Every table is generated from experiment reports by script rather than transcribed manually.This prevents tables from diverging from the runs behind them.
  • 58 of roughly 14,900 cached API calls were reissued in a confirmation pass, reproducing every reported figure exactly.Retrieval, tree construction, extractive compression, and statistics are deterministic given a fixed seed.
  • The harness blocks reference answers and evidence annotations from retrieval code paths and sequentially executes retrieval for valid latency measurement.

Ethics Statement

The paper addresses environmental and information-integrity risks from compression, reports content-policy exclusions, and uses only public QASPER data without private or personally identifiable information.

  • SCT-LLM construction uses LLM calls proportional to the number of tree nodes, while SCT-Extractive provides a zero-LLM-cost alternative.The paper reports the retrieval-quality cost of choosing between these indexing options.
  • Residual compression is lossy and can distort source information, so the paper measures both judge faithfulness and evidence-recall gaps.For exact source wording, the verbatim-versus-token-level evidence-recall gap is the relevant warning signal.
  • A small number of questions were excluded because the answer-generation provider refused prompts on content-policy grounds.The same exclusions were applied to every system because filtering responds to retrieved content.
  • All experiments use the publicly available QASPER dataset under CC BY 4.0, with no private or personally identifiable data.

About the Author

Junaid Farooq works across AI research and production systems, and has over a decade of experience in artificial intelligence, machine learning, and software engineering. He is Chief Architect and Vice President of Artificial Intelligence at Sprouts.ai.

  • Farooq has over a decade of experience spanning artificial intelligence, machine learning, and software engineering.
  • He earned a 2023 PhD in Artificial Intelligence from the National Institute of Technology Srinagar.His doctoral work developed hybrid deep-learning architectures for spatiotemporal forecasting.
  • He has published peer-reviewed papers and written two books on clean code, architecture, and cybersecurity.The supplied passage names The Art of Code: Tactics and Principles of Clean Code and Architecture and Zero Day: The Invisible War.
  • Farooq leads platform architecture and production LLM and agentic systems at Sprouts.ai.He serves as Chief Architect and Vice President of Artificial Intelligence.
Loading 2608.21610v1…