PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoIndex: Learning Representation Programs for Retrieval

Sam O'Nuallain, Nithya Rajkumar, Ramya Narayanasamy, Hanna Jiang, Shreyas Chaudhari, Andrew Drozdov

arXiv:2607.18603v1cs.IRcs.AIcs.CL

TL;DR

Document representation is often treated as fixed preprocessing rather than an optimization target in information retrieval. AutoIndex searches over executable representation programs using retrieval performance to guide updates, improving Recall@100 over static full-document BM25 on all 8 CRUMB tasks, with average gains of +8.4%.

  • Problem

    Document representation is often treated as fixed preprocessing rather than directly optimized, despite determining which cues, context, and structure retrieval systems expose.

  • Method

    AutoIndex iteratively analyzes retrieval failures, synthesizes executable document-representation programs, rebuilds indexes, and selects updates using validation retrieval performance with BM25 fixed.

  • Results

    +8.4% average Recall@100 over static full-document BM25, with improvements on all 8 CRUMB tasks.

  • Takeaways & Limitations

    The results support treating indexing representation as an explicit retrieval optimization target while keeping the underlying retriever fixed.

  • Takeaways & Limitations

    AutoIndex primarily optimizes Recall@100 with fixed BM25, limited iterations, and few seeds, leaving reliability and broader retriever interactions open.

Abstract

from arXiv · show

We present AutoIndex, a framework for learning representation programs: executable transformations that map raw documents into the representations exposed to a retrieval system. Rather than tuning retrievers, rerankers, or a small set of preprocessing hyperparameters, AutoIndex searches over programs that slice, enrich, normalize, reweight, or reorganize documents before indexing. At each iteration, AutoIndex performs validation-guided program search, in which agents diagnose failures of the current program and synthesize candidate updates, retaining only updates that improve retrieval quality under the resulting index. We evaluate AutoIndex on CRUMB, a benchmark of heterogeneous retrieval tasks, with BM25 held fixed across all experiments. The learned programs improve recall over a static full-document BM25 baseline on all 8 tasks, with average gains of +8.4% in Recall@100 and +8.3% in nDCG@10, and largest gains of +30.5% in Recall@100 and +43.6% in nDCG@10. These results suggest that document representation should not be treated as a fixed preprocessing choice made before retrieval begins, but as an explicit optimization target. Code to reproduce our results is available at https://github.com/auto-index/autoindex.

1 Introduction

AutoIndex treats document representation as an explicit retrieval optimization target rather than fixed preprocessing, searching executable programs while holding the retriever fixed. On CRUMB, learned programs improve fixed-BM25 retrieval across all 8 tasks, with average gains of +8.4% in Recall@100 and +8.3% in nDCG@10.

  • Motivation: Document representation determines which lexical cues, context, metadata, and structure are exposed to retrieval, but is often governed by fixed heuristics.These choices occur before documents are searched, ranked, or used in retrieval-augmented generation pipelines.
  • Approach: AutoIndex formulates retrieval indexing as code optimization over executable document representation programs, shifting attention from fixed preprocessing to retrieval-aware design.Programs can transform documents by slicing, enriching, normalizing, reweighting, or reorganizing them before indexing.
  • Results: +8.4% in Recall@100 was the average gain over a static full-document BM25 baseline across 8 of 8 CRUMB tasks.The corresponding average gain was +8.3% in nDCG@10.
  • Results: +30.5% in Recall@100 and +43.6% in nDCG@10 were the largest gains reported, without retriever fine-tuning, embedding updates, or online feedback.The evaluation used BM25 as a fixed retriever on the heterogeneous CRUMB benchmark.
  • Approach: The framework iteratively diagnoses retrieval failures, synthesizes candidate programs, executes them in a sandbox, indexes the results, and selects updates using offline validation metrics.The retriever, ranking function, and indexing backend remain fixed, isolating the representation program as the optimization target.

2 Related Work

Prior work improves retrieval by tuning document representations, enriching indexes, and optimizing RAG systems or executable programs. AutoIndex addresses the cost of broad representation search and relates to feedback-driven, verifiable program optimization.

  • Adaptive chunking and document representation: Document-representation tuning over chunking, overlap, titles, and metadata can improve retrieval, but broad exploration is expensive because each configuration requires rebuilding the index.AutoIndex targets this bottleneck while searching a broader space.
  • Index enrichment: Index-enrichment methods augment documents before indexing to improve lexical matching or add summaries, facts, rationales, and rewrites, but expansions can be hallucinated or low quality.Doc2Query-style expansion can inflate the index and hurt retrieval.
  • Agentic program optimization: AutoIndex relates to LLM-based program-optimization systems that propose executable artifacts and refine them through feedback.Related RAG optimization systems search over pipeline components such as retrievers, rerankers, prompts, and generators.

3 AutoIndex

AutoIndex optimizes executable document representation programs while keeping the retriever fixed. Its validation-guided loop diagnoses retrieval failures, synthesizes candidate programs, rebuilds indexes, and selects updates by retrieval performance.

  • 3 AutoIndex: AutoIndex transforms raw documents into indexable units searched by a fixed retriever, with BM25 used in the experiments.Each program induces the representation exposed to retrieval rather than tuning the retriever itself.
  • 3 AutoIndex: Representation programs are executable mappings fθ that convert each document d into units {c1, . . . , ck}, retaining each unit’s source-document identifier.Applying fθ across the corpus produces the indexed representation Cθ.
  • 3 AutoIndex: Programs are ranked by source-document score using MaxP over unit-level retriever scores, while the retriever, ranking rule, and indexing backend remain fixed.This isolates performance differences to the representation program θ.
  • 3 AutoIndex: AutoIndex evaluates candidates by executing each program on the corpus, rebuilding its index, and measuring a black-box validation objective J(θ).The next incumbent is selected according to validation performance, while the best-performing iterate is later evaluated on held-out queries.
  • 3 AutoIndex: Two specialized agents separate diagnosis from synthesis: the Analysis Agent summarizes concrete retrieval failures, and the Code Agent proposes executable programs conditioned on that summary and search history.The Analysis Agent uses read-only retrieval and document-inspection tools to ground its diagnosis in observed behavior.

4 Experimental Setup

AutoIndex is evaluated on CRUMB’s eight heterogeneous retrieval tasks using fixed BM25 retrieval and indexing infrastructure. The experiments measure retrieval quality against full-document and passage-corpus baselines while varying representation-program search settings.

  • Dataset and splits: CRUMB contains eight complex retrieval tasks stressing compositional queries, long documents, and heterogeneous evidence requirements.For each split, available queries are partitioned into validation and
  • Metrics and baselines: Evaluation reports Recall@100 and nDCG@10, aggregating passage-level scores to documents with MaxP over 10,000 retrieved candidate chunks per query.The main baseline is BM25 over each split’s full-document markdown corpus, with relevant headings preserved; CRUMB’s passage corpus is also compared.
  • Experimental design: The retriever and indexing backend remain fixed across experiments, isolating performance differences to the learned representation program.The primary setting uses the full AutoIndex loop with search history enabled; experiments use Claude Sonnet 4.6 with n = 2 seeds and qwen3-coder with n = 3 seeds.

5 Experimental Results

AutoIndex improves retrieval over BM25 full-document and uniform passage-corpus baselines across CRUMB evaluations, with the largest gains on tasks affected by vocabulary mismatch. A preliminary StackExchange experiment shows the learned representation also benefits dense retrieval.

  • Main results: AutoIndex improves Recall@100 across all CRUMB tasks over the BM25 full-document baseline.Held-out results use qwen3-coder with search history enabled.
  • Main results: The largest gains over the full-document baseline occur on TheoremRetrieval, SetOpEntity, and LegalQA.The passage attributes these gains to BM25 vocabulary mismatch leaving headroom for representation changes.
  • Comparison to uniform chunking: AutoIndex outperforms CRUMB’s uniform passage-corpus baseline on all reported splits.The learned programs adapt to each split’s document structure and vocabulary, unlike domain-agnostic chunking.
  • Preliminary dense retrieval result: Recall@100 rises from 0.7391 to 0.8741 with a reused AutoIndex representation in dense StackExchange retrieval, a relative gain of +18.3%.The experiment uses Qwen3-Embedding-0.6B; broader dense, hybrid, and reranking evaluations remain future work.

6 Framework Analysis and Discussion

AutoIndex’s gains depend on iterative, history-constrained search grounded in concrete retrieval failures rather than one-shot rewriting. Across datasets, learned programs make corpus-specific representation edits, while current evaluation leaves broader optimization and transfer questions open.

  • Ablations and search dynamics: Iterative search improves only 3 of 8 splits when limited to one iteration, showing that useful programs often require repeated proposal, evaluation, and selection.The full loop benefits from multiple rounds before identifying programs that generalize beyond validation queries.
  • Ablations and search dynamics: AutoIndex works best when candidate programs are grounded in retrieval failures and constrained by outcomes recorded in search history.Different splits exhibit early gains, gradual accumulation, or repeated rejection of near-threshold candidates.
  • Learned program behaviors: On TipOfTongue, reweighting Plot and Cast sections addresses the mismatch between concrete queries and abstract Wikipedia summaries while preserving full-document context.The edit increases term frequency for query-relevant narrative content without aggressively discarding other evidence.
  • Learned program behaviors: On LaTeX-heavy data, targeted markup stripping or normalization reduces retrieval noise only when corpus-specific analysis shows that repeated syntax dilutes useful natural-language terms.The resulting chunks contain a higher proportion of semantic content, rather than applying generic cleanup blindly.
  • Limitations and future work: AutoIndex currently optimizes primarily for Recall@100 with fixed BM25, limited iterations, and few seeds, leaving convergence, metric trade-offs, system interactions, and program transfer unresolved.Open questions include balancing recall against nDCG, latency, index size, preprocessing cost, and compatibility with dense or hybrid systems.

7 Conclusion

AutoIndex learns executable document representation programs that optimize corpus representations for retrieval rather than tuning the retriever. On CRUMB, it improves fixed-system BM25 retrieval across heterogeneous tasks without retriever training, embedding updates, or online feedback.

  • Contribution: AutoIndex learns executable document representation programs for retrieval.The framework optimizes the corpus representation exposed to retrieval systems.
  • Contribution: AutoIndex optimizes chunks, context, normalization, and transformations while keeping the retrieval system fixed.Its target is the corpus representation rather than the retriever itself.
  • Evaluation: AutoIndex improves BM25 retrieval across heterogeneous CRUMB tasks without retriever training, embedding updates, or online feedback.The reported evaluation demonstrates gains under a fixed retrieval system.

A Appendix · A.1 Claude Sonnet 4.6 Results

Appendix A.1 evaluates AutoIndex with Claude Sonnet 4.6 under the same search-based setting as the qwen3-coder experiments. Across held-out CRUMB splits, the method improves Recall@100 on 7 of 8 splits relative to the BM25 full-document baseline.

  • A.1 Claude Sonnet 4.6 Results: Claude Sonnet 4.6 is evaluated with search history enabled, five iterations, and one held-out test evaluation at the best validation checkpoint.Values are averaged over two seeds, with ∆ computed relative to the BM25 full-document baseline.
  • A.1 Claude Sonnet 4.6 Results: Recall@100 improves on 7 of 8 CRUMB splits relative to the BM25 full-document baseline.The results exhibit similar qualitative patterns to the qwen3-coder results.
  • A.1 Claude Sonnet 4.6 Results: Table 4 reports the Claude Sonnet 4.6 counterpart to the qwen3-coder results in Tables 1 and 2.The experimental setting is otherwise identical to those results.
  • A.1 Claude Sonnet 4.6 Results: Table 4 scores are multiplied by 100 and reported as mean ± standard deviation over two seeds.A dagger denotes a single seed.
  • A.1 Claude Sonnet 4.6 Results: AVG is the unweighted macro-average across datasets, and aggregate ∆ is computed from the corresponding macro-averaged scores.This aggregation applies to the reported Table 4 results.
  • A.1 Claude Sonnet 4.6 Results: Bold values indicate relative gains exceeding 10%.The table uses this formatting convention to mark larger relative improvements.
  • A Appendix: Figure 4 compares ∆Recall@100 relative to the BM25 full-document baseline across CRUMB splits for both Code Agent backbones.Bars show relative improvement, error bars show standard deviation, and AVG is shown without error bars.

A.2 Design Decisions and Implementation Notes

AutoIndex prioritizes Recall@100 and uses BM25-based validation to optimize document representation programs. Its design separates analysis from coding, controls validation-query access, and notes approximate evaluation on sampled subsets as future work for larger corpora.

  • Optimization target: Recall@100 is the primary optimization signal because downstream readers can recover from imperfect ranking but not missing evidence.Recall@10 was rejected because the hardest splits offered too little headroom for candidates to clear the acceptance threshold.
  • Validation/evaluation split ratio: The 1:2 validation/evaluation split provides enough queries to expose actionable failures while preserving representative held-out evaluation on unseen queries.Smaller validation pools stalled optimization when baseline retrieval already placed gold documents in the top 100.
  • Choice of BM25: BM25 is used as a transparent, efficient retriever whose sensitivity to segmentation, normalization, and term reweighting isolates representation-program effects.Experiments use bm25s v0.2.14 with k1=1.5, b=0.75, lucene scoring, MaxP aggregation, lowercasing, default tokenization, English stopword removal, and no stemming.
  • Why two agents: Two agents separate corpus diagnosis from code synthesis, avoiding low-signal generic hypotheses and keeping long excerpts out of the coding agent’s context.This separation particularly benefits weaker backbone models whose performance degrades with context length.
  • Analysis agent tools: Curated, balanced validation queries reduce search time and discourage fixation on idiosyncratic patterns that do not generalize.The analysis agent uses retrieval and file-reading tools to inspect failures under the current preprocessing program.
  • Limitations: For larger corpora, sampled-subset evaluation followed by full-corpus validation is proposed as future budget-aware work.The paper leaves these approximate evaluation strategies unevaluated.

A.3 Case Studies

This case study compares computational cost across two AutoIndex backbones. Sonnet 4.6 uses more tokens and incurs higher LLM latency than qwen3-coder, resulting in longer LLM-only runtime per run.

  • Computational cost: Sonnet 4.6 consumes more tokens per run and per call than qwen3-coder.Table 5 reports analysis, code, completion, and pooled per-request token averages.
  • Computational cost: Sonnet 4.6 has slower analysis and code calls than qwen3-coder.The comparison concerns per-request LLM latency for both call types.
  • Computational cost: Sonnet 4.6 yields a longer LLM-only wall-clock time per run than qwen3-coder.LLM-only wall/run excludes preprocessing execution, evaluation, and indexing time.

A.3.1 More information on the TipOfTheTongue Case Study … A.6 Case Study Generated Code Implementations

The case studies illustrate validation-guided preprocessing changes for heterogeneous documents while keeping BM25 fixed, and the prompts formalize metric-aware, corpus-wide optimization with regression and generalization safeguards. Generated implementations instantiate these principles through targeted LaTeX cleaning, Wikipedia markup normalization, and weighted document representations.

  • A.3.1 More information on the TipOfTheTongue Case Study: Wikipedia pages corresponding to the queried movies and TV shows form the TipOfTheTongue target-document corpus.
  • A.3.1 More information on the TipOfTheTongue Case Study: The TipOfTheTongue code preserves one full-document chunk while reweighting plot-related content to retain identifying context.The strategy explicitly reasons about BM25 term-frequency saturation and length normalization.
  • A.3.2 StackExchange Case Study: The StackExchange case study targets community questions and relevant answer pages, with failures involving surface phenomena matched to abstract conceptual reference articles.
  • A.3.2 StackExchange Case Study: A narrow, heavy-LaTeX-triggered cleaner was hypothesized after an earlier broad boilerplate stripper caused regressions.The cleaner was designed to remove LaTeX commands without repeating the earlier over-aggressive approach.
  • A.4.1 Code Agent Prompt: The prompts constrain agents to transform document chunks while BM25 remains fixed, optimizing Recall@100 primarily and nDCG@10 secondarily.They warn that added chunks can improve recall while diluting top-10 ranking quality, and that metadata is not indexed unless explicitly included.
  • A.4.2 Analysis Agent Prompt: The analysis prompt requires corpus-wide recommendations that investigate both failures and successes, preserve existing signal, and avoid validation-set overfitting.Validation queries guide hypothesis selection, but final judgment uses a separate held-out evaluation set.
  • A.5 Query Splits: The query-split materials provide full query IDs through query_splits.json and distinguish validation queries from evaluation queries.
Loading 2607.18603v1…