DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing

1 Introduction

• Long, heterogeneous records and complex analysis tasks challenge single-call LLM operations, which may exceed context limits, omit relevant instances, or add spurious information.The motivating police-record analysis spans documents that can be hundreds of pages long.Source: §1 · p. 1
• A split → gather → map → reduce pipeline can process long documents in chunks, add neighboring context, summarize each chunk, and then combine results globally.The described pattern uses neighboring chunks as background for per-chunk summaries.Source: §1 · p. 2
• The space of rewrites, prompts, and parameter choices makes manual pipeline design and evaluation require extensive experimentation.The motivating discussion notes that even choosing chunk size k can be challenging.Source: §1 · p. 2
• DocETL uses agent-guided rewrite directives to decompose complex operations and evaluate alternative pipelines, with opportunistic top-down optimization of newly synthesized operations.Agents generate task-specific validation prompts and focus further rewriting on operations that are not sufficiently accurate.Source: §1 · pp. 2–3, Figure 1 · p. 2
• 21 to 80% more accurate than baselines across four unstructured document analysis tasks, DocETL’s reported plans improve on the comparison systems.The introduction describes its initial evaluation across four tasks.Source: §1 · p. 3

2.1 Programming Model

• DocETL represents datasets as JSON document collections and uses YAML pipelines to specify operations, prompts, schemas, and inputs; operator validations can trigger retries.Pipelines can pass one operator’s output to the next, and validation failures provide context for subsequent attempts.Source: §2.1 · p. 3

2.2 LLM-Powered Operators

• The operator suite combines LLM-based semantic processing with auxiliary data-manipulation operators, including the resolve operator for canonicalizing attribute variations.Table 1 summarizes operator configuration and functionality.Source: §2.2 · p. 3, Table 1 · p. 4

2.2.1 Map

• Map applies an LLM-powered projection to each document, adding schema-defined outputs; parallel maps support independent transformations on the same documents.Prompts can reference document content, and outputs can contain varying numbers of extracted items.Source: §2.2.1 · p. 4

2.2.2 Reduce

• Reduce groups documents by specified keys and aggregates each group, while batched folding processes oversized groups in order and can preserve state in a scratchpad.DocETL automatically determines a fold batch size; folding also supports non-associative reductions.Source: §2.2.2 · p. 4, Figure 2 · p. 4

2.2.3 Resolve

• Resolve compares and canonicalizes values that may refer to the same entity, preserving document count and enabling consistent references before downstream aggregation.DocETL can synthesize resolve operations when needed.Source: §2.2.3 · pp. 4–5

2.3 Auxiliary Operators; 2.3.1 Unnest

• Auxiliary operators manipulate data without LLMs; unnest flattens arrays or dictionaries into individual elements for downstream processing.Unnest can also elevate attributes from nested dictionaries.Source: §2.3 · p. 5, §2.3.1 · p. 5

2.3.2 Split

• Split divides long text into chunks and assigns document IDs and chunk numbers that support reassembly or downstream context-aware processing.Chunks from one source document share its ID and receive sequential chunk numbers.Source: §2.3.2 · p. 5

2.3.3 Gather

• Gather enriches each chunk with configurable peripheral context while preserving document count, unlike reduce, which performs many-to-one aggregation.Context can include full or partial chunks, summaries, or transformations generated between split and gather.Source: §2.3.3 · pp. 5–6, Figure 3 · p. 6

3 Rewrite Directives

• Rewrite directives are agent-instantiated frameworks for decomposing map, reduce, and equijoin operations to improve output quality; map directives also support filters.Agents interpret directives for specific tasks and data, and assess whether each rewrite is likely to help.Source: §3 · p. 6

3.1.1 Document Chunking (Map)

• For documents that exceed context windows or reasoning capacity, split decomposes map into chunking, context gathering, per-chunk mapping, and reduction.Smaller directives add shared document metadata, header lineage or summaries, chunk filtering, and flattening nested results.Source: §3.1.1 · pp. 6–7

3.1.2 Multi-Level Aggregation (Reduce)

• Hierarchical aggregation first reduces data at finer-grained keys, then rolls those summaries up to the desired key, with effectiveness dependent on task and granularity.For example, city-level summaries can be combined into state-level summaries, retaining the option for intermediate validation.Source: §3.1.2 · p. 7

3.2 LLM-Centric Improvements

• LLM-centric improvements comprise gleaning and duplicate resolution, targeting iterative output refinement and semantic duplicates, respectively.Source: §3.2 · p. 7, §3.2.1 · p. 7, §3.2.2 · p. 7

3.2.1 Gleaning (Map and Reduce)

• Gleaning uses a separate validator to assess an initial output and give feedback for iterative refinement, with refinement applied to grouped data for reduce operations.The process can stop when no further refinement is needed or after the specified maximum iterations; Figure 4 illustrates an omission corrected in a second pass.Source: §3.2.1 · p. 7, Figure 4 · p. 8

3.2.2 Duplicate Key Resolution (Reduce)

• Resolve operations consolidate semantically equivalent reduce-key values, addressing LLM outputs that vary syntactically despite referring to the same entity.Without resolution, values such as “New York City,” “NYC,” and “The Big Apple” would be treated as separate keys.Source: §3.2.2 · p. 7

3.3 Projection Synthesis

• Projection synthesis pushes task- and data-specific projections before complex operations, including chaining, parallel isolated subtasks, pre-aggregation, and pre-joining.Agents use operator-specific criteria; the possible projections are open-ended, and directive effectiveness varies by context.Source: §3.3 · pp. 7–8

4 Optimizer

• DocETL uses generation agents to create candidate plans from logical rewrites and validation agents to assess them on sampled data, prioritizing accuracy while leaving cost and latency constraints for future work.Source: §4 · p. 8

4.1 Optimization Approach

• The optimizer traverses operations left to right, recursively rewrites sub-pipelines when validation finds room for improvement, and replaces them with selected plans.Agents synthesize new operations and configurations, which are recursively optimized before the current optimization continues.Source: §4.1 · p. 8
• Candidate plans are first rated on sampled outputs, then the top six undergo pairwise comparisons, and the plan with the most wins is selected.This two-stage evaluation avoids comparing every pair among potentially 100+ generated plans.Source: §4.1 · p. 8
• Sampling adjusts for upstream selectivity, but sampled documents may miss larger documents in the full dataset and lead to execution errors.The authors are developing methods to adapt plans during execution.Source: §4.1 · p. 9

4.2.1 Generation Agents

• Generation agents synthesize logical and physical operation choices, while heuristics explore plausible physical settings such as chunk and batch sizes.The authors describe their chunking and gathering search as a first attempt; optimal settings may vary by task, and batch-size evaluations likewise show task-dependent optima.Source: §4.2.1 · pp. 9–10

4.2.2 Validation Agents

• Validation agents formulate explicit criteria for accuracy, precision, and recall, rate outputs, and compare top-rated plans; the current top-k is six.The criteria are intended to provide a consistent basis for comparing pipeline variants.Source: §4.2.2 · p. 10

4.2.3 Implementation Details

• The engine caches sub-pipeline outputs and uses Rust for efficient resolve and equijoin execution, while supporting tool-calling LLMs for structured outputs.Source: §4.2.3 · p. 10

5 Evaluation

• DocETL’s optimized plans improve task-specific accuracy by 21% to 80%, while baseline approaches often achieve less than 40% accuracy.The evaluation aims to show that these gains require neither training labels nor developer intervention.Source: §5 · pp. 10–11

5.1.2 Results

• DocETL’s optimized legal plan improved F1 by 21.4% over LOTUS, the next-best LLM-based plan, and recall by 67% over unoptimized DocETL, with no hallucinations.Its runtime and cost are higher; optimizer cost is $1.58 using gpt-4o-mini for the optimizer’s agents.Source: §5.1.2 · pp. 11–12

5.2.2 Results

• 32.9% lower hallucination rate and 34.3% higher Kendall’s Tau: the optimized game-review pipeline improves extraction reliability and chronological ordering.Sentiment accuracy remained stable (66.4% vs 65.0%). The pipeline chunks documents, gleans positive and negative reviews separately, then reduces them in chronological order; it costs $1.48 versus $0.12 for the truncated-data baseline.Source: Table 4 · p. 12, §5.2.2 · pp. 12–13, §5.2.1 · p. 12

5.3.2 Results

• 100% precision across 435 locations: the optimized declassified-article pipeline extracts 46% more locations than the resolve-only variant.Resolve consolidates event types from 233 to 83, enabling aggregation; batched folding with batch size 41 supports incremental location tracking. The optimized pipeline costs $1.84 versus $1.16 for resolve-only.Source: Table 5 · p. 13, §5.3.2 · p. 13, §5.3.1 · p. 13

5.4.2 Results

• 80% improvement in RP@25 over reimplemented LOTUS: DocETL also improves RP@5 by 33% and RP@10 by 50%.Synthesized blocking rules, including requiring all reaction-label words to appear in the article, may improve recall; all systems have essentially zero hallucinations, with DocETL averaging 0.001. Manual inspection found ground-truth errors that may limit scores.Source: Table 6 · p. 14, §5.4.2 · p. 14, §5.4.1 · p. 14

5.5 Case Study: Police Misconduct

• DocETL synthesized an entity-resolution step for the police-misconduct case study after the initial pipeline produced very incorrect outputs without resolving officer names.The case study uses 227 documents averaging 12,500 tokens; the sensitive data cannot be open-sourced.Source: §5.5 · p. 14

5.6 Case Study: Rewrite Directives in Legal Contract Analysis

• 47% of generated plans beat the unoptimized baseline on precision, 67% on recall, and 40% on F1; 30% beat the optimized plan on precision, 10% on F1, and none on recall.The study evaluated 30 plans on 50 contracts; Figure 5 shows cost and precision, recall, and F1 across their implementations.Source: Figure 5 · p. 15, §5.6 · p. 16

5.5 Case Study: Police Misconduct

• Human assessment agreed with the LLM judge on 96% of officer-name validity judgments, 97% of date-and-location judgments, and 92% of extensive-description judgments.The authors manually assessed samples of outputs to validate the judge’s evaluations.Source: §5.5 · p. 15
• DocETL𝑂 was, on average, 1.34× more accurate than the baseline on police misconduct identification.DocETL𝑆 and DocETL𝑇 performed similarly, but DocETL𝑆 often omitted dates and locations; DocETL𝑇 and DocETL𝑂 were the most reliable, with DocETL𝑂 particularly suited to longer documents.Source: Table 7 · p. 15, §5.5 · p. 15
• For 227 police-misconduct documents, DocETL𝑂 cost $1.35 versus the baseline’s $2.24, while optimizer execution cost about $100 and took about 20 minutes.DocETL𝑂 ran in 364.97 seconds; the optimizer considered and evaluated over 200 pipeline variants, and its one-time cost was framed as amortizable across hundreds of thousands of documents.Source: §5.5 · p. 16

5.6 Case Study: Rewrite Directives in Legal Contract Analysis

• Generated directive plans varied substantially: 20% of chained-projection plans omitted document placeholders, while isolated projections consistently outperformed chaining on this task.Gleaning implementations had more consistent cost profiles because each used exactly three LLM calls per document; many generated plans nevertheless outperformed the unoptimized baseline.Source: §5.6 · p. 16
• LLM judge rankings correlated with actual F1 rankings at Kendall’s tau 0.642, indicating that the selection mechanism could identify more effective plans despite implementation variance.The judge was GPT-4o-mini; plan implementations varied in prompts, structures, and decomposition strategies.Source: §5.6 · p. 16

5.7 User Adoption and Impact

• Users report DocETL deployments in healthcare, legal work, scientific research, and enterprise tasks, and describe its natural-language plans as intuitive and transparent.Reported applications include medical-record analysis, regulatory compliance, literature analysis, financial-report insights, and processing forensic psychiatry records with handwritten notes and changing formats.Source: §5.7 · pp. 16–17

• DocETL output quality can vary with physical operator settings, and decomposing documents matters because LLM accuracy often decreases as document size increases.Processing order also matters: chunks cannot always be processed independently, motivating gather and folding operations.Source: §6 · p. 17
• LLM-instantiated rewrites can produce variable-quality prompts and an infinite space of plans, making rewriting affect correctness as well as performance.Unlike traditional database rewrites, DocETL rewrites are semantic rather than guaranteed to preserve identical outputs.Source: §6 · p. 17
• Because DocETL lacks a reliable accuracy model and LLM judges are uncalibrated, it evaluates plans empirically by scoring and ranking sample outputs with pairwise comparisons.Source: §6 · p. 17
• A plan’s best suffix depends on its chosen prefix, so DocETL cannot optimize subplans independently using dynamic programming or divide-and-conquer.Source: §6 · p. 17
• Natural-language specifications can omit processing corner cases, so output-schema failures and rate limits require robust fault-tolerance mechanisms.Source: §6 · p. 17
• Using LLMs for both rewrite implementation and validation risks shared biases, although the validation agent’s different perspective often identifies issues.Possible mitigations include different models, prompt diversification, human refinement, and hybrid validation with labeled ground truth.Source: §6 · p. 17

• LOTUS, Palimpzest, and Aryn focus mainly on extraction or queries that capable LLMs can handle without decomposition, whereas DocETL explores decomposition for complex documents.DocETL also supports documents longer than LLM context windows and makes gather, split, and entity resolution first-class capabilities.Source: §7 · p. 17
• Other systems target structured documents or specialized tasks, while DocETL supports flexible pipeline construction; inadequate context management remains common for long documents and cross-document reasoning.Source: §7 · p. 18
• Declarative crowdsourcing systems used human rather than machine intelligence, and DocETL extends that tradition to LLM processing through a flexible interface and agent-driven optimization.Source: §7 · p. 18

• Across four unstructured document analysis tasks, DocETL found plans with outputs 21–80% more accurate than baselines.Source: §8 · p. 18
• DocETL combines novel rewrite directives with agent-based plan rewriting and evaluation and an opportunistic optimization strategy.Source: §8 · p. 18
• The paper presents automated optimization as feasible and beneficial despite the difficulty of exploring the large space of possible plan decompositions.Future work will consider cheaper models for simpler subtasks and human feedback to refine plans.Source: §8 · p. 18

• Gather configuration specifies a group ID key, an order key, and a content key for chunk grouping, sequence, and content.Source: §A.1 · p. 20
• Peripheral chunk configuration specifies previous and next sections, each of which may contain head, middle, and tail subsections, with amounts to include.Source: §A.1 · p. 20
• Each peripheral subsection must specify a content_key identifying the field used as chunk content.Source: §A.1 · p. 20

A.2 Header Lineage Preservation

• Gather reconstructs the header path for a chunk by finding the level of its first header and including the most recent higher-level headers from preceding chunks in order.When configured with a doc_header_key, it examines preceding chunks and arranges the relevant headers to preserve structure across chunk boundaries.Source: §A.2 · p. 20
• For a 74-page legal contract, headers are extracted from chunks by map operations, and rendering Chunk 20 includes the latest headers above its first header.This supplies hierarchical context for LLM processing.Source: §A.2 · p. 20

• The appendix provides a detailed walkthrough of an LLM agent instantiating a rewrite directive for the legal contract analysis task in Section 5.1.Source: B Rewrite Directive Instantiation · p. 20

B.1 Initial Task and Baseline Operation

• The baseline uses a single map operation with a prompt listing all 41 clause types and an output schema of clause-type and text-span pairs.The prompt requests empty strings for absent or indeterminable categories and comma-separated spans when a category has multiple spans.Source: §B.1 · pp. 20–21
• The prompt covers contract identity, parties, dates, renewal, termination, governance, and commercial restrictions, among other clause categories.Its categories include change of control, assignment, intellectual property, liability, warranties, insurance, and third-party beneficiaries.Source: §B.1 · pp. 20–21
• Figure 6 shows headers extracted from chunks by map operations and the latest higher-level headers included when rendering a chunk of a 74-page contract.This provides hierarchical context for processing the chunk.Source: Figure 6 · p. 21

B.2 Optimization Process

• The optimizer first evaluates whether the operation needs improvement using an LLM-synthesized validator prompt based on the operation prompt and sample outputs.The agent is instructed to devise binary criteria around precision and recall.Source: §B.2.1 · p. 21

B.2.1 Determine If Optimization Is Needed

• The validator checks whether the original operation captures all relevant spans for every category and formats the extracted information correctly.Applied to sample outputs, it identified missed categories and recommended explicitly representing categories with no applicable text spans rather than omitting them.Source: §B.2.1 · p. 22

B.2.2 Select and Apply Rewrite Directive

• After assessing the operation, DocETL matches it to applicable rewrite directives and invokes an agent to decompose the complex map task into independent subtasks.The agent receives the original prompt and schema, sample contract documents, and decomposition instructions; the selected example follows Equation (11).Source: §B.2.2 · p. 22

B.2.3 LLM Agent Generates Decomposed Operations

• The agent generates prompts and schemas for subtasks such as basic contract information, renewal and termination, and legal governance clauses.DocETL also synthesizes an operation to convert the parallel_map's flat schema back to the user's expected output; in practice, it recursively applies rewrites to each new subtask.Source: §B.2.3 · p. 22
• DocETL executes alternative plans on a sample, has a validation agent rate and compare their outputs, and selects the top-ranked plan to replace the original operation.The walkthrough assumes newly synthesized operations perform their subtasks accurately, although the system recursively applies rewrites in practice.Source: §B.2.4 · p. 22, §B.2.3 · p. 22