Unsupervised Tool Evolution for Coding Agents in Data-Lake Analytics

1. Introduction

• Coding agents discard parsers, schema checks, and join scripts after each question, repeating data-lake exploration and preparation on later questions.The work can be costly under step budgets: finding and cleaning 52 irregularly headed state CSVs exhausts 30 steps and fails sooner under 15.Source: §1 · p. 1
• Existing tool generators target individual tasks or rely on costly ground-truth validation, while execution without errors cannot establish that a data-lake tool is useful.A tool may read data at the wrong granularity, drop parsed rows, or join on unaligned keys.Source: §1 · p. 2
• TEAL evolves executable tools and usage skills for a fixed Worker without labels, then freezes the library for answering new questions.Its Explorer proposes data-lake-wide tools, while its Critic uses Worker trajectories to propose targeted changes.Source: §1 · p. 2, §1.1 · p. 2

1.1. Contributions

• TEAL keeps a proposed change only when a rerun uses the changed tool, finishes without extra steps, and supports any changed answer with trajectory evidence.This verification uses no ground-truth answers.Source: §1.1 · p. 2
• TEAL-Q generates answerable synthetic questions from the data lake when real user questions are scarce.The evaluation reports that libraries evolved on 50 synthetic questions perform comparably to those evolved on real questions.Source: §1.1 · pp. 2–3
• 9.3 points on KramaBench and 21.7 points on LakeQA: TEAL improves Sonnet-5 at 15 steps, with gains for all four Workers on both benchmarks at 10 and 15 steps.Libraries evolved under tighter budgets also transfer to looser ones; Sonnet-5’s LakeQA library evolved at 20 steps scores 68.3 at 30 steps, versus 59.6 for baseline and 64.2 for a library evolved at 30 steps.Source: §1.1 · pp. 2–3

Tools and Agents for Data Analytics

• Earlier systems use human-prepared catalogs, indexes, views, plugins, or fixed search tools; EvoOntology refines an ontology using labeled validation, while DataCOPE discovers procedural skills from trajectory agreement.TEAL instead studies agents that decide from data exploration and unlabeled trajectories what callable tools and guidance to build.Source: §2 · p. 3

Tool Evolution for LLM Agents

• Tool-generation work has built helpers for single-program questions or long-horizon tasks, where acceptance can rely on execution or environmental completion; TEAL targets data-lake analytics where neither assumption suffices.Without labels, TEAL verifies changes through Worker behavior on reruns and investigates evolution under tighter step budgets.Source: §2 · p. 3

3.1. Problem Formulation

• The library contains callable programs retained across tasks, with optional persistent artifacts such as indexes or cached metadata built for reuse.The Worker answers training questions under a step budget; the frozen library is evaluated on disjoint test questions over the same lake.Source: §3.1 · pp. 3–4

3.2. Unsupervised Tool Evolution via TEAL

• TEAL coordinates five LLM agents around a Worker: Explorer, Critic, Toolsmith, Skill Writer, and Analyst.The Explorer proposes lake-specific tools, the Critic diagnoses trajectory bottlenecks, the Toolsmith implements and tests proposals, the Skill Writer maintains usage guidance, and the Analyst compares changed-answer trajectories.Source: §3.2 · p. 4

3.2.1. The tool evolution pipeline

• During offline evolution, TEAL processes question batches through trajectory collection, lake-wide exploration, trajectory criticism, skill updates, and verification.The Explorer runs periodically and may build persistent artifacts; the Critic targets defects, unused helpful tools, or capability gaps, and the Skill Writer incorporates implemented changes.Source: §3.2.1 · pp. 4–5
• The Critic targets immediate trajectory-specific fixes, while the Explorer proposes tools for the whole lake, such as search and lookup backed by catalogs or value indexes.The Explorer sees neither questions nor trajectories; the Critic uses trajectory evidence.Source: §3.2.1 · p. 5

3.2.2. Verification without ground-truth answers

• Verification reruns the motivating questions and checks tool use, task completion without extra steps, and trajectory support when an answer changes.A change is kept if a rerun passes all three checks and reverted otherwise; when the Worker ignores a tool, TEAL may first revise its usage guidance.Source: §3.2.2 · p. 6

3.3. Synthetic Question Generation via TEAL-Q

• TEAL-Q uses seed questions as demonstrations to generate synthetic questions from the data lake when real questions are unavailable.It drafts a question with a computed answer, hardens it with added processing or reasoning demands, builds a pipeline to check answerability, then rewrites the question; only questions are passed to TEAL.Source: §3.3 · pp. 6–7

4.1. Experimental Settings

• The evaluation uses KramaBench and LakeQA training and test tasks, four Worker backends, and step budgets from 10 to 30 that limit per-question work.KramaBench uses 51 training and 53 test tasks; the tables-only LakeQA split has 50 training and 80 test tasks.Source: §4.1 · p. 7, §4.2 · p. 7

4.2. Main Results

• At 15 steps, TEAL improves accuracy by 9.3 points on KramaBench and 21.7 points on LakeQA; gains hold for all four Workers on both benchmarks at 10 and 15 steps.In five of eight panels, TEAL at 15 steps is on par with or above the baseline at 30 steps.Source: §4.2 · pp. 7–8

4.3. Comparison with Existing Methods

• TEAL improves on a strong baseline across both datasets and both budgets, while execution-only ISE underperforms TEAL on both datasets at 15 steps.ISE is 5.7 points below TEAL on KramaBench and 10.0 points below the baseline on LakeQA; 121 of 126 test-time calls to its search_file_content tool timed out.Source: §4.3 · p. 9, Table 1 · p. 9

4.4. Results in the synthetic-question scenario

• A library evolved on 50 synthetic questions improves Sonnet-5 over baseline on both datasets and performs comparably to one evolved on real questions.The questions are generated from each lake and evaluated on the same held-out test tasks.Source: §4.4 · p. 9, Table 2 · p. 10

4.5. Ablation and Generalization

• Removing the Explorer leaves Critic-driven gains but loses artifact-backed tools, while removing evolved skills causes a larger drop because the Worker lacks guidance on tool use.The ablation supports evolving tools and their usage skills together.Source: §4.5 · p. 10, Table 3 · p. 10
• With ground-truth answers but not full solutions, TEAL performs comparably to its label-free version; full solutions enable more targeted tools, including joins that match area names differing in punctuation.The label-free library only standardizes state names with normalize_join_state_keys.Source: §4.5 · p. 10
• The library evolved under OpenHands improves Sonnet-5 over baseline when deployed unchanged under Google ADK, Prime Agent, and Codex CLI.This tests transfer across harnesses rather than across Worker models.Source: §4.5 · p. 10

• TEAL evolves tools and usage skills without labels, while TEAL-Q generates synthetic questions for tool evolution.Source: §5 · p. 11
• Libraries evolved under tight budgets improve all four Workers, remain effective under looser budgets, and perform comparably when evolved on synthetic or real questions.Source: §5 · p. 11
• The evolved library lengthens prompts at every step and leads the Worker to reason more at each step.The authors suggest cache reuse and internalizing skills as possible ways to reduce these overheads.Source: §5 · p. 11

A. Qualitative Analysis

• A LakeQA catalog tool stores file formats, headers, row-count estimates, column samples, quality flags, and directory names in a persistent 93 MB SQLite artifact.Building it takes about four minutes and roughly forty batched embedding calls; it is not rebuilt during a task.Source: A. Qualitative Analysis · p. 15
• The catalog tool was revised fourteen times from Worker trajectories, including a change that returns all editions of a matched table together.Source: A. Qualitative Analysis · p. 15
• Across evolution runs, artifacts include catalogs, value indexes, manifests, and decoded archive copies, with the specific artifact varying by run.One value index reaches 6 GB.Source: A. Qualitative Analysis · p. 15
• On an Albany County crime-rate task, catalog search plus a SQL query produced 359.8 within four steps in every TEAL repeat, while the baseline used nine to ten steps and was incorrect in two of three repeats.Source: A. Qualitative Analysis · p. 15
• For a paleoclimate workbook, header detection enabled correct answers in every TEAL repeat, while the baseline failed to parse or align the header correctly.Source: A. Qualitative Analysis · p. 15

B. Tool Statistics

• Three of the five most-used KramaBench tools support data discovery, while two load CSVs or Excel sheets that standard readers parse incorrectly.Source: B. Tool Statistics · pp. 15–16
• Table 4 reports coverage as the share of 159 test episodes, comprising 53 tasks with three repeats, that call each listed tool at least once.Source: Table 4 · p. 16
• On LakeQA, the larger lake is indexed ahead of time, and the Worker calls the catalog tool in nearly every episode.The library also includes full-text search over dataset names and headers and DuckDB-based filtering or aggregation that streams large files.Source: B. Tool Statistics · p. 16

C.1. Step counting

• The Worker’s actual step count can differ from its step budget.Source: §C.1 · p. 16
• Figure 4 plots mean steps per episode across three test repeats on the horizontal axis and accuracy on the vertical axis.Source: §C.1 · p. 16
• A curve nearer the upper left represents a better accuracy-versus-steps trade-off, and evolved tools reach higher accuracy with fewer steps.Source: §C.1 · p. 16

C.2. Cost accounting

• Episode cost is estimated from uncached-input, cached-input, and output token counts multiplied by their corresponding Worker prices.Assuming no cache misses makes the estimate a lower bound; it is used instead of billed cost because caching behavior differed across Workers.Source: §C.2 · p. 18
• With Sonnet-5, Gemini-3.1-Pro, and Gemini-3.8-Flash, TEAL generally reaches higher accuracy at the same cost or the same accuracy at lower cost.Opus-4.8 has more expensive input tokens, limiting TEAL’s cost advantage, but TEAL remains preferable when the goal is higher accuracy.Source: §C.2 · p. 18, Figure 5 · p. 19
• TEAL’s added tools and skills make its fixed prompt prefix roughly two to three times longer than the baseline’s, and the Worker reads it at every step.Prompt caching absorbs much of this overhead, so the authors recommend enabling caching when deploying TEAL.Source: §C.2 · p. 18
• On KramaBench at 15 steps, TEAL cuts uncached input per episode by 37% to 63% for Sonnet-5, Gemini-3.1-Pro, and Gemini-3.8-Flash.Tool outputs replace raw intermediate output passed through the shell; output shrinks less than step count, while output per step grows for most Workers.Source: §C.2 · pp. 18–19

D. Synthetic-Question Scenario Across Workers

• Table 7 compares accuracy for TEAL evolved on real or synthetic questions against the same Worker without evolved tools, using the same held-out tasks and a 15-step budget.The supplied passage identifies the table’s scope but does not provide its cell values.Source: D. Synthetic-Question Scenario Across… · p. 19, Table 7 · p. 20
• Figure 6 shows two trajectories on the Albany County task from Section A, both with Sonnet-5 as Worker under a 10-step budget; tool outputs and shell observations are abridged, and the final finish step is omitted.Source: Figure 6 · p. 21
• Figure 7 presents paleoclimate workbook task trajectories from Section A for Gemini-3.1-Pro as Worker under a 10-step budget; tool outputs and shell observations are abridged, and final finish step omitted.The caption does not state a trajectory outcome.Source: Figure 7 · p. 22