Measure Learning at Steady State: A BIRD-SQL Formula 1 Case Study

Motivation

• A short-schedule gain can mix early exploration with later reuse, so the paper defines steady-state learning as the baseline gap over the last 40 of 174 questions.The authors argue that a mean over the first 20 or 40 questions mixes the transient with later reuse.Source: §1 · p. 1

Measurement

• The evaluation separates exploration efficiency (SQL probes), task reward (binary correctness), and delivery cost (API dollars and prompt size).The authors note that these three measures can tell different stories on this task.Source: §1 · pp. 1–2

Study rationale

• The study uses ICL because CL-Bench found it strongest among the tested memories and it is easy to ablate, but does not claim it is the best learning mechanism.The authors argue that ranking ICL as best is unsafe when context can grow without bound.Source: §1 · pp. 1–2

Continual Learning Bench

• CL-Bench [Asawa et al., 2026] evaluates six stateful environments against each system’s stateless baseline; naive full-context ICL outperforms tested memory architectures on most tasks.The paper adopts its reset comparison and shared-world design, but changes the horizon and aggregation to late-window curves and three measures.Source: §2 · p. 2

Scope of comparison

• The authors do not rerun Mem0 or ACE and do not claim those systems would lose on Formula 1.Their claim is narrower: ICL’s ranking as best does not survive a long transcript because context growth raises input tokens and dollars.Source: §2 · p. 2

Related evaluation approaches

• AgentCL [Shu et al., 2026] motivates controls, fixed ordering, a per-question reset baseline, and separate reporting of late-window and whole-run means.The study applies the efficiency lens to repeated work in one database, rather than novel game episodes, unlike ARC-AGI-3 [ARC Prize Foundation, 2026].Source: §2 · p. 2

Context and benchmark

• Prompt caching bills shared prefixes as cache reads, but later turns still require sending a longer prefix; the environment is BIRD [Li et al., 2023]’s Formula 1 database and question bank.The evaluation uses the SQLite file from BIRD Mini-Dev and the formula_1 subset of the 2025-11-06 development dump, not a new text-to-SQL leaderboard.Source: §2 · p. 2

3.1 Task

• BIRD-SQL formula_1 uses one SQLite database and 174 natural-language questions in official order, with binary correctness and no mid-run schema change.The agent may run SQL only through db query and must write an answer file.Source: §3.1 · p. 3

3.2 Conditions

• The baseline starts a fresh chat for each question, while ICL resumes the same chat trajectory.A native pi compaction arm is reported only as a negative experiment, not as part of the main claim.Source: §3.2 · p. 3

3.3 Models and jobs

• Two OpenAI gpt-5.6-* models are evaluated, but rollout counts differ and Terra ICL has a resume seam at question 122.The Terra trajectory continued the q-121 session after a verifier fix; hits and probes are stitched, while dollars sum the two Harbor jobs.Source: §3.3 · p. 3

3.4 Metrics

• Steady-state learning is baseline minus learner over questions 135–174, with mean queries primary and hits and dollars also measured on that window.Context tokens estimate prompt size including input and cache-read; API cost is provider USD. Whole-run means are secondary, and wall time is not reported.Source: §3.4 · p. 3

4.1 Whole run versus late window

• 0.95 versus 5.14 mean late SQL probes per question for Luna ICL versus the mean of three baselines, a saving of about 4.2 probes per late question.Terra saves about 2.8 late probes per question (1.08 vs. 3.85).Source: Table 2 · p. 4, §4.1 · p. 4
• A 40-question mean already shows ICL helping but understates the late exploration gap and reports a hit story that barely moves.The authors argue that the last 40 questions are the relevant score when the product goal is to stop rediscovering the schema.Source: §4.1 · p. 4
• Late hits barely move on Luna (13 vs. 10–11), while Terra ICL gains late hits (19 vs. 9), making reward model-dependent despite reliable exploration-efficiency gains.On Luna, the whole-run hit lift is small (74 vs. 65), while SQL probes fall from 4.78 to 1.07; cumulative probes on Luna r1 are 186 vs. 832.Source: §4.1 · pp. 3–4

4.2 The curve, not the average

• Figure 1 shows mean SQL probes in successive ten-question blocks: Luna ICL is near one from the second block onward, while baseline does not fall.The late gap is a plateau against a system that keeps exploring, not a slow climb that a 40-question study would catch in miniature.Source: §4.2 · p. 4
• Luna ICL reaches about one probe by question 2; the late window measures persistence of savings, not when learning starts.The late window is the period in which the saving remains after a short evaluation would have stopped.Source: §4.2 · p. 4

5.1 Short-horizon gain hides the late saving

• A 40-question mean shows ICL helping but dilutes the late exploration gap; whole-run mean queries are 1.07 versus 4.78.Table 3 compares Luna r1’s first 40 and last 40 questions.Source: §5.1 · pp. 4–5, Table 3 · p. 5

5.2 Three measures, three conclusions

• Late ICL API dollars are $3.56 versus $1.16 baseline, while late SQL probes are 0.95 versus 5.58; whole-run hits are 74 versus 65.The measures give different conclusions about learning: substantial probe savings, small hit gains, and higher ICL spend.Source: §5.2 · p. 5

5.3 Unbounded context makes ICL a costly mechanism

• Late ICL blocks cost about $0.85–$0.92 per ten questions versus $0.17–$0.48 for baseline, as full-context prompts grow with each job.Terra shows the same qualitative spend result: $11.10 for ICL versus $3.63 for baseline over the run.Source: §5.3 · pp. 5–6

5.4 Experiment: native compaction does not fix delivery

• Native compaction held Luna context near 25–32k tokens (134 successful compactions, one summarizer failure) and Terra near 29k at the end (141 compactions), but did not cut dollars and cost more.Untuned summarization, broken cache prefixes after each compact, and extra summarizer calls were enough to explain the cost; fine-tuned or learned compactors might do better.Source: §5.4 · p. 6

• The evaluation covers one database and the official question order, with no mid-run schema change.Source: §6 · p. 7
• Each model has one ICL rollout; Luna baseline has three replicates, while ICL variance is unmeasured.Source: §6 · p. 7
• The main tables compare ICL as the only learning mechanism, using two OpenAI gpt-5.6-* models and provider usage in Harbor for dollar costs.Source: §6 · p. 7

• Late SQL probes fall to about one per question with ICL, versus four to six for a fresh chat, indicating schema reuse rather than an accuracy miracle.Source: §7 · p. 7
• A 40-question mean shows a probe cut but almost no hit lift, shrinks the late exploration gap, and does not track cost as the transcript grows.The API bill rises while probes stay flat.Source: §7 · p. 7
• The proposed measure is the late-window gap versus baseline, reported separately for exploration, reward, and delivery cost.The target is to retain the exploration benefit without the delivery-cost increase.Source: §7 · p. 7

• Traces are public on Harbor Hub, and an ancillary file lists every job.Source: A Jobs and traces · p. 7
• Figure 4 reports ICL context as input plus cache-read; Luna ends at ∼95k tokens and Terra at ∼103k, while baseline starts a new prompt each time.Source: Figure 4 · p. 8
• The listed traces include Luna baseline replicates, Luna ICL and compact runs, Terra baseline, and Terra ICL segments and compact run.The passage also identifies a Luna r1 interactive viewer and stitched Terra ICL metrics file.Source: A Jobs and traces · p. 8