Same Context, Different Agents: A Behavioral Study of Semantic Layers, Multi-Agent Orchestration and Memory Compaction in Data Agents

1 Introduction

• Agents must locate relevant evidence across heterogeneous files, tables, documents, and web pages, then align and compute results.Sources can have irregular layouts, multilingual entity names, obfuscated columns, and metrics defined elsewhere.Source: §1 · p. 1
• Growing histories and long prompts can lead agents to forget findings, state uncomputed values, or act without knowing where to look.With the same context, some agents inspect sources, some answer without looking, and others give up partway.Source: §1 · p. 1, Figure 1 · p. 1
• CETs target context by describing data with a semantic layer, distributing work through orchestration, or replacing growing history with bounded memory.Prior studies generally evaluate each technique alone, leaving unclear how they alter different models’ behavior alone and in combination.Source: §1 · pp. 1–2, Figure 2 · p. 2
• Across seven LLMs and five benchmarks (1,514 tasks), CET effects depend on the agent’s behavior: a technique can rescue one group while degrading another, and combined effects are smaller than their sum.The study tests all eight CET combinations; orchestration helps agents that answer without looking, while no CET raises accuracy for agents that already investigate.Source: §1 · p. 2
• The study evaluates accuracy, behavior, and cost across every combination of three CETs, addressing how they affect models and how their effects combine.Its contributions include tracing seven LLMs into three behavior groups and analyzing causes of wrong answers.Source: §1 · pp. 2–3

2 Framework

• Eight configurations switch each of the three CETs on or off while holding the model, tools, decoding settings, and step and call budgets constant.The baseline is a single tool-using agent.Source: §2 · p. 3
• The baseline agent alternates between a thought, one tool action, and an observation until it answers, with instructions to compute values from executed code and keep working.It can inspect sources, test intermediate results, and correct failed or suspicious actions; CETs change context and work organization, not its available functions.Source: §2 · p. 3
• A semantic layer supplies question-relevant descriptions of each source’s ontology, data profile, and contextualization, which the agent is instructed to verify.An LLM creates the descriptions offline once per benchmark, without answers or task-specific hints.Source: §2 · p. 3
• Orchestration assigns subtasks from a tool-less planner to specialist agents, whose condensed results a verifier accepts or sends back for more work.Specialists use the baseline agent’s tools in separate contexts; with a semantic layer, the planner passes relevant entries to them.Source: §2 · p. 3
• Compaction replaces the full interaction history with bounded working memory rewritten after every step to track the goal, completed work, evidenced values, and open items.Source: §2 · p. 3

3 Experimental Setup

• Five benchmarks contribute 1,514 tasks spanning irregular data lakes, noisy enterprise tables and policy texts, database-document-web evidence, and database-to-web clue chains.FDABench includes report and multiple-choice tasks; HybridDeepResearch requires table discovery without a schema.Source: §3 · p. 3
• The seven tested models are GPT-5.4, GPT-5.5, GPT-6 Astra, Claude Sonnet 5.5, Claude Opus 5.5, Gemini 2.5 Flash, and locally served GLM-4.7-Flash.GPT-5.5 wrote the semantic layers.Source: §3 · p. 3
• Each CET’s effect is its change when switched on, averaged over the settings of the other two; accuracy effects use percentage points and 95% bootstrap intervals over tasks.Cost is dollars per task at list prices, and group effects average the models in each group.Source: §3 · p. 3
• Behavior is measured from run traces using tool use, grounding, source coverage, persistence, recovery, repeated actions, hedging, give-ups, answers, and long loops.Long loops are defined as 100 or more model calls.Source: §3 · p. 3

4.1 Three Groups of Baseline Agents

• The seven baseline agents fall into investigators, premature answerers, and early stoppers; investigators lead benchmarks, while post-inspection failures mostly involve choosing the wrong data slice or misreading evidence.Answers without tool use are right in 7% of runs versus 44% after looking, and no CET removes the common post-inspection failure causes.Source: §4.1 · p. 4

4.2 CET Benefits Depend on Agent Behavior and Task Demands

• Orchestration adds 23 points for premature answerers; the layer gives early stoppers +4.6 versus orchestration’s +1.8 points, while every CET raises cost and investigators gain no accuracy.Orchestration loses accuracy on underspecified questions; the layer helps most with small, lightly messy data and less when three or four sources must be combined; compaction’s effects depend on task length.Source: §4.2 · p. 4, Figure 4 · p. 5, Figure 5 · p. 5
• Orchestration raises investigators’ process score by 11 points while their accuracy falls by 5, and shifts engagement, reliability, and commitment in different ways.Premature answerers act, cover sources, and persist in 36–50 points more runs; the layer raises reliability at the expense of engagement, while compaction adds repeated actions.Source: §4.2 · p. 5

4.3 Orchestration Helps Premature Answerers but Can Harm Investigators

• Orchestration gives premature answerers its largest gains by requiring evidence, but lowers investigators’ accuracy by 5 points at 1.7 times the dollars.A forced-action single agent closes at most 37% of GPT-5.5’s orchestration gap; investigators’ losses are concentrated on tasks combining databases with documents and web search.Source: §4.3 · pp. 5–6

4.4 Semantic Layers Help Early Stoppers but Can Discourage Investigation

• The semantic layer halves give-ups for agents that give up most, but investigators stop checking and their accuracy falls from 68% to 31% on runs where they stop looking.It reduces calls by 22–30% but raises dollars by 1.1–1.9 times; its ontology, data profile, and contextualization guide source selection, file reading, and joins.Source: §4.4 · p. 6, Figure 7 · p. 7

4.5 Compaction Bounds Context but Can Increase Cost and Repetition

• Compaction keeps prompts near three times the initial size instead of fourteen times, but raises dollars by 1.2–1.8 times and can trigger long loops.On the longest tasks, early stoppers give up less and score 2.5 points better; on short tasks, they lose 8 points, and 42% of GPT-5.5’s DataClaw runs reach 100 or more calls at eight times the cost.Source: §4.5 · p. 7

4.6 Combining CETs Offers Limited Added Gains but Can Mitigate Harms

• Combined CET effects are mostly subadditive because they repair overlapping behaviors, while the semantic layer mitigates compaction’s looping and grounding harms.With orchestration, compaction cuts grounded-answer gains from +12 to +4 points; adding the layer reduces compaction’s long-run increase from 9 to 6 points.Source: §4.6 · pp. 7–8, Table 2 · p. 8

Data agent benchmarks

• Data-agent benchmarks have expanded from single database queries to heterogeneous files, enterprise data, documents, and web evidence, while strong agents remain unreliable.The best published KramaBench agent scores 56%, compared with 77% for humans.Source: §5 · p. 8

Context-engineering techniques

• Prior studies report benefits and limits for all three CETs, including reduced accuracy for capable models, single-agent parity under equal budgets, and weak gains from plain summaries.The closest studies relate orchestration benefits to single-agent strength or compare semantic-layer delivery, but evaluate by accuracy.Source: §5 · p. 8
• This study evaluates the three CETs together across seven LLMs and characterizes them by changed behavior and repaired failures, relative to agent habits.Source: §5 · p. 8

6 Practical Guidance and Conclusion

• Match orchestration to agents that answer without looking, a semantic layer to agents that give up, and compaction to long runs; measure each combination because CET effects do not add predictably.The wrong-data-slice failure remains common after inspection, and none of the three techniques fixes it.Source: §6 · p. 8

Limitations

• The evaluation covers seven LLMs, three CET designs, and heterogeneous-data tasks; whether the observed behavior groups and interactions generalize to other domains remains open.Source: Limitations · p. 9
• The study tests fixed CET configurations; selecting or adapting techniques based on observed agent behavior is left for future work.Source: Limitations · p. 9
• The authors identify extending the analysis to more models, model families, CET designs, and techniques as future work.Source: Limitations · p. 9

Ethical Considerations

• The study does not evaluate whether context-engineering techniques are ready for autonomous deployment.Behavioral groupings describe patterns under the evaluated tasks and configurations, not fixed model properties.Source: Ethical Considerations · p. 9
• High-stakes analytical uses require task-specific validation, traceable evidence, appropriate human oversight, and safeguards for data access.The paper identifies medical, legal, financial, and other high-stakes decisions as settings where failures could have serious consequences.Source: Ethical Considerations · p. 9
• Practitioners should assess context-engineering choices against answer correctness and evidence provenance, not aggregate benchmark gains or efficiency alone.Source: Ethical Considerations · p. 9

AI Use Statement

• Generative AI assisted with coding and editing; the authors checked and tested AI-assisted code and reviewed and revised AI-assisted text.Source: AI Use Statement · p. 9
• The authors state they did not use generative AI for ideation, literature search, or results analysis and interpretation.LLMs also served as study subjects, annotators, and judges; the authors take responsibility for the paper.Source: AI Use Statement · p. 9

A Agent Configurations

• All eight configurations shared one harness, tools, decoding settings, and limits of up to 300 steps and 600 model calls per task.Source: A Agent Configurations · p. 12
• Semantic-layer entries were placed in the prompt up to a fixed size; compaction rewrites cost an extra model call and kept layer entries outside memory.Source: A Agent Configurations · p. 12

B Behavior and Task Features

• Baseline traces and run types are clustered from 12 features, while eight behavioral indicators summarize the resulting profiles; investigators score highly on most indicators, unlike GPT-5.5.The 12 features include tool use, sources touched, errors, repeats, and stopping behavior; the indicators include acting, grounded answers, source coverage, and persistence.Source: B Behavior and Task Features · p. 12

C Reasoning Effort

• Reasoning effort reduces GPT-5.4’s no-tool answers and raises accuracy, but barely changes GPT-5.5’s habit of answering without inspection.GPT-5.4 accuracy rises by 35 points on KramaBench and 35–36 points on HDR; at high effort GPT-5.5 still answers without a tool on 78% and 87% of tasks, respectively.Source: C Reasoning Effort · p. 15, Figure 12 · p. 16

D How Baseline Agents Fail

• Investigators mainly make wrong-value or partial-source errors, premature answerers often answer without tools, and early stoppers frequently use only some needed sources or give up.After inspection, wrong data slices (40.5%) and external evidence (26.4%) are more common causes of wrong values than computation (9.1%); the pattern remains broadly similar across CETs.Source: D How Baseline Agents Fail · pp. 15, 17, Figure 14 · p. 16

E Where and for Whom Each CET Helps

• Orchestration raises premature answerers’ accuracy on all 19 demand levels but lowers investigators’ accuracy on 18; task demands mostly change effect size, not its sign.For premature answerers, orchestration helps most when sources are given or the candidate pool is small (+26 points), and least on joins, name matching, and medium-size data (+17, +9, and +8).Source: E Where and for Whom Each CET Helps · p. 17, Figure 15 · p. 18
• The semantic layer helps early stoppers most on name matching and lightly messy data, while compaction’s standout losses occur for early stoppers on name matching and medium-size data.The paper cautions that name matching and medium-size data levels come mostly from DataClaw, so those comparisons partly reflect that benchmark.Source: E Where and for Whom Each CET Helps · pp. 17, 19, Figure 15 · p. 18

F Behavioral Fingerprint

• Orchestration can make Claude investigators less likely to flag uncertainty, while compaction increases repeated actions across groups.Opus and Sonnet flag uncertainty less with orchestration (−35 and −29 points); with compaction, no-repeat rates fall for GPT-5.5, Astra, and GLM by 16, 12, and 10 points.Source: F Behavioral Fingerprint · p. 19

• Premature answerers gain 27.5 accuracy points, while investigators lose 11.4 and regain 6.4; investigator losses cluster in tasks chaining databases, documents, and web search.Investigator losses include wrong values despite reaching the needed source and dropped or swapped multiple-choice options.Source: Figure 17 · p. 20, G Orchestration: Further Results · p. 21
• Forcing GPT-5.5 to act or match orchestration’s call count closes only 7–37% of its accuracy gain across three benchmarks.Matching the calls does not clearly outperform forcing one action on any benchmark.Source: G Orchestration: Further Results · p. 20, Figure 19 · p. 22
• Orchestration turns premature answerers’ runs into lookups and web investigations, but adds repetitive effort without new source types for early stoppers.Early stoppers make 39 more calls, while their FDABench reports grow by about 4,700 characters.Source: G Orchestration: Further Results · pp. 20–21, Figure 20 · p. 22
• 13 of 17 baseline pairs above .45 lose accuracy (mean −4.7 points), while 17 of 18 at or below .45 gain (mean +14.0).Accuracy change correlates −.65 with baseline accuracy and +.60 with the baseline share of answers without a tool.Source: Figure 21 · p. 22, G Orchestration: Further Results · pp. 22–23

H Semantic Layer: Further Results

• The ontology produces the most answer fixes: 47 fixes and 17 breaks per 1,000 runs, compared with at most 6 fixes and 2 breaks for either other layer part.It especially helps early stoppers, while investigators—who usually find sources themselves—show the opposite balance: 77 fixes against 448 breaks.Source: H Semantic Layer: Further Results · p. 23
• The layer’s parts have distinct effects: the ontology guides source choice, the data profile reduces failed reads, and contextualization supplies join keys where needed.On KramaBench, failed reads fall from 28% to 12% for investigators and from 28% to 8% for premature answerers; agents use supplied join keys in 14–27% of runs when a join is needed.Source: H Semantic Layer: Further Results · p. 23, Figure 23 · p. 24
• For early stoppers, the layer reduces long repetitive runs from 21% to 13%; for investigators, quick lookups rise from 31% to 45%.The direction of a model’s shift varies: Opus stops looking in 73 runs and starts in 11, while GPT-5.4 starts looking more often than it stops.Source: H Semantic Layer: Further Results · p. 23, Figure 22 · p. 24
• A 3B-authored layer gives GPT-5.4 nearly the same DataClaw accuracy gain as a GPT-5.5-authored layer: +12.7 versus +14.7 points.Across seven models, the layer narrows behavioral spread more than accuracy spread: accuracy standard deviation moves from 20 to 18 points, while the spread in giving up moves from 6.2 to 3.2.Source: Figure 24 · p. 24, H Semantic Layer: Further Results · p. 25
• In 300 sampled broken answers, the layer misled agents in 24; investigators more often stopped checking, while early stoppers were more often misled.In many investigator failures, the layer correctly described a common case but the agent treated it as complete and missed another relevant column, key, or source.Source: H Semantic Layer: Further Results · p. 25

I Compaction: Further Results

• Compaction gains have different sources: GPT-5.4 improves on DataClaw at every task length, while GLM’s HDR gain is concentrated in runs whose transcript overflowed its context window.The reported gains are +6.4 points for GPT-5.4 on DataClaw and +7.1 points for GLM on overflowed HDR runs; compaction also raises calls by 1.7 to 3.1 times and prevents prompt-cache reuse.Source: I Compaction: Further Results · p. 25
• Compaction loops arise through different mechanisms: premature answerers re-check results still in memory, while early stoppers repeat actions to recover information dropped by rewrites.Loops increase with task length; when full-transcript prompts exceed 60k tokens per call, compaction halves tokens but lowers accuracy by 3.3 points [−5.0, −1.6].Source: I Compaction: Further Results · pp. 25–26
• Compaction changes fewer than 2.5% of investigators’ runs between behavior types, but shifts 5% of premature answerers’ and 6% of early stoppers’ runs into long repetitive runs.Early stoppers’ long repetitive runs rise from 11% to 23%.Source: I Compaction: Further Results · p. 25, Figure 25 · p. 26

J Combinations and Task Difficulty

• The best configuration depends on the model: orchestration alone raises GPT-5.5 from 21 to 58, while all three CETs raise GLM from 15 to 23.Orchestration with the layer is best for GPT-5.4 (21 to 40) and Gemini (25 to 29).Source: Figure 26 · p. 26, J Combinations and Task Difficulty · p. 26
• For early stoppers, accuracy gains nearly add up, but the three-CET combination costs 3.3 times baseline dollars versus a 1.4-fold product of the parts.Investigators’ three-CET combination loses 8.1 points against a 10.6-point sum of losses; the authors advise measuring combinations rather than inferring them from component effects.Source: J Combinations and Task Difficulty · pp. 26–27
• For premature answerers, orchestration and the layer repair the same habit, so their combined accuracy gains fall short of the sum of their parts.Orchestration plus layer gains +27.7 points against a +31.7-point sum; all three gain +26.6 against +34.3, though all three cost 10.3 times baseline dollars rather than the parts’ 52.9-fold product.Source: J Combinations and Task Difficulty · pp. 26–27, Figure 27 · p. 27
• The layer reduces compaction’s increase in runs with 100 or more calls from 9.4 to 6.2 points, while orchestration and compaction together produce the worst interaction.Under orchestration, compaction raises 100-or-more-call runs by 10.0 points; the layer also reduces calls under orchestration to 0.85 of their level while increasing prompt tokens 1.9 times.Source: J Combinations and Task Difficulty · pp. 27, 29
• No CET adds more than 3 accuracy points for any group on tasks no baseline agent solves; task solvability matters more than gold-step or benchmark difficulty labels.For early stoppers, the layer adds 15 points on tasks solved by six or seven baseline agents, while compaction costs 20 points there.Source: Figure 28 · p. 28, J Combinations and Task Difficulty · p. 29

K Cost

• Orchestration multiplies dollars per task by 1.7 for investigators, 4.0 for premature answerers, and 2.3 for early stoppers; costs use provider list prices.The baseline costs are $1.97, $0.18, and $0.15 per task, respectively; cost estimates for Gemini and GLM are upper bounds because their usage records lack cache fields.Source: K Cost · p. 29
• Within a model, compaction raises GPT-5.5’s task cost from $0.34 to $2.67, while the layer lowers Gemini’s from $0.39 to $0.19.Orchestration with compaction raises GPT-5.5’s cost to $5.27.Source: K Cost · p. 29

L Benchmark Metrics

• On KramaBench, the layer raises average score by 4.3 points, while orchestration raises premature answerers’ pipeline-design score by 27.7 points.The layer improves pipeline design in six of seven models and cuts early stoppers’ calls by 26 per task.Source: L Benchmark Metrics · p. 30, Table 3 · p. 31
• On DataClaw, the layer raises execution efficiency by 25 points and temporal progress efficiency by 15, while compaction lowers execution efficiency by 8.6.Orchestration raises goal progress on incorrect tasks by 6.5 points across all seven models; the metrics apply to correct tasks for execution efficiency and incorrect tasks for goal and temporal progress.Source: L Benchmark Metrics · p. 30, Table 4 · p. 32
• On FDABench reports, orchestration raises the tool-orchestration score by 54 points in all seven models, even as investigators’ rubric scores fall by 5.1 points.The layer raises SQL accuracy by 1.5 points in six of seven models but lowers external integration by 3.1 points in all seven.Source: L Benchmark Metrics · p. 30, Table 5 · p. 33
• On FDABench multiple choice, orchestration raises exact match by 21.9 points for premature answerers and lowers it by 11.2 for investigators.The direction is consistent across difficulty levels, while tool-orchestration score rises by 30 points in all seven models.Source: L Benchmark Metrics · p. 30, Table 6 · p. 34
• On HDR, orchestration’s premature-answerer gain is largest on parallel tasks (+36.6 points), versus +21.6 when SQL comes first and +13.7 when search comes first.The HDR metrics also track calls, tool success, tokens, and runs exceeding the benchmark’s turn or time limits.Source: L Benchmark Metrics · p. 30, Table 7 · p. 35

M Implementation and Compute

• The evaluation fixes models, tools, decoding settings, and step and call budgets across eight configurations, using each benchmark’s own accuracy metric.The harness uses Python 3.10, read-only SQLite and BigQuery databases, and separate Python tool processes; all models except GLM-4.7-Flash are accessed through provider APIs.Source: M Implementation and Compute · p. 36
• Across all benchmarks and configurations, API-model tasks take a median 6.6 minutes and 12 calls; orchestration takes 9 minutes and 17 calls.A semantic layer is written once per benchmark, with larger layers taking several hours to produce for KramaBench and FDABench.Source: M Implementation and Compute · p. 36