Token Optimization and Context Window Management in Multi-Agent AI Workflows

1. Introduction

• Multi-step agent workflows compound token use through repeated context and review loops, while large or poorly arranged contexts can bury useful evidence and degrade quality.Apparently modest workflows can consume tens or hundreds of thousands of tokens internally, motivating decisions about what context to send, in what form, and when.Source: §1 · p. 2

1.1 Contributions

• Six token-optimization patterns connect multi-step LLM workflow practices to production implementations and observed cost, latency, or quality effects.The contribution emphasizes measurable changes in a real multi-stage workflow.Source: §1.1 · pp. 2–3
• Context-window management is framed as an engineering discipline of pruning, compressing, routing, and measuring context to make existing workflows efficient and measurable.The practitioner-layer contribution does not introduce new model internals, orchestration patterns, memory architectures, or model-level prompt caching.Source: §1.1 · pp. 2–3, §1 · p. 2
• 2,420 confirmatory trials across 11 model configurations show that same-domain low-relevance context can improve relevance-score concordance on target items through relevance contrast.The paper does not claim that same-domain noise helps every task.Source: §1.1 · p. 3
• Learned fusion did not beat the mechanical set union of repeated samples for recall-oriented extraction in the Fusion-of-N follow-up.This finding is limited to the studied setting, not a claim that learned fusion is generally useless.Source: §1.1 · p. 3

2. System and Optimization Patterns

• The internal production dashboard ingests meetings, email, and chat, extracts structured information, matches items to engineering workstreams, and generates status summaries.Its optimized architecture separates data acquisition, extraction, and assignment/summarization.Source: §2 · p. 3

2.1 Six Optimization Patterns

• Roughly 30K to roughly 3K tokens per extraction call is the estimated input reduction from context stratification.Each extraction call receives only the relevant data type, implementing the principle of supplying only information needed for that step.Source: §2.1 · p. 5
• 61-116 seconds cold-load latency and 3 acquisition API calls per run, down from 7-21 calls, were observed with fetch-once, process-locally architecture.Meetings, email, and chats are acquired once, then matched locally.Source: §2.1 · p. 5
• Token-aware fallback chains improve availability by retrying extraction with validated fallback models without repeating acquisition, the multi-call stage dominating cold-load latency.This addresses rate limits that otherwise trigger full pipeline retries.Source: §2.1 · p. 5
• Zero LLM tokens are used by cached repeat loads, which skip re-fetching and retokenization.Full pipeline results are cached and validated against current request parameters; near-instant operation was reported but not benchmarked.Source: §2.1 · p. 5
• An estimated 40-60% reduction in inter-stage wrapper tokens accompanies machine-message handoffs; compact JSON contracts also qualitatively lower output token usage and improve parse reliability.Schema-contracted messages replace prose handoffs, while extraction prompts require compact JSON and explicit empty-result conventions.Source: §2.1 · p. 5

2.2 Context-Window Management as an Engineering Skill

• Context-window management becomes an engineering skill encompassing context pruning, information density, memory offloading, parallel discovery, and context-pollution detection.These involve identifying harmful old state, conveying intent with fewer tokens and less ambiguity, and storing stable facts outside prompts. Independent context sources are collected concurrently to reduce latency, while pollution detection identifies irrelevant context changing model behavior.Source: §2.2 · pp. 5–6
• Context calibration balances too little against too much context; for extraction, same-domain low-relevance items can help models establish a relevance threshold.The empirical study refines calibration by showing that the danger is not always too much noise.Source: §2.2 · p. 6

3.1 Production Pipeline Analysis

• 61-116 seconds of optimized cold-load latency were directly observed, versus an estimated pre-optimization baseline of 3.5-10.5 minutes.The production analysis covers 25 recorded runs from February 2026, six with measured elapsed times. Historical token counts use character-count approximations validated against later API metadata, so reductions remain estimates; near-instant cache-hit loads were not separately benchmarked.Source: §3.1 · p. 6

3.2 Context-Composition Experiment

• 661 anonymized communication items come from one knowledge worker’s participant-centered stream over a single work week.The corpus contains 15 meeting transcripts, 627 email threads, and 19 chat threads, scored by the author on a 0-3 relevance scale. Scores 2-3 constitute signal, score-0 items supply low-relevance same-domain context, and score-1 items are excluded to sharpen the contrast.Source: §3.2 · pp. 6–7
• 2,420 confirmatory trials across 11 model configurations test context composition using ten-item prompts, with 20 trial blocks per condition for each configuration.Ratio conditions contain 100%, 80%, 50%, 30%, or 10% signal items. Six position conditions hold signal/noise at 50:50, placing signal at the start, middle, or end, or using three scatter patterns.Source: §3.2 · p. 7
• Relevance Accuracy (RelAcc) measures agreement with human 0-3 scores on signal items, awarding full credit for exact matches and partial credit for adjacent scores.Unparseable output scores 0; because RelAcc covers signal items, it measures target-item relevance scoring rather than full extraction precision. Secondary metrics are precision, JSON parse success, and fact-type recall: mean item-level recovery of annotated fact types, with partial credit for near-equivalent types.Source: §3.2 · p. 7
• Paired model x trial-block comparisons use Holm correction; contrasts originated in a pre-run design-review prompt, not a registry-backed preregistration.The prompt covered a smaller model set, and the final analysis set was elaborated from it before scoring.Source: §3.2 · p. 7

3.3 Fusion-of-N Follow-up

• 5 independent extraction samples per window support comparisons among a single sample, best-of-N judging, LLM fusion, and mechanical set union of extracted item IDs.This practical follow-up uses one corpus and DeepSeek-V3.2 at its default setting, with the same model generating and fusing samples. Its narrower scope tests deployment intuition rather than establishing a universal result.Source: §3.3 · p. 8

4.1 Production Pipeline Impact

• Orchestration-layer optimization produced large operational gains, but the production results are a source-system- and endpoint-dependent case study, not a benchmark.The transferable pattern is to eliminate redundant acquisition, reduce prompt surface area, and cache validated pipeline outputs.Source: §4.1 · pp. 8–9

4.2 Relevance-Contrast Context: Low-Relevance Same-Domain Items Help

• +0.083 on corrected versus +0.092 on original corpus preserved the positive relevance-contrast gain in a separate ten-configuration replicate.The rerun held human labels fixed after removing duplicate/out-of-scope items and affected export metadata; its −0.008 corrected-minus-original difference did not survive sensitivity checks. Separately, matching target items across the original ratio conditions retained a +0.076 effect (95% CI [+0.053, +0.099], p < .001).Source: §4.2 · p. 10
• +0.077 RelAcc improvement favored 50% signal over 100% signal, with positive raw effects across all 11 model configurations.Across 220 configuration-by-block pairs, the 95% CI was [+0.056, +0.098], Cohen’s d = 0.49, and Holm-adjusted p < .001. Accounting for configuration dependence, the nine-family estimate was +0.084 (95% interval [+0.064, +0.103]); block clustering gave [+0.032, +0.122], supporting a finite-corpus descriptive result rather than population inference.Source: §4.2 · pp. 10–11, Figure 2 · p. 11
• 0.661 RelAcc at both 50:50 and 30:70 signal/noise indicates a broad plateau, while fact-type recall keeps rising as signal decreases.The 10:90 mix scored 0.648; the proposed mechanism is threshold-setting through same-domain low-priority examples, not arbitrary irrelevant distractors. These target-item concordance results do not demonstrate improved overall extraction precision or explain complete precision/F1 behavior.Source: §4.2 · pp. 9–12, Figure 3 · p. 12

4.3 Position Effects Are Smaller and Model-Dependent

• +0.022 RelAcc favored end over start placement, but the effect did not survive Holm correction across five planned contrasts (p_adj = .098).Scattering did not beat contiguous placement, and middle placement was indistinguishable from average start/end placement, with low power for small penalties. Ratio mattered more than position in 10-item, roughly 2,000–3,000-token prompts; exploratory 30-item results suggested burial effects at larger sizes.Source: §4.3 · p. 12

4.4 Domain Filtering Is a Large Quality Lever

• Roughly +0.16 RelAcc favored meeting-only over mixed-source prompts in an exploratory, non-confirmatory comparison.Meetings contained 73.3% relevant items versus 5.1% for email, so routing changed the signal distribution before prompting rather than demonstrating a semantic advantage. This supports context stratification as a potential quality-improvement technique as well as a token-saving technique.Source: §4.4 · p. 13

4.5 Reasoning Budget and the Calibration Effect (Exploratory)

• +0.078 to +0.013 for Claude-Opus-4.7 versus +0.048 to +0.098 for GPT-5.5 showed opposite default-to-maximum reasoning trends, treated only as an exploratory hypothesis.High-reasoning gains were +0.050 for Claude and +0.058 for GPT-5.5; neither within-family trend was significant (p = 0.10 and p = 0.15, respectively). The interaction was nominally significant (p ≈ 0.015, n = 20 blocks), but involved only two families, a post-hoc GPT-5.5 sweep, and no correction for multiple reasoning contrasts.Source: §4.5 · p. 13

4.6 Fusion-of-N: The Set Union Is a Strong Baseline

• The set union of repeated extraction samples’ item IDs reached the observed recall ceiling, and learned LLM fusion did not beat it.For recall-oriented extraction, union provides a simple, strong baseline when multiple samples identify item IDs.Source: §4.6 · p. 14
• The fuser tended to drop lower-support true positives caught by only one extraction sample.Mechanical union retains these singly detected items.Source: §4.6 · p. 14
• A recall-leaning fuser prompt improved fusion and changed its comparison against best-of-N, so the findings do not establish that fusion is always bad.The narrower conclusion is that learned fusion did not beat union in this study.Source: §4.6 · p. 14

5. Practitioner Guidance

• Avoid over-filtering extraction prompts: include same-domain low-relevance items to establish a relevance threshold, and test the benefit separately for each source type.The gain concentrated in email and was near zero for meetings and chats. Extract source types separately when their signal/noise profiles differ; these are testable operational defaults, not universal prompting laws.Source: §5 · p. 14
• Optimize context arrangement before upgrading models, treating position as secondary to relevance ratio at short context sizes.Arrangement changes were cheap to test and produced measurable quality changes. Unlike Liu et al.’s “Lost in the Middle” findings, middle-position underuse was not visible at ten items; avoid burying important content in longer contexts until tested.Source: §5 · p. 14, §6 · p. 15
• context_arranger.py provides a post-retrieval arrangement layer, not a validated relevance judge.MIT-licensed and standard-library-only, it supports calibration-ratio batching, type routing, explicit position ordering, per-model position preferences, configurable long-context thresholds, and optional token-budget-aware batching. Replace its unvalidated keyword fallback with retrieval scores or a model call before production use; the study used human 0-3 relevance labels.Source: §5 · pp. 14–15

6. Related Work

• The framework complements existing workflow and memory architectures by measuring cost and context effects and asking how much available context each call needs.Anthropic’s “Building Effective Agents” supplies workflow patterns, while Weng, Park et al., and Shinn et al. emphasize memory and feedback architectures. Unlike provider prefix caching, the paper’s workflow-level semantic caching avoids a model call when request semantics match a previously computed pipeline result.Source: §6 · pp. 15–16
• Same-domain low-relevance context can support calibration, suggesting a compression-versus-calibration tradeoff when pruning on-topic content from relevance-sensitive tasks.Shi et al. show distractors can hurt reasoning, while Cuconasu et al. find random documents sometimes improve RAG answers; this study instead examines workplace relevance scoring and extraction. LLMLingua removes predictable tokens, LongLLMLingua adds question-aware compression and reordering, and LLMLingua-2 uses distilled token classification; locating the calibration tradeoff remains future work.Source: §6 · pp. 15–16
• 2,420 confirmatory trials across 11 model configurations, representing nine model families, show that same-domain low-relevance context can improve extraction calibration.The paper presents this measurement as part of an engineering framework for reducing token and latency costs, rather than as a new architecture or single benchmark.Source: §6 · p. 16
• Learned fusion did not beat mechanical set union in this recall-oriented extraction setup, making union the baseline to beat before deploying a fuser.Union item IDs from repeated extraction samples first, and add learned fusion only if it improves the target metric; this qualifies the deployment promise of sampling and aggregation approaches.Source: §5 · p. 14, §6 · p. 16

7. Limitations, Ethics, and Release Controls

• Author-only fixed labels isolate prompt-composition contrasts, while task-specific relevance scores and estimated historical token counts constrain interpretation.The signal-item-focused relevance metric is not directly comparable to benchmark accuracy or precision/recall. Historical token counts are estimates because full telemetry followed the initial optimization work.Source: §7 · p. 17
• +0.084 is the model-family-level effect (95% interval [+0.064, +0.103]), reported as a finite-corpus description rather than population inference.The 220 paired measurements cluster within nine model families and 20 trial blocks; the trial-block-clustered interval is [+0.032, +0.122]. Data cover one worker’s communication stream from one week and domain, and nested signal subsets still require replication with independently sampled item sets.Source: §7 · pp. 16–17
• About +0.083 was the reproduced relevance-contrast gain on the corrected corpus; the small decrease from the original result is not evidence of real change.Exclusion analyses of existing generations raised the gain from +0.077 across all sources to +0.112 without affected chat targets and +0.156 for email-only targets. The corrected rerun’s small decrease disappears under strict symmetric hold-out, shrinks about three-quarters under renormalization, and also appears among version-pinned models.Source: §7 · pp. 17–18
• Other participants’ content appears only in anonymized aggregates within the author’s communication corpus, and no attempt was made to re-identify individuals.No raw messages, names, or identifiers are released.Source: §7 · p. 18

appendix report only aggregate statistics, named models, and system descriptions.

• The paper is a practitioner systems study with controlled experimental support, not a universal benchmark or a claim covering all people, models, or extraction tasks.Source: appendix report only aggregate… · p. 18

8. Conclusion

• Orchestration reduced latency and eliminated repeated cached-path token use; same-domain contrast improved extraction relevance, while Fusion-of-N supports testing set union before additional model calls.The engineering levers are how context is acquired, routed, compressed, cached, and measured.Source: §8 · p. 18

Technical Appendix: Token Optimization and

• Context-window management is framed as an engineering skill, supported by appendix evidence covering aggregate run tables, telemetry design, named-model results, and robustness analysis.Source: §8 · p. 18, Purpose and Boundary · p. 20

Purpose and Boundary

• The appendix includes only aggregate or operational metadata, withholding message content, names, email addresses, private-content prompts, raw model outputs, and per-item identifiers.The underlying communication data remain unpublished.Source: §7 · p. 18, Purpose and Boundary · p. 20

Appendix A: Historical Pipeline Run Data

• Six of 25 logged runs had measured elapsed times because early runs were not fully instrumented.Latency measurements therefore cover only part of the logged production history.Source: Appendix A · pp. 20–21
• The historical run data support the paper’s production latency claims as live-pipeline operational evidence, not a controlled benchmark.This distinction limits how the latency comparison should be interpreted.Source: Appendix A · p. 20
• 61.4–116.0 seconds was the measured optimized cold-load latency, versus an estimated pre-optimization baseline of roughly 3.5–10.5 minutes.The baseline predates full telemetry, so the reported speed-up is approximate.Source: Appendix A · p. 21

• Run and stage events link runId to options, stage metadata, elapsed_ms, results, total_elapsed_ms, and a final summary.The design targets measurements missing from early runs.Source: Appendix B · p. 21
• LLM-call events record runId, model, input_tokens_est, and output_tokens_est.Source: Appendix B · p. 21
• Cache-hit events record cache_age_ms, while fallback events record from_model, to_model, and reason, all linked by runId.Source: Appendix B · p. 21

Appendix C: Optimization Impact Summary

• Roughly 90% per extraction call and roughly 40% from less data per call are additional reported optimization savings.The supplied fragments do not identify the associated optimization label; the table’s savings figures are operational estimates.Source: Appendix C · p. 22
• 57-86% fewer acquisition calls accompany fetch-once/local processing; token-aware fallback chains avoid full reruns after model failure and repeated acquisition.Fetch-once processing also avoids duplicated raw-data context.Source: Appendix C · p. 22
• Roughly 30% output-token reduction accompanies schema-contracted prompts; inter-agent communication compression provides estimated 40-60% wrapper reduction.Schema-contracted prompts lower parse/retry overhead, while communication compression lowers downstream parsing burden.Source: Appendix C · p. 22
• 100% token savings on semantic-cache hits accompany near-instant repeat loads, which were not benchmarked.These figures belong to the operational pipeline estimates rather than the controlled experiment.Source: Appendix C · p. 22
• +0.077 relevance-accuracy gain from relevance-contrast context is the central measured result; pipeline optimization savings constitute a separate, secondary case study.Compound operational estimates are roughly 60-70% total token reduction and roughly 2-10x faster cold loads versus the 3.5-10.5 min pre-optimization baseline, not controlled-experiment results.Source: Appendix C · p. 22

Dataset

• 55 signal items and 595 noise items remained after excluding 11 marginal-relevance items from the 661-item dataset.Signal means human relevance scores of 2–3; noise means 0. Excluding score-1 items avoids a fuzzy middle that would add label noise to RelAcc.Source: Dataset · p. 23

Relevance Rubric

• The author scored all items in a single first-impression pass using a fixed four-point relevance rubric.The supplied passage does not describe additional raters or repeated scoring.Source: Relevance Rubric · p. 23

Confirmatory Model Set

• Eleven model configurations were used in the confirmatory experiment.The supplied passage introduces the model set but does not identify its configurations.Source: Confirmatory Model Set · p. 24

Robustness, Dependence, and Contamination Sensitivity

• +0.084 is the family-level relevance-contrast gain, with a 95% interval of [+0.064, +0.103] across nine families.The effect is positive in 11 of 11 configurations, nine of nine families, and 16 of 20 blocks; the block-clustered interval is [+0.032, +0.122]. Because the 220 configuration-by-block pairs are dependent, these are finite-corpus descriptive results without a population-level p-value.Source: Robustness, Dependence, and… · pp. 24–25
• +0.070 under strict exact-match accuracy, +0.083 under off-by-one tolerance, and +0.059 under linear partial credit preserve the 50:50-versus-100% gain.Mean absolute error on the 0–3 relevance scale falls by 0.18 points, and every family-clustered interval excludes zero. Target scores match human scores exactly 39% of the time and within one point 83%; across targets and noise pooled, Spearman correlation is +0.48.Source: Robustness, Dependence, and… · p. 25
• Roughly tenfold variation in positive calibration gains accompanies opposite reasoning-sweep directions for Claude Opus 4.7 and GPT-5.5.The eleven prespecified configurations are distinguished from two post-hoc GPT-5.5 reasoning runs. The signed difference is only nominally significant and rests on two families, making it exploratory rather than an established finding.Source: Robustness, Dependence, and… · p. 25
• +0.112 after excluding chat targets and +0.156 for email-only targets exceed the +0.077 all-source gain, with both positive across all configurations and families.The gain concentrates in email, whose relevance base rate is 5.1%, and is near-zero for meetings and chats. Frozen data-quality artifacts are confined to chat, so this exclusion analysis of original generations places them where the effect is weakest rather than identifying them as its driver.Source: Robustness, Dependence, and… · p. 26
• Corrected-corpus reruns preserve positive 50:50-versus-100%-signal gains under both primary and matched-item scoring, despite slightly smaller gains after cleaning.The independent ten-model replicate used 4,400 calls, held human labels fixed, and reduced the corpus from 661 to 659 items (55 to 53 signal). The two scoring comparisons share the 50:50 condition and are correlated, not independent confirmations.Source: Robustness, Dependence, and… · pp. 26–27
• −0.00825 shrinks to −0.00202 under renormalization, showing that the corrected-versus-original gain difference is sensitive to parse handling.Symmetrically excluding positions with parse failures—five original and eight corrected—makes both gains and their difference not estimable under non-renormalizing scoring. Version-pinned and version-as-served differences are −0.00250 and −0.01400 without uncertainty bands; these diagnostics do not resolve the sign, and the unprespecified direction is descriptive.Source: Robustness, Dependence, and… · p. 27
• +0.125 is DeepSeek-V3.2’s exploratory position contrast (p=.0004), while five configurations are within ±0.03 of zero.The near-zero configurations are Claude Opus 4.7 default/high/xhigh, Gemini-3.1-Pro-Preview, and GPT-5.5. These single-model estimates are unadjusted for comparisons across eleven models and describe variation behind the pooled contrast, not a second confirmatory result.Source: Robustness, Dependence, and… · p. 28

Practical Noise-Ratio Guidance

• +0.077 for 50:50 versus 100% signal lies on a broad relevance-accuracy plateau beginning around 50:50, while recall continues increasing with noise.The gradual improvement has a slope of +0.000871 RelAcc per percentage point of noise on the nominal grid, or +0.000876 on the true item-count grid. Beyond roughly halfway, additional same-domain noise costs little relevance accuracy while continuing to help recall.Source: Practical Noise-Ratio Guidance · p. 28

Appendix E: Fusion-of-N Follow-Up

• Five independent extraction samples per window were generated by DeepSeek-V3.2, which also served as judge/fuser, with noise and window length varying together.The task was recall-oriented extraction over ecological dilution levels.Source: Appendix E · p. 29
• Using the same model to generate and fuse samples could favor its own fusion through self-preference, making union’s advantage conservative rather than inflated.The appendix argues that any such bias would favor learned fusion, not the mechanical union.Source: Appendix E · p. 29
• Learned fusion did not beat a free set union of extracted item IDs; union reached the recall ceiling, while fusion sometimes dropped true positives.Dropped true positives were sometimes found by only one sample. This measures item-identification recall—the fraction of target item IDs recovered—not the main study’s fact-type recall.Source: Appendix E · p. 29
• For recall-oriented extraction in this setup, use the set union of repeated samples before paying for learned fusion; do not generalize to all aggregation tasks.Source: Appendix E · p. 29