An Information Theoretic Perspective on Agentic System Design

1 Introduction

• The paper proposes a task-agnostic mutual-information measure of compressor efficacy and a rate-distortion analysis of downstream performance.It presents a simple, unbiased estimator computable via inference servers without full vocabulary log probabilities.Source: §1 · p. 2
• On LongHealth, scaling a Qwen-2.5 compressor from 1B to 7B improved accuracy by 60%, while scaling the predictor from 70B to 405B improved it by 12%.The authors conclude that compressor quality overwhelmingly governs performance in this comparison.Source: §1 · p. 2
• Larger compressors produce outputs up to 4.6× more concise; scaling Qwen-2.5 from 1.5B to 7B adds 1.3% more FLOPs-per-generation.The reported token efficiency yields sublinear scaling of FLOPs-per-generation with model size.Source: §1 · p. 2
• Compressor outputs carry up to 5.4× more mutual information, and information rate correlates with downstream performance and perplexity (r = −0.84, R2 = 0.71).The authors propose information rate as a practical proxy for predicting system performance without full end-to-end evaluation.Source: §1 · p. 2
• The reported design-factor hierarchy is compressor model family > compressor size > predictor size.The hierarchy comes from a meta-analysis across model families, sizes, and datasets.Source: §1 · p. 2
• A simplified Deep Research pipeline with local compressors as small as 3B reached 99% of frontier-LM accuracy on DeepResearch Bench [14] while reducing API costs by 74%.A single predictor aggregates outputs from multiple compressors.Source: §1 · p. 3

2.1 Related Work

• The paper defines AI agents as language models embedded in contexts that reason, plan, and act through tool use.Related work describes dividing complex tasks and contexts among agents with different roles.Source: §2.1 · p. 3
• Prior multi-agent work studies model size, agent counts, communication rounds, research depth, and planning, while information-theoretic analysis of multi-agent systems remains limited.Related work also applies information theory to neural representations, information flow, language generations, and training objectives.Source: §2.1 · p. 3

2.2 Information Theoretic Problem Setup

• The setup models the compressor as a noisy channel from context to summary and the predictor as the stage that uses the summary to generate the target.This extends information-theoretic analysis to communication between models.Source: §2.2 · p. 3
• Mutual information measures how much the compression retains about the context; the estimator uses language-model log probabilities and samples from the encoder and data distributions.The authors seek an estimator usable with inference-engine outputs rather than requiring an auxiliary model or the underlying distribution.Source: §2.2 · p. 4
• The proposed estimator is practical rather than theoretically optimal, is bounded by log(N), and does not require full vocabulary probabilities.The paper notes that proxy models can be used for log probabilities; its experiments condition the estimate on the query and clip small negative estimates to zero.Source: §2.2 · p. 4
• Rate is mutual information per output token, while distortion is defined as 1 − ACC(Z).Rate-distortion analysis describes the trade-off between information carried by the compression and prediction error.Source: §2.2 · p. 4

Datasets

• The evaluation spans five datasets, using accuracy for LongHealth, FinanceBench, and QASPER, and perplexity for WildChat and FineWeb.Experiments use five random seeds; a GPT-4o-mini judge assesses QA correctness, while a Llama-3.1-8B model supplies log probabilities for perplexity.Source: Datasets · p. 5

3.1 Where Do You Need the FLOPs: In Compressors or Predictors?

• 7–8B compressors are up to 3.1× more accurate than 1–1.5B models on LongHealth and up to 2.6× more accurate on FinanceBench.On LongHealth, they surpass the GPT-4o-only baseline by 4pp; on FinanceBench, they recover 97% of its accuracy.Source: §3.1 · p. 5, Figure 2 · p. 6
• 1.3% more FLOPs-per-generation: Qwen-2.5 compressors scale from 1.5B to 7B on LongHealth with this increase, as larger models produce fewer tokens while maintaining accuracy.Across model families, compression FLOPs-per-generation grow sublinearly with compressor size.Source: §3.1 · pp. 5–6

3.2 Which Compressors Maximize Communication Efficiency?

• Larger compressors carry more mutual information and are more bit-efficient, but MI can saturate at the largest sizes or already at 3B on FinanceBench.The largest Qwen-2.5 and Gemma-3 models saturate in mutual information on LongHealth; saturation is observed primarily on datasets with heterogeneous context documents.Source: §3.2 · pp. 7–8, Figure 4 · p. 8

3.3 Information Rate Correlates Strongly with Downstream Performance

• Mutual-information rate closely relates to distortion, while mutual information correlates with FineWeb perplexity (r = −0.84, R2 = 0.71).The rate-distortion analysis also finds only marginal distortion improvements from scaling predictors beyond 70B.Source: §3.3 · p. 8, Figure 6 · p. 9

3.4 Which Knobs to Turn?

• Compressor family and size outweigh predictor size as factors in QA accuracy, with Qwen-2.5 compressors outperforming Llama models.Predictors do not perform better when paired with compressors from the same model family.Source: §3.3 · p. 8, §3.4 · p. 10

3.5 Scaling Deep Research

• 2.3% higher RACE scores at 28.1% of the API cost: Qwen-2.5-14B with GPT-4o outperforms the uncompressed GPT-4o baseline on Deep Research.The study also reports that larger compressors improve RACE scores with minimal additional API costs.Source: Figure 7 · p. 10, §3.5 · p. 10

4 Discussion

• The analysis focuses mainly on GPT-style non-reasoning models and single-round communication; its 1–3B MI estimates use proxy models and log probabilities, which may introduce variance and bias.The authors call for broader model-family coverage and more principled evaluation of reasoning traces.Source: §4 · p. 11
• Larger compressors can scale at sublinear computational cost because they are more information-efficient, and running them locally can reduce cloud predictor-serving costs.The discussion also recommends front-loading compute into compressors because scaling them is more effective than scaling predictors.Source: §4 · p. 11
• Mutual information between input context and agent output is a task-agnostic indicator of compression quality, tightly linked to downstream performance and perplexity.Source: §4 · p. 11
• Qwen-2.5 compressors scale more compute-efficiently than Llama and Gemma-3 compressors, while Qwen-2.5 predictors yield higher accuracy than Llama predictors.Source: §4 · p. 11

A Extended Related Work

• Prior Deep Research systems decompose queries into parallel subtasks, then synthesize compressor results into reports, while existing benchmarks evaluate report quality and downstream utility.Source: A Extended Related Work · p. 17

B Extended Description of Methods

• The extended methods section provides a more detailed explanation and derivation of the paper’s information-theoretic approach and problem setup.Source: B Extended Description of Methods · p. 17

B.1 Compute Cost of Dense LMs

• The compute-cost measure counts FLOPs per token in each forward pass through the dense transformer-based language models.Source: §B.1 · p. 17
• FLOPs per token generated for dense models grow roughly linearly with model size.Source: §B.1 · p. 17
• The cost description identifies model size, input-context token count, layer count, and attention heads per layer as its variables.Source: §B.1 · p. 17

B.2 Theoretical Analysis: Bounds of Monte Carlo Estimator

• The Monte Carlo mutual-information estimator has a tight log N upper bound, where N is sampled contexts, when the matched summary likelihood dominates alternatives.Source: §B.2 · pp. 17–18

B.3 Rate-Distortion-Theory

• The Gaussian rate-distortion curve is used as a qualitative model, not an LM distribution; its exponential decay is fit with free C and b plus a distortion floor D0.The model accounts for unknown variance and modeling noise; the parameters are fit by least squares.Source: §B.3 · p. 18

C.1 Compressor Model Prompts

• Compressor prompts vary by task: query-specific summaries retain answer-relevant information without answering, while WildChat memory and FineWeb query-agnostic prompts preserve useful details without inventing facts.Source: §C.1 · p. 19

C.2 Predictor Model Prompts

• Predictor prompts answer from summaries or chat memory in JSON, while FineWeb uses distinct extractive and creative templates.Source: §C.2 · pp. 19–20

C.3 DeepResearch Prompts

• Deep Research prompts sequence strategic research planning, exactly eight targeted queries with subtasks, source extraction, and comprehensive synthesis integrating all findings.Source: §C.3 · pp. 21, 23

D.1 LongHealth, FinanceBench, and QASPER

• LongHealth uses 20 patients with 20 questions each; FinanceBench samples 20 evidence-centered contexts; QASPER samples 20 papers and queries, with three random seeds.LongHealth prediction removes multiple-choice options; FinanceBench contexts are 21,500 tokens, while QASPER evidence may span multiple paper sections.Source: §D.1.1 · p. 24, §D.1.2 · p. 25, §D.1.3 · p. 25

D.1 WildChat and FineWeb

• WildChat groups 1,000 chat conversations into 100 synthetic users, while FineWeb uses 100 samples of 15,000–28,000 tokens to generate two extractive and three creative QAs per sample.Source: §D.1.4 · p. 25, §D.1.5 · p. 26

D.2 Compressor Model Details; D.3 Predictor Model Details

• Compressor experiments use instruction-tuned Llama-3, Qwen-2.5, and Gemma-3 models, plus Qwen-3 reasoning and mixture-of-experts models; outputs are at most 4096 tokens, generated at temperature 0.7 except Gemma-3 at 1.0.Source: §D.2 · p. 27

D.4 Generalized Linear Model Analysis Setup; D.5 Mutual Information Proxy Model Details

• The generalized linear model setup predicts binary correctness using normalized input, prediction, and compression lengths, model sizes, and a Qwen-family indicator.Mutual information is estimated with Qwen-2.5-7B as proxy for Llama compressors, Llama-3.1-8B for Qwen-2.5 and Gemma-3, and internal log probabilities for Qwen-3.Source: §D.4 · p. 27, §D.5 · p. 27

D.6 Deep Research Setup Details

• Deep Research evaluation samples 20 English tasks from DeepResearch Bench, runs each configuration five times, and reports RACE scores across four dimensions.The dimensions are Comprehensiveness, Depth, Instruction-following, and Readability; Figure 7 costs use GPT-4o API rates from August 2025 and exclude a constant $0.12 SerpAPI cost per task.Source: §D.6 · p. 27, §D.6.1 · p. 28

D.6.1 Full Deep Research Workflow Setup

• In the Deep Research workflow, predictors split tasks into query-subtask pairs, compressors search and summarize retrieved evidence in parallel, and predictors aggregate summaries into reports.The system is evaluated using the DeepResearch Bench framework and RACE scores.Source: §D.6.1 · p. 28, Figure 8 · p. 28

D.6.2 Deep Research Compressor Model Details

• Deep Research compressors use Qwen-2.5-Instruct models from 0.5B to 14B, hosted on Modal Labs with SGLang, temperature 0.7, and a 2,000-token response limit.Each compressor independently processes assigned search results before the predictor aggregates their summaries.Source: §D.6.2 · p. 28

D.6.3 Deep Research Predictor Model Details

• Deep Research predictors range from Llama-3.1-8B to Llama-3.1-405B, alongside GPT-4o; Llama output is limited to 4,000 tokens versus GPT-4o’s 16,000.The stated output limit may constrain Llama predictors’ report comprehensiveness during synthesis.Source: §D.6.3 · pp. 28–29

D.7 Compression Failure Modes

• The section categorizes compressor errors into three recurring failure modes.These are incorrect answers, no answer, and omission or over-compression.Source: §D.7 · pp. 29–30
• Incorrect answers can arise from confusion between report time points; larger compressors reduce this error through better modeling of temporal dependencies.The example assigns a 1995 diagnosis date to coronary artery disease first diagnosed in 1989.Source: §D.7 · p. 29
• A no-answer error occurs when the compression contains no query-relevant information, as when a cardiac intervention question receives no answer.The intervention was recorded among historical diagnoses rather than the main findings.Source: §D.7 · p. 29
• Over-compression can retain the general finding while omitting details needed for a complete answer, as in the summary that collapsed metastases across several organs into “multiple unspecified locations.”The example’s compression omitted lung, liver, heart, muscle, and bone sites despite their presence in the context.Source: §D.7 · p. 30

E Extended Results

• The following appendix section presents extended results and ablations of key design choices in the compression-prediction setup.Source: E Extended Results · p. 30

• The analysis adds three datasets, two synthetic and one non-synthetic, to examine compressor scaling.Source: §E.1 · p. 30
• It evaluates compression accuracy and perplexity across compressor model sizes.Perplexity is measured using a Llama-3.1-8B model’s log probabilities on the target answer.Source: §E.1 · p. 30

E.1.1 Summarizing Scientific Papers on QASPER

• 8B compressors outperform the GPT-4o-only baseline in accuracy on QASPER across three model families.Source: §E.1.1 · p. 30, Figure 9 · p. 31
• Up to 1.5× more mutual information is retained by 8B compressors on QASPER.The scaling trend holds for non-reasoning Llama and Qwen-2.5 compressors and reasoning Qwen-3 compressors.Source: §E.1.1 · pp. 30–31, Figure 9 · p. 31

E.1.2 Constructing Chat Memory on Wildchat

• Chat memories are formed by summarizing each user chat interaction, then answering synthetic user queries from that memory.Source: §E.1.2 · p. 31
• Up to 1.14× lower log probabilities are reported for query-agnostic chat memories from the largest compressor of each model family versus 1B models.Source: §E.1.2 · p. 31
• Qwen-2.5 and Gemma-3 compressor scaling is more compute-efficient than Llama scaling on WildChat.Compression length scaling makes FLOPs-per-generation sublinear for Qwen-2.5 and Gemma-3, whereas larger Llama compressors generate longer compressions and steeper compute scaling.Source: §E.1.2 · p. 31

E.1.3 Varying Task Type on FineWeb

• Query-agnostic summaries tend to achieve lower perplexity than query-specific summaries on creative tasks, while extractive tasks have lower perplexity overall.Extractive answers are explicit in context; creative generation is more challenging and benefits from broader stylistic and semantic cues.Source: §E.1.3 · p. 32
• Qwen-2.5 and Gemma-3 compressors scale compute cost more cheaply than Llama compressors across task types and summary types.The compute-scaling trends are consistent across extractive and creative tasks and query-specific and query-agnostic summaries.Source: §E.1.3 · p. 32, Figure 11 · p. 33

E.1.4 Effect of Proxy Model on Mutual Information Estimation

• Proxy-model choice adds a fixed vertical offset to mutual-information estimates on LongHealth but does not change scaling rates or prior conclusions.Source: §E.1.4 · p. 32

E.1.3 Varying Task Type on FineWeb

• Larger compressors consistently reduce perplexity on extractive and creative tasks for both query-specific and query-agnostic summaries, approaching full-context baseline performance.Source: §E.1.3 · p. 32, Figure 11 · p. 33

E.1.5 Extended Results on Reasoning and Mixture-of-Experts Compressors

• Across Qwen-2.5, Qwen-3, and Qwen-3-30B-A3B, compressor scaling trends are similar; at 3B, MoE exceeds dense models in accuracy, conciseness, and mutual information.The same-scale comparison is between the mixture-of-experts model and dense models.Source: §E.1.5 · p. 33

E.1.6 Scaling of Mutual Information and Bit Efficiency on FinanceBench

• On FinanceBench, larger compressors retain more information about the original document while compressing more efficiently.The appendix reports that this scaling behavior remains consistent beyond LongHealth.Source: §E.1.6 · p. 33

E.1.7 What Are the Effects of Conciseness Instructions?

• Across 3-, 6-, and 9-sentence prompts, accuracy and mutual information are unaffected; output size and compute cost shift by an absolute offset while scaling trends hold.The authors interpret the persistent scaling trends as driven by compressor capacity.Source: §E.1.7 · p. 34

E.1.8 Multi-Turn Interactions

• In the Llama-3.1-405B/Llama-3.2-3B workflow, mutual information rises with additional queries through two turns, but further turns add no improvement.The predictor integrates accumulated compression with each new response and issues targeted follow-up queries.Source: §E.1.8 · p. 35

E.2.1 Scaling Predictor Model Size

• Downstream QA accuracy rises from 1B to 8–70B predictors, then saturates at 405B across LongHealth and FinanceBench.The comparison fixes the compressor as either Qwen-2.5 or Llama and plots accuracy against bit efficiency.Source: §E.2.1 · p. 35
• Larger compressor model sizes yield outputs with higher mutual information and bit efficiency.Bit efficiency is the bits of mutual information per token.Source: §E.2.1 · p. 36

E.2.2 Alternative Measurements of Distortion

• With semantic-embedding distortion measured as 1−cosine similarity, rate-distortion curves retain the same predictor-size ordering.The embeddings are produced with OpenAI text-embedding-3-small.Source: §E.2.2 · p. 37

• Even with k=6 (48 sources), bypassing compression achieved similar RACE scores but significantly higher API costs than the compression-based strategy.The uncompressed baseline gives GPT-4o the top search results from each of eight predictor queries.Source: §E.3 · p. 37
• Total FLOPs per generation rise as predictor size increases, while larger compressors generally provide more tokens of information in their compressions.Compressor output tokens remain relatively stable across predictor sizes and increase moderately with larger compressors.Source: §E.3 · p. 37, Figure 24 · p. 43