Source-linked AI summary
Squeez: Task-Conditioned Tool-Output Pruning for Coding Agents
Ádám Kovács
TL;DR
Coding agents repeatedly reread long tool observations even though only a small fraction matters for the next decision. Squeez trains a task-conditioned extractor on grounded spans from real and synthetic tool outputs, and its compact model achieves strong evidence preservation under aggressive compression. The benchmark evaluates this behavior directly while leaving downstream task completion outside its scope.
Problem
Coding agents need to identify the small amount of next-step evidence in long tool observations, but commonly reread the full mixed-format output.
Method
Squeez fine-tunes a compact Qwen 3.5 2B model with LoRA to extract contiguous, verbatim evidence spans from a queried tool observation.
Results
0.86 recall and 92% compression are achieved by Squeez-2B, which exceeds Qwen 3.5 35B A3B by 11 recall points and outperforms heuristic baselines.
Takeaways & Limitations
Narrow task-specific supervision is effective for pruning mixed-format tool output, where zero-shot generative models and retrieval heuristics perform less well.
Takeaways & Limitations
The benchmark measures evidence preservation on single observations rather than downstream end-to-end task completion, and span overlap cannot capture every valid pruning decision.
Abstract
from arXiv · showhide
Coding agents repeatedly consume long tool observations even though only a small fraction of each observation matters for the next step. We study task-conditioned tool-output pruning: given a focused query and one tool output, return the smallest verbatim evidence block the agent should inspect next. We introduce a benchmark of 11,477 examples built from SWE-bench repository interactions and synthetic multi-ecosystem tool outputs, with a manually curated 618-example test set. We fine-tune Qwen 3.5 2B with LoRA and compare it against larger zero-shot models and heuristic pruning baselines. Our model reaches 0.86 recall and 0.80 F1 while removing 92% of input tokens, outperforming zero-shot Qwen 3.5 35B A3B by 11 recall points and all heuristic baselines by a wide margin.
1 Introduction
Coding agents must repeatedly process long, mixed-format tool observations even though only a small fraction is relevant to the next decision. Squeez formulates this as task-conditioned extraction of the smallest verbatim evidence block from one observation.
- Problem: Squeez targets query-conditioned pruning of a single mixed-format tool observation for the next agent decision.The task preserves relevant evidence while discarding the rest rather than solving the issue from one observation.
- Problem: The setting differs from generic prompt compression and document-level retrieval compression because it operates on mixed-format tool output.These observations combine sources such as file reads, logs, stack traces, API responses, and version-control history.
- Contributions: The paper contributes a benchmark, a dataset spanning real SWE-bench interactions and synthetic multi-ecosystem outputs, and a compact fine-tuned model that outperforms larger zero-shot and heuristic baselines.The model and tooling are designed for insertion into existing coding-agent stacks with minimal surrounding changes.
2 Related Work
Prior pruning methods mainly compress prompts or retrieved prose documents, whereas Squeez focuses on query-conditioned, verbatim extraction from single mixed-format tool observations. Its closest agent-specific comparison still targets repository code context rather than heterogeneous tool output.
- Prompt and document compression: Prompt compression methods reduce long prompts at the token or prompt-block level, while abstractive summarization rewrites salient content instead of preserving verbatim evidence.These approaches address efficiency or rewriting, not query-conditioned extraction from one raw tool observation.
- Document-level pruning: Document-level pruners select relevant portions of retrieved textual context for downstream question answering or assign relevance scores to documents.Their inputs assume retrieved passages or document text rather than interleaved code, logs, shell traces, metadata, and structured outputs.
- Agent-specific pruning: Agent-specific systems prune web observations or repository code context, but SWE-Pruner does not cover single observations spanning files, logs, build outputs, and other modalities.This makes SWE-Pruner the nearest comparison while preserving a distinct task scope.
- Evidence-grounded extraction: Extractive QA and supporting-evidence supervision share Squeez’s emphasis on traceable source spans, but Squeez applies that principle to tool-observation pruning rather than prose answer production.The common concern is faithfulness to source evidence.
3 Task Definition
The task maps a short extraction query and one raw tool observation to contiguous evidence spans in the original output. It deliberately narrows the agent’s responsibility to selecting minimal next-step evidence rather than diagnosing or solving the underlying bug.
- Input and output: The benchmark input is a pair (q, o), consisting of a short task-conditioned extraction query and one raw tool observation.The query is tool-aware and expresses a localized information need.
- Input and output: The output is one or more contiguous spans over the observation, with each span referring to line indices in the original output.Evaluation compares predicted lines with gold spans using recall, F1, and compression.
- Task scope: The model extracts the minimal evidence block that helps the agent’s next reasoning step rather than inferring a correct patch or completing bug solving.Queries may be easy, medium, or moderately semantic, provided they define a plausible pruning decision.
4 Dataset
The benchmark combines SWE-bench-derived repository observations with synthetic multi-ecosystem outputs, labels them through focused query and span selection, and releases a heterogeneous manually reviewed dataset. Its targets remain verbatim subsets of raw tool output while training uses XML-linearized spans.
- Sources: The benchmark draws on SWE-bench repository snapshots and synthetic outputs covering multiple programming, build, and deployment ecosystems.The SWE-derived source executes 14 tool types, while the synthetic source extends coverage beyond SWE-bench’s Python-heavy distribution.
- Dataset composition: 11,477 examples cover 27 tool types, comprising 9,205 SWE-derived examples, 1,697 synthetic positives, and 575 synthetic negatives.Synthetic negatives pair mismatched queries and tool outputs so the correct pruning decision is empty output.
- Labeling: A two-stage teacher pipeline first writes a focused tool-aware query and then selects the smallest contiguous span or span set answering it.Positive examples unsupported by the observation are discarded.
- Label representation: Every released target is mapped back to the original raw text as a verbatim subset, while generative training linearizes gold spans with XML wrappers.The released benchmark stores span coordinates over the original observation.
- Evaluation split: The split contains 10,508 training examples, 240 development examples, and 618 manually reviewed test examples.The final test set excludes near-duplicates, trivial outputs, overly broad spans, and incorrect annotations.
- Heterogeneity: The data intentionally mixes short exceptions and test summaries with long file reads, type-check outputs, and container logs.Relevant evidence can occur at the beginning, middle, or end, and the useful pruning unit varies by tool type.
5 Model and Evaluation
The paper evaluates a LoRA-tuned Qwen 3.5 2B extractor against zero-shot models and heuristics for selecting verbatim evidence from tool outputs. Evaluation uses line-level overlap metrics on a held-out test set, with deployment designed as lightweight preprocessing for coding agents.
- Model: Qwen 3.5 2B is fine-tuned with LoRA to emit verbatim extracted text from a focused query and raw tool output.The model is served through vLLM after merging the LoRA adapter.
- Baselines: The comparison includes Qwen 3.5 35B A3B, Kimi K2, unfine-tuned Qwen 3.5 2B, BM25, First-N, Last-N, and Random.Heuristic baselines retain approximately 10% of input lines.
- Metrics: Recall, exact match, compression, tolerant line-matching F1, and strict exact-text overlap F1 are reported using lines as the matching unit.The main analysis emphasizes recall under strong compression, with F1 as the summary metric.
- Evaluation: All models are evaluated on the held-out 618-example test set using the same query/tool-output prompt format for generative models.The fine-tuned model uses vLLM, while larger zero-shot models use OpenAI-compatible APIs.
- Deployment: The model is intended as a lightweight preprocessing step that can consume piped tool output without changing an agent’s core planning loop.The release supports both a CLI and vLLM serving.
6 Results
Squeez-2B achieves the strongest held-out performance while retaining highly compressed evidence. Its advantages appear across precise extraction, compact failure blocks, and explicit negative cases, whereas competing systems fail through missed or adjacent selections.
- Overall results: 11 recall points separate Squeez-2B from the 18× larger Qwen 3.5 35B A3B, while the unfine-tuned 2B base trails by 33 points.The fine-tuned model is also reported as the most precise system.
- Baseline analysis: BM25 reaches only 0.22 recall, and heuristic baselines perform substantially worse than generative models.Relevant lines can occur anywhere in an observation, and usefulness depends on the query rather than lexical overlap alone.
- Qualitative analysis: On 59 held-out negative examples, Squeez-2B returns empty output 80% of the time compared with 7% for Qwen 35B.Its remaining errors are usually semantically adjacent selections or correct blocks with nearby extra context.
- Qualitative analysis: Figure 3 illustrates a two-line gold failure block embedded in a 250-line kubectl observation.The example concerns the OOMKilled reason and exit code for a container.
7 Limitations
The benchmark directly measures evidence preservation for individual tool observations, but not end-to-end task completion. Its usefulness metric and tool-family noise also limit what the evaluation establishes.
- Single-observation evaluation measures evidence preservation, not downstream effects on end-to-end task completion.
- Span overlap approximates usefulness but cannot capture every valid alternative pruning decision.
- Grep and lint outputs remain noisier than some other tool families.
8 Conclusion
Squeez is a compact task-conditioned pruner trained for mixed-format coding-agent observations. It reaches strong recall while removing most input tokens, and its model and evaluation artifacts are released for integration and reuse.
- 0.86 recall is achieved while removing 92% of input tokens.
- Squeez outperforms the 18× larger Qwen 3.5 35B A3B and all heuristic baselines by a wide margin.
- Mixed-format tool output responds well to narrow, task-specific supervision rather than zero-shot generative models or retrieval heuristics alone.
- The model and evaluation code are available on GitHub, while the released model and dataset are available on Hugging Face.