PaperlayerView exact revision on arXiv

Source-linked AI summary

Concise and Precise Context Compression for Tool-Using Language Models

Yang Xu, Yunlong Feng, Honglin Mu, Yutai Hou, Yitong Li, Xinghao Wang, Wanjun Zhong, Zhongyang Li, Dandan Tu, Qingfu Zhu, Min Zhang, Wanxiang Che

arXiv:2407.02043v1cs.CL

TL;DR

Tool documentation can occupy substantial context, while existing compression approaches risk losing tool and parameter names or cannot readily adapt summary length to documentation length. The paper combines selective preservation of key text with block-based compression, reaching performance comparable to an uncompressed baseline at up to 16x compression on two benchmarks.

  • Problem

    Tool documentation can occupy substantial context, while existing compression methods risk losing key information and do not readily support preset compression ratios across documentation lengths.

    Source: §1 · pp. 1–2, §5 · p. 9

  • Method

    The approach preserves tool and parameter names as raw text and divides the remaining documentation into blocks for fixed-length compression at a preset ratio.

    Source: §1 · p. 2

  • Results

    At up to 16x compression, the method reaches performance comparable to the uncompressed upper-bound baseline on API-Bank and APIBench.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The approach provides a training recipe for compressing tool-use context while controlling compression ratio and mitigating key-information loss.

    Source: §1 · p. 2, §5 · p. 9

    The strategies rely on human priors: key information and preset compression ratios must be tuned to the task.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Through reading the documentation in the context, tool-using language models can dynamically extend their capability using external tools. The cost is that we have to input lengthy documentation every time the model needs to use the tool, occupying the input window as well as slowing down the decoding process. Given the progress in general-purpose compression, soft context compression is a suitable approach to alleviate the problem. However, when compressing tool documentation, existing methods suffer from the weaknesses of key information loss (specifically, tool/parameter name errors) and difficulty in adjusting the length of compressed sequences based on documentation lengths. To address these problems, we propose two strategies for compressing tool documentation into concise and precise summary sequences for tool-using language models. 1) Selective compression strategy mitigates key information loss by deliberately retaining key information as raw text tokens. 2) Block compression strategy involves dividing tool documentation into short chunks and then employing a fixed-length compression model to achieve variable-length compression. This strategy facilitates the flexible adjustment of the compression ratio. Results on API-Bank and APIBench show that our approach reaches a performance comparable to the upper-bound baseline under up to 16x compression ratio.

1 Introduction

Tool documentation can consume substantial context, yet its names and formats must remain exact; the paper proposes selective and block compression to address key-information loss and variable-length needs. On API-Bank and APIBench, the approach reaches performance comparable to the uncompressed upper-bound baseline at up to 16x compression.

1 Introduction

  • Tool documentation can require hundreds of tokens per tool and exceed a thousand tokens when multiple tools are combined or retrieved.The documentation occupies the input window whenever a tool is used.Source: §1 · pp. 1–2
  • Tool names and parameter names and formats must remain unchanged because errors can cause generated function calls to fail.Other verbose descriptions can be summarized because the model needs to understand tool functionality, not its exact wording.Source: §1 · p. 2
  • Selective compression retains tool and parameter names as original text tokens in interleaved compressed and uncompressed subsequences.This mitigates key-information loss and makes shorter summary sequences less challenging.Source: §1 · p. 2
  • Block compression splits documentation according to a preset compression ratio and compresses each block to a fixed length, enabling variable-length summaries.This addresses the difficulty of setting one summary length for documentation of widely varying lengths.Source: §1 · p. 2
  • At up to 16x compression, the approach reaches performance comparable to the uncompressed upper-bound baseline on API-Bank and APIBench.Selective compression mitigates key-information loss, while block compression adds no compression loss compared with overall compression.Source: §1 · p. 2
  • The paper explores combinations of training objectives and compression strategies and provides a training recipe for tool-using language models.The proposed approach starts from a pre-trained model and uses continual pre-training followed by fine-tuning.Source: §1 · p. 2

2 Related Work

Tool-using language models may either rely on predefined built-in tools or read documentation to generate structured calls for arbitrary tools. The paper situates its approach within soft context compression and contrasts it with hard token pruning.

2 Related Work

  • Documentation-based tool use lets language models generate structured function calls from tool descriptions and instructions, supporting more dynamic tool use.The alternative is to support a limited set of predefined tools as built-in features.Source: §2 · pp. 2–3
  • Soft summary methods compress context with condensed representations but often lack adjustable compression ratios.They use trained language models as compressors, while hard token pruning removes text judged non-essential and can be applied to black-box models.Source: §2 · p. 3
  • The paper adapts soft summary methods to tool use by prioritizing controllable compression ratios and using tool-specific strategies.It presents a customized solution for tool-using language models, differing from Chevalier et al. (2023) and Ge et al. (2023).Source: §2 · p. 3

3 Method

The method builds on soft-token compression, then preserves tool and parameter names as raw text while compressing descriptive content. It uses fixed-size blocks to vary the overall compression ratio and integrates both strategies in one pipeline.

3.1 Basic Soft Context Compression

  • Basic soft context compression maps an input sequence to fixed-length soft tokens, which a decoder uses as a prompt to generate output.Training applies language-modeling loss to the decoder output to supervise the compressor and decoder indirectly.Source: §3.1 · pp. 3–4

3.2 Selective Compression Strategy

  • Compression loss grows with compression ratio, and errors in tool or parameter names can directly cause tool-use failure.Selective compression splits documentation into key-information and compressible subsequences, retaining key information as raw text tokens.Source: §3.2 · p. 4

3.3 Block Compression Strategy

  • Fixed-length summaries trade information retention on long documentation against compression efficiency on short documentation; selective compression can intensify the problem.Block compression chunks documentation according to a preset ratio and compresses each block to a fixed number of soft tokens.Source: §3.3 · p. 4

3.4 Concise and Precise Context Compression

  • The final method integrates selective and block compression, treating key-information subsequences as uncompressed blocks and chunking the remaining subsequences.Blocks are compressed in parallel, concatenated into compressed documentation, and used by the decoder; compressor and decoder are jointly trained with language-modeling loss.Source: §3.4 · pp. 4–5

4 Experiments

Experiments evaluate context-compression strategies on API-Bank and APIBench, testing compression ratios, training objectives, and the effects of selective and block compression. Selective compression maintains accuracy while reducing name errors, and block compression improves basic soft compression.

4.1 Basic Settings

  • Experiments compare no compression, overall compression, and selective compression, with block compression enabling controllable compression ratios in the compressed cases.The base model and compressor/decoder are initialized from LLaMA-7b; overall compression is basic soft compression integrated with block compression.Source: §4.1 · p. 5

Key Information for Selective Compression

  • Key information for selective compression is defined as tool names and parameter names, which the method preserves as original text tokens.The approach interleaves compressed and uncompressed subsequences.Source: Key Information for Selective… · p. 5

4.2 Pre-training Compression Models

  • Pre-training uses SlimPajama: a 1B-token subset sampled within 2k context length, with all compressed models trained on it for one epoch.The compression models are pre-trained before downstream fine-tuning; the no-compression upper-bound baseline omits pre-training.Source: §4.2 · p. 6

4.3 Tool-Using Benchmarks

  • API-Bank tests tool calls in multi-turn dialogues, while APIBench uses candidate model cards and API calls, with retrieval uncertainty creating possible cascading errors.The experiments use API-Bank's level1-api subset; APIBench candidates are expanded to up to five with the correct answer included.Source: §4.3 · pp. 6–7

4.4 Tool-Using Evaluation

  • Selective compression reaches similar or higher performance than no compression, while overall compression degrades as compression ratios rise.Table 2 reports test accuracy on API-Bank and APIBench.Source: Table 2 · p. 7, §4.4 · p. 7
  • Selective compression makes fewer name errors; on APIBench, its name errors remain below baseline as compression rises, unlike overall compression.Overall compression with reconstruction loss produces even more name errors, suggesting models may not infer the importance of names without priors.Source: §4.4 · pp. 7–8, Table 3 · p. 7

4.5 Effects of Block Compression Strategy

  • Block compression significantly improves basic context compression when documentation is compressed separately or concatenated as a whole.The comparison compresses documentation to 50 soft tokens and includes RMT and AutoCompressor.Source: §4.5 · p. 8, Table 5 · p. 8

4.6 Effects of Selective Compression Strategy

  • The best-performing strategy uses selective compression in both pre-training and fine-tuning; fine-tuning can adapt a model to selective compression, though with a performance gap.This conclusion holds with and without reconstruction loss.Source: §4.6 · pp. 8–9, Table 4 · p. 8

5 Conclusion

The paper compresses tool documentation using selective and block compression to address key-information loss and fixed-length summaries. On two tool-using benchmarks, it reports performance at least comparable to an uncompressed baseline at up to 16x compression.

5 Conclusion

  • Selective compression preserves key information as raw text tokens, while block compression chunks documentation to support variable-length compression with preset ratios.These strategies target key-information loss and the inability of existing approaches to support preset compression ratios.Source: §5 · p. 9
  • At up to 16x compression, the method reaches at least comparable performance to the uncompressed baseline on two tool-using benchmarks.The paper also explores the strategies and provides a context-compression training recipe.Source: §5 · p. 9

Limitations

The strategies rely on human priors, and the proposed approach requires pre-training.

Limitations

  • The definition of key information and the preset compression ratio need task-specific tuning.The authors identify reliance on human priors as the main limitation.Source: Limitations · p. 9
  • The approach requires pre-training, which entails relatively high computation cost and makes it unsuitable for black-box models.The authors note that this limitation is shared by soft context compression approaches.Source: Limitations · p. 9

A.1 The Necessity of Non-key Information

The appendix tests whether tool documentation needs information beyond tool and parameter names. The reported results support the necessity of non-key information.

A.1 The Necessity of Non-key Information

  • The experiment removes non-key information, leaving tool documentation composed almost entirely of names, to test whether names alone suffice.The motivation is that there was no evidence supporting the necessity of descriptions, and models might infer functionality from API and parameter names.Source: §A.1 · p. 10
  • API-Bank test tools are unseen during training, whereas APIBench uses the same tool library in training and testing, allowing possible memorization.The passage reports that a performance gap also remains in the latter setting.Source: §A.1 · p. 10
  • Deleting non-key information significantly hurts performance, supporting its necessity.Table 6 reports the deletion result; the test compares documentation with non-key information removed.Source: Table 6 · p. 11, §A.1 · p. 11

A.2 Comparison with Hard Summarization

The appendix compares soft context compression with hard summarization, such as prompting ChatGPT to summarize documentation. The reported comparison favors the proposed approach, though the authors describe these experiments as supplementary to their focus on soft compression.

A.2 Comparison with Hard Summarization

  • Hard summarization uses text tokens, which the authors say carry less information per context length than soft compression.They present this as the reason for initially considering soft compression.Source: §A.2 · p. 11
  • For the hard-summary comparison, GPT-4 Turbo summarized non-key documentation; API-Bank compressed description fields, while APIBench also compressed example_code.Other APIBench fields were very short.Source: §A.2 · pp. 11–12
  • Over 10× compression is possible when soft compression represents a short description with a single soft token, while plain text cannot compress concise input at a high ratio.The passage presents this as an illustrative example.Source: §A.2 · p. 12
  • GPT-4 Turbo hard summarization achieves lower accuracy and lower compression ratios than the proposed approach.The reported comparison includes GPT-4-summarized inputs and cases trained with summarized data; the latter perform even lower.Source: Table 7 · p. 12, Table 8 · p. 12, §A.2 · p. 12
  • The authors focus on soft context compression, not a task-specific comparison of soft and hard compression; hard-summary results are supplementary.They state that hard-summary performance has limited relation to the integrity of their work.Source: §A.2 · p. 12

Related papers

Loading 2407.02043v1…