PaperlayerView exact revision on arXiv

Source-linked AI summary

Batch Prompting: Efficient Inference with Large Language Model APIs

Zhoujun Cheng, Jungo Kasai, Tao Yu

arXiv:2301.08721v2cs.CLcs.AI

TL;DR

Inference over large sample volumes can be costly, and standard prompting requires a separate LLM call for each test sample. Batch prompting groups test samples into one inference run, placing them after grouped in-context demonstrations and using position identifiers to align responses. Across ten datasets, batch prompting achieves better or similar performance than standard prompting, with much lower token and time costs.

  • Problem

    Inference over large sample volumes can be costly, and standard prompting requires a separate LLM call for each test sample.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    Batch prompting groups test samples into one inference run, placing them after grouped in-context demonstrations and using position identifiers to align responses.

    Source: §2.2 · p. 2

  • Results

    Across ten datasets, batch prompting achieves better or similar performance than standard prompting, with much lower token and time costs.

    Source: §7 · p. 7

  • Takeaways & Limitations

    Users can consider batch prompting to save money and time while maintaining good performance in realistic applications.

    Source: §3.3 · p. 4

    Benefits depend on shared input tokens substantially exceeding outputs, while complex tasks or long contexts can reduce performance and no reliable prediction guideline is identified.

    Source: §6 · pp. 6–7

Abstract

from arXiv · show

Performing inference on large volumes of samples with large language models (LLMs) can be computationally and financially costly in industry and real-world use. We propose batch prompting, a simple yet effective prompting approach that enables the LLM to run inference in batches, instead of one sample at a time. Our method reduces both token and time costs while retaining downstream performance. We theoretically demonstrate that under a few-shot in-context learning setting, the inference costs decrease almost inverse linearly with the number of samples in each batch. We extensively validate the effectiveness of batch prompting on ten datasets across commonsense QA, arithmetic reasoning, and NLI/NLU: batch prompting significantly~(up to 5x with six samples in batch) reduces the LLM (Codex) inference token and time costs while achieving better or comparable performance. For state-of-the-art Chat-based LLMs, e.g., GPT-3.5 and GPT-4, we show the benefits of batch prompting also hold. Further analysis shows that the number of samples in each batch and the complexity of tasks affect its performance. Moreover, batch prompting can be applied across different reasoning methods using LLMs. Our code can be found at the site https://github.com/xlang-ai/batch-prompting.

1 Introduction

Inference over large volumes of LLM samples can be costly. Batch prompting handles multiple samples per inference run to reduce token and time costs while retaining downstream performance.

1 Introduction

  • Prior in-context learning work has explored eliciting reasoning steps, selecting representative exemplars, and designing prompt templates.The passage cites examples of each direction, including Wei et al. (2022), Liu et al. (2022), and Jiang et al. (2020).Source: §1 · p. 1

Batch Prompting

  • Batch prompting processes b samples in one inference run, reducing the N calls required for N samples to N/b without changing APIs.The paper frames this as an alternative to standard prompting, which handles one sample per run.Source: §1 · p. 2
  • Under few-shot in-context learning, most call tokens come from exemplars, so increasing batch size can reduce inference costs almost inverse linearly.The paper says only a small portion of the token budget is used for the particular inference samples.Source: §1 · p. 2
  • The approach is evaluated across commonsense QA, arithmetic reasoning, and NLI/NLU, with comparable or better performance reported on all ten datasets.The paper also tests the method with GPT-3.5 and GPT-4.Source: §1 · p. 2
  • Batch size and task complexity affect performance, and batch prompting is reported to work across end-to-end, Chain-of-Thought, and code-generation reasoning methods.The authors describe it as a potential drop-in substitute for conventional prompting.Source: §1 · p. 2

2 Approach

The approach replaces one-call-per-sample inference with batched prompting. Its cost analysis explains token savings under few-shot learning and qualifies time savings by API overhead and decoding time.

2.1 Problem Setup

  • For N test samples, standard prompting makes N separate inference calls.This is the paper’s stated large-volume setting, including dataset benchmarking and customer requests.Source: §2.1 · p. 2

2.2 Batch Prompting

  • Each run batches b test contexts after ordered demonstration contexts and outputs, adds [index] for response alignment, and cuts calls from N to N/b.The demonstrations are divided into K/b batches of b exemplars each.Source: §2.2 · p. 2

2.3 Token Cost

  • When K ≫1 and b < K, ηbatch scales almost inverse linearly with b because prompt tokens dominate call costs.The cost model includes input prompt and generated tokens, with prompt tokens usually accounting for most tokens.Source: §2.3 · pp. 2–3

2.4 Time Cost

  • Batching reduces practical time by cutting API calls, but longer decoding grows with b and can blunt gains when overhead and rate limits stop dominating.The paper says token-cost comparisons are more reliable over time because LLM infrastructure and services can change.Source: §2.4 · p. 3

3 Experiments

Experiments assess batch prompting’s inference efficiency and downstream performance across ten datasets, then examine its behavior with other LLMs and batch configurations.

3.1 Datasets

  • The ten datasets cover commonsense QA, arithmetic reasoning, and NLI/NLU; five use full dev/test sets and five use the first 300 test samples.The datasets include CommonsenseQA, StrategyQA, GSM8K, SVAMP, AQuA, AddSub, MultiArith, RTE, MNLI, and SST-5.Source: §3.1 · pp. 3–4

3.2 Experimental Setups

  • Experiments use Codex across ten datasets and also test GPT-3, GPT-3.5, and GPT-4 with 12 manually selected Chain-of-Thought exemplars.For budget reasons, GPT-4 is tested on the first 100 samples of each dataset.Source: §3.2 · p. 4

3.3 Main Results

  • Up to 5× with six samples per batch, batch prompting reduces Codex token and time costs while matching or exceeding standard prompting accuracy across ten datasets.The reported cost decrease scales almost inverse-linearly with batch size.Source: §3.3 · p. 4, Table 1 · p. 4

3.4 Results across More LLMs

  • Across GPT-3, GPT-3.5, and GPT-4, batch prompting retains downstream performance; Chat-based models tend to improve.The authors suggest system-message roles may help GPT-3.5 and GPT-4 follow batch input and output instructions.Source: §3.4 · p. 4, Table 2 · p. 4
  • Figure 3 reports declining accuracy with larger batch sizes across five datasets.Source: Figure 3 · p. 5
  • Similarity- and diversity-based batching methods do not achieve performance gains on five representative datasets.Source: Table 3 · p. 5

4 Analysis

The analysis examines how batch size and sample grouping affect performance, how task complexity and input length shape the tradeoff, and whether batching works with different reasoning methods.

4.1 Number of Batch Samples

  • At b = 6, performance drops significantly on four of five datasets; b = 3 or 4 often balances performance with token and time savings.The cost reductions diminish as b grows, and the authors suggest b < 6 with 12 in-context examples as a balance.Source: §4.1 · p. 5

4.2 Selection of Batch Samples

  • Similarity- and diversity-based grouping do not improve over random grouping.The authors suspect earlier in-batch samples without ground-truth outputs may propagate errors to later samples; they identify better selection strategies as future work.Source: §4.2 · p. 5

4.3 Complexity of Tasks

  • AQuA accuracy has the steepest reported drop, from 46.1 to 42.1, while longer WikiTQ table inputs show steeper declines as batch size increases.On WikiTQ, the three-row schema averages 216 tokens per table and has the steepest performance drop; shorter contexts are steadier.Source: §4.3 · pp. 5–6

4.4 Reasoning Methods

  • Batch prompting supports end-to-end and program-based reasoning with similar or better task performance while retaining efficiency benefits.The experiments use Binder on WikiTQ and Program-of-Thought on GSM8K and SVAMP.Source: Table 4 · p. 6, §4.4 · p. 6

5 Related Work

Related work improves in-context learning through reasoning, program generation, and exemplar selection, while this paper adds batching as a way to reduce budget and time costs. It also presents batching as an API-compatible alternative to efficiency methods requiring architectural changes.

Improve In-Context Learning

  • Prior in-context learning work uses reasoning methods such as chain-of-thought prompting and program generation.Other work selects better in-context exemplars.Source: §5 · p. 6
  • Batch prompting adds a dimension to in-context learning for large-scale applications by aiming to save budget and time while maintaining good or better performance.Source: §5 · p. 6

Efficient Language Generation

  • Unlike efficiency approaches that introduce alternative architectures, batch prompting is presented as usable with off-the-shelf language-model APIs without extra training or customized hosting.Source: §5 · p. 6

6 Limitation

The discussion identifies task and prompt-length conditions that can limit batch prompting, and recommends testing a small subset before scaling up.

6 Limitation

  • Batch prompting may not suit heavy-output tasks because input prompt tokens should be significantly greater than output tokens.Source: §6 · p. 6
  • Task complexity and lengthy input contexts can negatively affect performance, and the authors report no definitive guideline for predicting declines.They advise testing a smaller subset before larger-scale implementation.Source: §6 · p. 7

7 Conclusion

The conclusion presents batch prompting as a way to handle multiple samples per inference call, reducing token and time costs. Across ten datasets, it reports better or similar performance than standard prompting with much lower costs.

7 Conclusion

  • Batch prompting handles multiple samples in a batched fashion to reduce token and time costs.Source: §7 · p. 7
  • Across ten datasets spanning commonsense QA, arithmetic reasoning, and NLI/NLU, batch prompting achieves better or similar performance than standard prompting with much lower token and time costs.Source: §7 · p. 7
  • The authors hope batch prompting offers pragmatic value for efficient real-world LLM usage.Source: §7 · p. 7

A Time Cost Analysis Regarding Transformer Architecture

The analysis models decoder time for few-shot batch prompting and shows how batch size affects one-inference cost. Although larger batches increase decoding time per inference, reducing API calls can lower practical time costs when API overhead and rate limits are bottlenecks.

A Time Cost Analysis Regarding Transformer Architecture

  • The analysis assumes K few-shot exemplars averaging C tokens per sample and b samples per batch; standard prompting is the special case b = 1.It focuses on the autoregressive decoder used by decoder-only Transformer LLMs.Source: A Time Cost Analysis Regarding… · p. 10
  • Transformer decoding takes O(n^2) time for the first token and O(n) for each later token when previous tokens are cached.The embedding dimension d is omitted as a constant.Source: A Time Cost Analysis Regarding… · p. 10
  • Larger batches increase one-inference time, with a relatively marginal effect when b is small, especially when b ≪ K.The passage identifies b = 1 as common practice in few-shot in-context learning.Source: A Time Cost Analysis Regarding… · p. 10
  • Table 5 gives time-per-inference examples for K = 12 and C = 100 across varying batch sizes, but the supplied passage gives no cell values.The examples are calculated according to Equation 3.Source: A Time Cost Analysis Regarding… · p. 10, Table 5 · p. 10
  • Reducing API calls from N to N/b almost inverse-linearly reduces practical time costs when API overhead and rate-limit blocking dominate.If those bottlenecks are absent, the decoding-time increase should not be overlooked.Source: A Time Cost Analysis Regarding… · p. 10

B More Experimental Results

This section points readers to the reported experimental results and records the API settings used in those experiments.

B More Experimental Results

  • Results for all experiments are listed in Tables 6–9.The supplied passage does not state the tables’ cell values.Source: B More Experimental Results · p. 10
  • Experiments used temperature 0.4 for WikiTQ with Binder and 0 for the other experiments.The WikiTQ setting follows the Binder paper.Source: B More Experimental Results · p. 10
  • The experiments used five rotating OpenAI keys, and rate-limit errors were counted in time costs.The passage notes that using fewer keys should produce more rate-limit errors because each key’s request interval would be shorter.Source: B More Experimental Results · p. 10

C Prompts

The appendix documents prompt construction across the evaluated tasks and provides example prompts for Codex and chat-based models. It also identifies the accuracy and time comparisons reported in the experimental tables.

C Prompts

  • Prompts follow Chain-of-Thought for seven tasks, Binder and Program-of-Thought for program-generation tasks, and author-designed Chain-of-Thought prompts for RTE, MNLI, and SST-5.For prompts with fewer than 12 exemplars, training samples are added to reach 12; examples use b = 4 and regroup the same 12 exemplars for other batch sizes.Source: C Prompts · p. 10
  • Tables 6 and 7 report Codex Chain-of-Thought accuracy and time per sample across ten datasets and varying batch sizes.The captions identify the comparisons but provide no outcome values in these passages.Source: Table 6 · p. 11, Table 7 · p. 11
  • Tables 8 and 9 report program-based accuracy comparisons: Binder on WikiTQ and Program-of-Thought on GSM8K and SVAMP, each across varying batch sizes.The captions identify the evaluated methods and tasks, but no accuracy values are supplied here.Source: Table 8 · p. 11, Table 9 · p. 11
  • The cited table captions identify prompt material for CommonsenseQA (Table 10), StrategyQA (Table 11), GSM8K, SVAMP, AddSub, and MultiArith (Table 12), AQuA (Table 13), RTE (Table 14), and MNLI (Table 15).Source: Table 10 · p. 12, Table 11 · p. 13, Table 12 · p. 14
  • Table 17’s chat prompt places instructions in the system message, exemplars in user-assistant rounds, and test contexts in the final user turn.The model outputs answers after receiving the test samples.Source: Table 17
Loading 2301.08721v2…