BatchPrompt: Accomplish more with less

1 INTRODUCTION

• Batching can stretch input contexts and lower performance, while accuracy varies significantly with data position and order.The authors attribute the position effect possibly to autoregressive decoding, which gives each answer a different context and distance from the task specification and data.Source: §1 · pp. 1–2
• Figure 1’s rotated Boolq batches show varying average accuracy by position, with overall accuracy decreasing as batch size increases.The example compares batch sizes 16 and 32 after rotating data across batch indices.Source: Figure 1 · p. 2
• BPE permutes the same batch data into diverse orders and uses majority voting to combine the resulting outputs.It works off-the-shelf with pretrained LLMs and requires no extra human annotation.Source: §1 · p. 2
• SEAS uses repeated confident labels to stop voting early on individual samples, keeping rounds and token use low.The introduction reports that around 80% of data can be answered within two voting rounds, with a side benefit of improved prediction accuracy.Source: §1 · p. 2
• The evaluation covers Boolq, QQP, and RTE, plus a small side experiment applying BatchPrompt to arithmetic reasoning on GSM8K.The paper presents BatchPrompt with BPE and SEAS as achieving promising performance on benchmark tasks.Source: §1 · pp. 2–3
• At batch size 32/five rounds, input tokens fell 72.6% (Boolq), 81.4% (QQP), 69.2% (RTE); SinglePrompt accuracy pairs were 90.6%→90.9%, 87.2%→88.4%, 91.5%→91.1%.LLM calls fell 84.4%, 90.6%, and 84.4%, respectively; reported token counts exclude generated output tokens.Source: §1 · p. 3

2 BATCHPROMPT

• BPE permutes data across voting rounds and uses majority voting to combine the resulting answers for each batch.The method relies on correct answers tending to agree more across positions than incorrect answers.Source: §2 · p. 3
• Different permutations change the preceding-answer context for each data item, so its predicted answer can vary across rounds.The paper attributes this variation to answers being conditioned on prior outputs as well as the task specification.Source: §2 · p. 3
• Self-weighted majority voting (sw-mv) assigns weight 1 to “confident” outputs and α to “not confident” outputs, with α set to 0.2 in experiments.The LLM generates a binary confidence label as part of its response.Source: §2 · p. 3
• The paper focuses on regular and self-weighted majority voting as compatible strategies for combining predictions.Figure 2 depicts the BatchPrompt and BPE flow, followed by self-weighted voting and SEAS.Source: §2 · p. 4, Figure 2 · p. 4

3 SEAS

• Repeated “confident” labels allow SEAS to truncate voting for individual samples, reducing the effective batch size and potentially the number of rounds.Voting can stop early for those samples while maintaining high accuracy, according to the paper.Source: §3 · p. 4
• SEAS can cut token use and boost accuracy by removing easy samples earlier, leaving fewer and harder samples for later rounds.The remaining samples may be easier to predict in smaller effective batches.Source: §3 · pp. 4–5
• Later rounds may handle only a few unresolved samples instead of the original full batch, rather than refilling batches with completed samples.The paper does not select the refill alternative because it sacrifices the potential benefit of smaller batches for harder samples.Source: §3 · p. 5

4.1 EXPERIMENTAL SETTINGS

• The experiments cover BoolQ, QQP, and RTE using GPT-4 and gpt-3.5-turbo in few-shot settings without training or fine-tuning.They use randomly selected validation examples, with 320 examples per dataset except 277 for RTE.Source: §4.1 · pp. 5–6

4.2 RESULTS FOR BATCHPROMPT + BPE

• BPE sharply reduces LLM calls, but its token savings depend on voting rounds: at one round tokens are one-fifth of SinglePrompt, while 9+ rounds can exceed it.Without SEAS, token count increases linearly with voting rounds.Source: §4.2 · pp. 6–7, Figure 3 · p. 6
• At batch sizes below 64, more BPE voting rounds can outperform SinglePrompt; at 64 or higher, more rounds recover accuracy after a weak one-round result.For BoolQ at batch size 64 or higher, accuracy rises from 72.8% to 86.3%, which the authors describe as competitive with SinglePrompt.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 RESULTS FOR BATCHPROMPT + BPE + SEAS

• The SEAS ablation compares confidence-guided stopping with random dropping and plain BPE to assess self-reflection's contribution.The comparison includes an accuracy ablation and a separate comparison of LLM-call counts.Source: Figure 3 · p. 6, §4.3 · pp. 7–8
• SEAS with BPE uses far fewer tokens and calls while achieving accuracies competitive with SinglePrompt.The figure reports accuracy near or above SinglePrompt, especially for GPT-4 at lower batch sizes.Source: §4.3 · p. 7, Figure 4 · p. 8

Related batching work

• A related batching method used batch sizes below 6; this paper presents techniques to counteract the performance degradation associated with naive batching (Cheng et al. (2023)).Source: §5 · p. 8

LLM prompting

• Prompt research includes discrete-template search and other approaches, while this paper instead batches sample data to improve prompting performance.The discussion cites prompt research on paraphrasing Haviv et al. (2021), mining Jiang et al. (2020), gradient-based search Wallace et al. (2019), and scoring Davison et al. (2019).Source: §5 · p. 8

Long context prompting

• Prior long-context studies report degraded conditioning and position-sensitive performance, including the middle input region performing worst (Liu et al. (2023a)).The cited work also reports answers drawing on nearby context and simple summary statistics for distant context (Sun et al. (2021a); Sharan et al. (2018)).Source: §5 · p. 9

Consistency in LLMs

• Consistency methods use repeated generations and voting or search to select answers, but can increase time, token, call, and cost burdens.Examples include MathPrompter (Imani et al. (2023)), self-consistency (Wang et al. (2022)), and Tree-of-Thought (Yao et al. (2023)).Source: §5 · p. 9

6 FUTURE WORK

• Batch prompting may become increasingly important as token limits grow, given the cost and power consumption associated with greater LLM use.Source: §6 · p. 9
• Voting, confidence, permutation, and batch-size strategies require tuning to task-description length, task-specific performance, and the LLM used.Automating these choices is left for future work.Source: §6 · p. 9
• The authors suggest that selecting among LLMs could be combined with BatchPrompt.Source: §6 · p. 9
• The framework is mainly used for binary-classification and arithmetic-reasoning tasks; its applicability to summarization or generation is uncertain because repeated answers vary substantially.Source: §6 · p. 9

A DETAILS FOR FIG.4

• Table 2 compares BatchPrompt + BPE + SEAS on BoolQ, and Table 3 does so on QQP, with batch size and voting rounds identified as settings.The detailed numbers for these comparisons are referenced in the appendix.Source: Table 2 · p. 13, Table 3 · p. 13, A DETAILS FOR FIG.4 · p. 13

• The experiment used 1,280 filtered test samples and two few-shot examples; token limits required GPT-4 and batch sizes below 32, and SEAS was not used.Source: B EXTENSION TO ARITHMETIC REASONING TASK · p. 13
• At batch size 32, GSM8K accuracy rose from 89.1% (1 voting round) to 92.0% (5 voting rounds).Source: B EXTENSION TO ARITHMETIC REASONING TASK · p. 13
• SinglePrompt accuracy on GSM8K was 94.9%, reported as ranking 5th on the leaderboard.Source: B EXTENSION TO ARITHMETIC REASONING TASK · p. 13
• The authors suggest that earlier samples in a batch may act as few-shot examples for later ones, helping a task that relies heavily on few-shot logical deduction.Source: B EXTENSION TO ARITHMETIC REASONING TASK · pp. 13–14

C EXTENSION TO CAUSAL REASONING AND NATURAL LANGUAGE INFERENCE TASKS

• The extension uses all validation data (>300) for long-answer tasks and directs readers to Table 5 for COPA and Table 6 for MNLI results.Source: C EXTENSION TO CAUSAL REASONING AND… · p. 14, Table 5 · p. 15, Table 6 · p. 15

D PROMPTS

• The prompts require one [class X] label for each batched input and explicitly remind the model to generate the batch-size number of labels.Source: D PROMPTS · p. 16
• For GSM8K, the prompt requests intermediate reasoning steps, a final sentence in the form “The answer is xxx.”, and a numeric answer.Source: D PROMPTS · pp. 17–18

• The authors filtered sensitive-content data with gpt-3.5-turbo, then randomly selected 320 samples each for BoolQ, QQP, and RTE and 1,280 for GSM8K.Source: E DATA INDEX · p. 18

F EXPERIMENTAL RESULT OF BATCH GPT+BPE ON RTE AND QQP

• Table 7 presents RTE comparisons, while Table 8 presents QQP comparisons.The supplied passages identify the tables’ scope but provide no cell values or winning method.Source: Table 7 · p. 20, F EXPERIMENTAL RESULT OF BATCH GPT+BPE… · p. 20

G EXAMPLES FOR NEGATIVE FEW-SHOT SAMPLES (GSM8K, BOOLQ)

• The examples also show arithmetic questions alongside generated worked answers.One example calculates the result for a tennis-ball question, and another works through an apples question.Source: G EXAMPLES FOR NEGATIVE FEW-SHOT… · p. 20
• BoolQ examples pair questions with passages and return class labels marked with confidence tags.The examples include property-tax and pardon questions, with outputs labeled by class and confidence.Source: G EXAMPLES FOR NEGATIVE FEW-SHOT… · p. 22

H CALCULATION FOR TOKEN

• For BoolQ, estimated token use is 184,331 at batch size 1, 34,031 at 16, and 29,031 at 32.The calculation uses a 501-token task specification and 24,011 data tokens.Source: H CALCULATION FOR TOKEN · p. 22
• Doubling batch size halves LLM calls, although token reductions become less pronounced above batch size 16.The authors emphasize call savings because token savings are not as obvious beyond that batch size.Source: H CALCULATION FOR TOKEN · p. 22

I PUTTING THE DATA AT THE VERY BEGINNING OR END

• Data placed at the very beginning or end can achieve better performance.The passage says this pattern is shown in Fig. 1.Source: I PUTTING THE DATA AT THE VERY… · p. 22
• Ignoring outputs for middle-position data wastes tokens and entails more repeated experiments.The authors also say the middle-position range depends on data length and batch size and cannot be defined exactly.Source: I PUTTING THE DATA AT THE VERY… · pp. 22–23
• Because BatchPrompt is already competitive with SinglePrompt on all three datasets, position alone is not expected to easily boost accuracy further.The authors therefore retain their random permutation strategy.Source: I PUTTING THE DATA AT THE VERY… · p. 23