PaperlayerView exact revision on arXiv

Source-linked AI summary

Cascaded Batch Prompting

Sho Hoshino, Peinan Zhang

arXiv:2608.27038v1cs.CL

TL;DR

Batch prompting makes LLM inference more efficient but can unpredictably affect downstream task performance when reasoning and symbol grounding are combined. The paper proposes a two-stage cascaded approach that separates these tasks, and reports better performance than single prompting while retaining speed proportional to batch size.

  • Problem

    Batch prompting improves LLM inference efficiency but introduces unpredictable downstream performance because formatting multiple instances can alter generated outputs and conflate reasoning with symbol grounding.

  • Method

    Cascaded batch prompting uses a reasoning stage for free-form answers followed by a symbol-grounding stage that maps them to constrained outputs.

  • Results

    Cascaded batch prompting outperforms the single prompting baseline on MCQA and NLI while maintaining speed proportional to batch size.

  • Takeaways & Limitations

    The task-decomposition principle extends to classification tasks requiring free-form reasoning to be mapped to constrained output formats.

  • Takeaways & Limitations

    The evaluation focuses on classification tasks, while extending the approach to open-ended generation remains future work.

Abstract

from arXiv · show

Although batch prompting makes large language model inference more efficient by processing multiple instances simultaneously, it suffers from unpredictable downstream task performance. We propose cascaded batch prompting, a two-stage approach designed to resolve the unpredictability of conventional batch prompting by disentangling complex reasoning from symbol grounding. Experiments on multiple-choice question answering and natural language inference demonstrate that the proposed method outperforms the standard single prompting baseline while achieving a speedup proportional to batch size, establishing a new state of the art on the Pareto frontier.

1 Introduction

Batch prompting improves LLM inference efficiency but can unpredictably alter downstream performance because reasoning and symbol grounding are conflated. Cascaded batch prompting addresses this issue with a two-stage process that separates free-form reasoning from constrained output mapping, outperforming single prompting while preserving speed proportional to batch size.

  • Motivation: Batch prompting processes multiple instances simultaneously, improving inference efficiency but sometimes improving or degrading downstream task performance unpredictably.This unpredictability creates a trade-off between inference speedup and performance reliability.
  • Motivation: Conventional batch prompting conflates complex reasoning with procedural symbol grounding under strict output constraints.The model must both solve the task and express the solution in a constrained format within one step.
  • Method: Cascaded batch prompting separates reasoning from symbol grounding through a two-stage batching approach.The first stage generates a free-form answer, and the second maps that answer to the required constrained output.
  • Method: The task decomposition allows the LLM to handle reasoning and procedural formatting as separate subtasks.This division of labor is intended to address the cognitive bottleneck associated with conflated processing.
  • Results: Experiments on multiple-choice question answering and natural language inference show higher performance than single prompting with speed proportional to batch size.The results are presented as resolving the unpredictability of conventional batch prompting and establishing a new Pareto-frontier state of the art.

2 Prompting Methodology

The methodology formalizes classification and prompting strategies, then introduces cascaded batch prompting to separate reasoning from symbol grounding. The approach targets batching's efficiency while addressing unpredictable performance.

  • 2.1 Problem Definition: A classification instance contains an input x and class labels, with answer a identifying the correct class.The answer may be represented by a symbol s, its class-label name n, or a comprehensive format containing both.
  • 2.1 Problem Definition: LLM calls map inputs to outputs, and batching represents multiple inputs with bold notation for efficient processing.
  • 2.2 Single Prompting: Single prompting predicts each instance separately, with output variants determined by the desired format.In the Multiple Choice variant, a class-label name can be recovered deterministically from the predicted symbol.
  • 2.3 Conventional Batch Prompting: Conventional batch prompting processes multiple inputs simultaneously to improve throughput.However, prior studies report unpredictable performance and possible degradation when inputs are batched.
  • 2.4 Cascaded Batch Prompting: Cascaded batch prompting is a two-stage approach that disentangles complex reasoning from simple symbol grounding.
  • 2.4 Cascaded Batch Prompting: The reasoning stage generates class-label names for a batch of inputs, allowing the model to focus on solving the tasks.
  • 2.4 Cascaded Batch Prompting: The symbol-grounding stage maps generated class-label names to their corresponding symbols, completing the cascaded process.The complete method is expressed as a composition of the two stages.
  • 2.4 Cascaded Batch Prompting: Symbol grounding is performed individually via single prompting, while Table 1 compares prompting strategies using a conventional baseline batch size of 32.The implementation reports this procedural stage as robust with minimal computational overhead, and the table marks highest scores and statistical significance.

3 Experiments

Experiments on MCQA and NLI across multiple models and batch sizes show that cascaded batch prompting improves accuracy robustness while preserving batching efficiency. Its two-stage design also improves performance without batching, though large batches introduce mechanical output-alignment issues and modest cost overhead.

  • Experimental Setup: Experiments evaluate prompting strategies on MCQA and NLI using MMLU and MNLI across GPT-4.1, GPT-4.1-mini, and Phi-4.Batch sizes vary from 1 to 128, and classification accuracy is reported.
  • Main Results: 86.81% MMLU accuracy and 86.20% MNLI accuracy are the highest reported scores achieved by cascaded batch prompting across the evaluated results.With GPT-4.1, it leads on MMLU; on MNLI, conventional batch prompting performs slightly better.
  • Main Results: Cascaded batch prompting maintains robust accuracy as batch size increases, whereas conventional batch prompting is less consistent and can underperform single prompting.The contrast is exemplified by Phi-4 on MMLU.
  • Sanity Check: At batch size 128, both batch methods show slight degradation because some inputs may lack corresponding outputs, while a sanity check can detect and rerun affected instances.The main results omit these post hoc corrections; the sanity check only slightly improves performance and is not statistically significant.
  • Scalability Analysis: Higher accuracy and instance-level throughput jointly establish a new Pareto frontier for cascaded batch prompting compared with single prompting.Conventional batch prompting is faster but has unpredictable performance, while single prompting has high accuracy at lower speed.
  • Ablation Study: Cascaded single prompting consistently outperforms standard single prompting across MMLU and MNLI, isolating a benefit from task decomposition rather than batching alone.The ablation supports the role of disentangling reasoning from symbol grounding.
  • Cost Analysis: Cascaded batch prompting costs approximately 1.2 times more than conventional batch prompting in token-level throughput because it adds a symbol grounding stage.The additional stage is presented as a necessary trade-off for resolving unpredictability.

4 Discussion

The discussion attributes cascaded batch prompting’s effectiveness to separating reasoning from procedural symbol grounding. This task decomposition is presented as pragmatically valuable because batch pressure can make their combined cognitive load difficult for current models.

  • The discussion moves from empirical advantages to the mechanisms and broader implications of decomposing these tasks.
  • Cognitive Division of Labor: The method divides processing into a reasoning stage for problem-solving and a symbol-grounding stage for procedural formatting.
  • Pragmatic Value of Task Decomposition: The discussion frames task decomposition as a mechanism for addressing the combined cognitive load of reasoning and symbol grounding under batch pressure.

5 Conclusion

The conclusion identifies conflated reasoning and symbol grounding as the source of conventional batch prompting’s unpredictability and presents cascaded batch prompting as the remedy. Experiments show better performance than single prompting with speedup proportional to batch size, while the principle is positioned for broader classification use.

  • Conventional batch prompting’s unpredictability results from conflating distinct cognitive tasks into a single step.
  • Cascaded batch prompting disentangles complex reasoning from simple symbol grounding in a two-stage approach.
  • The method outperforms the single prompting baseline while achieving a speedup proportional to batch size.
  • The experiments focus on MCQA and NLI, while the core principle extends to classification tasks mapping free-form reasoning to constrained outputs.

Limitations

The evaluation is scoped to classification tasks, and the two-stage design introduces additional inference overhead. The authors describe both boundaries as acceptable trade-offs or targets for future work.

  • Task Applicability: The evaluation intentionally focuses on classification tasks as a controlled setting for validating reasoning–symbol-grounding disentanglement.
  • Task Applicability: Extending the approach to open-ended generation is left for future work, potentially using methods such as universal self-consistency.
  • Inference Overhead: For dataset size N and batch size b, implementation requires N/b batched reasoning calls plus N individual symbol-grounding calls.
  • Inference Overhead: The authors regard this additional inference overhead as a necessary and worthwhile trade-off for resolving conventional batch prompting’s unpredictability.

A Setup Details

The appendix details dataset splits, inference settings, model versions, prompts, statistical testing, licenses, and the prompt formats used for MCQA and NLI.

  • Data Splitting: MMLU uses validation for development and test for testing, while MNLI uses matched validation for development and matched test for testing.
  • Hyperparameters: The experiments use max tokens of 20 for single prompting and 1,000 for other prompting strategies.
  • Model Versions: Models are GPT-4.1, GPT-4.1-mini, and Phi-4, with all LLM API calls made asynchronously to maximize throughput.
  • Prompts: Figures 5, 6, and 7 provide the MCQA and NLI prompts, including cascaded and conventional batch formats.
  • Statistical Testing: Paired bootstrap resampling finds statistically significant cascaded gains on MMLU, while MNLI gains are nonsignificant for Phi-4 and worse than conventional batching for GPT-4.1.
  • Licenses: The experiments use MMLU, MNLI, GPT-4.1 API, and Phi-4 under the licenses listed by the authors.
Loading 2608.27038v1…