PaperlayerView exact revision on arXiv

Source-linked AI summary

Intra-Prompt Parallel Decoding for Common-Context Question Answering

Theodore Glavas, Nikhita Vedula, Dushyanta Dhyani, Antonios Valkanas, Yilun Zhu, Shervin Malmasi

arXiv:2609.05707v1cs.CL

TL;DR

CCQA inference wastes computation by repeatedly processing shared contexts in separate prompts, while attention is constrained by GPU memory access. IPPD combines questions within a prompt and parallelizes answer-token decoding through position IDs and attention masks. It achieves up to 7X speedup without sacrificing model performance, while its advantage narrows for some long-context and long-generation regimes.

  • Problem

    CCQA repeatedly recomputes shared prompts and passages for separate questions, wasting GPU resources during autoregressive decoding.

  • Method

    IPPD stacks questions within a prompt, uses virtual position IDs and attention masks, and decodes one token for every question in parallel without modifying the LLM architecture.

  • Results

    IPPD achieves up to 7X speedup without sacrificing model performance and outperforms batched autoregressive decoding across tested benchmark datasets and model sizes.

  • Takeaways & Limitations

    IPPD provides a high-throughput CCQA inference method compatible with batched inference and effective across prompts with different contexts.

  • Takeaways & Limitations

    IPPD targets offline workloads with independently answerable questions, and its advantage narrows for very long contexts with single-token answers and for long generations.

Abstract

from arXiv · show

In common-context question answering (CCQA) tasks, multiple input questions share a common context to base their answers from. However, Large Language Models typically generate each answer using an independent prompt. While existing batching and caching techniques help improve parallelism and reduce repeated computations, the separation of questions across prompts limits the achievable speedup, as modern GPUs are underutilized due to a memory bottleneck during attention. We present Intra-Prompt Parallel Decoding (IPPD), a novel inference method that answers multiple common-context questions in parallel within a single prompt. IPPD directly addresses the bottleneck by efficiently sharing both memory and computation during the attention process, as the next token for every question is decoded in a single inference step. IPPD uses virtual position IDs and attention mask manipulation to generate the same output as standard prompting without requiring fine-tuning or any changes to the LLM architecture. Since all parallelism occurs within a prompt, IPPD is fully compatible with batched inference, even when each prompt features a different context. Our experiments show that IPPD delivers up to 7X the effective throughput as standard decoding without quality degradation, and outperforms prefix caching with PagedAttention in most settings.

1 Introduction

CCQA asks models to answer multiple questions about a shared context, but conventional independent prompting repeats context computation and limits efficient GPU use. IPPD addresses this by decoding shared-context answers within one prompt, reducing attention-related inefficiency while preserving compatibility with existing inference.

  • CCQA produces multiple answers to different questions about a shared document or passage.
  • 0.5?

2 Related Work

Prior work accelerates LLM inference through token, batch, and tree-based parallelism, but these approaches generally separate work across prompts or cache shared computation. IPPD instead combines common-context questions within one prompt to share attention computation and improve GPU utilization.

  • Token parallelism: Speculative decoding and related methods decode consecutive tokens in parallel, using a draft model or propose-then-verify process.These methods target token parallelism within a prompt.
  • Token parallelism: Independent sub-task decomposition can reduce single-task latency but requires creating and processing multiple new prompts.Intra-prompt parallelism produces denser attention computations and higher offline throughput than batching separate prompts.
  • Batch parallelism: Prefix caching and PagedAttention share Key-Value caches across batched prompts to reduce memory usage and avoid recomputing common-prefix matrices.These methods reduce attention FLOPs for common prefixes.
  • Tree-based decoding: Tree-based decoding stores shared contexts in a KV-cache tree, performs cross-attention in chunks, and merges partial attention scores.This shares memory and computation across partially overlapping queries.
  • Intra-Prompt Parallel Decoding: IPPD combines multiple questions and contexts within one prompt, jointly decoding answers while reducing repeated computation and GPU attention memory bottlenecks.The figure states that IPPD produces the same answers in a fraction of the time.

3 Problem Formulation: CCQA

CCQA inference involves independently answering multiple questions that may share contexts, while conventional batching performs attention separately for each prompt. The formulation measures candidate methods by both throughput and answer quality.

  • Inputs and assumptions: A CCQA input consists of a common instruction p, a context c, and a question x, forming tokenized triplets that may share contexts.The formulation represents realistic workloads with multiple context-question triplets.
  • Inputs and assumptions: Each question is assumed answerable independently, with its output depending on its instruction, context, question, and previously generated tokens.An answer cannot depend on the answers generated for other questions.
  • Traditional inference: Traditional autoregressive generation issues one independent prompt per triplet and generates each answer token through a separate forward pass.Generated tokens are fed back into the input sequence at each step.
  • Traditional inference: Batched inference parallelizes some computation across prompts, but attention remains independent for each prompt, creating a bottleneck when prompts share a context.The proposed method seeks to share computation across these common-context prompts.
  • Evaluation: Evaluation compares batched autoregressive generation using throughput and answer quality, with quality measured by Accuracy, F1, and ROUGE-L.Throughput is logical triplets answered per unit time per GPU, while quality compares generated and ground-truth answers.
  • Attention structure: The attention diagram contrasts standard batched prompts with IPPD's stacked prompt containing multiple contexts and queries.Green cells denote unmasked query-key pairs, and arrows indicate which query-row output generates a token.

4 Method: Intra-Prompt Parallel Decoding (IPPD)

IPPD combines common-context questions into one structured prompt and decodes their answers in parallel while preserving the independent autoregressive computation path. Virtual positions and attention masks prevent cross-question leakage, and shared attention computation improves throughput when decoding is memory-bound.

  • 4 Method: Intra-Prompt Parallel Decoding (IPPD): IPPD replaces independent forward passes with one structured pass that stacks contexts and questions, assigns hierarchical positions, masks attention, and decodes answers in parallel.The method is designed to preserve each question’s conditional distribution while processing multiple triplets within a single prompt.
  • 4.1 Input pre-processing: Answers are appended after the prompt header, while token metadata records virtual positions, context identifiers, and question identifiers.Virtual positions encode each triplet’s local autoregressive ordering rather than its absolute location in the concatenated sequence.
  • 4.1 Input pre-processing: The final attention mask combines causal, context, and question constraints so each answer token attends only to shared instructions, its context, and its own prior tokens.Tokens from other questions or contexts fail at least one mask condition and are blocked.
  • 4.2 Parallel Decoding: IPPD produces the same attention support and computational path as independent autoregressive decoding for each question.Masked logits outside the valid key set are removed from the softmax, yielding an identical distribution under the stated construction.
  • 4.2 Parallel Decoding: Each forward pass generates the next token for every unfinished answer, and standard Hugging Face Transformers support the required multi-token outputs and custom masks.Completed answers stop receiving appended tokens while the remaining answers continue.
  • 4.3 Computational Efficiency: IPPD increases arithmetic intensity by reusing shared instruction and context keys and values within one attention operation, reducing repeated memory traffic per output token.This adds FLOPs per output token but can improve throughput when attention is memory-bound.
  • 4.3 Computational Efficiency: IPPD is most effective for short-context single-token tasks and for multi-token tasks across short and long contexts, while long shared contexts can favor other methods.For long shared contexts with short unique suffixes, IPPD is described as better suited than tree-based decoding because tree-method score-combination overhead can dominate.
  • 4.4 Batch Inference Compatibility: IPPD remains compatible with batched inference by allowing multiple stacked prompts to be processed together.

5 Experimental Setup

The evaluation uses four CCQA benchmarks, several open-source language models, and standard batched inference plus prefix caching with PagedAttention as baselines. It also profiles throughput and quality across controlled workload variations.

  • Datasets: The study evaluates NarrativeQA, SQuAD 2.0, RACE, and LongHealth to cover varied context lengths, answer lengths, domains, and task formats.The public benchmarks serve as proxies for larger industrial CCQA workloads.
  • Models: The models include Qwen-3 variants, Phi-4 14B, and OLMo-2-0325-32B-Instruct, with larger models quantized to 4 bits.
  • Baselines: Standard Transformers batched inference is the primary baseline, while prefix caching plus PagedAttention provides the additional comparison method.The comparison adjusts vLLM settings to reduce confounding differences between backends.
  • Profiling and controlled ablations: Controlled experiments vary shared-question counts from 1 to 32 and context length from 1× to 8× while separately profiling prefill and decode phases.These experiments use controlled variants of real benchmark examples.
  • Quality evaluation: Table 2 compares quality metrics for standard batched inference and IPPD, including exact-match agreement between their answers.
  • Throughput evaluation: Figure 3 reports questions answered per second and normalized throughput relative to standard batched inference across datasets and models.The figure distinguishes standard batching, prefix caching plus PagedAttention, and IPPD by shade.

6 Results

IPPD improves throughput across models and benchmarks while preserving near-identical task performance. Its advantage is strongest when many questions share a context and attention remains memory-bandwidth bound, though long contexts can favor PC+PA.

  • Within one percentage point, IPPD matches batched inference across all tested models and benchmarks, with 22 of 23 combinations above 95% exact match.Exact match rates range from 91.3% to 100%.
  • Up to 32X throughput increases over standard batched inference occur on NarrativeQA with Qwen3-8B, partly because IPPD reuses the dataset-wide instruction prefix.IPPD still outperforms PC+PA by 2.2X in this setting.
  • Across models, IPPD reaches 4.1-5.6X relative throughput, rising to 7.2X with Phi4-14B on SQuAD 2.0.Its advantage is stronger for models with 8 billion parameters or more, while Qwen3-1.7B slightly favors PC+PA.
  • For multiple-choice datasets, IPPD consistently beats standard batched inference, including a 2.1-2.7X increase on RACE.With OLMo2-32B, stacking six contexts yields a 5.8X greater throughput increase than PC+PA, +171% versus +29%.
  • IPPD reduces both prefill and decode time, with decode reductions generally tracking the reduction in decoding steps.The profiling is consistent with decode remaining memory-bandwidth bound despite additional FLOPs per IPPD step.
  • IPPD’s win rate against PC+PA rises from 0% with one question per context to 73% with 16 or 32 questions, and remains 75% at 8X context length.LongHealth accounts for five of the six remaining losses in the question-count ablation.

7 When to Use IPPD

IPPD is most useful for throughput-oriented workloads with many independently answerable questions sharing a context. Its advantage narrows for long answers, long contexts, or workloads dominated by question-specific decoding.

  • IPPD performs best when many independently answerable questions are available together in throughput-oriented workloads.The number of questions per context is the clearest practical indicator of its benefit.
  • IPPD’s advantage narrows when answers are long relative to the shared context and question-specific suffixes dominate the stacked prompt.
  • For single-token outputs, short contexts such as RACE favor IPPD, whereas long contexts such as LongHealth can favor PC+PA.
  • When short contexts pair with long generations, standard batched inference or speculative decoding may be preferable because neither method targets the dominant question-specific decoding work.
  • IPPD suits extracting many fields from unstructured documents, product and web data, and validating documents against requirements.The method can also apply to decomposed reasoning tasks before cross-question dependencies arise.

8 Conclusion

The paper introduces IPPD to accelerate shared-context LLM inference without modifying the model architecture. It reports substantial speedups while preserving performance and compatibility with batched inference across different contexts.

  • IPPD decodes the next token for all questions in parallel by manipulating position IDs and attention masks, without modifying the LLM architecture.
  • IPPD is fully compatible with batched inference and can process multiple prompts with different contexts simultaneously.
  • IPPD achieves up to 7X speedup without sacrificing model performance and outperforms batched autoregressive decoding on every tested benchmark and model size.
  • IPPD outperforms prefix caching with PagedAttention in most evaluated settings and may extend to recommendation and multi-aspect information-extraction tasks.

Limitations

IPPD is evaluated primarily for offline workloads with independently answerable questions sharing contexts. Its advantage narrows for very long contexts, short answers, or generations long relative to the shared context, and several implementation and comparison boundaries remain.

  • IPPD targets offline workloads where the full question set is available and questions sharing a context can be answered independently.Tasks whose answers condition one another fall outside its evaluated scope.
  • Very long contexts with single-token answers can favor prefix caching with PagedAttention over IPPD.This regime increases prefill arithmetic intensity; LongHealth is the reported example.
  • Long generations relative to the shared context reduce IPPD’s advantage, although this regime is not measured.
  • IPPD is implemented with HuggingFace Transformers rather than a high-throughput serving backend, so reported speed may understate attainable efficiency.Integration with vLLM is left for future work.
  • The evaluation omits several specialized shared-prefix and tree-based systems and does not test production batching policies.The omitted systems lack compatible implementations for the model families and inference platforms used.

A Method Details

IPPD stacks questions and contexts into structured prompts, using virtual positions and attention masks to preserve independent causal behavior while decoding multiple answers together. Its efficiency comes from sharing attention work and reducing repeated memory traffic, especially during memory-bound decoding.

  • Position IDs and Attention Mask: Virtual positions map tokens to their own instruction, context, and question blocks rather than their physical locations in the concatenated sequence.This preserves the illusion of isolated autoregressive decoding across physically distant sequences.
  • Position IDs and Attention Mask: The attention mask enforces which keys each query may attend to, preventing cross-question leakage while retaining parallel execution.The additive mask assigns −∞ to disallowed keys.
  • Complexity Analysis: IPPD reduces memory accesses by a factor of Mj by processing questions through a shared attention operation.This exchanges the memory reduction for increased FLOPs, with benefits when shared prefixes dominate or workloads are memory-bottlenecked.
  • Decode: Decode is more memory-access bottlenecked than prefill, so IPPD benefits even with very large contexts and prompts during multi-token generation.The method outperforms PC+PA on the reported multi-token SQuAD 2.0 and NarrativeQA settings.
  • Method: IPPD replaces independent prompts with stacked contexts and questions, then decodes multiple answer tokens within one structured prompt.The method preserves per-question causal inference while enabling intra-prompt parallelism.
  • Evaluation: The evaluation measures throughput as logical triplets processed per unit time per GPU and answer quality with Accuracy, F1, and ROUGE-L.Table 4 contrasts larger matrix dimensions but fewer attention calculations for IPPD.

B.2 Dataset Details

The evaluation spans four CCQA datasets, multiple inference backends, and controlled throughput analyses. Results show that IPPD accelerates prefill and decode, with gains increasing with questions per context but weakening on very long, single-token workloads.

  • Datasets: NarrativeQA uses human-written book and movie-script summaries as contexts, with about 30 question–answer pairs per document and 5-shot prompting.
  • Datasets: SQuAD 2.0 contains answerable and unanswerable Wikipedia questions, with the model instructed to output “null” for unanswerable cases.
  • Experimental Setup: The primary setup uses one Nvidia L40S 48GB GPU, greedy decoding, non-thinking hybrid models, and dataset-specific maximum output lengths.RACE and LongHealth use one-token output limits, while NarrativeQA and SQuAD 2.0 use longer limits.
  • Prefill and Decode Speedup: IPPD accelerates both prefill and decode relative to standard batched autoregressive inference.Prefill gains come from fewer forward passes, while decode benefits from reduced memory-bandwidth pressure.
  • Question Count Ablation: IPPD speedup grows with questions per context and is generally largest at 32 questions per context, whereas PC+PA scales more slowly.With one question per context, IPPD provides little or no shared-context benefit.
  • Question Count Ablation: LongHealth consistently favors PC+PA because extremely long contexts and one-token answers leave no iterative decode phase.The comparison is also affected by PC+PA’s FlashAttention backend versus IPPD’s standard SDPA implementation.

D.2.2 Context Length Ablation

This ablation tests how extending shared contexts affects IPPD’s throughput relative to PC+PA. Although longer contexts generally reduce IPPD’s relative speedup, IPPD retains substantial benefits and usually outperforms PC+PA at 8× context length.

  • Experimental setup: The ablation constructs 1×, 2×, 4×, and 8× context-length variants for three datasets and reports speedup relative to batched autoregressive inference.LongHealth is excluded because it already has long contexts.
  • Results: Longer contexts generally reduce IPPD’s speedup relative to autoregressive inference, although the trend is not monotonic for every model.
  • Results: The reduced relative benefit occurs because longer contexts shift more inference time toward the FLOP-intensive prefill phase, diminishing the impact of IPPD’s memory-access savings.
  • Results: Even at 8× context length, IPPD retains substantial speedups on NarrativeQA, SQuAD 2.0, and RACE.
  • Results: At 8× context length, IPPD outperforms PC+PA in 12 of 16 supported model–dataset comparisons.Table 9 reports throughput speedups relative to batched autoregressive inference, with higher values indicating better performance.
Loading 2609.05707v1…