PaperlayerView exact revision on arXiv

Source-linked AI summary

Generalized Parallel Scaling with Interdependent Generations

Harry Dong, David Brandfonbrener, Eryk Helenowski, Yun He, Mrinal Kumar, Han Fang, Yuejie Chi, Karthik Abinav Sankararaman

arXiv:2510.01143v3cs.AIcs.LG

TL;DR

Parallel generations for one prompt are usually independent, leaving potentially useful information in one response unused by others. Bridge lets same-prompt generations share information while remaining parallel, improving response and response-set quality across models and reasoning tasks.

  • Problem

    Parallel sampling produces responses that remain unaware of one another, leaving potentially useful information from other responses unused when a high-quality response set is needed.

    Source: §1 · p. 1

  • Method

    Bridge treats batched hidden states as a shared tensor and lets tokens from sequences with the same prompt share information during parallel generation.

    Source: §1 · p. 2, §5 · p. 12

  • Results

    Across multiple models and reasoning tasks, Bridge improves single-sample accuracy and response-set quality; on DeepSeek-R1-Distill-Qwen-7B, it improves RLVR’s relative benefit by 39% over the next-best method across 7 math tasks.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    Trained once, Bridge supports varying generation widths and can be used with post-generation aggregation techniques.

    Source: §1 · p. 2

Abstract

from arXiv · show

Parallel LLM inference scaling involves sampling a set of $N>1$ responses for a single input prompt. However, these $N$ parallel responses tend to be generated independently from each other, partitioning compute resources and leaving potentially useful information in one generation untapped by others. This is in contrast to response length scaling where past computation is used in all future steps. For higher quality responses and response sets, we propose Bridge to generate interdependent responses in parallel by rethinking batched LLM hidden states as holistic tensors rather than independent slices. With only a small amount (2.8%-5.1%) of new parameters, Bridge improves the relative mean accuracy gains from reinforcement learning with verifiable rewards by up to 39% and boosts consistency of correct responses. Trained once, Bridge scales to any generation width, all with greater performance than independent generations, unlocking a more general mode of parallel scaling that effectively leverages information between sequences, compatible with any post-generation aggregation technique.

1 Introduction

Bridge addresses the isolation of parallel responses by allowing generations for the same prompt to share information while remaining parallel. The paper reports improved response and response-set quality with a small parameter increase and flexibility across generation widths.

1 Introduction

  • Independent parallel generations leave potentially useful information from other responses unused, unlike sequential scaling where sampled tokens can contribute to the final output.Earlier parallel methods that interact between paths funnel computation into one output; the challenge is to generate N interdependent responses in parallel without extensive post-training.Source: §1 · p. 1
  • Bridge lets tokens from sequences sharing a prompt depend on tokens in other generation threads, enabling information sharing while preserving parallel generation.The method rethinks batched hidden states as a tensor rather than independent slices and evaluates both response accuracy and response-set quality.Source: §1 · p. 2
  • Adding 2.8% to 5.1% additional parameters, Bridge improves the effectiveness of RLVR after warming up on a small SFT dataset.Bridge significantly increases final RLVR performance on 12 benchmarks using multiple reasoning models.Source: §1 · p. 2
  • Bridge is robust to train-time and test-time width discrepancies; all tested widths outperform independent generations in accuracy, coverage, and consistency.It can be trained once, has no generation-time heuristics or interventions, and can be used with post-generation aggregation techniques.Source: §1 · p. 2
  • 39% greater relative benefit from RLVR on DeepSeek-R1-Distill-Qwen-7B, averaged over 7 math tasks, than the next-best method.For the same model, the rate of competition-math problems with all responses correct rose from 15.0% to 17.8%.Source: §1 · p. 2

2 Background & Related Works

The related work frames high-quality interdependent response sets as an understudied goal. It contrasts independent sampling, post-generation synthesis, and mid-generation interaction methods, and situates Bridge among work on higher-order tensors.

2 Background & Related Works

  • Generating high-quality sets of interdependent responses remains understudied, although some related methods combine parallel generations into one response.Source: §2 · p. 3

2.1 Test-time Scaling

  • Independent generations divide compute and remain oblivious to one another, while longer chains of thought use earlier computation in later steps.Pass@k training is noted as a promising direction for parallel exploration and a possible extension.Source: §2.1 · p. 3

2.2 Post-generation Synthesis

  • Majority voting mainly targets discrete answers, while synthesis of reasoning traces remains unclear; other approaches concatenate responses for an LLM to combine.Bridge focuses on generation and can be integrated with post-generation aggregation techniques.Source: §2.2 · p. 3

2.3 Mid-generation Synthesis/Pruning

  • Prior mid-generation methods share caches, concatenate parallel outputs, or prune unpromising generations; several funnel parallel resources into one output rather than generating N outputs.Bridge instead supports inter-output dependence while producing multiple responses simultaneously.Source: §2.3 · p. 3

2.4 High Order Tensors

  • Research on richer tensor representations spans theory and practice, and Bridge draws on this work to diffuse information across the LLM hidden-state tensor rather than restricting it to matrix slices.Source: §2.4 · p. 4

3 Bridge: Connecting Generation Paths

Bridge connects parallel generation paths with small attention-like blocks, letting responses from the same prompt share information during generation. The section describes its tensor-based architecture, optional SFT warm-up, and RLVR objective.

3 Bridge: Connecting Generation Paths

  • Bridge uses small attention-like blocks that fit into LLMs to share information across parallel generations while supporting any test-time generation width.The blocks are designed to avoid worsening attention and feedforward memory bottlenecks.Source: §3 · p. 4

3.1 Bridge Architecture

  • Bridge applies attention between tokens from sequences sharing a prompt, at each timestep, rather than treating batch entries as independent slices.The batch hidden states are interpreted as a 3-D B × S × D tensor, analogous to an image, with decoding like autoregressive generation of additional columns.Source: §3.1 · p. 4, §3.1.1 · p. 5

3.1.1 Connection to Efficient Attention for Tensors

  • Bridge’s tensor view applies attention along different input axes, connecting it to axial attention introduced for computer vision.The section frames the batch, sequence, and hidden dimensions as a holistic tensor rather than a stack of independent 2-D slices.Source: §3.1.1 · p. 5

3.1.2 Generation Interdependence

  • Bridge preserves independence between tokens sampled at the same timestep, so next-token sampling can remain parallel despite interdependent generations.Its attention mask excludes sequences from different prompts and sequences that have completed generation.Source: §3.1.2 · p. 5, §3.1 · p. 5

3.2 SFT Warm up

  • Optional SFT warm-up trains Bridge blocks on correctness-filtered traces grouped by prompt while freezing all other parameters.The traces are generated from prompts in an existing math dataset; Figure 3 also describes a parameter-matched MLP baseline.Source: §3.2 · p. 6, Figure 3 · p. 6

3.3 RLVR Objective

  • Bridge makes the RLVR objective inter-sample dependent: ratios and KL terms link relevant samples, and each output’s loss and gradients are intertwined with others sharing its prompt.Gradients from sequences with both positive and negative advantages are backpropagated through each sequence because of Bridge blocks.Source: §3.3 · p. 6

4 Experiments

Across multiple models and benchmarks, Bridge improves individual accuracy and response-set quality, with experiments also testing width, length, warm-up, feature contribution, and output stability.

4.2.1 Accuracy

  • Bridge with SFT and RLVR has the highest Pass@1 in nearly all cases, with average improvements over the original model 26%, 39%, and 34% relatively more than the next-best method for DS-Qwen-1.5B, DS-Qwen-7B, and DS-Llama-8B.The comparison covers math benchmarks; parameter-matched P-Match improves accuracy over pure RLVR in many cases but is less consistent.Source: Table 1 · p. 7, §4.2.1 · p. 8

4.1.1 Models and Baselines

  • Bridge adds 5.1%, 2.8%, and 3.4% parameters to DS-Qwen-1.5B, DS-Qwen-7B, and DS-Llama-8B, respectively; training uses GSM8K SFT warm-up and 1,000 RLVR steps, with evaluation on seven math and five non-math benchmarks.SFT warm-up trains only the added parameters; RLVR then trains the full model.Source: §4.1.1 · p. 7, §4.1.2 · p. 7, §4.1.3 · p. 8

4.2.2 Set Evaluations

  • Bridge has higher G-Pass@8τ for nearly all τ values and models, improving response-set coverage and the frequency of correct answers without spreading responses across many incorrect answers.Qwen-7B and DS-Llama-8B also generalize from training width 4 to evaluation width 8.Source: §4.2.2 · p. 9, Figure 4 · p. 9

4.3.1 Generation Width

  • For DS-Qwen-7B trained at width 4, Bridge outperforms P-Match in accuracy at tested widths above 1 and generally improves G-Pass, while width 1 is equivalent to independent generation.The results support generalization to widths both wider and thinner than the training width.Source: §4.3.1 · p. 9, Table 3 · p. 10, Figure 5 · p. 10

4.3.2 Generation Length

  • Beyond its 4,096-token training length, Bridge scales smoothly and usually beats the baselines; it has the highest accuracy across tested lengths and 6.0% more all-correct sets than the next best at 16K.The length experiment generates four responses per input.Source: §4.3.2 · p. 10, Figure 6 · p. 10

4.3.3 Cold Start vs. Warm up

  • SFT warm-up before RLVR improves Bridge performance, although applying RLVR directly from initialization is not catastrophic.The comparison is for DS-Qwen-7B.Source: §4.3.3 · p. 11, Table 4 · p. 11

4.3.4 Feature Contribution

  • Bridge blocks contribute relatively little to residual feature norms compared with P-Match despite their significant performance impact.The analysis compares block-output norms with corresponding residual norms across DS-Qwen-7B layers.Source: §4.3.4 · p. 11, Figure 7 · p. 11

4.3.5 Output Stability

  • Bridge slightly increases MATH-500 response similarity without collapsing output diversity, and produces the lowest variance in summarization quality across responses to the same prompt.The similarity measure is pair-wise BERTScore; summarization quality is measured with Rouge.Source: §4.3.5 · p. 11, Table 5 · p. 11

5 Conclusion

Bridge lets parallel generations for the same input share information during decoding. The paper reports gains in individual-response accuracy and response-set quality across multiple models and reasoning tasks.

5 Conclusion

  • Bridge improves both single-sample accuracy and set-wise quality across multiple models and reasoning tasks.It is an architectural addition that enables parallel generations for the same input to share information throughout decoding.Source: §5 · p. 12
  • The approach reinterprets hidden states in parallel scaling as higher-order tensors rather than disjoint slices.Source: §5 · p. 12
  • The paper identifies pretraining or mid-training, global post-training objectives, and other modalities and tasks as directions for future investigation.The paper notes that these settings may exhibit different levels of output homogeneity.Source: §5 · p. 12

A Training Hyperparameters

Appendix A identifies the hyperparameters used for RLVR and SFT.

A Training Hyperparameters

B Parameter Count Breakdown

Bridge adds 2.8% to 5.1% parameters to the original LLMs; Table 7 organizes model parameters by component.

B Parameter Count Breakdown

C Further GRPO Considerations

Bridge makes sequence rewards dependent, while permutation invariance keeps unconditional rewards identically distributed. The authors use a single-group setup and leave a more independent multi-group alternative for future work because it would be computationally expensive.

C Further GRPO Considerations

  • Bridge makes sequence rewards dependent, but permutation invariance keeps unconditional rewards identically distributed and preserves advantage unbiasedness.Source: C Further GRPO Considerations · p. 16
  • The authors do not use multiple Bridge groups per prompt to compute advantages between groups, because that would be computationally expensive.They report that their single-group setup is already empirically performative.Source: C Further GRPO Considerations · p. 16

D Bridge Placement

Bridge placement after attention blocks or feedforward blocks yields no significant MATH-500 accuracy difference, suggesting flexibility; the experiments use the placement with higher warmed-up performance.

D Bridge Placement

E Self-consistency

Bridge can be paired with post-generation methods such as self-consistency. Its pseudocode describes Bridge blocks and their masking, and the blocks can be extended to multiple heads.

E Self-consistency

F Bridge Pseudocode

  • The Bridge pseudocode computes Q, K, and V for each generation and constructs a mask for each token position.Source: F Bridge Pseudocode · p. 18
  • The mask allows interactions between generations with the same prompt that are incomplete at token s, and Bridge blocks can extend to multiple heads.Other generation pairs are masked with −∞.Source: F Bridge Pseudocode · p. 18

Related papers

Loading 2510.01143v3…