Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs

Guinan Su; Yanwu Yang; Xueyan Li; Jonas Geiping

Source-linked AI summary

Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs

Guinan Su, Yanwu Yang, Xueyan Li, Jonas Geiping

arXiv:2605.12460v1May 12, 2026cs.LGcs.CL

TL;DR

Language models remain constrained by sequential message-based interaction, preventing simultaneous reading, thinking, and acting. This paper introduces parallel token streams and reports substantially reduced latency, improved prompt-injection robustness, and a basis for monitoring parallel reasoning.

Problem
Single-stream, sequential message formats force each user, system, thought, and tool exchange to finish before another begins, limiting overlapping interaction.
Method
The paper fine-tunes language models to attend over and generate multiple causally interdependent token streams in each forward pass.
Results
Large reductions in time-to-first-token and end-to-end latency are achieved while task performance is largely preserved, alongside improved prompt-injection robustness and monitoring support.
Takeaways & Limitations
Parallel streams offer an alternative interaction paradigm for language models that supports lower latency, stronger robustness, and more legible parallel reasoning.
Takeaways & Limitations
The models are relatively small and trained on tiny amounts of instruction data compared with modern instruction-tuning pipelines, so larger-scale studies remain needed.

Abstract

from arXiv · show

The continued improvements in language model capability have unlocked their widespread use as drivers of autonomous agents, for example in coding or computer use applications. However, the core of these systems has not changed much since early instruction-tuned models like ChatGPT. Even advanced AI agents function on message exchange formats, successively exchanging messages with users, systems, with itself (i.e. chain-of-thought) and tools in a single stream of computation. This bottleneck to a single stream in chat models leads to a number of limitations: the agent cannot act (generate output) while reading, and in reverse, cannot react to new information while writing. Similarly, the agent cannot act while thinking and cannot think while reading or acting on information. In this work, we show that models can be unblocked by switching from instruction-tuning for sequential message formats to instruction-tuning for multiple, parallel streams of computation, splitting each role into a separate stream. Every forward pass of the language model then simultaneously reads from multiple input streams and generates tokens in multiple output streams, all of which causally depend on earlier timesteps. We argue that this data-driven change remedies a number of usability limitations as outlined above, improves model efficiency through parallelization, improves model security through better separation of concerns and can further improve model monitorability.

1 Introduction

Modern intelligent systems remain constrained by chat models’ single sequential text stream, which prevents reading, thinking, and acting from overlapping. The paper proposes multi-stream instruction-tuning, enabling parallel token inputs and outputs while improving efficiency, security, and monitorability.

Motivation: LLM-based agents, assistants, and orchestrators still process and generate a single sequential text stream inherited from instruction-tuned chat models.Standard chat templating encodes user and model roles sequentially, while chain-of-thought and tool use are retrofitted into the same format.
Motivation: Single-stream chat models cannot read, think, or act concurrently, must finish consuming inputs before responding, and cannot ingest new information mid-generation without interruption.Long-running tool calls, thinking blocks, and subagent communications are funneled through the same sequential stream.
Approach: Multi-stream instruction-tuning separates user, system, model, and thinking roles into parallel streams with interdependent attention.Each forward pass reads multiple input streams and predicts tokens across multiple output streams; the paper also provides recipes for converting message data and generating stream data.
Results: Overlapping reading, thinking, and acting produces large reductions in time-to-first-token and end-to-end latency while largely preserving task performance.The approach is designed for memory-bound inference and reports that finetuning for the multi-stream format is not harder than standard instruction-tuning.
Results: Explicit stream separation strengthens prompt-injection robustness by structurally distinguishing input content from the model’s own generation.The cleaner separation of concerns provides a stronger signal for identifying which content is input versus generated.
Results: Additional internal streams make model awareness and intention easier to monitor by exposing considerations that may not appear in user-facing messages or functional chain-of-thought.The model can sub-vocalize these considerations in dedicated internal streams.

2 The Advantages of Multiple Parallel Streams

Multiple parallel streams let language models overlap roles rather than follow strictly sequential messages, enabling more natural interaction, interruptions, and lower-latency parallel actions. The format predicts all streams in a row with one forward pass, supporting efficient multi-stream execution.

Format: Parallel streams separate roles into columns and predict each row jointly, allowing role overlap while avoiding format tokens.Each row is one forward pass over prior rows, with each column representing a separate role.
Simultaneous Speech: Overlapping streams unblock interactions between roles, enabling more fluent conversations that resemble natural turn-taking with frequent overlap.The benefit applies even to a simple user-model setup.
Interrupting Users: Parallel streams let models interrupt users while inputs stream in, rather than waiting for the complete message before thinking and responding.The example also illustrates concurrent model thinking during user input.
Gains in Efficiency through Parallel Streaming: Parallel action streams reduce latency by reading, searching, checking, and formulating an answer while the user continues describing their ideas.With 5 streams, the model predicts the entire row in one forward pass.

3 Method

The method trains and runs language models over multiple causally linked streams rather than isolated parallel branches. It combines synthetic causal data construction with stream-aware Transformer modifications and synchronous interleaved inference.

Multi-Stream Parallel Generation: Multi-Stream Parallel Generation produces H token sequences concurrently, with each stream autoregressive over its own history and able to attend to other streams’ earlier positions.This preserves intra-stream and cross-stream causality while distinguishing the method from fully isolated parallel-reasoning streams.
Data Construction: Training samples are synthesized through stream-like generation, causal verification, and quality filtering.Judges discard chunks informed by future user tokens, while per-stream and cross-stream checks assess fluency, redundancy, completeness, and role fulfillment.
Architecture: The decoder-only Transformer is adapted with per-stream position encoding and a cross-stream causal attention mask.Each stream has its own RoPE position counter and learnable stream embedding; the mask permits attention only to positions strictly earlier than the query time.
Implementation: Interleaved packing preserves the intended attention connectivity while arranging tokens into the contiguous lower-triangular structure favored by standard causal traversal.The method uses interleaved rather than straightforward end-to-end stream concatenation for efficient implementation of the structured mask.
Inference: At inference, one forward pass synchronously emits one token per stream, with latency determined by the longest stream and a theoretical H× speedup over sequential decoding.Empty slots use masked ‘-’ tokens and allocate no KV-cache entries, incurring zero memory overhead.

4 Efficiency: Reduced Latency via Parallel Streaming

This section investigates whether overlapping sequential reasoning stages through parallel streams and cross-stream access can reduce latency without sacrificing quality. Multi-stream models run solver and auditor streams concurrently with incoming input, targeting lower first-token and overall latency while preserving accuracy.

Approach: Parallel streams overlap sequential reasoning stages with cross-stream access instead of merging fully isolated branches only at the end.The approach is evaluated across two settings with increasing parallelism.
Evaluation: The evaluation reports accuracy, Token Number to First Target Token, generated tokens, end-to-end latency, and Maximum independent proxy for responsiveness across reasoning and QA benchmarks.Experiments use Qwen3-1.7B and Qwen3-4B in Base, Vanilla, and Ours settings.
Mechanism: Multi-stream LLMs run solver and auditor streams concurrently with incoming input, reducing Token Number to First Target Token and overall latency.Vanilla LLMs wait for the complete input before responding.
Results: The method maintains comparable accuracy while improving efficiency across benchmarks, with gains scaling with the degree of parallelism.The reported efficiency measures include first-target-token timing, generated tokens, and end-to-end delay.

5 Security: Separation of Concerns through Stream Separation

Stream isolation separates system, user, and assistant computation to improve robustness against direct and indirect prompt injection without adversarial training. The multi-stream models generally preserve or improve safety-helpfulness and instruction-following performance, though Gandalf remains difficult.

Experimental setup: The study compares Vanilla single-stream and Stream multi-stream variants of Qwen2.5-7B-base and Qwen3-4B-base trained for three epochs on matched Alpaca-derived data.Vanilla concatenates system prompt, user input, and assistant response, while Stream uses separate streams.
Prompt-injection robustness: Stream lowers attack success rates on nearly every direct- and indirect-injection benchmark at both model scales, without adversarial training.The largest indirect-injection improvement is StruQ-ID, which drops by more than 33 ASR points for both models.
Safety-helpfulness: NESSiE’s combined Safe & Helpful score rises substantially, indicating better discrimination between helpful and unsafe requests rather than simply more refusals.The result supports separation of concerns through architecturally isolated input streams.
Limitations: Gandalf remains difficult for both variants because the attacker can iterate freely against a single hidden secret.This benchmark is the stated exception to the broader safety improvements.
General capability: Stream performs on par with or better than Vanilla across all instruction-following metrics, showing that the safety gains do not degrade general capability.Table 3 summarizes higher-is-better safe-and-helpful and instruction-following performance alongside lower-is-better attack success rate.

6 Monitorability: Legible Parallel Reasoning via Internal Streams

Parallel internal streams are designed to make a model’s considerations legible to outside monitors, including concerns omitted from its visible answer. In evaluations, stream models were substantially more monitorable than their base models across four metrics.

Motivation: Parallel internal streams expose considerations that may be omitted from the visible answer, supporting direct monitoring of the model’s reasoning.The motivation includes surfacing unraised medical concerns and awareness that the model is being tested.
Experimental setup: The experiment finetunes Qwen3.5-27B into a 10-stream model with eight role-assigned internal thinking streams and trains an 8B counterpart for comparison.Training uses 3,864 synthetic 10-stream conversations generated by frontier models in tabular form to preserve causality between streams.
Results: Stream models are substantially more monitorable than their bases on all four evaluated metrics.Evaluations use an LLM judge reading either the visible output or internal reasoning across eval-awareness, monitoring accuracy, and concern sub-vocalization settings.
Results: 0.42 concern sub-vocalization at 8B and 0.44 at 27B rise from 0.07 and 0.06, respectively, in the stream models.Concern sub-vocalization measures everyday pressure scenarios where an objective concern is raised internally but absent from the visible answer; higher is better.
Results: The internal streams enable outside readers both to classify the model’s behavior and to recover concerns hidden by the visible answer.The authors state that the internal streams are not merely decorative and report that monitor accuracy nearly doubles for the 27B model.

7 Discussion … B.2 Data Construction

The paper argues that replacing single-stream, message-based interaction with parallel streams enables models to read, think, act, and respond concurrently. It presents this format as a promising paradigm for improving latency, security, monitorability, and agentic interaction, while constructing training data through synthetic generation, causal verification, and quality filtering.

7 Discussion: Parallel streams address the limitations of sequential message-based agents by enabling concurrent reading, thinking, acting, and responding.The single-stream paradigm forces models to read, then think, then respond, preventing intervention during long tasks and limiting autonomous interaction.
7 Discussion: The finetuned models reduce latency, improve robustness to prompt injection, and support monitoring through legible parallel reasoning.The authors report these benefits even at the relatively small scales explored in the work.
A Impact Statement: The proposed instruction-tuning change may improve deployed-system efficiency, security, and monitorability through reduced latency, stream separation, and internal oversight streams.The impact statement also notes potential misuse as a general possibility accompanying advances in LLM efficiency.
B.1 Related Work: Prior instruction-tuning research progressed from instruction-following demonstrations and zero-shot generalization to RLHF, synthetic data, self-critique, and preference optimization.These developments provide the broader training context for the paper’s proposed instruction-tuning format.
B.1.1 Detailed Comparison to Multiverse: The framework differs from Multiverse by applying densely connected parallelization across all model roles rather than only branching independent thinking channels.It uses a fixed set of streams, making the format a more constrained version of free-form branching structures.
B.1.2 Detailed Comparison to StreamingThinker: Unlike StreamingThinker’s sentence-boundary splitting and message-based global reasoning, this work applies an always-streaming approach to all input and output channels.StreamingThinker overlaps reading and thinking but decodes only the reasoning stream at each step before returning to message-based reasoning and reflection.
B.2 Data Construction: Because naturally occurring simultaneous data is scarce and expensive, the authors synthesize training samples through stream-like generation, causal verification, and quality filtering.The pipeline is designed specifically to construct multi-streaming training data.
B.2 Data Construction: The data pipeline generates wait-k-style samples with system, user, and assistant streams, verifies temporal causality, and filters malformed, incoherent, redundant, or repetitive content.Assistant streams begin with bridging utterances while user input is still arriving; target chunks depend only on the source prefix available at that point, and causal checks tokenize all streams with the target tokenizer.

B.3 Additional Examples · B.4 Detailed Experimental Settings

The additional examples illustrate how parallel streams support mid-response redirection, concurrent auditing, prompt-injection defense, monitorable thinking, and multiple actions within one forward pass. They also show these interactions in concrete token-level layouts and a longer Stream-27B generation.

B.3 Additional Examples: B.3 Additional Examples: Token-level layouts represent each forward pass as rows and separate input or output streams as columns, with “-” marking idle slots.These layouts motivate the multi-stream interaction format through illustrative examples.
B.3 Additional Examples: B.3 Additional Examples: Parallel user and model streams let the model observe a mid-response redirect and switch explanations without finishing its previous sentence.In the gradient-descent example, the user requests a simple, nonmathematical explanation while the technical answer is underway.
B.3 Additional Examples: B.3 Additional Examples: A dedicated Audit stream can consume generated code in real time while a Think stream coordinates auditing, exposing security issues before completion.The example flags raw SQL injection risk and missing rate limits on failed logins, unlike sequential auditing after output completion.
B.3 Additional Examples: B.3 Additional Examples: Separating untrusted documents from user instructions lets the thinking stream identify prompt injection and summarize legitimate content instead of treating both as one stream.The example ignores an injected instruction and reports that revenue grew 15% while profit fell 3%.
B.3 Additional Examples: B.3 Additional Examples: Six concurrent thinking streams can externalize otherwise hidden considerations, including the “genuine or test” eval-awareness signal detectable by a monitor.Figure 7 depicts streams reacting to “How old are you?” and one stream identifying the question as self-referential.
B.3 Additional Examples: B.3 Additional Examples: The format can dispatch an email draft, calendar entry, and ticket creation concurrently in one forward pass while a Think stream coordinates the actions.The request is to schedule a design review Friday with the team.
B.3 Additional Examples: B.3 Additional Examples: In a longer Stream-27B generation, four thinking streams register a user’s redirect from a caffeine-and-sleep essay to a cat haiku and switch tasks.The model abandons the essay, acknowledges the change, and begins writing the haiku.

B.4.1 Datasets.

The evaluation spans mathematical reasoning, logical inference, biomedical and reading comprehension, instruction following, and prompt-injection safety. Training data are constructed from several source datasets, often with reduced subsets, while evaluations use established benchmarks and held-out test splits.

Mathematical reasoning: GSM8K contains 8,500 human-written grade-school math word problems requiring 2–8 basic-arithmetic steps.Its split includes 7,473 training problems and 1,319 test problems, with natural-language solutions.
Mathematical reasoning: MATH-500 evaluates mathematical reasoning on 500 challenging high-school competition problems across five core mathematical domains.Its problems are sourced from competitions including AMC and AIME.
Mathematical reasoning: MetaMathQA augments GSM8K and MATH training data with 40k samples, yielding approximately 8k final training samples for GSM8K and MATH500 evaluation.The dataset combines symbolic reasoning with natural-language explanations for multi-step arithmetic and algebraic tasks.
Logical reasoning: ProofWriter tests multi-step logical reasoning and theorem proving from natural-language hypotheses and supporting facts, using approximately 25k training samples after sampling 1k test instances.The task may require determining entailment, contradiction, or neutrality and optionally generating an explicit reasoning chain.

B.4.2 Evaluation Metrics. · B.4.3 Training Hyperparameters. · B.5 Extra Training Details: Stream-Contrastive Training.

The paper evaluates task performance, responsiveness, generation length, security, safety, and instruction following using complementary metrics, while recording training hyperparameters in Table 5. Stream-contrastive training upweights tokens whose predictions benefit most from cross-stream context, using probability shifts between complete and single-stream contexts with normalization and capped weights.

B.4.2 Evaluation Metrics.: Accuracy is the primary task-performance metric, measuring the percentage of reasoning and question-answering instances answered correctly.
B.4.2 Evaluation Metrics.: TNFT counts tokens generated before the first target output token, while Delay measures real-time latency from the last input token to the first answer token.TNFT is reported instead of wall-clock latency because interface speed and prefill computation affect time-to-first-token; lower Delay indicates greater responsiveness.
B.4.2 Evaluation Metrics.: MSL is the longest individual stream, MSL = max_h |y(h)|, and directly proxies forward passes waited for because streams decode synchronously.Tokens instead counts total tokens emitted across streams and reflects compute work rather than the relevant multi-stream latency proxy.
B.4.2 Evaluation Metrics.: Security evaluation reports Direct Attack Success Rate and Indirect Attack Success Rate, measuring successful direct prompt overrides and indirect behavior hijacking, respectively.Lower Direct Attack Success Rate indicates stronger robustness, while indirect attacks embed adversarial instructions in input data rather than directly targeting the system prompt.
B.4.2 Evaluation Metrics.: Instruction Following uses IFEval’s prompt-level and instruction-level strict and loose accuracy metrics.Strict metrics require exact compliance; loose metrics apply response transformations to reduce false negatives before checking compliance.
B.4.3 Training Hyperparameters.: Training hyperparameters for all experiments are recorded in Table 5.
B.5 Extra Training Details: Stream-Contrastive Training.: Stream-contrastive training upweights tokens benefiting most from cross-stream context by contrasting complete-context predictions with gradient-free single-stream predictions.The log-probability shift measures cross-stream dependence; weights undergo per-stream mean normalization, and γ caps extreme values for stability.

B.5.1 Additional results on efficiency

Parallel auditing improves streaming efficiency while preserving comparable accuracy. On MATH500, it eliminates TNFT and reduces MSL, though gains are mainly due to format correction and may not generalize to deep global reasoning.

Real-Time Latency Reduction in Streaming Inference: 1.63× throughput speedup over sequential solving-then-auditing is achieved when a third audit stream monitors input and solution in real time, with comparable accuracy.This measures end-to-end efficiency in streaming inference.
Auditing While Solving on MATH500: On MATH500, Auditing While Solving reduces TNFT to zero and MSL by roughly 20% relative to Vanilla + Reflection across both model scales.The setting also produces modest accuracy gains.
Auditing While Solving on MATH500: The MATH500 improvements are largely attributable to format correction rather than substantive reflection, making parallel auditing potentially less suitable for tasks requiring deep global reasoning.This limitation concerns the form of parallel auditing evaluated on MATH500.

B.5.2 Ablations: What Is the Best Way to Encode Position Across Streams? · B.6 Security Experiment Details

The ablation compares positional encoding strategies for multi-stream inputs, where concurrent streams create attention and positional conflicts. The paper reports that its method outperforms alternatives across four IFEval metrics, while B.6 expands the security experiments’ setup and capability-check details.

B.5.2 Ablations: What Is the Best Way to Encode Position Across Streams?: Concurrent interleaving creates attention contention because tokens from different streams compete for shared softmax-normalized attention capacity.This competition makes cross-stream dependency ordering ambiguous and dilutes attention to relevant within-stream context.
B.5.2 Ablations: What Is the Best Way to Encode Position Across Streams?: Positional conflicts arise when tokens from multiple asynchronous streams require positional representations that preserve their distinct structures.
B.5.2 Ablations: What Is the Best Way to Encode Position Across Streams?: 2D RoPE encodes intra-stream position and stream identity on separate axes, with Axial and Mixed variants controlled by frequency scaling α.Axial applies independent rotary embeddings, whereas Mixed blends both axes into one rotation.
B.5.2 Ablations: What Is the Best Way to Encode Position Across Streams?: Offset separates streams by shifting non-primary positional indices by a fixed constant d, while Rotate creates stream-specific positional subspaces through angular rotation.
B.5.2 Ablations: What Is the Best Way to Encode Position Across Streams?: NoPE removes positional embeddings after training and relies solely on causal attention for implicit position awareness.
B.5.2 Ablations: What Is the Best Way to Encode Position Across Streams?: Our method consistently outperforms all positional-encoding alternatives across all four IFEval metrics.Offset with d = 128 and Rotate are competitive on instruction-level metrics but weaker on prompt-level accuracy; Mixed at α = 1 degrades notably, while NoPE lacks full cross-stream expressiveness.
B.6 Security Experiment Details: B.6 expands Section 5’s experimental description by detailing the data, per-benchmark setups, and IFEval capability check.

B.6.1 Data and training

The study reconstructs Alpaca into a multi-stream format using Qwen3-Next-80B for data construction and verification, while deriving the Vanilla baseline by collapsing the same data into one stream. Both settings use matched training conditions without adversarial or prior instruction/safety tuning, isolating architectural effects.

Data and training: Alpaca is reconstructed into the multi-stream format with Qwen3-Next-80B serving as the backbone for data construction and quality/causality verification.The reconstruction follows Section 3.2.
Data and training: The Vanilla baseline uses the same data collapsed into a single-stream sequence, so no separate baseline dataset is required.The collapsed multi-stream format remains a valid causal language-modeling input.
Data and training: Both settings omit adversarial training and use identical optimizers, schedules, step budgets, and three training epochs.These controls are intended to isolate the safety effect of the architecture itself.
Data and training: Training starts from pretrained base models rather than instruction- or safety-tuned variants, preventing safety-aligned post-training from confounding the comparison.The comparison therefore avoids introducing prior instruction or safety alignment through model initialization.

B.6.2 Benchmark details … B.8 Instruct templates of Dataset Building.

The appendix specifies benchmark protocols, training procedures, and monitorability evaluations for parallel-stream models, then provides wait-k streaming instructions for reconstructing the Alpaca dataset. It also defines how stream and chat models are normalized for judging and how stream roles, architecture handling, and evaluation scenarios are constructed.

B.6.2 Benchmark details: All benchmarks use sampling temperature 0.7, maximum sequence length 1024, and three random seeds.
B.6.2 Benchmark details: The benchmark suite covers indirect prompt injection, Safe & Helpful behavior, and instruction following with four IFEval accuracy metrics.Indirect prompt injection evaluates four attack variants across in-domain and other settings; NESSiE reports a combined Safe & Helpful score.
B.7 Monitorability Experiment Details: Monitorability experiments use conversational stream models trained on Qwen3-8B and Qwen3.5-27B and summarize three evaluations.
B.7.1 Training data: The stress test generates 3,864 synthetic 10-stream conversations, averaging about 100 rows with 10 tokens per parallel-generation step.Eight internal streams receive dedicated roles including analytical thinking, checking, intuitive thinking, curiosity, and synthesis.
B.7.2 Training procedure: Qwen3.5-27B is finetuned for 2 epochs on 4×B200 GPUs with learning rate 2×10^-5, weight decay 10^-3, attention dropout 0.2, masked user-column loss, and random-concatenation augmentation.Hybrid attention and Gated DeltaNet architectures require separate parallelization handling, while shared parameters and per-stream channel embeddings identify streams.
B.7.3 Evaluation methodology: All three evaluations use gpt-5.4-mini as judge after stream reasoning and chat thinking traces are converted into comparable named-block formats.
B.7.3 Evaluation methodology: The evaluations measure alignment-faking awareness and sub-vocalization, monitor-as-classifier accuracy over six covert-behavior labels, and concern sub-vocalization across 50 pressure scenarios and 56 concerns.The scenarios span health, financial, relationship, safety, legal, emotional, and parenting categories.
B.8 Instruct templates of Dataset Building.: The wait-k dataset template requires the streaming assistant to begin after the first k user tokens, repeat the bridging utterance verbatim, and extend responses token by token without anticipating unseen input.The final response must retain key content and be returned as a single JSON object containing bridging and response fields.