PaperlayerView exact revision on arXiv

Source-linked AI summary

Reactive Transformer (RxT) -- Stateful Real-Time Processing for Event-Driven Reactive Language Models

Adam Filipek

arXiv:2510.03561v1cs.CLcs.AIcs.LG

TL;DR

Conversational Transformers reprocess dialogue history at each turn, with costs that grow quadratically. RxT instead processes turns as events with integrated memory and asynchronous updates; the paper reports that a comparable-size RxT model dramatically outperforms a stateless baseline. The authors identify larger-scale evaluation and comparisons with leading efficient architectures as future work.

  • Problem

    Stateless conversational models reprocess the growing dialogue history at each turn, making long conversations costly and slow.

    Source: §1 · p. 1

  • Method

    RxT processes each query as an event, generates a response from the previous memory state, and asynchronously updates integrated memory from the interaction.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    The 26M RxT-Alpha Micro model dramatically outperforms a stateless baseline with a comparable parameter count.

    Source: §6.5 · p. 23

  • Takeaways & Limitations

    The authors conclude that RxT's asynchronous cycle and integrated memory lead to linear cost scaling and enable real-time interaction.

    Source: §7 · p. 24

    The authors identify scaling RxT to larger parameter counts, benchmarking on complex real-world datasets, and comparing it directly with leading efficient architectures as future work.

    Source: §7 · p. 24

Abstract

from arXiv · show

The Transformer architecture has become the de facto standard for Large Language Models (LLMs), demonstrating remarkable capabilities in language understanding and generation. However, its application in conversational AI is fundamentally constrained by its stateless nature and the quadratic computational complexity ($O(L^2)$) with respect to sequence length $L$. Current models emulate memory by reprocessing an ever-expanding conversation history with each turn, leading to prohibitive costs and latency in long dialogues. This paper introduces the Reactive Transformer (RxT), a novel architecture designed to overcome these limitations by shifting from a data-driven to an event-driven paradigm. RxT processes each conversational turn as a discrete event in real-time, maintaining context in an integrated, fixed-size Short-Term Memory (STM) system. The architecture features a distinct operational cycle where a generator-decoder produces a response based on the current query and the previous memory state, after which a memory-encoder and a dedicated Memory Attention network asynchronously update the STM with a representation of the complete interaction. This design fundamentally alters the scaling dynamics, reducing the total user-facing cost of a conversation from quadratic ($O(N^2 \cdot T)$) to linear ($O(N \cdot T)$) with respect to the number of interactions $N$. By decoupling response generation from memory updates, RxT achieves low latency, enabling truly real-time, stateful, and economically viable long-form conversations. We validated our architecture with a series of proof-of-concept experiments on synthetic data, demonstrating superior performance and constant-time inference latency compared to a baseline stateless model of comparable size.

1 Introduction

Conversational Transformers reprocess the full dialogue history at every turn, making long conversations costly and slow. The paper introduces RxT, which handles turns as events and updates an internal memory asynchronously.

1 Introduction

  • Full-history reprocessing makes conversational Transformers quadratically costly as dialogue turns accumulate, creating economic and latency problems for long-running dialogues.Increasing context windows postpone rather than solve the scaling problem, at escalating cost.Source: §1 · p. 1
  • RxT shifts conversational processing from a data-driven, stateless paradigm to event-driven, stateful computation.It processes each query against internal memory, generates a response, and then updates that memory asynchronously.Source: §1 · p. 2
  • A cyclical, asynchronous workflow separates user-facing response generation from internal memory consolidation to reduce perceived latency.The architecture redefines the encoder and decoder roles around these separate tasks.Source: §1 · p. 2
  • RxT maintains fixed-size internal memory and processes only the current interaction, so total conversation cost scales linearly with the number of turns.The paper presents this scaling as making long, coherent dialogues computationally feasible.Source: §1 · p. 2
  • The authors frame RxT as a step toward systems that maintain a persistent, evolving understanding of an interaction.The introduction compares that goal to the behavior of a true conversational partner.Source: §1 · p. 2

2 Background - the evolution of conversation modeling

The background traces the shift from recurrent sequence models to stateless Transformers and reviews memory-based alternatives. It positions RxT as an approach combining inter-sequence state with event-driven processing.

2 Background - the evolution of conversation modeling

  • The background frames RxT as a synthesis of prior concepts intended to address the challenges of maintaining sequence-model state and context.It locates that challenge within a history of conversation-modeling research.Source: §2 · p. 2

2.1 From Recurrence to Attention

  • RNNs maintain compressed hidden state across tokens but face vanishing gradients and sequential-processing bottlenecks; Transformers enable parallelism while losing inter-sequence statefulness.The Transformer replaced recurrence with self-attention for parallel processing and modeling long-distance relationships within a sequence.Source: §2.1 · p. 2

2.2 Augmenting Neural Networks with Memory

  • Neural Turing Machines and Differentiable Neural Computers paired neural controllers with addressable memory, but were difficult to train and not widely adopted.They learned memory read and write operations through differentiable attention.Source: §2.2 · p. 2

2.3 System-Level Memory for LLMs

  • System-level RAG and agent memory keep memory external to the LLM, adding text to prompts and retaining context-window and reprocessing limitations.The passage also characterizes their memory as limited in expressiveness and compression.Source: §2.3 · pp. 2–3

2.4 Alternative Architectures for Long Sequences

  • SSMs such as S4 and Mamba scale linearly or near-linearly with sequence length, but their state targets intra-sequence dependencies rather than persistent inter-sequence dialogue memory.The passage says dialogue use still relies on processing the full, growing history.Source: §2.4 · p. 3

2.5 Stateful Memory-Augmented Transformers

  • MemBART and Memformer use synchronous memory operations that add to response latency and update memory from input without the previous answer.Their encoder reads and writes memory before passing results to the decoder.Source: §2.5 · p. 3

2.6 Next generation of the stateful processing - Reactive Transformer

  • RxT is presented as natively stateful, parallel within interactions, integrated with implicit memory, and asynchronously updated using both query and answer.The authors contrast this with synchronous memory operations that contribute to user-perceived latency.Source: §2.6 · p. 3

3 The Reactive Transformer Architecture

RxT treats each query-response pair as a discrete interaction and maintains a multi-layer short-term memory. Its cyclical operation generates a response from prior memory, then encodes the completed interaction and consolidates memory asynchronously.

3 The Reactive Transformer Architecture

  • RxT combines a Generator-Decoder, Memory Encoder, and Memory Attention network to manage a multi-layer Short-Term Memory state over discrete query-response interactions.The architecture is designed around the Event-Driven AI paradigm rather than processing the whole conversation as one sequence.Source: §3 · p. 4

3.1 Architectural Components

  • The Generator-Decoder produces each response from the current query conditioned on the previous memory state through Memory Cross-Attention.Its layers use masked self-attention, Memory Cross-Attention, and a feed-forward network; decoder FFNs use Mixture-of-Experts layers.Source: §3 · p. 4, §3.1 · p. 4
  • After response generation, the Memory Encoder processes the concatenated query and response into Encoded Data representing the completed interaction.The encoder uses standard layers with self-attention and a dense feed-forward network.Source: §3 · p. 4, §3.1 · p. 4
  • The Memory Attention network combines previous memory with Encoded Data to produce the state carried into the next interaction.Its variants use memory self-attention, interlayer memory attention, or both, with residual gates controlling use of current and previous data.Source: §3 · p. 4, §3.1 · p. 4

3 The Reactive Transformer Architecture

  • Because memory updates occur after the response has been generated and streamed, that work does not block response generation.The stated effect is minimized user-perceived latency.Source: §3 · p. 4

3.1 Architectural Components

  • The encoder, decoder, and memory system share a single token embedding layer to operate in a unified vector space and reduce parameters.This is the architecture's shared-embeddings design.Source: §3.1 · p. 4

3.2 The Attention-Based Memory System (ABMS)

The Attention-Based Memory System uses fixed-size, learnable STM vectors organized by encoder and decoder layer. It reads context during generation and writes the completed interaction into the next memory state.

3.2 The Attention-Based Memory System (ABMS)

  • The STM consists of fixed-size, learnable memory slots organized into layers corresponding to encoder and decoder layers.Source: §3.2 · p. 4
  • During memory reads, decoder hidden states query the corresponding layer’s STM slots, which provide keys and values; positional encodings apply only to queries.Source: §3.2 · pp. 4, 10
  • During memory writes, previous STM slots query the interaction’s encoded data, and a residual connection, typically gate-controlled, produces the final memory state.Source: §3.2 · p. 10

3.3 Memory Attention Variants

The section presents variants for consolidating memory, ranging from direct retrieval from encoded interaction data to additional within- or across-layer attention and gated information flow.

3.3 Memory Attention Variants

  • Simple Memory Attention has the STM query the Encoded Data directly.Source: §3.3 · p. 10
  • Memory Self-Attention first lets previous-state memory slots attend to each other, enabling internal reorganization before they attend to Encoded Data.Source: §3.3 · p. 10
  • Interlayer Memory Attention lets slots in one layer attend to an aggregated representation of other STM layers, which can help reduce redundancy.Source: §3.3 · p. 10
  • Gated variants use a Residual Gate to control information flow from self-attention or interlayer-attention steps.Source: §3.3 · p. 10

3.4 Residual Gates

Residual Gates manage memory updates through a dynamic interpolation rather than simple addition, controlling memory plasticity and helping prevent catastrophic forgetting.

3.4 Residual Gates

  • The Residual Gate replaces simple addition with a dynamic interpolation to manage memory plasticity and prevent catastrophic forgetting.Source: §3.4 · p. 10
  • A small network computes gating vector G, typically with sigmoid activation, to weight the update and previous memory as a weighted average.Source: §3.4 · p. 10
  • Gating may be dynamic or a learned constant; using tanh or skipping the gate requires stronger regularization of memory updates during training.Source: §3.4 · p. 10

3.5 Comparison with Stateful Memory-Augmented Transformers

RxT differs from stateful Memory-Augmented Transformers (MATs) in its event handling, component responsibilities, and training design. It is built to extend decoder-only LLMs for conversational interaction rather than to modify encoder-decoder models for stateful sequence-to-sequence processing.

3.5 Comparison with Stateful Memory-Augmented Transformers

  • Earlier stateful models are reported to outperform comparable stateless models, especially in long multi-turn conversations, but RxT and those models began from different goals and baselines.Stateful MATs extend encoder-decoder Transformers; RxT targets stateful decoder-only LLMs for conversational tasks.Source: §3.5 · p. 10, §3.5.3 · p. 14

3.5.1 Compatibility with event-driven paradigm

  • RxT processes only the current query as input, whereas encoder-first stateful MATs process the previous answer together with the current query.RxT accesses memory in the decoder and updates it from the query-answer interaction; MATs access and update memory in the encoder based on the previous answer and current query.Source: §3.5.1 · p. 13

3.5.2 Single Responsibility Principle

  • RxT separates responsibilities: the decoder handles interaction dependencies and memory access, the encoder transforms the interaction, and Memory Attention updates memory.The encoder has no memory access or update responsibility, while Memory Attention can have multiple variants.Source: §3.5.2 · pp. 13–14

3.5.3 Design choices for architecture and training pipelines

  • RxT extends decoder-only LLMs for conversational interaction, while stateful MATs modify encoder-decoder models for stateful sequence-to-sequence processing.RxT was created from scratch and requires a redesigned training pipeline; the MAT approach retains a more slightly modified training pipeline.Source: §3.5.3 · p. 14

4 Inference and Computational Analysis

RxT generates each response from the current query and existing memory, then updates memory asynchronously. Its user-facing conversation cost scales linearly with the number of interactions, while standard Transformer LLM cost scales quadratically.

4.1 Inference Process

  • Before generation, RxT processes the query through the decoder to populate a self-attention KV cache, then generates the answer token by token using that cache and the memory cross-attention cache.The memory cache's key and value projections can be computed before the prompt phase because memory is static during a response.Source: §4.1 · p. 14
  • RxT generates a response from the current query and memory, then updates memory asynchronously from the completed query-answer interaction without adding user-perceived latency.The update has cost quadratic in fixed STM size, but does not affect user-perceived latency; the memory includes answers, preserving full dialogue context.Source: §4.1 · p. 15, §4.2 · p. 15

4.2 Computational Cost Analysis

  • Generation cost per token remains constant with conversation length because the self-attention cache holds only the current interaction and the memory cache has fixed size.Prompt processing is limited to the current short query and memory; standard LLM per-token generation cost grows linearly with dialogue length.Source: §4.2 · p. 15
  • O(N · T) user-facing cost for RxT contrasts with O(N^2 · T) for a standard Transformer LLM over N interactions.RxT processes roughly T tokens per interaction, while the standard LLM reprocesses the growing conversation history.Source: §4.2 · p. 15

5 Supervised Training Curriculum

The curriculum stages training to establish a shared language foundation, prepare the memory network, and then teach the full memory-dependent dialogue cycle. It is designed to address instability and cold-start problems before integrating the components.

5.1 Overview of the Training Pipeline

  • Four supervised stages progress from joint language pre-training and interaction fine-tuning, through memory pre-training, to full memory-aware training.The sequence is intended to build language capabilities first, prepare the memory system independently, and then integrate the complete conversational flow.Source: §5.1 · pp. 16–17

5.2 Stages 1 & 2: Joint Language Model Pre-Training and Fine-Tuning

  • Detached, lightly noised encoder states condition autoregressive decoding, while masked-language modeling and response prediction form the weighted joint objective.The noise regularizes decoder conditioning, reducing the risk of overreliance on the encoder’s context.Source: §5.2 · p. 17

5.3 Stage 3: Self-Supervised Memory Attention Pre-Training

  • The Memory Attention network learns from a dynamically weighted average of prior memory and new encoded data, with the new-data weight annealed across interactions.Negative cosine similarity encourages semantic alignment without requiring an exact match; pre-training aims to prevent noisy memory outputs from disrupting later training.Source: §5.3 · pp. 17–18

5.4 Stage 4: Supervised Memory-Aware Training

  • Multi-step dialogue training updates memory from each completed interaction and conditions the next response on that updated state.The final stage trains the decoder to use accumulated memory from past interactions; encoder and Memory Attention parameters may initially be frozen and gradually unfrozen.Source: §5.4 · p. 18

6 Experiments and Results

The experiments compare RxT variants with a similarly configured stateless baseline on multi-turn dialogue tasks, evaluating language modeling, conversational coherence, and prompt latency. Across these tests, the reported results favor RxT, while the authors describe the findings as a proof of concept for small-scale models.

6.1 Experimental Setup

  • RxT variants ranged from 12M to 160M parameters; TinyStories-derived multi-turn MRL datasets were used for fine-tuning and evaluation, and the 22M baseline shared the models’ SQA and decoder configuration.Conversational quality was assessed with a composite MRL reward score scaled from 0 to 10.Source: §6.1 · pp. 18–19

6.2 Performance on Memory-Aware Language Modeling

  • 2.74 PPL for RxT-Alpha Nano (12M) versus 4.37 for the 22M LLM baseline; all RxT variants outperform, with 160M Synthetic at 2.18.The updated Micro model’s PPL fell from 2.56 to 2.31 after training-pipeline improvements.Source: Table 3 · p. 19, §6.2 · p. 19, Figure 10 · p. 20

6.3 Conversational Coherence Evaluation

  • Mean MRL reward rose from 3.1 for Nano to 3.8 for Synthetic, and all RxT models outperformed the baseline on the 8+1-step dialogue benchmark.Larger RxT variants also had higher minimum reward scores than the LLM baseline, which the authors interpret as suggesting greater robustness.Source: Table 4 · p. 21, §6.3 · p. 21, Figure 12 · p. 22

6.4 Prompt Phase Latency in Memory-Based Dialogue Benchmark

  • RxT prompt latency stayed near 0.06s across dialogue steps, while the reference LLM increased from 0.09s at step 1 to over 0.22s at step 8.The benchmark covered up to eight conversational steps; the authors attribute the difference to RxT’s fixed-size memory mechanism.Source: §6.4 · pp. 21, 23, Figure 13 · p. 23

6.5 Analysis and Discussion

  • The authors attribute RxT’s comparable-size performance advantage to specialized memory components that offload long-term context management from the decoder.They characterize the results as a small-scale proof of concept and motivate evaluation on larger models and more complex, real-world datasets.Source: §6.5 · p. 23

7 Conclusion and Future Work

RxT’s asynchronous operational cycle and attention-based memory are presented as supporting linear cost scaling and real-time interaction. Future work includes training development, memory extensions, and broader benchmarking.

7 Conclusion and Future Work

  • RxT’s asynchronous operational cycle and attention-based memory are presented as leading to linear cost scaling and genuine real-time interaction.The conclusion frames this architecture as addressing computational-complexity and latency bottlenecks in dialogue applications.Source: §7 · p. 24
  • The current Short-Term Memory is a stepping stone toward models with persistent Long-Term Memory, which the authors associate with live learning and infinite context retention.The paper presents these memory capabilities as future direction, not as current RxT features.Source: §7 · p. 24
  • Supervised training is only the first stage of a larger curriculum, with memory reinforcement learning and reinforcement learning from human feedback planned for future work.Source: §7 · p. 24
  • Future work prioritizes scaling RxT to larger parameter counts, benchmarking on complex real-world datasets, and direct comparison with efficient architectures such as Mamba.Source: §7 · p. 24

License and Patent Notice

The notice states that the Reactive Transformer patent is pending and that commercial usage is regulated by a license.

License and Patent Notice

Related papers

Loading 2510.03561v1…