PaperlayerView exact revision on arXiv

Source-linked AI summary

HeteroReason: Heterogeneous FPGA-GPU Acceleration for Disaggregated Speculative Reasoning

Zehuan Zhang, Quan Deng, Zibo Ren, Hao Mark Chen, Guoyu Li, Xuchun Hu, Jose G. F. Coutinho, Ce Guo, Wayne Luk, Zhiqiang Que, Hongxiang Fan

arXiv:2609.28717v1cs.AR

TL;DR

Forward-only speculative reasoning can leave suboptimal steps unrecoverable, while heterogeneous drafting and verification workloads challenge homogeneous GPU execution. HeteroReason combines backtracking, FPGA-GPU placement, shadow synchronization, and step-ahead scheduling; it reports a 1.01×–1.42× latency reduction and a +4.2% average accuracy improvement in Config 2 with a 1.5B target model.

  • Problem

    Forward-only speculative reasoning lacks recovery from suboptimal early steps, and homogeneous platforms struggle with the disparate workloads of drafting, verification, and refinement.

    Source: §I · p. 2

  • Method

    HeteroReason combines backtracking with FPGA drafting, GPU-based PRM and target models, shadow synchronization, and step-ahead speculation and refinement.

    Source: §I · p. 2

  • Results

    HeteroReason reports a 1.01×–1.42× latency reduction across diverse reasoning benchmarks and a +4.2% average improvement for Config 2 with a 1.5B target model.

    Source: B. Algorithmic Experimental Results · p. 9, §IX · p. 12

  • Takeaways & Limitations

    Backtracking's accuracy gain is more pronounced with a 1.5B target model than with a 7B target model: +4.2% versus +1.7%.

    Source: B. Algorithmic Experimental Results · p. 9

    Some results combine simulation and on-board evaluation, and broader evaluation across model architectures remains needed to demonstrate generalizability.

    Source: §VIII · p. 12

Abstract

from arXiv · show

Large Reasoning Models (LRMs) have achieved state-of-the-art performance in reasoning tasks by utilizing Chain-of-Thought (CoT) reasoning. To achieve fast execution speed, speculative reasoning techniques adopt a lightweight draft model for candidate token generation followed by process reward models (PRMs) for verification and a strong target model for refinements. This paper identifies that the existing speculative reasoning paradigm follows a strictly forward-only reasoning trajectory, which lacks robustness and can lead to severe error propagation if early reasoning steps are suboptimal. Furthermore, executing these disparate inference schemes, including sequential drafting and parallel verification on homogeneous GPU platforms, can lead to severe resource underutilization. To address this, we propose HeteroReason, an algorithm-hardware co-designed heterogeneous FPGA-GPU inference paradigm specifically tailored for LRM speculative reasoning. At the algorithmic level, we introduce a backtracking-enhanced workflow that enables the system to recover from low-quality states and explore alternative reasoning trajectories, significantly improving reasoning robustness. At the system level, the draft model is offloaded to the FPGA while deploying the PRM and target models on GPUs. A specialized workflow is optimized to achieve prefill-decode disaggregation, which exploits shadow synchronization to overlap GPU-side refinements with FPGA-side token updates to effectively hide synchronization latency. To mitigate inherent sequential constraints, we propose a step-ahead speculation and refinement scheduling scheme, transitioning the system from a sequential execution scheme to a parallel pipeline. Experimental evaluations show an average 4.2% accuracy improvement, with 1.01x-1.42x latency speedups and 1.25x-1.57x improvements in energy efficiency compared to homogeneous GPU baselines.

I. INTRODUCTION

Speculative reasoning can accelerate long LRM reasoning, but forward-only trajectories and mismatched workloads on homogeneous GPUs leave robustness and efficiency challenges. HeteroReason combines backtracking, FPGA-GPU execution, and step-ahead scheduling to address them.

II. BACKGROUND AND MOTIVATION

  • LRMs use Chain-of-Thought reasoning, while speculative reasoning uses a lightweight draft model to generate candidates for PRM or target-model verification and refinement.The target confirms or adjusts draft candidates with lower invocation frequency.Source: §I · p. 1

I. INTRODUCTION

  • Existing speculative reasoning cannot recover from suboptimal early steps, while heterogeneous model workloads and sequential stages constrain robustness and execution efficiency.Drafting is sequential; PRM verification is parallel, and target refinements are invoked conditionally.Source: §I · pp. 1–2
  • HeteroReason adds backtracking so the system can recover from suboptimal early generations and switch to alternative reasoning trajectories.The paper presents this as improving robustness and overall accuracy.Source: §I · p. 2
  • The system places the draft model on an FPGA and the PRM and target models on GPUs, using shadow synchronization for prefill-decode disaggregation.Shadow synchronization overlaps GPU refinements with FPGA-side draft-state updates.Source: §I · p. 2
  • Step-ahead speculation and refinement overlaps subsequent drafting with current-step verification, shifting sequential execution toward a parallel pipeline.The scheme is integrated with system optimizations for GPU and FPGA architectures.Source: §I · p. 2

A. Background

LRMs use extended reasoning chains, and speculative reasoning seeks to reduce their inference latency by drafting and verifying intermediate steps. The workflow invokes the draft model and PRM each step, while target-model refinements depend on reward scores.

1) Large Reasoning Models (LRMs):

  • LRMs use intermediate reasoning steps to solve complex problems, increasing computational cost and latency compared with direct response generation.The supplied background describes LRMs as using deliberate reasoning before producing an answer.Source: 1) Large Reasoning Models (LRMs) · pp. 2–3

2) Speculative Reasoning:

  • Speculative reasoning drafts intermediate steps and uses semantic alignment rather than exact token reproduction, with a PRM or target model assessing, confirming, or refining them.The passage also states that speculative reasoning can complement speculative decoding.Source: 2) Speculative Reasoning · p. 3

B. Algorithmic Workflow and Analysis

  • The draft model proposes a reasoning step and the PRM scores it; low scores trigger target-model refinement, while accepted steps update the context.The process repeats until completion.Source: B. Algorithmic Workflow and Analysis · p. 3
  • Draft and target account for 34.6%–55.4% and 22.4%–47.5% of step latency, versus PRM's 16.1%–22.2%.The profiling covers four reasoning datasets; frequent unconditional draft invocations require both prefill and autoregressive decoding.Source: B. Algorithmic Workflow and Analysis · p. 4

C. Motivation for Heterogeneous System

  • The section identifies homogeneous GPU clusters as a challenge for stepwise speculative reasoning's distinct computational paradigm.The following motivation examines heterogeneous execution.Source: C. Motivation for Heterogeneous System · p. 4

1) Inherent Workload Asymmetry:

Speculative reasoning combines sequential, memory-intensive drafting with parallel, compute-intensive verification, while refinements are score-dependent. The section motivates hardware specialization, step-level communication, and recovery from low-quality reasoning paths.

1) Inherent Workload Asymmetry:

  • Drafting is sequential and memory-intensive, PRM verification is parallel and compute-intensive, and target refinements are parameter-heavy but score-dependent.These asymmetric workloads challenge homogeneous GPU platforms.Source: 1) Inherent Workload Asymmetry · p. 4

2) Distinct Hardware Characteristics:

  • FPGAs can be customized for the small draft model's memory-bandwidth-bound autoregressive decoding, while GPUs suit parallel, arithmetic-intensive workloads.The passage contrasts FPGA energy efficiency and reconfigurability with GPUs' high throughput and power consumption.Source: 2) Distinct Hardware Characteristics · p. 4

3) Amortized Communication:

  • Step-level transfers amortize communication overhead because reasoning steps typically contain fewer than 200 tokens and PCIe transfer latency remains at microsecond scale.The passage contrasts this with frequent token-level communication.Source: 3) Amortized Communication · p. 4

D. Challenge Analysis

  • The section identifies missing error recovery and dedicated hardware as challenges for efficient speculative inference.It frames the need as spanning algorithm design and hardware support.Source: D. Challenge Analysis · p. 4

1) Robustness Bottlenecks in Forward-Only Reasoning Trajectory:

  • A low-reward draft and a still-low-reward target refinement can leave the forward-only workflow forced to commit a suboptimal step.Because reasoning steps are interdependent, committed suboptimal steps can propagate errors through the chain, and the vanilla workflow cannot change earlier steps.Source: 1) Robustness Bottlenecks in… · pp. 4–5

2) Intra-Device Prefill-Decode Interleaving of the Draft Model:

The draft model’s repeated local prefill and the serial FPGA–GPU workflow create design challenges and inter-device idle time. HeteroReason separates drafting from GPU-side prefill and verification, while shadow synchronization and backtracking support state updates and recovery.

2) Intra-Device Prefill-Decode Interleaving of the Draft Model:

  • Each reasoning step requires a local draft-model prefill, interleaving operations with distinct computational characteristics.This interleaving poses hardware design challenges.Source: 2) Intra-Device Prefill-Decode… · p. 5

3) Inherent Sequential Dependencies Limiting Pipeline Parallelism and Resource Utilization:

  • Serial FPGA drafting followed by GPU verification and refinement leaves each device idle while waiting for the other, adding execution and communication times to iteration latency.The proposed pipeline aims to overlap drafting with verification and refinements.Source: 3) Inherent Sequential Dependencies… · p. 5

A. System Overview

  • The system places the draft model on an FPGA and the PRM and target model on separate GPUs, exchanging states, speculative sequences, and feedback over PCIe.The FPGA is optimized to minimize draft-decoding per-token latency.Source: A. System Overview · p. 5

B. Prefill-Decode Disaggregation Paradigm

  • The workflow has prefilling and state migration, speculation and verification, and bifurcated continued decoding phases; accepted tokens continue drafting, while rejected tokens trigger target-model refinement.The initial draft KV cache is transferred to FPGA HBM once per reasoning task; the one-time transfer latency is described as negligible relative to total generation time.Source: B. Prefill-Decode Disaggregation… · p. 6
  • Shadow synchronization overlaps GPU refinement with FPGA KV-cache updates, while backtracking searches backward for a previously draft-accepted anchor within K steps and caps attempts at five per problem.The workflow dedicates FPGA decoding to drafting and GPU work to parallel prefilling and verification, allowing independent scaling of drafting, verification, and refinements.Source: §III · p. 5, B. Prefill-Decode Disaggregation… · p. 6

V. STEP-AHEAD SPECULATION AND REFINEMENT

HeteroReason advances drafting and target-model refinement while current speculative tokens are being verified. Asynchronous target prefetching and cached partial outputs support this overlap across acceptance, refinement, and backtracking scenarios.

V. STEP-AHEAD SPECULATION AND REFINEMENT

  • After submitting current speculative tokens for PRM verification, the FPGA draft model and GPU target model decode the subsequent step concurrently.This step-ahead strategy uses idle periods awaiting verification to move sequential execution toward a parallel pipeline.Source: §V · p. 6

A. Asynchronous Target Prefetching & Caching

  • The target model prefetches the next potential reasoning step before verification finishes, and partial outputs without KV cache are reused when later contexts share prefixes.The target may produce only a partial generation because it is typically slower than the draft model.Source: A. Asynchronous Target Prefetching &… · p. 7

B. Scheduling

  • When the PRM accepts a step, step-ahead drafting proceeds while GPU refinement continues, with per-step latency set by the maximum of drafting and verification time plus communication overhead.Source: B. Scheduling · p. 7
  • On rejection, speculative tokens are discarded and refined tokens update the FPGA cache through overlapped shadow synchronization; if refinement fails verification, backtracking reuses cached partial target outputs.The scheduling scheme reports latency reductions across all possible scenarios.Source: B. Scheduling · p. 7

C. Design of FPGA-based Accelerator

The FPGA accelerator handles draft-model decoding with streaming dataflow and pointer-based KV-cache control. The evaluation describes its hardware and model configurations, workloads, metrics, and simulator, including a projected FPGA clock frequency.

C. Design of FPGA-based Accelerator

  • The FPGA accelerator accommodates the draft model’s weights and dynamic KV cache in off-chip memory and uses fully streaming decoding without buffering intermediate activations there.The draft model is described as typically having around 1B parameters.Source: C. Design of FPGA-based Accelerator · p. 7
  • The KVMU tracks validated, current, and step-ahead KV-cache boundaries with three pointer registers.The pointers are named Ptrvalid, Ptrcurr, and Ptrahead, respectively.Source: C. Design of FPGA-based Accelerator · p. 8
  • KVMU initialization, speculation, accept, and rollback modes reset or advance pointers to restore state, track speculative entries, accept verified entries, or discard rejected history.Source: C. Design of FPGA-based Accelerator · p. 8
  • The input MUX uses local sampled tokens during speculation and acceptance, and PCIe target-model tokens during initialization and rollback.The KVMU is described as a modular, plug-and-play component for FPGA-based LLM accelerators.Source: C. Design of FPGA-based Accelerator · p. 8

A. Experimental Setup

  • Experiments use AMD U280 and V80 FPGAs and NVIDIA RTX 3090 GPUs; the FPGA designs use TAPA and W8A8 quantization.U280 on-board measurements run at 200 MHz; performance is projected to 300 MHz and V80 performance is estimated.Source: A. Experimental Setup · pp. 8–9
  • The evaluated Qwen2.5-Math-Instruct configurations pair a 0.5B draft with a 7B PRM and either a 7B or 1.5B target model.Source: A. Experimental Setup · p. 8
  • Experiments use batch size 1, set PRM acceptance and backtracking trigger thresholds to 0.7, and cap backtracking at five attempts per problem.Source: A. Experimental Setup · p. 8
  • The evaluation covers Math500, GSM8K, Gaokao2023EN, and OlympiadBench, measuring accuracy, latency, goodput, and energy efficiency.A trace-driven simulator replays 3-GPU baseline reasoning trajectories and models FPGA, GPU, and PCIe execution latencies.Source: A. Experimental Setup · pp. 8–9

B. Algorithmic Experimental Results

Backtracking improves accuracy over forward-only RSD, and the fully optimized system retains average accuracy gains over that baseline. The reported gain is larger with the 1.5B target model than with the 7B target model.

B. Algorithmic Experimental Results

  • +2.0% and +5.0% accuracy improvements over the forward-only RSD baseline are reported for backtracking across all tested datasets.The complete system’s accuracy degradation versus backtracking alone is reported as less than 0.8%.Source: B. Algorithmic Experimental Results · p. 9, Table III · p. 9
  • The fully optimized system reports a +4.2% average improvement with a 1.5B target and +1.7% with a 7B target; the paper attributes the disparity to weaker-target refinements being more susceptible to logical deviations.The authors state that backtracking compensates for target-model capacity limitations by allowing alternative reasoning paths.Source: B. Algorithmic Experimental Results · p. 9
  • +1.7% and +4.2% average accuracy gains are reported for the fully optimized system over RSD with the 7B and 1.5B target models, respectively.The accuracy table covers two model configurations and various reasoning benchmarks.Source: Table III · p. 9, B. Algorithmic Experimental Results · p. 9

C. Hardware Experimental Results

HeteroReason’s FPGA–GPU system improves latency, goodput, and energy efficiency over homogeneous-GPU baselines, while backtracking-enhanced execution on HeteroReason reduces the speed penalty observed on a homogeneous GPU setup. The comparisons cover four reasoning datasets and include the vanilla RSD framework.

1) End-to-end Results:

  • 1.01×–1.12× (U280) and 1.26×–1.42× (V80) latency speedups over weak baselines; V80 latency is comparable to the strong baseline.The speedup is consistent across all benchmarks.Source: Figure 8 · p. 10, 1) End-to-end Results · p. 10
  • 1.01×–1.12× (U280) and 1.25×–1.42× (V80) goodput improvements over weak baselines; V80 goodput is comparable to the strong baseline.Preemptively advancing the next reasoning step mitigates inter-device idle states and improves hardware utilization.Source: 1) End-to-end Results · p. 10
  • About 1.3× and 1.1× higher energy efficiency than the weak and strong baselines, respectively.The paper attributes these gains primarily to the FPGA’s lower operating power.Source: 1) End-to-end Results · p. 10

2) Comparisons with RSD:

  • On homogeneous GPUs, backtracking runs at 0.73×–0.93× and 0.68×–0.93× the RSD speed in Config 1 and Config 2, respectively; HeteroReason reduces this speed degradation across benchmarks and configurations.The three-GPU setup falls below RSD, while HeteroReason accelerates backtracking-enhanced inference relative to that GPU setup.Source: Figure 9 · p. 10, 2) Comparisons with RSD · p. 10

D. Ablation Study

The ablations separate the effects of step-ahead scheduling, FPGA-based prefill-decode disaggregation, and backtracking. The communication and wasted-token analyses report low overheads alongside the speed benefits of step-ahead execution.

D. Ablation Study

  • 1.25×–1.33× speedup in Config 1 and 1.35×–1.40× in Config 2 versus the baselines with T2; T2 overlaps speculation and refinement with verification.The scheme uses prefetching and caching to reduce inter-device idle states and applies across diverse platforms.Source: D. Ablation Study · p. 11, Figure 10 · p. 11
  • Adding T1 prefill-decode disaggregation to T2 slightly degrades speedup, while the heterogeneous system retains higher energy efficiency due primarily to FPGA power.Source: D. Ablation Study · p. 11
  • Without backtracking, HeteroReason reduces latency by 1.03×–1.13× in Config 1 and 1.05×–1.07× in Config 2; backtracking improves accuracy but adds latency.The paper states that enabling backtracking lets HeteroReason outperform GPU baselines and shifts the speed-accuracy trade-off frontier.Source: D. Ablation Study · p. 11

E. Communication Analysis

  • The maximum FPGA–GPU communication latency is only 0.0123% of end-to-end latency across configurations and datasets.Transferring a batch of tokens per step rather than the KV cache minimizes communication overhead.Source: E. Communication Analysis · p. 11
  • The speculation stall portion ranges from 4.95% to 26.57%, averaging 17.50%; faster speculation can increase this share even as end-to-end latency decreases.The stall ratio measures when FPGA speculation finishes before GPU verification returns.Source: E. Communication Analysis · p. 11

F. Energy Overheads of Wasted Tokens

  • Step-ahead prefetching provides 1.25×–1.33× latency speedup with only 0.480%–0.719% additional energy consumption.The paper reports the wasted-token overhead as consistently small across evaluated datasets; HeteroReason can incur lower energy overhead due to FPGA power.Source: Table IV · p. 11, F. Energy Overheads of Wasted Tokens · p. 11

G. Discussion on Edge GPU/FPGA/ASIC for Speculation

HeteroReason reports 1.01×–1.42× latency reduction and 1.25×–1.57× energy-efficiency gain over homogeneous-GPU baselines while improving reasoning accuracy. The discussion contrasts draft-model hardware options, situates the system among related work, and notes implementation and generalizability limits.

G. Discussion on Edge GPU/FPGA/ASIC for Speculation

  • Replacing the FPGA draft engine with a Jetson Orin Nano edge GPU is estimated to slow the system by approximately 5×–6×, despite reducing power consumption.The paper attributes the edge GPU’s limited drafting efficiency to memory-bound decoding and mismatch with sequential token generation.Source: G. Discussion on Edge GPU/FPGA/ASIC… · p. 12

VII. RELATED WORK

  • The related-work discussion distinguishes HeteroReason’s training-free approach from SCoT’s task-specific LoRA alignment and describes its verifier-agnostic mechanisms as complementary to DREAM-R’s verifier-side advances.Source: §VII · p. 12

VIII. LIMITATIONS

  • The current implementation combines simulation and on-board evaluation; broader evaluation across linear-attention and diffusion language models remains future work.Source: §VIII · p. 12

IX. CONCLUSION

  • 1.01×–1.42× latency reduction and 1.25×–1.57× energy-efficiency gain versus homogeneous-GPU baselines, with reasoning accuracy improved.The conclusion describes an FPGA–GPU system that assigns drafting to the FPGA and verification and refinement to GPUs.Source: §IX · p. 12

APPENDIX

The appendix describes an artifact for reproducing algorithmic accuracy results and hardware latency, goodput, and energy results. It lists the packaged workflows, model and dataset setup, and reproduction expectations.

A. Abstract

  • The artifact separates algorithmic reproduction of Table III accuracy results from a trace-driven hardware simulator and plotting workflow for Fig. 8 latency, goodput, and energy results.The hardware workflow uses packaged CSV/JSONL inputs and does not rerun model inference.Source: A. Abstract · pp. 12–13

B. Artifact Check-list

  • The reference algorithmic setup uses three RTX 3090 GPUs, Qwen2.5 0.5B draft, Qwen2.5-Math-PRM-7B, Qwen2.5 7B and 1.5B targets, and four listed reasoning datasets.The datasets are Math500, GSM8K, Gaokao2023EN, and OlympiadBench; model checkpoints are not redistributed.Source: B. Artifact Check-list · p. 13

C. Description

  • The appendix names the algorithm variants rsd, brsd, and brsd_optimized for vanilla RSD, RSD with backtracking, and BRSD with optimizations, respectively.The corresponding implementation names are beam1, bbeam1, and bbeam1_prefetch_cache.Source: C. Description · p. 13

D. Installation

The README.md documents installation, algorithmic reproduction levels, hardware-simulator and plotting workflows, expected outputs, and platform or checkpoint notes.

D. Installation

  • Installation instructions describe Python dependencies, local model paths, and required environment variables; the simulator and plotting workflow needs neither checkpoints nor GPU inference.Download the artifact and follow the README.md installation instructions.Source: D. Installation · p. 13

F. Hardware Simulator and Figure Workflow

  • The README.md documents a hardware-simulator and plotting workflow that regenerates results without rerunning model inference or hardware experiments.This workflow uses packaged CSV/JSONL inputs.Source: F. Hardware Simulator and Figure… · p. 13

E. Algorithmic Experiment Workflow

  • The key-result workflow reproduces Config1/Math500 Table III entries, while the full workflow covers all Table III algorithmic accuracy values across Config1 and Config2.The smoke test reports accuracy and latency on a fixed eight-example subset; brsd_optimized is the recommended key-result artifact.Source: G. Evaluation and Expected Results · p. 13

G. Evaluation and Expected Results

H. Notes

  • Algorithmic latency depends on GPU type, while the hardware simulator and plotting workflow are deterministic for the supplied CSV/JSONL inputs.Model checkpoints are referenced by local paths or Hugging Face identifiers and are not redistributed with the artifact.Source: H. Notes · p. 13
Loading 2609.28717v1…