Source-linked AI summary
RoofLang: Enabling AI-Driven Architecting of LLM Inference Systems
Ziyue Yang, Yuting Jiang, Lei Qu, Peng Cheng
TL;DR
Profiling-bound optimization confines LLM inference search to the capabilities of the current software stack. RoofLang provides a general workload representation, verifiable architecting actions, and implementation-independent evaluation; its evaluation reports model-level throughput differences and improvements from agent-discovered architectures.
Problem
Profiling-based optimization is constrained by the existing software stack, which can prevent fundamentally different LLM inference architectures from being realized.
Source: §1 · p. 1Method
RoofLang combines a general workload representation, verifiable semantics-preserving architecting actions, and roofline-based discrete-event evaluation with iterative agent feedback.
Source: §1 · pp. 1–2Results
DeepSeek V4-series models could achieve 3.5–39.5× higher peak decode throughput than representative models, while agent-discovered DeepSeek V4 Pro architectures improved throughput and interactivity by 6.23–50.1% on NVIDIA B300.
Source: §1 · p. 2Takeaways & Limitations
A verification-constrained analytical model can expose useful model-level insights and guide architectural exploration before implementation-specific optimization.
Source: §5 · p. 12Current models assume ideal MoE load balance and omit KV-cache appending, speculative-decoding structures, DeepSeek V4’s mHC, and Kimi K3’s multimodal components.
Source: §5 · p. 12
Abstract
from arXiv · showhide
AI is beginning to make substantive contributions to LLM inference optimization. Existing AI optimizations are predominantly profiling-based. Profiling-bound feedback confines the search to the capabilities and performance of an existing software stack, preventing a fundamentally better architecture of LLM inference systems from being identified. To enable the AI-driven LLM inference system architecting loop, we argue that a general workload representation, a verifiable mutation space, and an implementation-independent evaluator are required. We present the RoofLang domain-specific language (DSL) that provides these features. In our evaluation, RoofLang reveals that DeepSeek V4-series models could achieve 3.5-39.5$\times$ higher peak decode throughput than other representative models. This gap is disproportionate to their total parameter counts and arises largely from compact KV-cache designs that support larger batches and reduce memory traffic. A persistent optimizer agent further discovered several new architectures that improved both throughput and interactivity of DeepSeek V4 Pro on NVIDIA B300 by 6.23-50.1%.
1 INTRODUCTION
Profiling-based optimization is constrained by the software stack being profiled, motivating an AI loop that can architect inference systems more broadly. RoofLang provides a workload representation, verifiable architecting actions, and an evaluator; its evaluation reports model-level throughput differences and gains from agent-discovered architectures.
1 INTRODUCTION
- Profiling-based optimization is constrained by the current software stack, which can prevent fundamentally different architectures from being realized.A better serving architecture may require features absent from the profiled stack or implementations that are not sufficiently optimized.Source: §1 · p. 1
- RoofLang combines general workload representation, semantics-preserving architecting actions, and roofline-based discrete-event evaluation in an AI feedback loop.An agent proposes valid actions for a verified initial workload and architecture, then iteratively improves them using simulation feedback.Source: §1 · pp. 1–2
- DeepSeek V4-series models could achieve 3.5–39.5× higher peak decode throughput than representative models, while agent-found V4 Pro architectures improved throughput and interactivity by 6.23–50.1% on NVIDIA B300.The gap arises largely from compact KV-cache designs that support larger batches and reduce memory traffic, rather than total parameter counts alone.Source: §1 · p. 2
- The paper analyzes model, workload, and hardware effects on architecting trade-offs and compares AI-driven exploration with heuristic grid search.It also examines how an agent uses simulation feedback to generate architecting choices.Source: §1 · p. 2
2 ROOFLANG DESIGN
RoofLang frames AI-driven LLM inference architecting as a feedback loop over general workload and hardware representations, valid architecting actions, and implementation-independent evaluation. Its design combines graph transformations and placement with roofline-based simulation.
2.1 OVERVIEW
- RoofLang closes an autonomous architecting loop by evaluating valid actions on a verified workload and feeding results back to an AI agent for revision.The approach combines graph-based representations, constrained actions, and roofline-based simulation without requiring profiled operator implementations.Source: §2.1 · pp. 2–3
2.2 COMPUTE GRAPH
- The externally supplied compute graph represents the LLM inference workload, and its provider is responsible for ensuring that representation is faithful and correct.Its nodes describe operations and their tensors, analytical attributes, and dependencies.Source: §2.2 · p. 3
2.3 HARDWARE GRAPH
- The hardware graph describes the target platform through compute and memory components and the fabrics connecting them.Compute devices specify peak FLOPs by precision, memory devices specify capacity, and fabrics have communication properties including bandwidth and latency.Source: §2.3 · pp. 3–4
2.4 PLACEMENT
- Placement lowers a compute graph onto a hardware graph by assigning operators to compute devices and tensors to memory devices, while preserving semantics.The placement primitives are summarized in Table 1.Source: §2.4 · p. 4, Table 1 · p. 4
2.5 SEMANTICS-PRESERVING TRANSFORMATION
- RoofLang’s structurally constrained graph rewrites preserve modeled observations and can be composed with placement to express inference architecting actions.Examples include prefill–decode disaggregation, KV-cache lifecycle policies, and sharding; transformations recompute affected analytical attributes such as FLOPs, bytes, tensor shapes, and communication volumes.Source: §2.5 · pp. 4–5, Table 2 · p. 4
2.6 SIMULATION
- The topology-aware discrete-event simulator schedules dependency-ready kernels, tracks live memory and capacity, and models operator duration as the bottleneck of compute, local-memory, and network costs under ideal full overlap.Collective communication uses a theoretical α+β model, and concurrent resource sharing reduces available bandwidth and extends simulated cost.Source: §2.6 · pp. 5–6
3 EVALUATION
The evaluation compares models, hardware, and workloads, then examines throughput–interactivity tradeoffs and agent-discovered architectures. Decode results highlight large V4-series throughput advantages, while three optimization cases improve both throughput and interactivity under distinct conditions.
3.1.1 SETUP
- The evaluation covers four models and four NVIDIA platforms, with workloads spanning 8 or 64 GPUs, 64K or 1M contexts, and batch sizes swept to OOM.The setup also omits KV-cache appending and speculative decoding, and omits Kimi K3 multimodal components.Source: §3.1.1 · p. 6
3.1.2 RESULTS
- Smaller per-request KV-cache footprints sustain larger batches and reduce memory traffic, while prefill’s large-batch endpoint is primarily compute-bounded.Decode interactivity-oriented endpoints are primarily bounded by activated-parameter bytes per step divided by HBM bandwidth.Source: §3.1.2 · p. 7
- Larger scale-up domains help most when MoE routing dominates; as KV-cache traffic grows, routing becomes a smaller share of decode overhead.The paper notes that attention time rises and sustainable batch size falls as the KV cache grows, reducing the room for scale-up networking to improve performance.Source: §3.1.2 · pp. 7–8
- On 64 NVIDIA GB300 GPUs, DeepSeek V4 Pro’s peak decode throughput is 3.5×/5.3× GLM-5.3 and 9.3×/12.4× Kimi K3 at 64K/1M context.The reported ratios are for the 64K- and 1M-context settings, respectively.Source: §3.1.2 · p. 7, Figure 4 · p. 9
3.2.1 SETUP
- The optimizer agent iteratively searched placements and semantics-preserving graph transformations using RoofLang simulation feedback until it chose to stop.Each session provided the agent the RoofLang repository, DeepSeek V4 Pro implementation, and baseline results.Source: §3.2.1 · p. 10
3.2.2 RESULTS
- For decode-64K on 256 B300 GPUs, a cost- and head-aware contiguous pipeline-parallel partition improved throughput and interactivity by 20.3%.The agent accounted for embedding and output-head work to reduce the maximum stage span.Source: §3.2.2 · p. 10
- For prefill-8K on eight B300 GPUs, DP–CP routed-expert splitting improved throughput and interactivity by 6.23% at batch 512.The split was enabled only with at least 1,024 routed tokens per copy and EP no greater than eight, because it adds expert-weight reads.Source: §3.2.2 · p. 10
- For prefill-1M on 16 B300 GPUs, node-local expert replication improved throughput and interactivity by 50.1% in a communication-bound case.The change traded additional expert-weight reads and HBM capacity for lower exposed routing communication.Source: §3.2.2 · p. 10
4 RELATED WORK
Prior work models inference performance at varying levels of fidelity and develops verified graph optimization or implementation-grounded agent loops. RoofLang combines constrained architecture mutations with an implementation-independent evaluator, while relying on human verification of initial graphs and transformation contracts.
4.1 INFERENCE PERFORMANCE MODELING
- Inference performance models range from analytical bounds to detailed simulators, but performance estimation alone does not establish semantic equivalence between architectures.RoofLang combines a constrained mutation space with an evaluator based on workload attributes and hardware specifications; humans still verify initial graphs and transformation contracts.Source: §4.1 · p. 11
- Unity jointly optimizes algebraic transformations and parallelization for distributed DNN training using a unified parallel computation graph.Source: §4.1 · p. 11
4.2 KERNEL AGENTS AND OTHER EFFORTS FOR RSI
- Kernel and inference agents generally optimize executable implementations or use real-hardware performance samples, coupling search to the available software stack.RoofLang instead evaluates graph organizations using hardware specifications and roofline relations before implementation quality influences comparison; kernel and framework tools can then optimize a selected architecture.Source: §4.2 · p. 11
5 CONCLUSION, LIMITATIONS, AND FUTURE DIRECTIONS
ROOFLANG is an initial step toward AI-driven LLM inference architecting, with evaluation findings on model-level throughput and agent-discovered architectures. The authors identify limitations in autonomy, modeling fidelity, and simulation scalability.
5 CONCLUSION, LIMITATIONS, AND FUTURE DIRECTIONS
- DeepSeek V4-series models could achieve 3.5–39.5× higher peak decode throughput than other representative models, largely because compact KV-cache designs support larger batches and reduce memory traffic.The gap is disproportionate to differences in total parameter counts.Source: §5 · pp. 11–12
- A persistent optimizer agent discovered architectures that improved both throughput and interactivity of DeepSeek V4 Pro on NVIDIA B300 by 6.23–50.1%.Source: §5 · p. 12
- ROOFLANG assumes the initial compute and hardware graphs are correct, so human verification remains necessary.The authors propose automating construction and validation, including comparison against execution traces.Source: §5 · p. 12
- Current models simplify behaviors that can materially affect performance, including idealized MoE load balance and omitted KV-cache appending, speculative decoding, mHC, and multimodal components.Source: §5 · p. 12
- The discrete-event simulator scales poorly to the largest configurations; simulating Kimi K3 on 128 GPUs can take hours and hundreds of gigabytes of memory.This makes repeated agentic optimization impractical.Source: §5 · p. 12