PaperlayerView exact revision on arXiv

Source-linked AI summary

Forge-UGC: FX optimization and register-graph engine for universal graph compiler

Satyam Kumar, Saurabh Jha

arXiv:2604.16498v1cs.ARcs.AIcs.DC

TL;DR

Existing accelerator frameworks have opaque compilation pipelines and limited pass-level visibility, while transformer deployment also faces lossy export requirements and super-linear compilation time. Forge-UGC uses transparent, composable compiler infrastructure, and across six model families it achieves faster compilation, lower latency, and lower energy consumption than both baselines.

  • Problem

    Existing frameworks provide limited pass-level visibility and require lossy intermediate-format exports, while their compilation times scale super-linearly with model depth.

  • Method

    Forge-UGC is a transparent, composable, hardware-agnostic compiler that separates graph capture, optimization, intermediate-representation lowering, and backend scheduling.

  • Results

    Across six model families on WikiText-103 and GLUE, Forge-UGC achieves 6.9–9.2× faster compilation, 18.2–35.7% lower inference latency, and 30.2–40.9% lower energy consumption per inference versus both baselines.

  • Takeaways & Limitations

    The results support transparent, composable compiler infrastructure as a practical alternative to proprietary black-box frameworks for accelerator deployment.

  • Takeaways & Limitations

    Fusion Gain Ratio is a cost-model diagnostic rather than wall-clock latency, and its values are not linearly proportional to measured speedup.

Abstract

from arXiv · show

We present Forge-UGC (FX Optimization and Register-Graph Engine for Universal Graph Compilation), a four-phase compiler for transformer deployment on heterogeneous accelerator hardware, validated on Intel AI Boost NPU. Existing frameworks such as OpenVINO and ONNX Runtime often use opaque compilation pipelines, limited pass-level visibility, and weak buffer management, which can lead to higher compilation cost and runtime overhead. Forge-UGC addresses this with a hardware-agnostic design that separates graph capture, optimization, intermediate representation lowering, and backend scheduling. Phase 1 captures graphs with torch.export at the ATen operator level, supporting modern transformer components such as rotary position embeddings, grouped-query attention, and SwiGLU without manual decomposition. Phase 2 applies six optimization passes: dead code elimination, common subexpression elimination, constant folding, attention fusion, operator fusion, and layout optimization, reducing graph node count by 14.2 to 21.9%. Phase 3 lowers the optimized graph into a typed intermediate representation with explicit virtual register assignments. Phase 4 performs liveness analysis, linear-scan buffer allocation, reducing peak buffer count by 30 to 48%, and device-affinity scheduling, reducing NPU-CPU transitions by 42 to 65%. Across six model families ranging from 125M to 8B parameters, evaluated on WikiText-103 and GLUE, Forge-UGC delivers 6.9 to 9.2x faster compilation than OpenVINO and ONNX Runtime, 18.2 to 35.7% lower inference latency, and 30.2 to 40.9% lower energy per inference. Fidelity is preserved, with max absolute logit differences below 2.1e-5 and KL divergence below 8.4e-9. We also introduce Fusion Gain Ratio, Compilation Efficiency Index, and per-pass execution profiling for systematic evaluation of NPU compilation pipelines.

1 INTRODUCTION

Heterogeneous AI workloads expose gaps in existing opaque deployment frameworks, motivating FORGE-UGC’s transparent, four-phase compiler validated across modern transformer models and NPU deployment settings.

  • Motivation: Heterogeneous agents combine NPU, GPU, and CPU execution, making hardware–software co-design central to efficient inference.The compiler must translate high-level models, manage device-boundary data movement, and support iterative development.
  • Existing-framework limitations: OpenVINO and ONNX Runtime require lossy intermediate exports that can fail on dynamic control flow, tied weights, and modern transformer operators.FORGE-UGC instead captures PyTorch graphs at the ATen operator level with torch.export.export().
  • Existing-framework limitations: Neither baseline exposes individual optimization passes or sufficient diagnostics for inspecting fusion, quantifying pass contributions, or conducting ablations.Their opacity limits performance debugging and principled optimization.
  • Existing-framework limitations: 58–62 seconds of compilation time for 8B-parameter models makes the baselines costly for iterative development and just-in-time deployment.The cited bottlenecks are OpenVINO’s monolithic IR conversion and ONNX Runtime’s execution-provider initialization.
  • FORGE-UGC: FORGE-UGC introduces a backend-agnostic four-phase pipeline with pluggable hardware-specific lowering, enabling reuse across accelerator targets.Its contributions include direct FX compilation, typed IR, formal buffer management, and systematic evaluation metrics.

2 MOTIVATION: THE COMPILER AS A SYSTEM-LEVEL ORCHESTRATOR

Modern edge systems require orchestration across complementary accelerators rather than reliance on one device. FORGE-UGC addresses this role by separating reusable optimization from hardware-specific compilation and serving as an NPU backend for heterogeneous orchestrators.

  • System-level orchestration: No single accelerator efficiently executes an entire multi-stage agent graph, which may route vision, attention, embeddings, and control logic across devices.These pipelines combine NPU, GPU, and CPU strengths within one system-on-chip.
  • System-level orchestration: The compiler must partition graphs by accelerator cost characteristics while minimizing latency, energy, and data movement across device boundaries.This expands the compiler’s role from device-specific code generation to system-level orchestration.
  • FORGE-UGC’s role: FORGE-UGC can serve as the NPU compilation backend for orchestrators that route transformer layers across CPU, GPU, and NPU devices using workload-specific energy models.The combined system selects layer placement and compiles selected layers with NPU-oriented optimizations.
  • FORGE-UGC’s role: Separating hardware-agnostic optimization from backend-specific lowering allows new accelerator targets to reuse the optimization pipeline while extending only backend modules.The frontend likewise operates on universal PyTorch FX graphs.

3 BACKGROUND & RELATED WORK

FORGE-UGC is positioned within compiler systems that use staged lowering and inspectable passes, while differing through direct PyTorch FX processing and Intel NPU-specific compilation. The paper motivates this design by identifying export gaps, backend limitations, and insufficient NPU-aware scheduling and memory management in existing approaches.

  • TVM established a three-stage compiler design using Relay for graph transformations and TIR for low-level tensor operations.
  • IREE offers composable passes and explicit buffer management but requires conversion through torch-mlir or StableHLO and lacks an Intel NPU backend.
  • FORGE-UGC is designed for backend portability, while its current validation targets Intel AI Boost NPU and its architecture is contrasted with Qualcomm- and MLIR-based alternatives.
  • FORGE-UGC operates directly on PyTorch FX graphs at the ATen level, preserving PyTorch operator semantics and avoiding framework-specific re-export.
  • torch.compile custom backends cannot expose the composable IR-pass structure and NPU-specific liveness-aware buffer management required by FORGE-UGC.
  • OpenVINO and ONNX Runtime have export and operator-coverage limitations for modern PyTorch models, while their pipelines provide limited pass-level visibility and NPU-specific control.

4 THE FORGE-UGC METHODOLOGY

The FORGE-UGC methodology captures PyTorch graphs, applies composable FX optimizations, and fuses transformer computation patterns before backend lowering. Its passes target dead computation, duplicate expressions, constants, attention subgraphs, and linear-activation chains.

  • Phase 1: FX Graph Capture: FORGE-UGC captures computation graphs with torch.export.export() at the PyTorch ATen operator level.
  • Phase 1: FX Graph Capture: Tied parameters are detected by tensor identity and resolved to one canonical tensor, allowing placeholders to share a single physical buffer.
  • Phase 2: Graph Optimization: Six composable passes are applied sequentially, with a fixpoint loop repeating passes until convergence so earlier transformations can expose later opportunities.
  • Phase 2: Graph Optimization: Dead code elimination removes nodes unreachable from graph outputs, including debugging artifacts, gradient branches, and dead sub-expressions.
  • Phase 2: Graph Optimization: Common subexpression elimination replaces duplicate operations on identical inputs with references to a canonical result using hash-consing.
  • Phase 2: Graph Optimization: Attention fusion replaces the traced scaled-dot-product attention chain with one NPU fused module, while operator fusion combines linear projections with ReLU, GELU, or SiLU.

4.4 Phase 3: Lowering to NPUIR

Phase 3 lowers the optimized FX graph into NPUIR through a topological traversal, assigning typed instructions, virtual registers, resolved callables, and deterministic device routing.

  • The optimized FX graph is lowered to NPUIR, where each FX node becomes an instruction with explicit metadata.
  • NPUIR uses integer virtual-register identifiers for abstract tensors and opcodes distinguishing NPU modules, CPU ATen functions, and CPU tensor methods.
  • The deterministic routing rule sends named NPU linear, fused, matrix-multiplication, and addmm modules to the NPU; other nodes run on the CPU.
  • Lowering freezes FX references as register markers, enabling runtime resolution from the register file without graph traversal or attribute lookup.
  • Topological lowering classifies operations, routes them to CPU or NPU, assigns output virtual registers, and records inputs and callables.

4.5 Phase 4: IR Analysis & Optimization

Phase 4 analyzes NPUIR lifetimes, reuses physical buffers through linear-scan allocation, and schedules instructions with device affinity. These steps reduce memory usage and host-device transition overhead while producing a flat executable stream.

  • Phase 4 comprises liveness analysis, linear-scan buffer allocation, and instruction scheduling over the flat NPUIR instruction list.
  • Liveness Analysis: Live intervals identify when each virtual register is written and last read, allowing nonoverlapping registers to share physical buffer slots.
  • Buffer Allocation: Linear-scan allocation maps N virtual registers to M physical buffers in O(N log N) time by reusing slots from expired intervals.
  • Buffer Allocation: 30–48% fewer physical buffers than virtual registers are achieved for transformer models in the experiments.
  • Instruction Scheduling: The scheduler performs a dependency-respecting priority topological sort that favors instructions on the most recently scheduled device.
  • Instruction Scheduling: On Llama-3.1-8B, transitions fall from 264 to 93, eliminating 50–130 ms of per-inference overhead and contributing 11.2% of total latency improvement.

4.6 NPU Cost Model

The NPU cost model enables hardware-free comparison of compiler configurations, and autotuning searches this space before selecting a configuration for compilation.

  • 4.6 NPU Cost Model: A heuristic cost model estimates NPU execution cost without hardware profiling.Its score combines operation count, weight-tensor count, linear-operation fraction, graph depth, and parameter size.
  • 4.6 NPU Cost Model: The autotuner searches fusion aggressiveness, layout strategy, NPU precision, and fixpoint iterations.The configuration space contains 45 candidates compiled using the cost model without hardware execution.
  • 4.6 NPU Cost Model: Under 200ms per model, autotuning completes in negligible time relative to a single compilation.The selected configuration is chosen after evaluating the candidate space with the heuristic model.

5 NOVEL EVALUATION METRICS

The paper introduces metrics for evaluating compiler behavior beyond raw latency, including fusion impact, compilation-time return, and pass-level overhead.

  • 5 NOVEL EVALUATION METRICS: Per-pass execution profiling isolates which optimization passes contribute to compilation overhead versus speedup.This supports informed pass-selection decisions for latency-sensitive deployments.
  • 5.2 Metric 2: Fusion Gain Ratio (FGR): FGR measures fusion’s effect on the cost model’s estimated execution cost independently of layout optimization and constant folding.Values above 1.0 indicate lower estimated cost from fusion, but are not proportional to wall-clock latency.
  • 5.3 Metric 3: Compilation Efficiency Index (CEI): CEI measures inference speedup relative to a baseline per second of compilation time.Separate CEI values use OpenVINO and ONNX Runtime as baselines because their baseline latencies differ.
  • 5.3 Metric 3: Compilation Efficiency Index (CEI): CEI is more informative for iterative or just-in-time deployment than for compile-once-run-millions production deployment.In the latter regime, absolute latency improvement is the primary metric because compilation costs are amortized.

6 EXPERIMENTAL SETUP

Experiments evaluate Forge-UGC and two deployment baselines on a single workstation across six model families, multiple precision settings, and language-modeling and NLU workloads.

  • 6 EXPERIMENTAL SETUP: Experiments span six model families covering 125M–8B parameters on a single workstation.The hardware platform is documented in Table 2, while model specifications are documented in Table 3.
  • 6.2.1 Precision Strategy for Llama-3.1-8B: Llama-3.1-8B uses symmetric int8 weights with fp16 activations, while models ≤2.6B use fp16 weights without quantization.The 8B strategy reduces weight memory to approximately 8GB and is applied during Phase 4 dispatch after graph optimization.
  • 6.3 Baselines: Both baselines use their default NPU precision settings, including int8 quantization for the 8B model.OpenVINO exports models through ONNX into OpenVINO IR, while ONNX Runtime uses torch.onnx.export with opset 17 and its OpenVINO Execution Provider.
  • 6.4 Evaluation Workloads: WikiText-103 measures perplexity and generation latency, while GLUE measures batch-size-one latency on SST-2 and MNLI.The workloads use 128-token inputs; WikiText-103 additionally uses 64-token generation.
  • 6.5 Measurement Protocol: Latency uses 50 iterations after 10 warmups, with mean, P50, P90, and P99 reported across three fixed-seed runs.Compilation time includes graph capture, optimization, lowering, and code generation.
  • 6.6 Numerical Fidelity: Numerical fidelity is assessed using perplexity agreement, maximum absolute logit difference, and KL divergence.The protocol combines coarse-grained semantic checks with fine-grained output-level comparisons.

7 RESULTS

FORGE-UGC compiles substantially faster than OpenVINO and ONNX Runtime while reducing latency, energy use, and device-management overhead across transformer models. Its optimization and backend stages are lightweight, preserve numerical fidelity, and scale mainly with graph depth.

  • Compilation time: 6.9–9.2× faster compilation than OpenVINO and ONNX Runtime is reported across model families.FORGE-UGC scales approximately linearly with layer count, while the baselines scale super-linearly.
  • Compilation phase breakdown: 78.4% of GPT-2 compilation time is spent in FX Capture, while optimization requires 208ms and backend phases require 8ms.The authors attribute end-to-end gains partly to avoiding additional ONNX/TorchScript export and partly to lightweight compiler stages.
  • Numerical fidelity: Numerical fidelity is preserved, with max-absolute logit differences below 1.2 × 10^-5 for fp16 models and 2.1 × 10^-5 for Llama-3.1-8B.The larger 8B discrepancy is attributed to int8 weight quantization rather than graph optimization passes.
  • Inference performance: 18.2–35.7% lower inference latency is achieved across WikiText-103 and GLUE, with the largest improvement reaching 35.7% on Llama-3.1-8B versus ONNX Runtime.The reported gains are consistent across benchmarks, with relative-improvement standard deviation below 1.2%.
  • Energy efficiency: 30.2–40.9% lower energy per inference than OpenVINO is reported, exceeding corresponding latency savings by 5–12 percentage points.Lower energy is associated with reduced device transitions, tighter buffer allocation, and shorter active inference time.

8 ABLATION STUDIES

Ablations identify attention fusion as the dominant optimization, while buffer allocation and instruction scheduling provide increasingly large benefits as transformer depth grows. Aggressive fusion, autotuning, and deterministic execution further improve cost, reproducibility, and deployment behavior.

  • Pass-Level Ablation: 27.6× cost-model degradation follows attention-fusion removal, establishing it as the most critical pass.Operator fusion contributes a modest 2.3% improvement, while lightweight passes have minimal GPT-2 impact.
  • Cross-Model Ablation: Attention Fusion Impact: 16.6% latency reduction occurs for 12-layer models, increasing to 28.6–29.6% for 32-layer models with attention fusion.The measured benefit scales with model depth as more attention blocks are fused.
  • Buffer Allocation: 47.8% buffer reduction is achieved on 8B models, where deeper graphs provide more overlapping live intervals for reuse.The scheduler benefits from the tighter buffer layout, and device-transition reduction correlates with buffer reduction.
  • Fusion Aggressiveness: Aggressive fusion consistently improves the NPU cost-model score, unlike the register-pressure trade-off reported for GPU targets.NNFactory dispatches entire fused subgraphs in single calls, eliminating per-operation dispatch overhead.
  • Autotuning: Autotuning improves cost-model score by 4.2–8.7% with less than 200ms compilation overhead.The overhead is reported to amortize after a single inference iteration for all models.
  • Instruction Scheduling Ablation: 42–65% fewer device transitions are reported, with the reduction rising from 41.9% on GPT-2 to 64.8% on Llama-3.1-8B.On Llama-3.1-8B, reducing transitions from 264 to 93 eliminates approximately 50–130ms of per-inference overhead.
  • Execution Stability: FORGE-UGC reports a 1.20 P99/P50 latency ratio versus 1.27–1.28 for both baselines.The tighter tail distribution is associated with pre-allocated buffers and deterministic instruction scheduling.

9 ANALYSIS & DISCUSSION

FORGE-UGC’s compilation gains arise from direct FX processing, composable optimization, and efficient allocation and scheduling, while its evaluations show task-robust improvements and defined hardware and measurement boundaries.

  • Compilation Efficiency: 6.9× to 8.7× compilation speedup over OpenVINO grows with model size, from GPT-2 to Llama-3.1-8B.The reported explanation combines direct FX processing, composable passes, and linear-scan allocation.
  • Compilation Efficiency: 78% of FORGE-UGC’s compilation time is spent in torch.export graph capture, while its optimization and backend phases take approximately 216ms for GPT-2.The remaining speedup is attributed to avoiding additional ONNX/TorchScript export and keeping later phases lightweight.
  • Runtime Mechanisms: 42–65% fewer NPU–CPU transitions and 30–48% lower peak buffer pressure result from instruction scheduling and linear-scan allocation.Attention and operator fusion further reduce dispatches and transitions, with the mechanisms described as complementary.
  • Scaling: 30–35% lower per-parameter cost is an approximate trend based on six data points spanning a 64× parameter range.The authors caution that the fit is not a precise predictive model.
  • Scope and Limitations: Extending beyond Intel AI Boost NPU requires new Phase 4 backend dispatch modules, while the frontend and middle-end pipeline can be reused.The current prototype uses single-batch inference.
  • Robustness: Standard deviation below 1.2% across datasets confirms task-agnostic latency improvements, while graph-level optimization explains the low variation.The cited analysis reports standard deviation below 0.3% for latency in one comparison and exactly 0% for compilation time.
  • Fidelity: At 8B parameters, int8 weight quantization introduces a max-absolute logit difference of 2.1 × 10^-5, unlike the semantics-preserving fp16 results through 2.6B parameters.The paper attributes the error to NNFactory dispatch-level quantization and proposes quantization-aware controls.
  • Measurement Limitations: FGR is based on a heuristic cost model rather than calibrated wall-clock latency, and energy uses system-level RAPL readings rather than component-level sensors.The authors identify hardware-calibrated cost modeling and per-component power sensing as future work.

10 CONCLUSION & FUTURE WORK

The conclusion presents FORGE-UGC as a transparent, universal graph compiler that improves NPU deployment across performance, energy, and compilation analysis, while future work extends it toward portable and adaptive compilation.

  • Conclusion: 14.2–21.9% fewer graph nodes, 30–48% lower peak buffer count, and 42–65% fewer NPU↔CPU transitions summarize the compiler’s measured transformations.The conclusion also reports max-absolute logit difference below 2.1 × 10^-5 and KL divergence below 8.4 × 10^-9.
  • Conclusion: 6.9–9.2× faster compilation, 18.2–35.7% lower inference latency, and 30.2–40.9% lower energy per inference were achieved versus both baselines.These results cover six model families from 125M to 8B parameters on WikiText-103 and GLUE.
  • Evaluation: Fusion Gain Ratio, Compilation Efficiency Index, and per-pass profiling enable systematic ablation of NPU compilation for transformer workloads.The metrics respectively support fusion comparison, iterative-development analysis, and pass-level evaluation.
  • Future Work: Hardware-agnostic optimization passes and typed IR can support additional NPU backends by reusing the pipeline and replacing target-specific dispatch modules.Named targets include Qualcomm Hexagon, AMD XDNA, Apple ANE, and Samsung NPU.
  • Future Work: Triton integration is intended to lower custom NPU kernels through FORGE-UGC’s optimization passes and NPUIR backend.This would extend the system from whole-model compilation to kernel-level custom operator development.
  • Future Work: A self-evolving compiler module is being developed to refine pass ordering, fusion aggressiveness, and autotuning configurations across successive compilations.The proposed module uses runtime telemetry to adapt to workload and hardware characteristics.
  • Implications: The paper positions transparent, composable infrastructure on open standards as an alternative to proprietary black-box deployment frameworks.Its stated scope spans compilation speed, inference latency, energy efficiency, and heterogeneous accelerator targets.

A RAW PER-RUN LATENCY DATA

The raw latency table reports per-run mean inference latency on WikiText-103 across three independent FORGE-UGC runs.

  • Raw Measurements: Table 25 contains raw per-run mean inference latency values in milliseconds for WikiText-103 across three independent runs.The values support variance statistics reported in Table 19.
Loading 2604.16498v1…