PaperlayerView exact revision on arXiv

Source-linked AI summary

Programming Heterogeneous Systems from an Image Processing DSL

Jing Pu, Steven Bell, Xuan Yang, Jeff Setter, Stephen Richardson, Jonathan Ragan-Kelley, Mark Horowitz

arXiv:1610.09405v1cs.SE

TL;DR

Image accelerators can improve performance and energy efficiency, but fixed accelerator functions and the programming and integration burden constrain their use. The paper extends Halide and its compiler to generate FPGA hardware and supporting software, then reports performance and energy gains that increase with imaging-computation complexity.

  • Problem

    Fixed accelerator functions restrict programmers to predefined calls, and programming and integrating configurable hardware remains difficult.

    Source: §1 · p. 1

  • Method

    The system extends Halide scheduling and compilation to generate FPGA accelerators and the CPU, driver, and software components that connect them to applications.

    Source: §1 · p. 2, §2.3 · p. 3

  • Results

    The authors report significant performance and energy-efficiency gains, with gains increasing as imaging computation becomes more complex.

    Source: §7 · p. 11

  • Takeaways & Limitations

    The extended Halide system generates both accelerator designs and the software that communicates with the hardware.

    Source: §7 · p. 11

    Generated HLS code inserts unnecessary FIFOs, adding 30% FF and 20% LUT overhead; fine-grained blocking is inefficient overall for most evaluated applications given launch overheads and recomputation.

    Source: §6.2 · p. 9, §6.4 · p. 10

Abstract

from arXiv · show

Specialized image processing accelerators are necessary to deliver the performance and energy efficiency required by important applications in computer vision, computational photography, and augmented reality. But creating, "programming,"and integrating this hardware into a hardware/software system is difficult. We address this problem by extending the image processing language, Halide, so users can specify which portions of their applications should become hardware accelerators, and then we provide a compiler that uses this code to automatically create the accelerator along with the "glue" code needed for the user's application to access this hardware. Starting with Halide not only provides a very high-level functional description of the hardware, but also allows our compiler to generate the complete software program including the sequential part of the workload, which accesses the hardware for acceleration. Our system also provides high-level semantics to explore different mappings of applications to a heterogeneous system, with the added flexibility of being able to map at various throughput rates. We demonstrate our approach by mapping applications to a Xilinx Zynq system. Using its FPGA with two low-power ARM cores, our design achieves up to 6x higher performance and 8x lower energy compared to the quad-core ARM CPU on an NVIDIA Tegra K1, and 3.5x higher performance with 12x lower energy compared to the K1's 192-core GPU.

1. Introduction

Image-processing hardware can deliver substantial efficiency, but fixed accelerators and the expertise needed to program and integrate configurable hardware limit flexibility. The paper extends Halide and its compiler to generate FPGA accelerators, supporting software, and variable-throughput mappings.

1. Introduction

  • Processing 1080p/60fps raw video requires 120 gigaops/sec, and matching hardware to image-data patterns can yield enormous energy savings.Image-processing tasks are increasingly important as cameras become ubiquitous and image analysis improves.Source: §1 · p. 1
  • Fixed accelerator functions limit programmers to predefined calls, while programming and integrating configurable hardware remains difficult and C-based HLS still requires hardware expertise.Application demands are evolving, motivating a more flexible approach.Source: §1 · p. 1
  • Variable-throughput acceleration yields energy savings even on small FPGA fabrics, while fixed unit-rate designs need substantial resources before savings appear.As FPGA resources grow, the system can tune throughput and add pipeline stages.Source: Figure 1 · p. 1, §1 · p. 2
  • The compiler generates the FPGA configuration, CPU algorithm, Linux kernel drivers, and software glue mapping Halide calls to hardware access.This integration supports exploration of CPU/FPGA workload partitioning.Source: §1 · p. 2
  • Halide’s scheduling language enables FPGA generation with only two added commands, and the compiler extends support to affine indices and data-dependent reductions.The system builds on the popular open-source DSL and its existing image algorithms.Source: §1 · pp. 1–2

2. Background and Prior Work

Prior hardware-generation tools leave microarchitecture design to users, while image-processing DSLs encode domain knowledge to generate hardware. The section reviews image locality and line buffers, prior DSLs, and systems that integrate accelerators with host software.

2. Background and Prior Work

  • Domain-specific hardware templates can reduce the microarchitecture burden left by high-level synthesis tools, which still require users to understand their target hardware.The section situates image-processing DSLs among prior synthesis and accelerator-language efforts.Source: §2 · p. 2

2.1. Image Processing

  • Image pipelines trade data locality against redundant computation; line buffers can retain the minimum intermediate working set, about two pixel rows at matched rates.Combining tiling and kernel fusion can help, but their best parameters are difficult to determine.Source: §2.1 · p. 3

2.2. Image DSLs

  • Prior DSLs such as Darkroom and HIPAcc demonstrated DSL-to-hardware generation; this system extends line-buffered templates for variable throughput, blocking, higher-dimensional stencils, affine indices, and data-dependent reductions.The compiler generates synthesizable C and uses HLS tools to create Verilog.Source: §2.2 · p. 3

2.3. End-to-end Systems

  • CUDA/OpenCL and Lime provide host-device programming or runtime support; here, compiler-known CPU/accelerator boundaries enable automatic drivers and software integration, with scheduling primitives controlling placement.The scheduling primitives extend relocation brackets to specify whether code runs on the CPU or accelerator.Source: §2.3 · pp. 3–4

3. Language

The language extends Halide scheduling to describe heterogeneous hardware mappings while retaining familiar scheduling primitives for software. Two new commands and line-buffer syntax support accelerator scope, task granularity, throughput, and buffering choices.

3. Language

  • The accelerate primitive defines the accelerator’s scope and interface, while its block and inner-loop variables set task granularity and pixel throughput.In the example, a 256×256 tile is the task, and the inner loop produces one pixel per cycle.Source: §3 · p. 4
  • The fifo_depth primitive sets inter-function FIFO depth to balance latency when downstream functions consume multiple streams.Optimal depths can be formulated as an integer linear programming problem, but the paper specifies and tunes them by hand.Source: §3 · p. 4
  • The linebuffer syntax instantiates a line buffer for a function, avoiding direct fusion that could recompute values reused downstream.It is syntactic sugar for existing Halide primitives.Source: §3 · p. 5
  • Hardware unrolling duplicates datapath compute units and can raise throughput; unrolling the unsharp example’s color loop changes it from 1/3 to one pixel per cycle.The example instantiates three multipliers to process three color channels simultaneously.Source: §3 · p. 5
  • Existing tile, vectorize, and parallel primitives remain available for software scheduling, while hardware mapping reuses existing primitives where possible and adds only two new primitives.Halide scheduling exposes tradeoffs among locality, parallelism, and redundant recomputation.Source: §3 · pp. 4–5

4. Compiler Implementation

The compiler analyzes scheduled Halide programs, transforms accelerated regions into a dataflow representation, and generates hardware and software through separate back ends. Its pipeline template, IR transformation, and loop optimization support accelerator designs across varied computation and throughput targets.

4. Compiler Implementation

  • The compiler extracts architecture parameters, rewrites hardware IR into dataflow form, applies loop perfection and scalar optimizations, then sends the final IR to HLS and LLVM back ends.The back ends produce hardware designs and software programs, respectively.Source: §4 · p. 5

4.1. Architecture Parameter Extraction

  • The accelerator template is a DAG of stencil kernels connected by pixel-window streams, with bounds inference deriving stream parameters from the output backward.A producer computes the union of stencils required by its consumers; line buffers capture reuse between kernels.Source: §4.1 · p. 5, Figure 5 · p. 5

4.2. IR Transformation

  • IR transformation allocates local stencil storage, replaces original storage with streams and push/pop operations, and inserts image-domain loops and needed line-buffer or dispatcher calls.The resulting dataflow IR explicitly separates pipeline stages; the unsharp example has four kernel stages, two line buffers, and two stream dispatchers.Source: §4.2 · p. 6, Figure 7 · p. 6

4.3. Loop Perfection Optimization

  • Loop perfection recursively moves outer-loop operations into the innermost loop so HLS can pipeline imperfect loop nests at sub-pixel-per-cycle rates.For the 5-point convolution scheduled at 1/5 pixel per cycle, the original loops are imperfect and would otherwise execute portions sequentially.Source: §4.3 · p. 6, Figure 8 · p. 6

4.4. Code Generation

  • The HLS back end emits synthesizable C with pipelining and array-partitioning directives, while LLVM generates the CPU portion and platform-specific calls, allocations, and transfers for accelerator access.Hierarchical line-buffer templates support up to four dimensions, and constant functions can be evaluated into code that synthesizes to ROMs.Source: §4.4 · p. 7

5. Platform Development

The implementation targets a Xilinx Zynq-7000 SoC, combining ARM Cortex A9 cores, Linux, and FPGA fabric. Generated Halide code manages accelerator access, while DMA, a kernel driver, and threaded tile processing coordinate data movement and execution.

5. Platform Development

  • The system runs Linux on a Zynq-7000 with two ARM Cortex A9 cores and FPGA fabric, and presents the generated program as a single C ABI function callable from ARM userland.Interaction with the FPGA is managed within that function.Source: §5 · p. 7
  • AXI DMA connects accelerator streams to the CPU cache hierarchy through the accelerator coherency port and transfers image rows, including sub-images, without copying them.A parametrized Linux kernel module provides asynchronous accelerator launch and a synchronization barrier.Source: §5 · p. 7
  • Linux threads overlap CPU work and accelerator execution across image tiles, with active threads limited to three to avoid cache thrashing.Threads waiting for accelerator completion allow other active threads to use the CPU cores.Source: §5 · p. 7

6. Evaluation

The evaluation compares generated hardware with prior implementations, measures loop perfection and heterogeneous-system performance, and assesses programmability and extensibility. Across applications, the Zynq implementation improves throughput and energy efficiency over the TK1 CPU and GPU, with results shaped by cache behavior and workload characteristics.

6.1. Experimental Setup

  • The setup compares Zynq implementations against TK1 ARM and CUDA implementations, with the latter generated from the same Halide algorithms using different schedules.For stereo, the comparison uses NVIDIA's Tegra-optimized OpenCV CUDA kernel because Halide-generated CUDA performs poorly.Source: §6.1 · p. 8

6.2. Hardware Generation

  • Generated conv2d and Harris hardware reaches quality similar to the HLS library and HIPAcc, while Halide application code is 2× shorter than HIPAcc’s.Shared line buffers save BRAM; multi-rate designs use more LUTs and DSPs while line-buffer resources change little. Unnecessary interstage FIFOs add 30% FF and 20% LUT overhead.Source: §6.2 · pp. 8–9, Table 2 · p. 9

6.3. Impact of Loop Perfection

  • For histogram and stereo, loop perfection improves performance by 7% and 300%, respectively, with low resource overhead.Stereo benefits more because its four-iteration innermost loop incurs substantial entry and exit overhead when not pipelined with outer loops.Source: §6.3 · p. 9, Table 3 · p. 9

6.4. Heterogeneous System Performance

  • With three threads processing three stereo tiles concurrently, per-tile cost falls to 7.96 ms as most CPU work overlaps accelerator execution.Cache misses make the 768KB block configuration's actual runtime 77% longer than simulated; fine-grained blocks also bring recomputation and launch overhead.Source: §6.4 · pp. 9–10, Table 4 · p. 10

6.5. Programmability and Efficiency

  • The compiler evaluation uses six applications plus a camera-and-unsharp combination, and runs 8-megapixel images through each application.For most applications, the CPU handles tiling and accelerator scheduling; stereo also requires padding, while bilateral grid shuffles data into 8×8 grid order.Source: §6.5 · p. 10
  • On average, Zynq applications achieve 2.6× higher throughput and 14.2× higher energy efficiency than TK1 CPUs, and 1.9× higher throughput and 6.1× higher energy efficiency than its GPU.Harris has the largest reported gains: 6× and 38× versus TK1 CPUs, and 3.5× and 12× versus the GPU, respectively.Source: §6.5 · p. 10, Figure 12 · p. 11
  • The FPGA camera accelerator is the exception to the other Zynq applications reaching simulated peak throughput, because it needs more read bandwidth and suffers from L2-miss bandwidth problems.Bilateral-grid parallelism is limited by data dependencies, favoring the TK1's higher-frequency, higher-bandwidth processors; deeper FPGA pipelines instead run at the slowest pipeline's throughput.Source: §6.5 · p. 11

6.6. Extensibility

  • Porting the dataflow IR to Catapult HLS required changing around 1,000 lines of code, and the resulting 14nm ASIC pipelines achieve approximately 12× higher throughput and 8–20× better energy efficiency than the 28nm Zynq FPGA designs.Source: §6.6 · p. 11

7. Conclusion

The authors extend Halide to generate image-processing accelerators and the software that communicates with them, reporting performance and energy-efficiency gains. They plan further optimization and support for additional image-processing engines.

7. Conclusion

  • The system generates both image-processing accelerator designs and the software that communicates with the hardware, addressing an implementation barrier for designers.The authors report performance and energy-efficiency gains that increase as imaging computation becomes more complex.Source: §7 · p. 11
  • Planned work includes more automatic optimization of low-level buffering and blocking to help designers find optimal settings.Source: §7 · p. 11
  • The authors plan to target custom hardware, specialized SIMD, and coarse-grain reconfigurable image-processing architectures, and hope the system helps the community develop and use efficient programmable ISPs.Source: §7 · p. 11
  • The system’s source is available on GitHub.Source: §7 · p. 11
Loading 1610.09405v1…