PaperlayerView exact revision on arXiv

Source-linked AI summary

Accelerating Data Preprocessing for Efficient Vision Model Inference on Jetson Edge Device

Tian Chen, Nawras Alnaasan, Jinghan Yao, Aamir Shafi, Hari Subramoni, Dhabaleswar K., Panda

arXiv:2608.27655v1cs.PF

TL;DR

JPEG decoding can dominate vision-model preprocessing on edge devices, limiting throughput, particularly for large images. The paper combines NVJPEG-based decoding with multi-instance execution across CPU, GPU, and DLA, achieving up to 30.02% higher throughput than the best design without NVJPEG in large-image settings. It also characterizes hardware behavior and provides tuning guidance for pipeline configurations.

  • Problem

    JPEG decoding is compute-intensive and can occupy most of preprocessing, creating a throughput bottleneck for edge vision inference.

  • Method

    The paper integrates Jetson AGX Orin’s NVJPEG decoder with multi-instance pipelines that concurrently use CPU, GPU, and DLA resources.

  • Results

    30.02% speedup is achieved for large image sizes compared with the most optimized design without NVJPEG.

  • Takeaways & Limitations

    NVJPEG can improve large-image edge inference throughput, while pipeline tuning must account for image size and hardware-unit characteristics.

Abstract

from arXiv · show

Data preprocessing is a crucial part of deep learning workflows on edge devices. However, decoding data saved in JPEG format is very compute-intensive and occupies a major portion of the preprocessing pipeline. Therefore, increasing the decoding speed is vital for improving overall throughput, especially for inputs with large image sizes, which are often subject to preprocessing bottlenecks. On the other hand, edge devices are equipped with specialized hardware units to accelerate media processing and image decoding. For instance, the NVIDIA Jetson platform possesses a dedicated NVJPEG unit. These units can be used to enhance the performance of the preprocessing pipeline. This paper introduces the utilization of such specific hardware acceleration units for offloading decoding tasks. By combining this with a multi-instance approach, it allows for the parallelization of all compute resources including CPU, NVJPEG, GPU, and DLA in Jetson devices. In this work, we compare various potential pipeline designs. On ResNet18, ResNet50, and ResNet152, three models with different sizes, we evaluate the impact of batch sizes and image sizes, as well as the characteristics of GPU/DLA inference. Finally, a fine-tuning experiment for multi-instance design has been conducted. The multi-instance design with a specific hardware decoding unit involved offers up to 30.02% speedup for large image sizes, compared with the most optimized design without it. Based on these findings, we demonstrate the benefits of using the NVJPEG unit in deep learning workflows and provide guidelines for tuning and optimizing edge inference workflows.

I. INTRODUCTION

Edge vision inference depends on preprocessing, yet JPEG decoding can dominate that stage and constrain throughput. The paper motivates combining specialized decoding hardware with concurrent use of the device’s compute units.

  • A. Problem Statement: JPEG decoding consumes a significant portion of preprocessing time, especially for large images.In ResNet50 on ILSVRC 2012, preprocessing reaches 64% of workflow time, while decoding accounts for 52.54%–87.30% of preprocessing time depending on image size.
  • A. Problem Statement: 64% of the workflow can be preprocessing when ResNet50 runs on ILSVRC 2012.
  • B. Motivation: Specialized hardware can offload image decoding, but CPU resources may remain underutilized during that processing.
  • B. Motivation: Existing approaches lack a systematic treatment of preprocessing bottlenecks and specialized decoding across varying models and image sizes.

C. Contributions

The paper integrates NVJPEG with multi-instance preprocessing and evaluates Jetson AGX Orin hardware behavior to guide edge inference optimization.

  • C. Contributions: The proposed design combines NVJPEG with multiple pipeline instances to use NVJPEG, CPU, GPU, and DLA together.
  • C. Contributions: 30.02% speedup is achieved for large image sizes versus the most optimized design without NVJPEG.
  • C. Contributions: The evaluation varies batch size and image size and compares CPU versus NVJPEG decoding and GPU versus DLA inference.
  • C. Contributions: Jetson AGX Orin provides CPU, GPU, two DLAs, and an NVJPEG unit for the evaluated workflow.

C. Edge Inference

Edge inference uses optimized TensorRT engines and evaluates pipeline designs across hardware lanes. The designs separate preprocessing from inference while assigning decoding and inference to available units.

  • C. Edge Inference: TensorRT builds separate optimized engines for GPU and DLA inference, commonly using lower-precision datatypes such as INT8.
  • C. Edge Inference: The evaluated workflows divide processing into image decoding on CPU or NVJPEG and inference on GPU or DLA.
  • C. Edge Inference: Six pipeline designs are outlined, including CPU preprocessing, DALI-based preprocessing, DLA variants, and multi-instance configurations.
  • C. Edge Inference: Figure 1 represents single-instance pipeline workflows with hardware units as lanes and time progressing from left to right.

B. Preprocessing Pipelines

Preprocessing converts image files into model-ready tensors through reading, decoding, and transformation. The paper compares CPU-based, DALI-based, and NVJPEG-offloaded pipeline designs.

  • B. Preprocessing Pipelines: A preprocessing pipeline reads image files, decodes them, transforms them, and produces tensors for inference.
  • B. Preprocessing Pipelines: PyTorch executes reading, decoding, and transformation on the CPU while worker threads and prefetching overlap preprocessing with GPU inference.
  • B. Preprocessing Pipelines: DALI uses GPU acceleration for image transformation and outputs a GPU tensor that can directly feed inference.
  • B. Preprocessing Pipelines: NVJPEG offloads JPEG decoding and runs it in parallel with other devices using worker threads and an intermediate buffer.

C. Multi-Instance

The multi-instance design coordinates CPU, GPU, DLA, and NVJPEG resources concurrently, while instance selection and batch sizing must match the workflow bottleneck. Its preprocessing models account for reading, decoding, transfers, transformation, and NVJPEG-specific conversion overhead.

  • Multi-instance pipelines concurrently use the Orin’s CPU, GPU, DLA, and NVJPEG units to enhance overall throughput.
  • Duplicating instances enables interleaved execution, allowing one device to work while another instance waits on a different stage.For example, GPU inference on one instance can overlap with transformation for another.
  • Instance combinations should reflect the bottleneck: NVJPEG and CPU instances suit preprocessing-bound large-image workloads, while inference-bound workloads should use all available inference devices.
  • Excessive instance counts can add overhead and force smaller per-instance batches, potentially causing memory exhaustion and lower throughput.
  • CDPre time includes image reading, CPU decoding, host-to-device copying, and GPU transformation, whereas NDPre replaces CPU decoding with NVJPEG decoding and adds transpose and color-space conversion.

2) Timeline Comparison over Different Designs:

The designs differ in how they overlap preprocessing and inference across hardware units. NDPre offloads JPEG decoding to NVJPEG, while multi-instance variants interleave work and can engage more inference and preprocessing resources.

  • PyPre uses an intermediate buffer so GPU inference on the current batch overlaps with CPU fetching the next batch.The optimization target is reducing inference-device idle time while waiting for preprocessing.
  • CDPre-DLA offloads inference to DLA, allowing GPU transformation of the next batch to overlap with inference.
  • NDPre decouples NVJPEG decoding from the DALI pipeline with an intermediate buffer, enabling decoding to overlap with other operations.
  • NDPre can be less efficient than CDPre for GPU-bounded or small-image workloads because color-space conversion adds overhead.
  • Multi-instance execution interleaves instances and can include more GPU and DLA inference units than a fine-grained single pipeline.
  • MI-NVJ overlaps CPU, NVJPEG, GPU, and DLA activity, improving preprocessing-bound performance, especially for large images.

IV. EVALUATION AND ANALYSIS

The evaluation uses a Jetson AGX Orin software stack, ImageNet validation data, synthetic resolutions, and three ResNet models. GPU and DLA engines are built separately with INT8 quantization for edge-oriented inference.

  • Experiments run on a Jetson AGX Orin in MAXN power mode with JetPack 5.1.1, TensorRT 8.5.2.2, and CUDA 11.7.
  • DALI support on Jetson is experimental, and NVJPEG access uses a customized Python library wrapping the Jetson Linux Multimedia API.
  • The evaluation uses the ILSVRC 2012 validation set for batch-size analysis and a synthetic dataset with varying resolutions for image-size analysis.
  • ResNet18, ResNet50, and ResNet152 represent models with different computational demands.
  • GPU and DLA TensorRT engines are built separately, with INT8 quantization used to align inference with edge-device usage.

B. Comparing Preprocessing Pipelines

The comparison examines PyPre, CDPre, and NDPre across batch sizes and analyzes preprocessing-time composition. GPU transformation favors CDPre, while NVJPEG’s decoding advantage depends on image size and can be offset by conversion overhead.

  • CDPre consistently delivers better throughput than PyPre and NDPre, with GPU transformation helping amortize kernel-launch overhead as batch size increases.This improvement applies before memory capacity is exhausted.
  • NDPre is less sensitive to batch-size changes because its parallelism and intermediate buffer provide a fine-grained pipeline, but extreme batch sizes perform poorly.
  • NVJPEG does not improve performance for datasets with substantial image-size and aspect-ratio variation.
  • Decoding accounts for 53% of preprocessing time at 480x320 and reaches 87% at 3840x2160.
  • NDPre provides up to a 54% decoding-speed advantage over CPU decoding above 1024x768, but added conversion overhead removes a single-instance latency advantage over CDPre.

D. Inference Devices Comparison

The comparison examines how GPU- and DLA-based inference respond to batch size, image size, and model latency. GPU throughput generally saturates with larger batches, while DLA throughput peaks and can decline beyond its optimal batch size.

  • For GPU inference below 1280x720, increasing batch size raises throughput until gradual saturation.The CDPre preprocessing pipeline also favors larger batch sizes in this regime.
  • For GPU inference above 1280x720, large batches can cause out-of-memory errors and preprocessing bounds throughput.Across the three models, similar throughput is observed under this preprocessing-bound condition.
  • For DLA inference, increasing batch size does not necessarily improve overall throughput.The DLA has different performance characteristics because it is less powerful than the GPU and uses its own microcontroller and hardware buffers.
  • For ResNet18 and ResNet50, DLA throughput peaks at batch size 64 before decreasing.

E. Multi-Instance Design with NVJPEG Involved

The multi-instance evaluation studies pipeline combinations across GPU, DLA, CPU, and NVJPEG resources. NVJPEG-based multi-instance designs are especially beneficial for large, preprocessing-bound images, although device combinations and instance counts require tuning.

  • 1) Characteristic of Multi-Instance: Two CDPre instances reached 382.98 images/sec versus 358.02 images/sec with one, while six instances reached only 382.34 images/sec.Adding instances initially improved utilization, but further additions provided no additional gain.
  • 1) Characteristic of Multi-Instance: 498.14 images/sec with two CDPre and one NDPre instance improved throughput by approximately 30.02% over 383.10 images/sec without NDPre.A single NDPre instance reached 375.7 images/sec versus 358.02 images/sec for a single CDPre instance.
  • 2) Multi-Instance Tunning: For ResNet18 at 3840x2160, MI provided an 18.0% speedup over CDPre, particularly in large-image, preprocessing-bound scenarios.
  • 2) Multi-Instance Tunning: For ResNet152 at 640x480, MI provided a 13.3% advantage over CDPre by paralleling two DLAs and a GPU for inference.
  • 2) Multi-Instance Tunning: For ResNet50 at 2560x1440, MI-NVJ outperformed MI by 30.02%, whereas smaller image sizes may reduce overall throughput.MI-NVJ mandates NVJPEG decoding offload and generally performs better than MI for large image sizes.

V. RELATED WORK

Prior work addresses edge inference, GPU- or FPGA-assisted JPEG decoding, and parallel inference, but the paper identifies a gap in integrated preprocessing and inference optimization for Jetson edge devices. It combines NVJPEG offload with parallel use of CPU, GPU, and DLA resources and supplies tuning guidance.

  • Related studies examine TensorRT-based GPU/DLA parallel inference, GPU- or FPGA-offloaded decoding, and GPU-accelerated JPEG decompression.
  • The cited prior work focuses on server platforms with powerful GPUs rather than integrated preprocessing and inference on edge devices.
  • The paper introduces NVJPEG offload with a fine-grained pipeline and multi-instance design to use CPU, GPU, DLA, and NVJPEG concurrently.
  • The study evaluates Orin hardware characteristics and recommends profiling candidate multi-instance combinations after tuning DLA and GPU batch sizes.
Loading 2608.27655v1…