PaperlayerView exact revision on arXiv

Source-linked AI summary

Seeing at the Edge: Energy Characterization of Vision-Language Model Inference on Edge Platforms

Junlin Liao, Harshit Kondle, Di Liu, Zonghua Gu

arXiv:2610.20136v1cs.LG

TL;DR

Energy evidence for multimodal inference on edge single-board computers remains limited. The paper measures complete-system energy across CPU and accelerator paths and separates request-fixed from output-dependent costs. It reports substantial Pi-NPU energy savings and finds that, at typical response lengths, decode energy exceeds the image increment.

  • Problem

    Published energy data for multimodal inference on battery-powered or passively cooled edge platforms remain scarce.

    Source: §I · p. 1

  • Method

    The study measures complete-system energy for paired vision-language and text workloads across Pi and Jetson CPU and accelerator paths, separating fixed request energy from output-length-dependent energy.

    Source: D. Three-condition modality ablation · p. 4, §I · p. 7

  • Results

    At typical ≈120-token responses, the no-image decode term is 4–36× the whole image/no-image increment; Pi-NPU reduces VLM energy per generated token by 65.5% gross versus Pi-CPU.

    Source: §VI · p. 9, §VII · p. 10

  • Takeaways & Limitations

    Pi-NPU offloading improves energy efficiency and thermal headroom on the tested Pi stack, while the findings do not isolate silicon effects from software-stack differences.

    Source: §VI · p. 9, §VII · p. 10

    Results cover one 2B VLM family and one unit of each platform, and quantization and serving differences prevent attributing energy or accuracy differences to silicon.

    Source: §VI · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) are moving onto edge devices, but little is known about how much energy they draw on single-board computers with NPU or GPU accelerators, or where that energy goes. We measure total-system energy, latency, and thermal behavior of Qwen2-VL-2B-Instruct and its text-only sibling on a Raspberry Pi 5 with a Hailo-10H NPU and on a Jetson Orin Nano Super, on both CPU and accelerator, and separate the fixed energy of each request from the energy that grows with output length. Because the accelerators use different quantization and serving software, we compare deployment stacks rather than silicon. Three findings stand out. First, NPU offloading makes the Pi a much more efficient VLM platform: it cuts energy per generated token by about two-thirds, leaves the host CPU largely free, and avoids the thermal throttling that the CPU backend reaches in sustained use. Second, an image adds both fixed and per-token energy, but at typical response lengths generating the answer costs far more than processing the image. Third, the Jetson GPU favors short responses through its lower fixed cost per request, whereas the Pi NPU favors longer ones through its lower cost per output token; their energy curves cross near 40 output tokens.

I. INTRODUCTION

Energy data for multimodal inference on edge single-board computers remain scarce. This work measures complete-system energy across four deployment paths and examines image costs, output length, and quality–energy trade-offs.

I. INTRODUCTION

  • Battery-powered and passively cooled devices may struggle to sustain multi-watt CPU inference, while published multimodal energy data on these platforms are scarce.Prior work named VLM workloads on the Pi 5 and Hailo-10H as an open problem.Source: §I · p. 1
  • The measurement plan compares Pi-CPU, Pi-NPU, Orin-CPU, and Orin-GPU energy and latency, image costs, and the effect of output length.An INA228 measures paired workloads on both complete SBC systems.Source: §I · p. 1
  • The study separates fixed request energy from output-dependent decode energy to explain where the NPU and GPU stacks cross.It compares deployed stacks because the Hailo and llama.cpp paths differ in quantization, compiler, runtime, and serving software.Source: §I · p. 1
  • The contributions include a quality–energy study across image resolutions and output caps using deterministic grading.The paper also quantifies fixed and output-dependent image-conditioned inference costs and compares four deployment paths under contention and sustained operation.Source: §I · p. 1

II. RELATED WORK

Prior work covers edge LLM benchmarking, VLM energy profiling, and efficient inference, but multimodal total-system energy on NPU-equipped SBCs remains unreported. The study measures that setting using paired model workloads and complete deployment stacks.

II. RELATED WORK

  • Existing mobile multimodal benchmarks do not report hardware-integrated total-system energy on NPU-equipped SBCs; this study measures it alongside task quality and latency.Prior VLM energy work used GPU-class hardware, while earlier Pi 5 + Hailo-10H benchmarking was text-only.Source: §II · p. 2

A. Raspberry Pi 5 and AI HAT+ 2

  • The Pi AI HAT+ 2 pairs the Raspberry Pi 5 with a Hailo-10H NPU that supports generative workloads through HailoRT and GenAI serving interfaces.The Hailo-10H delivers approximately 40 TOPS (INT4) with 8 GB of dedicated LPDDR4X.Source: A. Raspberry Pi 5 and AI HAT+ 2 · p. 2

B. Workload pair

  • Qwen2-VL-2B-Instruct and Qwen2-1.5B-Instruct enable image-conditioned, no-image VLM, and text-only comparisons, while their remaining differences make VLM–LLM results model-level comparisons.The Pi-NPU and Orin-GPU labels refer to complete hardware-and-software stacks, and energy and accuracy differences are properties of those stacks.Source: B. Workload pair · p. 2

C. Inference phases

The paper distinguishes input prefill from sequential output decoding, measures complete-system request energy at platform inputs, and uses a paired 30-prompt suite to compare vision-language and text workloads.

C. Inference phases

  • Inference consists of prefill over the full input context and autoregressive decode, with decode dominating wall time for long outputs.For VLMs, prefill includes vision encoding and multimodal projection; decode is memory-bandwidth-bound.Source: C. Inference phases · p. 2

A. Instrumentation

  • An INA228 at each platform’s input measures host, DRAM, accelerator, and peripheral energy jointly; request energy spans request construction through the complete response.Jetson internal rails are telemetry only and exclude carrier-board, fan, and regulator losses; they under-report input power by up to approximately 50% before calibration.Source: A. Instrumentation · pp. 2–3
  • The energy monitor was not calibrated against a traceable reference load, limiting calibrated claims about absolute Joules.Shared proportional errors cancel to first order in within-platform and Pi/Jetson ratios, but not in absolute Joules.Source: A. Instrumentation · p. 3

B. Prompt suite

  • The prompt suite uses 30 paired tasks across short, medium, and long inputs, matching each vision-language prompt with a text-described scene for the LLM.Only the final response is generated and measured.Source: B. Prompt suite · p. 3

C. Execution conditions and protocol

The protocol standardizes latency measurement, initial thermal state, and statistical comparisons across backends. Separate modality and output-length probes estimate the image-conditioning and decode-energy components.

C. Execution conditions and protocol

  • Client-observed wall time is the primary latency metric across backends, while backend-reported prefill and decode fields are secondary.Forced-output wall-time decode rates match backend-reported rates within 0.2%; Hailo prefill comparisons remain approximate because its timing field may include vision encoding.Source: C. Execution conditions and protocol · p. 3
  • Each prompt receives one warm-up and three measured trials, with every trial starting below 60 °C and the prompt order fixed across blocks.CPU and GPU server processes start and stop with each block, while the Hailo daemon remains resident throughout the Pi campaign.Source: C. Execution conditions and protocol · p. 3
  • Prompt-level means are compared with paired tests across 30 prompts, with Holm correction across 15 hypotheses and 95% confidence intervals.Wilcoxon signed-rank tests provide a distribution-free check.Source: C. Execution conditions and protocol · p. 3

D. Three-condition modality ablation

  • The modality ablation compares image and no-image VLM requests using the same model, prompt, and decoding settings; the image increment includes more than vision encoding.It also includes image loading and preprocessing, projection, visual-token prefill, request overhead, and any output-length change; VLM-T responses were not retained.Source: D. Three-condition modality ablation · p. 4

E. One-token proxy and vision-token scaling

  • A one-token proxy estimates energy per additional output token from full-run energy and generated length, while separate sweeps vary image resolution and image count.The full-run and one-token campaigns were separate executions, making their paired contrast noisy; negative differences are retained and flagged.Source: E. One-token proxy and vision-token… · p. 4
  • A fixed-length probe forces outputs from 1 to 256 tokens on Pi-NPU and Orin-GPU, then regresses energy on achieved length.It holds one medium prompt and image fixed and tests VLM-V at three resolutions.Source: E. One-token proxy and vision-token… · p. 4

F. Quality–energy protocol

The protocol pairs objective quality grading with energy measurements across controlled prompts, image-resolution and output-cap sweeps, contention tests, and sustained thermal sessions. Measurements were checked for validity, with the reported campaign spanning both platforms and backend conditions.

F. Quality–energy protocol

  • Twenty-five of 30 prompts have objective answers, and a prompt is correct only when every required component matches; five generative prompts are checked for format only.Energy per correct answer sums mean request energy for gradable prompts and divides by the number of correct prompts.Source: F. Quality–energy protocol · p. 4
  • Two single-run sweeps vary image long-side resolution and output cap, with intervals resampling the 25 gradable prompts and reflecting prompt sampling only.The resolution sweep uses 448/672/1120 px images and a 256-token cap; the output-cap sweep uses 32/64/128/256 tokens.Source: F. Quality–energy protocol · p. 4
  • Three 30-minute sessions per backend log power, temperature, clock frequency, and completions, while stress-ng measures host contention during inference.Pi throttling is read from the hardware flag; no reliable Jetson throttling flag was available.Source: F. Quality–energy protocol · p. 4

V. RESULTS

  • Most results come from one measurement occasion per platform, and the fixed-length probe ran the following day under the same protocol and power mode.The Jetson was run in verified MAXN SUPER mode.Source: §V · p. 4
  • All per-request trials passed the validity gate; accumulator/software disagreement averaged 2.67% and reached at most 4.77%.The campaign included 540 ablation and 540 one-token trials per platform, and the Pi and Jetson GGUF and mmproj files were byte-identical.Source: §V · p. 5

A. Headline energy and throughput

On the Pi, NPU offloading substantially reduces VLM energy per generated token while increasing decoding speed and reducing completion time. Energy per token varies with prompt length and modality, with image conditioning increasing it on the Pi.

A. Headline energy and throughput

  • 65.5% lower VLM energy per generated token (1959→676 mJ) and 28.9% lower mean power (9.22→6.56 W) accompany 2.75× faster decoding and 2.34× shorter end-to-end completion with NPU offloading.All 15 paired CPU–NPU contrasts remained significant after Holm correction (adjusted p < 0.001; n = 30 prompts).Source: A. Headline energy and throughput · p. 5

B. Per-category behavior

  • For VLM-V, energy per token falls 25% on CPU (2340→1751 mJ) and 50% on NPU (998→495 mJ) from short to long prompts.Fixed per-request and per-image costs are amortized over longer generations.Source: B. Per-category behavior · p. 5
  • The VLM-to-LLM energy ratio contracts from 2.05× to 1.49× on CPU and from 1.70× to 1.53× on NPU between short and long categories.The ratio is largest in the short category, where fixed request costs are least amortized.Source: B. Per-category behavior · p. 5

C. Modality ablation

  • Adding an image raises Pi energy per token by 574±120 mJ on CPU (+41%) and 185±48 mJ on NPU (+38%).The LLM-T→VLM-T step is positive on the Pi but negative on the Jetson; image conditioning increases energy on every backend.Source: C. Modality ablation · p. 5, Figure 1 · p. 5

D. One-token request decomposition

One-token request energy separates fixed request costs from energy associated with visual tokens and additional output tokens. Vision-token energy scales nearly linearly on the accelerated paths, while host contention is less severe with accelerator offload.

D. One-token request decomposition

  • The one-token VLM-V−VLM-T contrast is the cleaner image-attributable fixed-energy proxy because VLM-T did not terminate earlier than VLM-V.One-token request energy is 11.8% of per-inference VLM-V energy on CPU and 24.1% on NPU; marginal image-conditioning energy is 18.64 J on Pi CPU versus 6.54 J on NPU.Source: D. One-token request decomposition · p. 5

E. Vision-token energy scaling

  • One-token request energy rises near-linearly with vision-token count at 5.90 mJ/vision-token on the Orin GPU and 4.82 mJ/vision-token on the Pi NPU.The slopes are within 1.22×; reducing image long-side from 1120 to 448 px cuts total one-token request energy by about 4.1× on GPU and 1.5× on NPU.Source: E. Vision-token energy scaling · p. 6, Figure 2 · p. 6

F. Host contention

  • Under host CPU stress, Pi CPU-only VLM decode throughput falls 45.4% and energy per token rises 110.5%, versus 12.3% and 37.3% with NPU offload.Mean host CPU utilization is 87–99% for CPU backends versus 17% on Pi NPU and 3% on Orin GPU; stress-ng retains 52.3% of isolated throughput alongside CPU inference and 95.1% alongside NPU inference.Source: F. Host contention · p. 6

G. Sustained thermal behavior

Across the accelerator stacks, which platform uses less energy depends on response length, while the Pi NPU avoids the sustained thermal throttling observed on the Pi CPU.

G. Sustained thermal behavior

  • The Pi CPU reached 83.0–84.9 °C, triggered the throttle flag in all three sessions, and its decode rate fell 23.8–24.3%; the Pi NPU stayed at 53.0–55.0 °C without throttling.NPU decode rate changed only −0.2% to +0.3% over the session.Source: G. Sustained thermal behavior · p. 6

H. Cross-platform comparison: Jetson Orin Nano Super

I. Image-ingestion/decode and platform crossovers

The fitted energy model separates request intercepts from per-token decode costs, revealing distinct within-stack image/decode equalities and a cross-platform energy crossover near 40 output tokens.

  • The fixed-length experiment fits request energy as an intercept plus marginal energy per achieved output token, splitting the intercept into non-image base and image-attributable terms.It forces seven output lengths from 1 to 256 tokens at three resolutions, with 21 trials per fitted path.Source: §I · p. 7
  • Across 448–1120 px, Orin-GPU fixed request intercepts are 51–81% lower (1.71–6.59 vs. 9.23–13.41 J), while Pi-NPU decode costs are 30–31.5% lower (384–404 vs. 556–583 mJ/token).Source: §I · p. 7
  • The fitted NPU–GPU total-request energy crossover is about 44, 42, and 38 tokens at 448, 672, and 1120 px; across 30 prompts, the median crossover is 40 tokens.Below the crossover Orin-GPU uses less energy; above it Pi-NPU uses less.Source: §I · pp. 7–8, Figure 5 · p. 8
  • At 120 tokens, no-image decode energy is 4.3–12.5× (Pi-NPU) and 6.6–35.8× (Orin-GPU) the image/no-image energy difference.Within a stack, image-attributable fixed energy equals decode energy at about 2–12 tokens, a different equality from the cross-platform crossover.Source: §I · p. 8
  • For Orin-GPU fits, one-token residuals reach 11.7% relative error; the resulting intercept bias could shift the crossover about 1.4 tokens, beyond bootstrap half-widths of about 0.6 tokens.Those intervals capture trial-resampling uncertainty only.Source: §I · p. 8

J. Exploratory quality–energy evaluation

Accelerators used less VLM energy per correct answer than their CPU counterparts in the native pass. Exploratory sweeps found non-monotonic energy per correct answer, with observed minima at 672 px and a 64-token cap, but adjacent-setting intervals overlap.

VI. DISCUSSION

The discussion compares the measured deployment paths with prior results, separates gross from idle-subtracted energy, and relates image and output-length costs to accelerator choice. Its conclusions are bounded by the tested models, devices, measurement conditions, and deployment stacks.

Comparison with the literature

  • 12.7 vs. 7.8 tok/s was Le and Durmaz Incel’s CPU-faster Qwen2-1.5B result; this campaign reversed the ordering (8.1 vs. 22.3 tok/s, 2.77× paired).Other measurements reported CPU-faster decode (9.72 vs. 7.05 tok/s for Qwen2.5-1.5B) and vendor figures reported 9.45 tok/s for Qwen2-1.5B-Instruct on the NPU.Source: §VI · p. 9

Implications for deployment

  • Vision-token scaling was near-linear at ≈5–6 mJ/token, while adding an image raised energy per token by 36–67% and weighed most on short queries.The image-conditioned request used 2.05× the text sibling’s energy on the Pi CPU; minimizing ingestion energy alone could move away from the observed lowest energy per correct answer.Source: §VI · p. 9
  • The Pi-NPU and Jetson stacks favor different response lengths: the GPU decoded 1.53× faster and used less energy below the 38–44-token crossover, while the Pi-NPU used less energy for longer outputs.Mean per-token energy was similar; the Pi-NPU drew about half the input power, and offloading left ≈83% of the Pi host CPU and ≈97% of the Jetson host CPU free.Source: §VI · p. 9

Limitations

  • The study covers one 2B VLM family, one unit per platform, and mostly one measurement occasion, so fitted constants and crossover lengths may shift with models, hardware, or toolchains.Sequential blocks also leave slow drift unrandomized, and reported intervals cover prompt-level variation only.Source: §VI · p. 9
  • Quantization and serving differences prevent attributing energy or accuracy differences to silicon; absolute Joules are uncalibrated, and the crossover rests on one prompt/image probe plus a natural-suite consistency check.The image cost includes more than the vision encoder, and the Hailo path accepts only complete image-plus-text requests.Source: §VI · pp. 9–10

VII. CONCLUSION

The paper measures total-system energy, latency, and thermal behavior for VLM inference across Raspberry Pi and Jetson CPU and accelerator paths. It finds that the Pi NPU benefits VLM deployment, image costs are outweighed by decoding at typical lengths, and the preferred accelerator depends on response length.

VII. CONCLUSION

  • 65.5% lower VLM energy per generated token (1,959→676 mJ) and 2.75× higher decode throughput came with Pi-NPU offloading versus the Pi CPU backend.Across three 30-minute sessions, the NPU never set the throttle flag set by the CPU in every session and ran ≈30 °C cooler.Source: §VII · p. 10
  • At typical ≈120-token responses, the no-image decode term was 4–36× the whole image/no-image energy increment.Adding an image raised energy per token by 36–67%; one-token energy scaled near-linearly with vision-token count at 4.82–5.90 mJ/token (R2 ≥0.99).Source: §VII · p. 10
  • The Orin-GPU stack’s 51–81% lower fixed request intercept and the Pi-NPU stack’s 30–31.5% lower marginal decode cost explain their crossover near 38–44 output tokens.Mean per-token energy was 676 vs. 717 mJ, while the Orin-GPU stack decoded 1.53× faster.Source: §VII · p. 10
  • An exploratory single-run sweep found the lowest energy per correct answer at 672 px image resolution and a 64-token output cap; repeated quality sweeps are a next step.The paper also identifies encoder-level energy on stacks that expose the vision encoder, and other model families and sizes, as future work.Source: §VII · p. 10
Loading 2610.20136v1…