PaperlayerView exact revision on arXiv

Source-linked AI summary

HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications

Inesh Chakrabarti, Zejun Xiong, Pragya Sharma, Mani Srivastava

arXiv:2609.12412v1cs.DCcs.LGcs.PF

TL;DR

Existing benchmarks assess model capability apart from hardware-specific deployment costs, while profiling tools lack unified cross-platform coverage. HoliBench combines measurement across foundation-model modalities and devices with constraint-aware configuration selection. In sequential co-resident CPS deployment, standalone profiles predict pipeline latency and power within 1.2% and 2.5%.

  • Problem

    Capability benchmarks omit device-specific latency and energy, while profiling efforts use incompatible setups and cover limited portions of the model-device space, complicating deployment comparisons.

    Source: §1 · p. 1

  • Method

    HoliBench provides cross-platform foundation-model profiling and constraint-aware configuration selection, using a platform abstraction layer to standardize measurement across heterogeneous devices.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    Standalone profiles predicted sequential co-resident CPS pipeline latency and power with errors of 1.2% and 2.5%, respectively.

    Source: §8 · p. 12

  • Takeaways & Limitations

    Across the evaluated models, quantizations, and devices, deployment is a hardware-software co-design problem: the best configuration maps its execution profile to target hardware capabilities and constraints.

    Source: §8 · p. 12

    Composability validation used the HuggingFace Transformers backend on one device; concurrent execution, dynamic batching, and optimized-runtime validation remain future work.

    Source: §8 · p. 12

Abstract

from arXiv · show

Foundation models, including large language models, vision-language models, and time-series foundation models, are increasingly deployed on embedded and edge platforms for CPS and IoT applications, where energy, latency, and memory are as critical as task accuracy. Existing benchmarking tools evaluate model capability in isolation, reporting accuracy assuming sufficient compute, while hardware profiling tools remain platform-specific and mutually incompatible. As a result, users lack a unified workflow for making deployment decisions across heterogeneous devices. We present HoliBench, a modular benchmarking and deployment toolkit that jointly characterizes accuracy, latency, and energy across platforms from single-board computers to GPU servers. Its platform abstraction layer calibrates cross-device measurement, and the toolkit supports multiple model modalities, inference engines, concurrencies, and existing evaluation harnesses. An interactive interface exposes constraint-aware configuration selection over a design space that is profiled once and reused across studies. Using HoliBench, we characterize 20 models across 7 device types, 3 quantization levels, 8 inference backends, and over 30 tasks, surfacing tradeoffs that existing tools miss: quantization reduces latency only on hardware with low-precision support, accuracy gains show diminishing returns relative to energy, and for autoregressive workloads, average inference power is approximately constant across output lengths. We further find that single-model profiles compose under sequential co-resident execution. In a multi-model CPS deployment, standalone profiles predict combined-pipeline latency and power within 1.2% and 2.5%, enabling deployment exploration without exhaustively profiling every pipeline configuration. We release HoliBench as open-source infrastructure for deployment-aware evaluation of foundation models.

1 Introduction

Existing benchmarks separate capability from deployment measurement, leaving users without a unified way to compare foundation-model configurations under real device constraints. HoliBench combines cross-platform profiling and constraint-aware selection, and its standalone profiles predict sequential multi-model deployment measurements closely.

1 Introduction

  • LLM decoding, VLM vision encoding and generation, and TSFM single-pass windows have different hardware demands, so aggregate model measurements can obscure deployment bottlenecks.Profiling must account for modality-specific workload structure for comparisons to generalize across modalities.Source: §1 · p. 2
  • Platform APIs differ in interfaces, sampling granularity, and measurement semantics, so cross-device measurements require normalization, calibration, and temporal alignment.The differences span ARM SBCs, GPU-capable devices, and unified-memory platforms.Source: §1 · p. 2
  • HoliBench supports LLMs, VLMs, and TSFMs across SBCs to GPU servers, with multiple inference engines, quantization backends, and evaluation harnesses.Its architecture is modular and extensible.Source: §1 · p. 2
  • 1.2% latency error and 2.5% power error show that standalone profiles predict a sequential co-resident multi-model CPS pipeline.This supports deployment exploration without exhaustively profiling each candidate pipeline configuration.Source: §1 · p. 2
  • Its platform abstraction layer unifies power, energy, memory, and utilization profiling through synchronized measurement, temporal alignment, and consistent metrics.This is intended to make measurements comparable across heterogeneous hardware.Source: §1 · p. 2
  • Across 20 models, 7 devices, and 3 quantization levels, quantization benefits depend on hardware and runtime support, while accuracy gains diminish relative to energy cost.Average inference power is largely determined by the target device rather than model size.Source: §1 · p. 2
  • The interactive interface identifies feasible model-device-quantization configurations under joint accuracy, latency, energy, and memory constraints.This addresses a design space that can reach 200 configurations even for 10 models, 4 quantization levels, and 5 devices.Source: §1 · p. 2

2 Related Work

Prior work covers edge deployment characterization, capability benchmarks, and hardware profiling, but each leaves gaps in modality breadth, cross-platform comparability, or deployment decision support. HoliBench combines these functions in a cross-platform framework spanning foundation-model modalities and constraint-aware configuration exploration.

2.1 Foundation Model Deployment and Characterization

  • Edge characterization studies provide useful but disjoint snapshots: MELT uses one backend and excludes VLMs and TSFMs, while other studies focus on selected devices or sub-3B models.Their independent infrastructure, metrics, and baselines make cross-study comparison difficult.Source: §2.1 · pp. 2–3

2.2 FM Capability Benchmarks

  • Capability benchmarks report accuracy, perplexity, and calibration but do not measure the latency or energy cost of running configurations on specific devices.This limits their use for assessing viability on embedded and edge platforms.Source: §2.2 · p. 3

2.3 Hardware-Aware Profiling Infrastructure

  • HoliBench combines cross-platform measurement, LLM/VLM/TSFM and inference-engine coverage, and constraint-aware configuration selection—capabilities absent from prior profiling frameworks.Existing hardware profiling efforts are limited by scope, device class, or lack of unified measurement infrastructure.Source: §2.3 · p. 3

3 System Architecture

HoliBench uses a modular architecture to profile foundation-model workloads across heterogeneous platforms and select configurations under deployment constraints. Its measurement and timing layers account for platform telemetry and modality-specific inference behavior.

3 System Architecture

  • The four-layer architecture integrates user-specified models, quantization, tasks, devices, and engines through fixed interface contracts that allow extensions without changing orchestration.Figure 1 shows configuration selection as the broader concept, with deployment optimization as one solver-based realization.Source: §3 · p. 3, Figure 1 · p. 3

3.1 Platform Abstraction Layer (PAL)

  • The Platform Abstraction Layer normalizes telemetry into consistent physical units, aligns measurements with inference windows, and records peak and average memory.Peak memory determines whether a configuration fits; average memory characterizes sustained pressure under concurrent workloads.Source: §3.1 · pp. 3–4
  • HoliBench reports gross energy and net inference energy, deriving energy from either sampled power or cumulative counters and subtracting the post-load idle baseline.Gross energy reflects system draw for power budgeting, while net inference energy isolates marginal computational cost for cross-device comparison.Source: §3.1 · pp. 3–4

3.2 Inference Timing Model

  • LLMs are timed across prefill and decode, VLMs across vision encoding and decode, and TSFMs as single-pass inference, with power sampled within the profiled window.This phase-level decomposition supports latency interpolation and time-weighted power estimation in the multi-model case study.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3 Modality-Aware Benchmarking

  • Modality-aware scenarios adapt evaluation protocols to output semantics, while actual quantized execution is profiled on each device rather than inferred from bit width.Inference is supported through native forward passes and eight higher-level runtimes or engines, with batching also supported.Source: §3.3 · pp. 4–5

3.4 Constraint-Aware Configuration Selection

  • A pluggable solver ranks configurations that pass hard feasibility checks, including peak-memory limits, using weighted normalized objectives and user-defined deployment constraints.The interface also permits strategies such as random search or greedy weighted-utility ranking.Source: §3.4 · p. 5

4 Technical Implementation

HoliBench is implemented as modular Python infrastructure for configuring experiments, profiling hardware, running model scenarios, and presenting deployment selections.

4 Technical Implementation

  • The implementation comprises approximately 15,000 lines of Python organized around the toolkit’s four modular layers.The section describes implementation choices that realize the architecture.Source: §4 · p. 5

4.1 Configuration and Experiment Orchestration

  • Hydra composes YAML configurations across model, device, and scenario axes, supporting command-line overrides and reproducible sweeps without editing files.Hydra records the configuration with each captured result.Source: §4.1 · p. 5

4.2 Hardware Profiling

  • Hardware profilers register availability predicates and are instantiated for the host environment, so adding a device requires a conforming profiler class rather than core-framework changes.Each profiler implements start(), sample(), and stop().Source: §4.2 · p. 5

4.3 Model Loading

  • The model loader routes first by modality and then by inference engine and quantization or concurrency settings, unloading models after each scenario to free device memory.Backend-specific libraries are imported only when their loader is instantiated.Source: §4.3 · p. 5

4.4 Benchmark Suite

  • The benchmark suite runs valid test-matrix combinations in sequence, skips configurations that fail pre-flight memory checks, and aggregates run summaries into comparative outputs.The hardware-testbed table distinguishes VRAM, unified CPU/GPU memory, and system RAM by platform type.Source: §4.4 · p. 5, Table 1 · p. 6

4.5 LLM-as-a-Judge

  • For open-ended generation tasks, an offline re-evaluation tool uses Qwen2.5-32B-Instruct as an LLM judge with a structured rubric for task-specific criteria.The rubric covers exact match, semantic equivalence, hierarchical label matching, multiple choice, and classification.Source: §4.5 · p. 6

4.6 Configuration Selection Front-End

  • The Flask interface filters profiled results by device and scenario, lets users adjust objective weights and hard constraints, and displays ranked configurations alongside a feasible-design-space plot.The merged table includes latency, energy, and accuracy; users may also constrain power, memory, or minimum accuracy.Source: §4.6 · p. 6

5 Experimental Setup

The evaluation exercises HoliBench’s platform abstraction and modality-aware drivers on diverse platforms and foundation-model workloads. It uses single-query sequential inference and validates accuracy measurements against an established harness on overlapping tasks.

5.1 Hardware Testbed

  • The seven-platform testbed spans four deployment tiers and distinct profiler backends, with telemetry normalization evaluated for commensurability across platforms.The devices represent power-constrained, GPU-accelerated, cloud, and unified-memory workstation settings.Source: §5.1 · p. 6

5.2 Model and Task Selection

  • The model set covers LLMs, VLMs, and TSFMs, while evaluation scenarios span classification, generation, and regression with 100 samples per setting.Models range from small time-series models to LLMs with up to 8B parameters.Source: §5.2 · pp. 6–7

5.3 Measurement Protocol

  • Profiling uses single-query sequential inference, samples power at 100 ms for LLMs and VLMs and 10 ms for TSFMs, and crosses models, up to three quantization levels, and seven platforms.Accuracy measurements are validated against EleutherAI’s Model Evaluation Harness on overlapping models and tasks.Source: §5.3 · p. 7

6 Evaluation

HoliBench’s evaluation finds that deployment outcomes depend on energy as well as accuracy, and vary with device, quantization, and runtime. Its cross-platform profiles reveal these tradeoffs and show that several characterization patterns persist across inference backends.

6 Evaluation

  • HoliBench organizes its evaluation around deployment tradeoffs that accuracy-only frameworks do not capture.The evaluation presents three characterization findings followed by cross-platform validation of the platform abstraction layer.Source: §6 · p. 7

6.1 Diminishing Accuracy Returns Under Energy Constraints

  • Above approximately 80% accuracy, the summarization Pareto frontier steepens, and configurations at comparable accuracy can differ by as much as 12-fold in energy.The energy cost per accuracy point in the upper range exceeds that in the lower range by over an order of magnitude.Source: §6.1 · p. 7, Figure 4 · p. 7

6.2 Latency-Energy Linearity and Predictive Power Estimation

  • Autoregressive inference energy scales linearly with latency, and average inference power is approximately stable for a given device across models and workloads.A reasoning model that generated hundreds of intermediate tokens for a classification task deviated from the trend, with output inflation raising its energy cost.Source: §6.2 · pp. 7–8, Figure 5 · p. 8

6.3 Hardware-Dependent Quantization Effects

  • INT4 quantization does not uniformly reduce latency: its effects depend on model size, runtime, and hardware, and can reverse into substantial penalties.On the RTX 3070, 4B models speed up 54–60%, while sub-1B models slow by 43–131%; on the Jetson, Qwen3-0.6B incurs a 478% penalty.Source: §6.3 · p. 8, Figure 5 · p. 8

6.4 Cross-Platform Measurement Validation

  • The A5000 can complete the same VLM inference faster yet consume more energy than the Jetson, so platform choice changes the latency-energy tradeoff.Task accuracy remained stable across platforms, and HoliBench’s abstraction layer made measurements from different telemetry sources commensurable.Source: Figure 6 · p. 8, §6.4 · p. 9

6.5 Cross-Backend Generalization

  • Across backends, energy-latency linearity persists, but vLLM speeds every tested model on the A5000 while slowing models at 2B parameters and above on the Jetson.Adding vLLM and other backends required no changes to HoliBench’s platform abstraction layer, modality-aware driver, or orchestration logic.Source: §6.5 · pp. 9–10, Figure 7 · p. 9
  • On Falcon3-1B, SGLang outperforms HuggingFace in prefill and decode throughput, while decode throughput scales more with batch size than prefill throughput.At higher concurrency, SGLang’s relative prefill throughput advantage converges to approximately 1.5×.Source: §6.5 · p. 10, Figure 8 · p. 10

7 Case Study: Profile Composability for Multi-Model CPS Deployment

In a CARLA perception-planning deployment, HoliBench uses standalone profiles to select and predict the cost of a co-resident VLM–LLM pipeline. The solver-selected configuration meets latency and power constraints more reliably than the accuracy-greedy choice, under the tested sequential-execution conditions.

7 Case Study: Profile Composability for Multi-Model CPS Deployment

  • The case study tests sequential, co-resident perception and planning on a Jetson AGX Orin 64 GB, with a 1.6 s cycle deadline and 30 W sustained-power budget.The models execute in a CARLA urban-driving simulation, and all candidate pairings fit in memory.Source: §7 · pp. 10–11

7.3 Runtime validation

  • The solver-selected pipeline’s predicted latency and power were within 1.2% and 2.5%, respectively, while the accuracy-greedy configuration missed the 1.6 s deadline every cycle and drew 32.0 W.The solver-selected configuration met the deadline on 88.8% of cycles; its tail violations were attributed to VLM input variance.Source: Figure 9 · p. 11, §7.3 · p. 12

7.1 Estimating deployment cost without task-specific profiling

  • For the approximately 25-token navigation command, HoliBench interpolates LLM decode latency and transfers measured power from other output lengths.Average power deviated by less than 16% across classification and summarization tasks for 9 of 10 configurations; VLM measurements came directly from image-captioning profiles.Source: §7.1 · p. 11

7.2 Configuration selection

  • The solver chooses Qwen2.5-VL-3B with SmolLM2-1.7B to satisfy latency and power constraints, accepting lower VLM accuracy than the accuracy-greedy pairing.The accuracy-greedy pairing exceeded the 30 W power budget and was estimated to reduce mission duration by approximately 24% relative to a power-feasible alternative.Source: §7.2 · pp. 11–12

7.3 Runtime validation

  • The observed profile composability depends on sequential execution, static GPU memory allocation, and temporal separation of compute phases.The authors note these conditions may not hold under concurrent execution or dynamic batching, where models contend for compute and memory bandwidth.Source: §7.3 · p. 12

8 Discussion and Conclusion

HoliBench shows that deployment behavior depends on hardware-runtime interactions, while standalone profiles can predict sequential co-resident pipeline behavior closely. The discussion frames deployment as hardware-software co-design and identifies validation boundaries.

8 Discussion and Conclusion

  • INT4 and INT8 latency benefits depend on hardware support and kernel implementation; dequantization overhead can offset or exceed memory-bandwidth savings.Hardware generation alone did not predict quantization behavior, so decisions require hardware-in-the-loop profiling.Source: §8 · p. 12
  • 1.2% latency and 2.5% power prediction errors under sequential co-resident execution on the embedded Jetson device show that standalone profiles tracked the solver-selected configuration.The time-weighted power model estimates pipeline energy from standalone measurements, while phase-level latency interpolation extends profiles to unprofiled tasks.Source: §8 · p. 12
  • On 24% of input frames, unexpectedly verbose outputs drove the accuracy-greedy configuration beyond deadlines on every cycle, behavior curated-input benchmarks missed.The failure occurred under sustained, diverse-input CPS conditions, rather than from resource exhaustion.Source: §8 · p. 12
  • Composability validation used HuggingFace Transformers on one device; concurrent execution, dynamic batching, and validation under optimized runtimes remain future work.The paper also notes that its standardized measurement semantics do not establish absolute equivalence among telemetry interfaces with different measurement scopes.Source: §8 · p. 12
  • HoliBench frames deployment as hardware-software co-design: the best configuration maps task execution profiles to target-hardware capabilities and constraints.The toolkit is open-sourced, with future work planned for multi-model concurrency and distributed execution across devices.Source: §8 · pp. 12–13
Loading 2609.12412v1…