PaperlayerView exact revision on arXiv

Source-linked AI summary

Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels

Amir Taherin, Sana Taghipour Anvari, Charles Amante, Yixiao Chen, Ruben Noroian, Zlatan Feric, Nicolas Bohm Agostini, Pu Zhao, José Cano, Bin Ren, Yanzhi Wang, David Kaeli

arXiv:2608.25053v1cs.ARcs.AIcs.DCcs.PF

TL;DR

Edge LLM inference on SoCs requires evidence beyond model size and precision because backend, platform, memory, and power behavior shape deployment. Hydra provides phase-aware, cross-backend measurement, showing that latency, resource utilization, and efficiency vary across backend structure, quantization, and SoC generation.

  • Problem

    Comparisons across SoC generations remain limited, despite the need to understand how evolving edge architectures change LLM inference behavior.

  • Method

    Hydra combines shared per-prompt timing across HuggingFace Transformers and llama.cpp with fused hardware telemetry aligned to prefill and decode phases.

  • Results

    Across three Jetson generations, 13 LLMs, five formats, and length sweeps, backend structure shifts latency, quantization reduces memory traffic and energy, and SoC generation affects utilization interpretation.

  • Takeaways & Limitations

    Edge LLM deployment should evaluate phase-level performance, utilization, and efficiency jointly across backends, quantization formats, SoC generations, and sequence lengths.

  • Takeaways & Limitations

    The evaluation uses single-stream sweeps, leaving longer contexts and concurrent serving for future work.

Abstract

from arXiv · show

Edge LLM deployment is shaped by more than model size and precision: inference backend, hardware platform, memory traffic, and power management all affect latency and efficiency. We present Hydra, a common-schema, phase-aware workload characterization framework for LLM inference on edge SoCs. Hydra instruments HuggingFace Transformers and llama.cpp with a shared per-prompt timing schema and fuses those records with hardware telemetry, enabling a multi-dimensional characterization of performance, system-resource utilization, and efficiency across prefill and decode phases. Using Hydra, we evaluate three consecutive edge System-on-Chip (SoC) generations (AGX Xavier, AGX Orin, and AGX Thor), 13 instruction-tuned LLMs from seven families, five execution formats, and consider input/output-length sensitivity. The resulting artifact contains roughly 107K per-prompt records and is publicly released with Hydra. Our analysis shows that aggregate latency alone hides key deployment effects: backend structure changes where latency is introduced, quantization reduces memory traffic and energy but does not predict power monotonically, and SoC generation changes how utilization and efficiency should be interpreted. By connecting phase-level timing with system-resource utilization and efficiency metrics, Hydra enables reproducible, phase-aware characterization of edge LLM inference. Hydra's source code and the collected per-prompt trace corpus are available open-source at: https://github.com/amirtaherin/hydra

I. INTRODUCTION

Hydra addresses three gaps in edge-LLM characterization by providing a common-schema, phase-aware framework spanning backends, SoC generations, and full-stack telemetry. Its evaluation shows that backend structure, quantization, SoC generation, and sequence length jointly shape performance, utilization, and efficiency.

  • Methodology: Hydra instruments HuggingFace Transformers and llama.cpp under a shared per-prompt schema and aligns timing with CPU, GPU, memory, power, energy, and thermal telemetry.The schema supports direct comparison of tokenization, prefill, generation, de-tokenization, TTFT, ITL, and end-to-end timing.
  • Key findings: Latency depends jointly on backend, model architecture, precision, and sequence length, with phase timing exposing effects hidden by end-to-end latency.Hydra distinguishes backend overhead, quantized decode throughput, and input/output-length scaling across inference phases.
  • Key findings: Resource utilization differs by backend, quantization, and SoC generation: quantization reduces DRAM traffic without necessarily reducing GPU occupancy, while raw utilization counters require normalization.Backend choice changes CPU-side orchestration and GPU effective utilization.
  • Key findings: Lower-bit formats often reduce energy per token, but power and thermal behavior depend on quantization implementation, runtime, and platform rather than bit-width alone.Faster configurations may draw more power while still using fewer joules per generated token.
  • Evaluation scope: Hydra evaluates 13 models across seven families, three SoC generations, two inference backends, five execution formats, and input/output-length sweeps.The study spans AGX Xavier, AGX Orin, and AGX Thor, representing Volta, Ampere, and Blackwell edge-GPU classes.
  • Artifact: Hydra releases unified per-prompt traces and analysis artifacts, including timing, system-resource, and efficiency aggregates, for reproducible cross-platform comparison.The source code and collected trace corpus are available open-source.

II. BACKGROUND AND MOTIVATION

LLM inference on edge SoCs is divided into prefill and decode phases that stress coupled hardware resources differently. Hydra addresses the resulting measurement gap with a common phase-aware schema spanning timing, resource utilization, and efficiency across inference backends.

  • Background: Prefill processes the input prompt and populates the KV cache, while decode generates tokens autoregressively.These phases can stress CPU cores, GPU cores, memory controllers, and power-management logic differently within a shared memory and thermal envelope.
  • Background: HuggingFace Transformers offers broad model coverage, whereas llama.cpp and GGML reduce software overhead and support efficient low-bit execution.Execution format includes 16-bit floating-point and lower-bit weight-only quantized representations; lower-bit formats can reduce model footprint and traffic.
  • Motivation: Aggregate reporting hides how performance, resource utilization, and efficiency interact across inference phases and coupled deployment dimensions.These dimensions include SoC generation, backend, execution format, and prompt/output length, while standard measurements often expose only one metric at a time.
  • Motivation: Hydra aligns token-level and phase-level timing with SoC resource signals and power, energy, and thermal behavior through a common per-prompt schema.The schema enables the same analysis pipeline across HuggingFace Transformers and llama.cpp.

III. HYDRA: PHASE-AWARE WORKLOAD CHARACTERIZATION · IV. HYDRA: WORKLOAD CHARACTERIZATION DESIGN

Hydra provides a phase-aware methodology that fuses backend timing with SoC telemetry into common per-prompt records. Its workload design separates SoC generation, model architecture, backend or format, and prompt/output-length effects across three Jetson platforms.

  • III. HYDRA: PHASE-AWARE WORKLOAD CHARACTERIZATION: Hydra converts heterogeneous backend execution and platform telemetry into common per-prompt records analyzable by phase, backend, precision, and SoC generation.The workflow includes shared timing instrumentation, telemetry fusion, and canonical-schema normalization.
  • III. HYDRA: PHASE-AWARE WORKLOAD CHARACTERIZATION: Hydra records tokenization, prefill, per-token generation, de-tokenization, ITL, TTFT, and end-to-end latency, while grouping events into prefill and decode telemetry windows.The shared windows align timing records and hardware counters for phase-attributed comparisons.
  • III. HYDRA: PHASE-AWARE WORKLOAD CHARACTERIZATION: HuggingFace Transformers and llama.cpp emit common per-prompt timing records despite their Python and C++ implementations and structurally different execution backends.Hydra synchronizes CUDA-dependent regions to measure completed GPU work and records llama.cpp performance counters for validation.
  • III. HYDRA: PHASE-AWARE WORKLOAD CHARACTERIZATION: Hydra samples CPU, GPU, memory-controller, RAM, power, and thermal signals, parses them onto the runtime profiler’s time base, and fuses them per prompt.Telemetry is aggregated over full-prompt, prefill, and decode windows.
  • III. HYDRA: PHASE-AWARE WORKLOAD CHARACTERIZATION: Hydra maps platform-specific telemetry into shared fields while preserving measurement asymmetries across GPU-utilization channels and power domains.Cross-platform analysis uses the canonical schema but interprets platform-specific telemetry differences explicitly when they affect results.
  • IV. HYDRA: WORKLOAD CHARACTERIZATION DESIGN: The experimental matrix separates SoC generation, model architecture, execution backend or format, and prompt/output length, which are often conflated in edge-LLM studies.The design includes instruction-tuned model families, multiple formats, and fixed-workload and length-sensitivity prompt sets.
  • IV. HYDRA: WORKLOAD CHARACTERIZATION DESIGN: The evaluation covers Xavier with Volta, Orin with Ampere, and Thor with Blackwell across three NVIDIA Jetson AGX generations.All three platforms use unified memory, so CPU and GPU activity share one DRAM pool without explicit host-to-device transfers in inference.

A. Models and Execution Formats · B. Evaluated Prompts and Sequence Lengths

Hydra evaluates model scale, architecture, backend, precision, and workload length across three edge SoC generations using a common execution policy. The study combines 13 instruction-tuned models, five formats, controlled IFEval workloads, and targeted input/output-length sweeps into roughly 107,000 per-prompt records.

  • A. Models and Execution Formats: Hydra characterizes 13 instruction-tuned decoder-only LLMs from seven families, spanning 1.24–8.54 B parameters.The families are LLaMA, Qwen, Granite, Gemma, Phi, Mistral, and Moxin.
  • A. Models and Execution Formats: The format axis includes HuggingFace bf16 and llama.cpp F16, Q8_0, Q6_K, and Q4_K_M, representing roughly 16, 8, 6.6, and 4.5 bits per weight.F16 is the 16-bit quality reference for llama.cpp quantization comparisons.
  • A. Models and Execution Formats: Native 16-bit cross-generation comparisons use llama.cpp F16 because Xavier lacks Ampere and Blackwell’s native BF16 tensor-core path.Xavier bf16 results therefore reflect backend and software behavior.
  • A. Models and Execution Formats: Q8_0 and Q6_K remain close to F16, while Q4_K_M stays within 1 PPL point and about 2.4 accuracy points in the worst case.Quality is evaluated on WikiText-2 perplexity and zero-shot Winogrande and HellaSwag accuracy.
  • B. Evaluated Prompts and Sequence Lengths: The main IFEval corpus contains 541 prompts across 25 verifiable instruction types, with 13–345-token inputs and a fixed 500-token decode budget.Early stopping is disabled so models execute a uniform decode workload.
  • B. Evaluated Prompts and Sequence Lengths: The main sweep spans three SoC generations, 13 models, and five formats, producing 195 cells; five Xavier F16 configurations fail to load, leaving roughly 103,000 records.The failures involve 7–8 B models and the JetPack 5 cuBLAS/NVMAP interaction.
  • B. Evaluated Prompts and Sequence Lengths: Sensitivity sweeps test 1k, 3k, and 5k input tiers or output budgets while controlling the other phase, adding roughly 4,300 records and bringing the study to about 107,000.They run on Orin and Thor using llama.cpp Q4_K_M and HuggingFace bf16 for six LLaMA/Qwen models; Xavier is excluded from long-context sweeps.
  • B. Evaluated Prompts and Sequence Lengths: All runs use batch size 1, greedy token-by-token decoding, KV-cache reuse, integrated-GPU CUDA execution, and full llama.cpp GPU offload.Configurations unable to allocate under n_gpu_layers=99 are marked failed rather than partially offloaded.

V. EVALUATION · A. Performance Analysis

Hydra evaluates edge LLM inference through performance, resource utilization, and efficiency, using phase-aligned timing and telemetry across SoC generations, backends, formats, models, and sequence lengths. Its performance analysis shows that platform, backend, quantization, and phase-specific effects are obscured by aggregate latency alone.

  • V. EVALUATION: Hydra connects per-prompt timing, phase-aligned telemetry, and power, energy, and thermal metrics to characterize performance, resource utilization, and efficiency.These dimensions expose latency and throughput behavior, explain CPU, GPU, and memory activity, and quantify deployment cost across backend, precision, model, and SoC choices.
  • A. Performance Analysis: Hydra’s performance analysis separates end-to-end latency, quantized decode throughput, phase-level latency attribution, and input/output-length sensitivity.Together, these views isolate backend, execution-format, model-architecture, and sequence-length effects.
  • A. Performance Analysis: 2.8–2.9× faster: Thor versus Xavier for HuggingFace bf16, while the Thor/Orin gap narrows from roughly 2.9× on LL-1B to about 1.4× on GE-7B.Newer SoC generations reduce latency substantially, but gains vary by model and platform.
  • A. Performance Analysis: llama.cpp consistently improves end-to-end latency over HuggingFace at similar 16-bit execution, with the largest gains on smaller models.Model families also show noticeably different latencies among similarly sized 7–8 B models.
  • A. Performance Analysis: 63.5 ms: QW-7B ITL on Thor under HuggingFace, versus 33.0 ms generation, because each token step also incurs CPU-side orchestration and de-tokenization.Hydra uses common timing stages to attribute backend latency rather than treating model execution as the only cause.
  • A. Performance Analysis: llama.cpp can deliver lower end-to-end latency despite higher raw per-token generation latency, because its runtime overhead is lower.Its tighter TTFT and prefill variation also suggest more stable prompt-side execution.
  • A. Performance Analysis: 5.0–5.7×: TTFT increase when input tokens grow from 1k to 5k, whereas decode throughput changes only a few percent as output budgets rise from 1k to 5k tokens.Long prompts can push first-token latency into the multi-second range for larger models, making prefill responsiveness the limiting factor for interactive use.

B. System Resource Utilization

Hydra attributes system-resource behavior to CPU orchestration, GPU effective utilization, and DRAM traffic, showing that backend, quantization, model scale, and SoC generation shape edge inference differently. These effects clarify why aggregate latency alone misses important deployment mechanisms.

  • Backend effects: Backend choice changes GPU feeding: HuggingFace leaves utilization and DRAM bandwidth below peak for small Qwen models, while llama.cpp approaches decode-utilization saturation at the same precision.llama.cpp also reduces CPU-side per-token orchestration, helping explain its end-to-end latency advantage.
  • Quantization effects: In llama.cpp, quantization lowers effective DRAM bandwidth while decode effective utilization remains high, improving tokens/s by reducing memory traffic rather than GPU occupancy.The runtime continues streaming token work efficiently even as weight precision decreases.
  • Model-scale effects: For QW-7B at 16-bit precision, HuggingFace and llama.cpp achieve high GPU utilization and similar DRAM-bandwidth utilization, narrowing the backend gap as model-scale work amortizes overhead.Per-token model work becomes large enough that Python-side overhead is a smaller portion of each decode step.
  • Length sensitivity: Increasing output budgets from 1k to 5k tokens leaves decode effective utilization nearly flat because each generated token repeats the same per-token loop.Longer outputs extend steady-state decode without changing per-token utilization behavior.
  • SoC-generation effects: Thor maintains high decode effective utilization across the sweep, whereas Orin has lower utilization on smaller models and approaches saturation only for QW-7B.The contrast shows that GPU frequency, memory bandwidth, and utilization must be interpreted together with SoC platform behavior.

C. Efficiency

Hydra shows that efficiency depends on phase-aware interactions among quantization format, backend, and SoC generation, not latency alone. Lower-bit formats and newer SoCs can reduce energy per generated token, while power, thermal behavior, and total prompt energy remain configuration- and workload-dependent.

  • Quantization: Lower-bit formats reduce energy per generated token across Orin and Thor, with Q4_K_M consistently lowest among llama.cpp formats.The reduction follows from lower DRAM traffic while maintaining high GPU occupancy.
  • Quantization: Power is not monotonic with bit-width: Q6_K often draws more power and runs hotter than Q8_0 and Q4_K_M.Unpacking, scaling, and dequantization work make power and thermal behavior dependent on format implementation rather than precision alone.
  • SoC generation: Thor usually draws more instantaneous power than Orin but achieves lower energy per token through higher throughput across matched model/format cells.Platform choice therefore changes the energy operating point as well as the latency operating point.
  • Thermals: Temperatures identify the active subsystem without binding single-stream inference: CPU temperatures can exceed GPU temperatures under HuggingFace on Orin, whereas llama.cpp makes the GPU warmer.All measured single-stream runs remain below throttling-relevant levels, making energy/token more useful than peak temperature as a constraint.
  • Length sensitivity: Output length dominates total prompt energy: increasing generated tokens from 1k to 5k scales energy almost proportionally by 5.1 to 5.4×, while longer inputs raise it 11 to 34%.Decode dominates runtime, whereas prefill is paid once and decode once per generated token.
  • Deployment tradeoffs: Backend, precision, model size, sequence length, and platform jointly determine deployment cost, and combined choices can more than double prompts served under the same energy budget.Moving from HuggingFace bf16 to llama.cpp Q4_K_M reduces runtime overhead and weight traffic, while Orin-to-Thor improves token rate at higher instantaneous power.

VI. DISCUSSION, LIMITATIONS, AND FUTURE DIRECTIONS

Hydra turns phase-aware measurements into deployment guidance while documenting bounded measurement overhead and platform-specific validity limits. Its extensible schema supports future adaptive control and studies of longer-context, concurrent serving.

  • Practical deployment implications: Latency-bound small models should use backends with low orchestration overhead, while energy-bound deployments should optimize mJ/token rather than nominal bit-width.Q4_K_M is often best for energy, whereas Q6_K can regress.
  • Practical deployment implications: Power- or thermal-capped systems may not gain power benefits from lower precision; long inputs should optimize prefill/TTFT, and long outputs decode mJ/token.The recommended objective depends on workload phase and output length.
  • Practical deployment implications: Platform selection must account for native precision support and allocator/runtime constraints, which can more than double prompts served under a fixed energy budget.Deployment choices therefore affect capacity as well as efficiency.
  • Measurement overhead and validity: < 0.1 ms per phase bounds Hydra’s timing overhead, while 0.06–0.08 ms per-token logit-read/argmax cost remains part of decoding.Measurements agree with llama.cpp counters at 27.04 vs. 27.00 ms prefill; telemetry is out-of-process and each record is ∼0.4 KB.
  • Limitations and future directions: The study’s trends are specific to three NVIDIA Jetson SoCs and weight-only GGUF quantization, while future work targets adaptive per-phase control and concurrent longer-context serving.Future studies will examine precision, offload, or scheduling changes during prefill and decode, plus KV-cache fragmentation, memory pressure, and DRAM contention.

VII. RELATED WORK · VIII. CONCLUSION

Hydra is positioned as a phase-aware edge-LLM characterization framework that unifies backend instrumentation, cross-generation SoC analysis, quantization scope, telemetry, efficiency metrics, and released traces. Its evaluation shows why aggregate latency or model size alone cannot explain edge inference behavior.

  • VII. RELATED WORK: Hydra is compared with prior work across hardware coverage, backend support, model and precision scope, phase-aware timing, telemetry, efficiency metrics, and released artifacts.The comparison distinguishes common-schema instrumentation from conventional benchmarking dimensions.
  • VII. RELATED WORK: Prior edge LLM studies cover mobile devices, Raspberry Pi-class systems, Jetson platforms, and analytical models, with MELTing Point and PalmBench providing especially close comparisons.These studies include runtime, energy, or thermal measurements for compressed or mobile LLM execution.
  • VII. RELATED WORK: Server-side work establishes the value of separating prefill and decode, while Hydra extends phase-aware analysis to measured single-request edge inference.Related efforts address phase-level power, serving, KV-cache management, request dynamics, scheduling, and accelerator behavior.
  • VII. RELATED WORK: Hydra complements generic benchmarks and quantization methods by characterizing deployment behavior rather than proposing a new quantizer or benchmark score.Generic efforts standardize top-line latency, throughput, power, and model-size reporting, while quantization methods reduce footprint and improve deployability.
  • VII. RELATED WORK: Hydra’s key distinction is combining cross-generation edge SoC behavior, dual-backend instrumentation, 16-bit and GGUF formats, phase-attributed telemetry, and an open per-prompt trace corpus.The instrumented backends are HuggingFace Transformers and llama.cpp.
  • VII. RELATED WORK: Its canonical records link token-level timing to system-resource behavior and efficiency, enabling comparisons across backend, precision, model-family, and SoC-generation effects.These comparisons occur within one workload-characterization framework.
  • VIII. CONCLUSION: Hydra aligns HuggingFace Transformers and llama.cpp timing with system telemetry so performance, utilization, and efficiency can be attributed to prefill and decode windows.The methodology spans three Jetson generations, thirteen LLMs, five execution formats, and input/output-length sweeps.

ARTIFACT APPENDIX · A.1 Abstract · A.2 Artifact Meta-Information Checklist •

The artifact releases Hydra’s cross-backend profiling and analysis code alongside a 107,110-record measurement corpus. It supports corpus-based reproduction on ordinary Linux or macOS systems and pipeline validation on three Jetson platforms using phase-aware performance and system-resource metrics.

  • A.1 Abstract: Hydra releases cross-backend profilers, telemetry fusion, a canonical schema, and an analysis pipeline as research code.The profilers target HuggingFace Transformers and llama.cpp and combine tegrastats/NVML telemetry.
  • A.1 Abstract: 107,110 per-prompt records comprise the complete Hydra measurement corpus, distributed across 286 unified CSVs.The corpus includes 190 main-corpus configurations plus S1/S2 length-sensitivity sweeps and is shipped as a split xz tarball.
  • A.1 Abstract: Evaluation I reproduces the paper’s computational results from the released corpus on any Linux or macOS machine without GPU or Jetson hardware.It covers Figs. 1, 3–7 and Tables 4–7.
  • A.1 Abstract: Evaluation II validates the measurement pipeline by re-measuring the flagship model on NVIDIA Jetson AGX Xavier, Orin, and Thor testbeds.SSH access to all three boards is provided, and fresh measurements are compared against the corpus.
  • A.2 Artifact Meta-Information Checklist •: The released program consists of Python and C++ Hydra profilers, a unifier, and a Python analysis pipeline.Evaluation I requires Python ≥3.10 with pandas, numpy, matplotlib, and seaborn; Evaluation II uses pre-provisioned board environments.
  • A.2 Artifact Meta-Information Checklist •: The artifact records 25 timing fields plus phase-attributed telemetry aggregates spanning 349–419 columns, depending on platform and backend.Reported metrics include TTFT, ITL, E2E latency, throughput, utilization, DRAM bandwidth, power, energy/token, and temperature.
  • A.2 Artifact Meta-Information Checklist •: The artifact outputs PDF figures, Markdown/CSV tables, and a spot-check comparison report, and is publicly available through GitHub and an archival DOI.Evaluation I requires ∼1 GB of disk space and takes ∼5 min; Evaluation II takes ∼0.5–1 h per board, longer on Xavier.

A.4 Evaluation I: Reproducing the Paper’s Results (no Jetson GPU needed) · A.5 Evaluation II: Validating the Measurement Pipeline (Jetson testbeds, SSH)

Evaluation I reproduces the released corpus, figures, and tables automatically without requiring a Jetson GPU. Evaluation II validates the end-to-end measurement pipeline on all three SoC generations using fresh hardware measurements and reproducibility checks.

  • A.4 Evaluation I: Reproducing the Paper’s Results (no Jetson GPU needed): About five minutes on a laptop are sufficient for the full reproduction evaluation, which can run on Linux or macOS without a Jetson GPU.On provided boards, the pre-installed environment and checkout allow reviewers to run only the driver.
  • A.4 Evaluation I: Reproducing the Paper’s Results (no Jetson GPU needed): 107,110 per-prompt records across 286 unified CSVs are verified before regenerating Figs. 1 and 3–7 and Tables 4–7.The driver checks both the release tarball SHA-256 and an integrity manifest, stopping on any mismatch.
  • A.4 Evaluation I: Reproducing the Paper’s Results (no Jetson GPU needed): Every regenerated table value is identical to the shipped reference tables, and six figure PDFs render successfully.The automated run also reports “CORPUS VERIFICATION PASSED.”
  • A.4 Evaluation I: Reproducing the Paper’s Results (no Jetson GPU needed): Evaluation I must not run concurrently with Evaluation II on the same board because its analysis load perturbs recorded telemetry.This restriction protects the validity of measurements collected during the hardware evaluation.
  • A.5 Evaluation II: Validating the Measurement Pipeline (Jetson testbeds, SSH): Evaluation II re-measures Qwen2.5-7B across HF bf16 and llama.cpp Q8_0, Q6_K, and Q4_K_M on all three SoC generations.The spot-check uses approximately 20 IFEval prompts and a 500-token decode, then fuses timing with telemetry.
  • A.5 Evaluation II: Validating the Measurement Pipeline (Jetson testbeds, SSH): 20 prompts × 4 configurations produce a fresh mini-corpus and a comparison table ending in “Result: INVARIANTS PASS.”The run reports fresh decode power, ITL, energy/token, and throughput beside corresponding released-corpus values; each board takes about 30–60 minutes.
  • A.5 Evaluation II: Validating the Measurement Pipeline (Jetson testbeds, SSH): The profiler can be rebuilt from source, with platform-specific CUDA architectures selected as Xavier sm_72, Orin sm_87, and Thor sm_110.This rebuild requires cmake, nvcc on PATH, and a C++17 compiler; full-scale collection is not expected of reviewers because it takes days of device time.

A.6 Interpreting Results · A.7 Customization

Hydra’s evaluations verify both deterministic analysis reproduction and fresh-corpus agreement, while three qualitative invariants capture key deployment findings. Its repository supports customization through input edits, platform-specific telemetry parsers, and additional analyses over the shared corpus.

  • A.6 Interpreting Results: Evaluation I automatically verifies the released corpus and requires regenerated tables to match the shipped reference tables exactly.The checks cover the corpus checksum, record manifest, and deterministic analysis outputs.
  • A.6 Interpreting Results: Manual review finds regenerated tables match published Tables 4–7 at printed precision, with occasional one-digit rounding differences, and figures should look identical.An example rounding difference is 86.3 versus 86.2.
  • A.6 Interpreting Results: Evaluation II typically finds fresh mini-corpus values within ±15% of corpus means, while thermal state and background load make the band guidance rather than a hard gate.The comparison prints each fresh value beside its corpus reference and percentage deviation.
  • A.6 Interpreting Results: Together, the evaluations cover Hydra’s collection, fusion, and analysis stages, while diagrams, handwritten summaries, and the independent lm-eval-harness table remain outside coverage.Evaluation II covers collection and fusion, whereas Evaluation I covers analysis.
  • A.6 Interpreting Results: Three qualitative invariants are binding: Q6_K draws more decode power than Q8_0, Q4_K_M has the lowest energy per token among llama.cpp formats, and llama.cpp ITL < HF ITL.These findings represent bit-width nonmonotonicity, energy efficiency, and the runtime-overhead gap, respectively.
  • A.7 Customization: The repository mirrors the paper’s architecture, with inputs for prompts and model registries, inference for profilers and telemetry, and analysis for unified figures and tables.The architecture is organized across inputs/, inference/, and analysis/.
  • A.7 Customization: Adding models, precisions, or prompt sets requires edits under inputs/, whereas adding platforms requires a tegrastats parser under inference/telemetry/.This separates workload customization from platform-specific telemetry support.
  • A.7 Customization: analysis/main.py exposes additional distribution, scaling, thermal, and memory-pressure plot families over the same corpus.The command-line interface can generate all analyses with --all, a corpus results root, and an output directory.
Loading 2608.25053v1…