PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Models for Small Devices: Recent Advances and Empirical Analysis of Edge AI Deployment

Subhransu Das, Jiaming Cheng, Arnav Kumar, Sadia Afrose, Mingzhe Han, Michael Silagy, Shreya Palande, Brijesh Soni, Rajiv Ramnath

arXiv:2608.15693v1cs.AIcs.LG

TL;DR

Edge deployment depends on whether models fit resource constraints, yet compression results may not predict deployment behavior. The paper surveys deployment evidence and evaluates compressed language and image models across hardware and tasks, finding that effectiveness depends on the task, model, and hardware.

  • Problem

    Edge deployment requires models to fit memory, compute, and energy constraints, but compression methods can show limited speedup or regressions across tasks.

  • Method

    The paper combines a deployment-oriented survey with empirical evaluation of compressed language and image models across GPU, CPU, and Raspberry Pi platforms.

  • Results

    Compression effectiveness is strongly task-, model-, and hardware-dependent: shrinking a stored checkpoint can still increase deployed latency, and post-compression rankings can change.

  • Takeaways & Limitations

    Deployment choices should be evaluated for the specific task, model, and hardware rather than inferred from checkpoint size or pre-compression performance.

  • Takeaways & Limitations

    The authors note that their workflow can fail to produce a compressed artifact for complex image segmentation architectures.

Abstract

from arXiv · show

Running large AI models on resource-constrained edge devices requires model compression to reduce model size and computation. What compresses well, however, need not deploy well. We survey dozens of recent works that report compression results on real hardware and extract practical deployment guidelines from them. Following these guidelines, we deploy compact language and image models on GPU, CPU, and Raspberry Pi platforms across question answering and image segmentation. No single technique wins across tasks. For question answering, Qwen3.5 0.8B reaches 93.85 SQuAD F1 and 92 EM under Q5_K_M GGUF quantization, while structured pruning at the same precision costs 16 F1 at a 1% ratio. For segmentation, the ranking reverses: default quantization leaves parameters and MACs unchanged, whereas pruning cuts model size by nearly 80% at near-constant mIoU. Pruning can even inflate the deployed artifact by 21-49% by breaking k-quant super-block alignment; combined with longer, less format-compliant outputs, this raises Raspberry Pi latency up to 3.4x. Compression can also manufacture the appearance of competence rather than destroy it visibly: one LoRA-recovered variant stays fully parseable and holds 71% strict BoolQ accuracy while sending 97 of 100 predictions to a single class, at 52.6% balanced accuracy. We explain these effects through neural-flow graph analysis and prefill-decode-level latency decomposition, and condense them into task-specific deployment research directions. The right technique depends on the task, the model, and the hardware. Our experiment code and artifacts are open-sourced at https://github.com/Arnavvvkumar/deployment

1 Introduction

Edge deployment depends on fitting models within device memory, compute, and energy budgets, yet compression benefits vary by task and hardware. This work addresses gaps in deployment-oriented evidence through measured cross-platform experiments and prefill–decode analysis.

  • Motivation: Edge deployment is constrained by whether a model’s memory, compute, and energy demands fit the target device.Promising models often lack evaluation on edge hardware, including SegFormer’s 3.7M-parameter smallest model.
  • Motivation: Compression methods reduce model size, memory usage, or computation, but their benefits do not transfer uniformly across tasks.The paper motivates evaluating pruning, quantization, tensor decomposition, and knowledge distillation for resource-constrained devices.
  • Contributions: The survey prioritizes studies reporting measured deployment results on resource-constrained hardware, often including real-platform acceleration numbers.Existing surveys commonly summarize theory or compare results reported under different protocols rather than measuring costs on a given device.
  • Contributions: The empirical study measures pruning, quantization, and LoRA recovery on GPU, CPU, and Raspberry Pi across language and segmentation tasks.Reported tasks include SQuAD, BoolQ, Natural Questions, and semantic segmentation, using task-relevant metrics.
  • Analysis: 18 matched baseline-versus-pruned pairs showed higher per-token decode cost, and every internally controlled CPU comparison slowed despite smaller stored checkpoints.The paper explains this behavior by separating compute-bound prefill from memory-bandwidth-bound decode.

2 Contributions •

This section pairs a real-hardware survey with cross-task deployment experiments to show when compression improves device-level deployment and when pruning instead harms quality, output behavior, or latency. It introduces neural-flow and phase-level analyses to explain these outcomes.

  • Cross-task deployment study: A survey of more than twenty real-hardware studies is paired with experiments spanning LLMs, segmentation models, compression methods, and GPU, CPU, and Raspberry Pi platforms.The study evaluates approximately 1B-parameter LLM families under pruning, GGUF quantization, and LoRA recovery, plus six segmentation models under pruning, quantization, and distillation.
  • Pruning-induced forgetfulness: 1% MLP-channel pruning already causes substantial quality loss across all evaluated model families, while post-pruning LoRA fine-tuning recovers only part of it.The paper names this measured loss of task quality and output discipline pruning-induced forgetfulness.
  • Neural-flow damage analysis: 77% of Qwen3.5 pruning distortion occurs in the gate projection, versus 67% on Qwen2.5’s down projection, while the flow-graph measure rank-matches SQuAD F1 loss.The first-order, layer-additive measure orders damage within each model family and identifies where pruning damage lands.
  • Prefill-decode latency analysis: 18 matched baseline-versus-pruned run pairs showed that added latency concentrates in generation: pruned models produce up to 2.7× more tokens, while broken k-quant alignment enlarges weights 21–49%.The prefill-decode analysis links compression, deployable artifact size, memory bandwidth, and platform latency; per-token increases remain below byte growth.
  • Prior-collapse failure mode: 71% strict BoolQ accuracy can coexist with 97 of 100 predictions assigned to one class and 52.6% balanced accuracy, revealing prior-collapse that strict accuracy hides.Collapse direction depends on the model, so strict accuracy can rank collapsed configurations opposite to balanced accuracy.

3 Overview of Compression Techniques and Recent Advances

This section reviews pruning, quantization, tensor decomposition, and knowledge distillation through deployment-relevant edge studies. Across techniques, measured benefits depend on hardware execution, task performance, and deployment-aware design.

  • Pruning: Pruning removes less important weights or structural components, reducing parameter count and multiply–accumulate operations.Saliency scores such as weight magnitude, Taylor importance, or empirical Fisher information guide which components are removed first.
  • Adaptive structured pruning: 50%+ latency reduction and up to 20% retained-class accuracy improvement are reported for OCAP on NVIDIA Jetson platforms.OCAP removes channels less relevant to locally observed classes.
  • Adaptive structured pruning: 3× smaller MobileNetV2 models and 2.2× lower inference latency are achieved by LightPrune on Jetson Nano.LightPrune uses a differentiable latency estimator to guide structured pruning.
  • Fine-grained unstructured pruning: 98% weight removal yields 1025.6 inferences/J on Raspberry Pi Pico, corresponding to more than a 46× energy-efficiency improvement with custom C inference.This result comes from fine-grained unstructured pruning using auxiliary Morphological Neural Networks.
  • Compound compression: 30× VGG16 compression and up to 24.5× better energy efficiency are achieved by combining mixed pruning, 8-bit quantization, and FPGA acceleration.The deployment target is the Xilinx ZCU102.
  • Quantization: Quantization reduces memory by 3.5–6.4× across Xavier NX and by 55% on an Orin Nano, while INT8 shrinks model size by more than 70%.The cited studies show quantization delivering measured deployment gains and composing with pruning and distillation, although aggressive low-bit post-training quantization can be problematic.
  • Tensor decomposition and deployment constraints: FLOP reductions become real speed-ups only when rank or structure is selected against on-device measurements rather than proxy counts.This deployment constraint is emphasized for tensor decomposition and structured methods.
  • Knowledge distillation: 127× compression and 8.7× speed-up are reported when distilling a Transformer into a self-explainable Bi-LSTM.Other studies report 6× smaller YOLOv4 models at 37.6 ms per frame on Jetson Orin Nano and 2.5% mAP50 gains without extra cost for weed detection.

4 Empirical Study

The empirical study evaluates language and image-segmentation compression across GPU, CPU, and Raspberry Pi deployments, measuring task quality, artifact size, and latency together. It compares structured pruning, LoRA recovery, and block-wise GGUF quantization while examining deployment-specific effects that aggregate performance reports can hide.

  • Experimental scope: The study deploys language and image-segmentation models on GPU, CPU, and Raspberry Pi platforms, followed by platform specifications and result analysis.The empirical scope spans several model families and edge hardware types.
  • Implementation: The primary LLM platform comparison uses llama.cpp with Q4_K_M GGUF artifacts, while PyTorch supports structured pruning, LoRA recovery, and image-model experiments.The reported FP16 column denotes pre-quantization artifact size rather than inference precision.
  • Compression methods: Structured pruning changes neural-flow topology, while quantization preserves topology and changes numerical precision.Pruning is restricted to transformer-block MLP widths, coupling gate, up, and down projections through shared intermediate channels.
  • Deployment constraints: A dense-model-optimal pruning ratio need not remain optimal after quantization because unaligned widths can trigger higher-precision storage and increase the deployed model size.The GGUF k-quant format stores weights in 256-element super-blocks, making alignment a deployment constraint.
  • Evaluation design: Task quality, artifact size, and latency are measured together across compression methods and hardware to expose deployment-level behavior hidden by aggregate performance reports.The study also runs a prompt sweep because instruction formatting can independently move extractive QA scores.

5 Domain-Specific Result Analysis · 5.1 LLM

Across small LLMs deployed on GPU, CPU, and Raspberry Pi, compression outcomes depend on model, task, prompt, and hardware: Qwen3.5 leads baseline QA, pruning harms quality and can slow inference, LoRA recovery is task-specific, and quantization has model-dependent optima.

  • 5.1.1 Baseline Results: Qwen3.5 0.8B provides the strongest overall unpruned baseline, reaching 84.08 SQuAD F1 on GPU and 86.30 on CPU.The evaluated models span approximately 1B parameters across GPU, CPU, and Raspberry Pi platforms using SQuAD, NQ, and BoolQ metrics.
  • 5.1.1 Baseline Results: 70×: mean latency rises from 0.13 s/sample on GPU to 8.91 s/sample on Raspberry Pi despite similar 508–769 MB Q4_K_M artifacts.The corresponding GPU-to-CPU slowdown is approximately 4.8×, showing that model size alone does not predict deployed latency.
  • 5.1.1 Baseline Results: 38.16 of the 38.85-point TinyLlama GPU-to-Raspberry-Pi SQuAD difference is reproduced by changing prompt templates alone.TinyLlama scores 55.29 under no_extra_words versus 17.13 under one_word_or_phrase, so its Raspberry Pi comparison is confounded by prompting.
  • 5.1.2 Pruning Results: 2.1×: Qwen3.5 response length grows from 29 to 61 characters at 7% pruning, closely tracking its 2.2× GPU latency increase.Pruning drives both generated-token count and per-token cost upward by making outputs more verbose and less format-compliant.
  • 5.1.2 Pruning Results: 21–49%: pruning increases converted Q4_K_M artifact sizes because removed MLP channels break 256-element k-quant super-block alignment.Qwen3.5 grows from 507.85 to 612.06 MB, Gemma from 768.72 to 1020.14 MB, and TinyLlama from 637.81 to 948.34 MB.
  • 5.1.3 LoRA Recovery: LoRA recovery is task-specific: for Qwen3.5 at 1% pruning, BoolQ rises from 57 to 72 while SQuAD F1 falls from 64.56 to 51.27.For TinyLlama at 1% pruning, LoRA instead raises SQuAD F1 from 9.55 to 36.59 while BoolQ falls from 26 to 11.
  • 5.1.4 Quantization Results: 93.85 F1 and 92 EM: Qwen3.5 0.8B performs best under Q5_K_M, with 551.22 MB GGUF size and 75% BoolQ accuracy.Q4_K_M reduces size to 507.85 MB but lowers the quality mean from 59.68 to 54.92, while TinyLlama and Gemma achieve their best aggregate scores under Q4_K_M.

5.2 Image Model Results

For image segmentation, structured pruning substantially reduced deployable model resources and Raspberry Pi latency while preserving task performance in selected models, whereas default quantization barely compressed these architectures. Compression and deployment outcomes varied by architecture: EfficientNet-B3 models were faster, but ResNet-50-backed models were generally more pruning-tolerant.

  • Full-size baselines: FPN–EfficientNet-B3 achieved the strongest full-model baseline, with 0.6728 mIoU and 75.59% mean accuracy using 12.476M parameters and 47.19 MB storage.It also required 16.33G MACs.
  • Structured pruning: 61.98% average model-size reduction followed structured pruning at S=0.5, alongside 61.55% fewer parameters and 43.14% fewer MACs across six models.U-Net–ResNet-50 reached a 79.62% storage decrease with mIoU changing from 0.5612 to 0.5517 and accuracy from 61.91% to 62.86%.
  • Structured pruning: DeepLabV3+–ResNet-50 provided the strongest pruned accuracy–compression trade-off, retaining 0.6258 mIoU and 70.97% accuracy after reducing size from 106.71 MB to 30.30 MB.Its parameter count fell from 26.678M to 7.943M and MACs from 46.29G to 20.78G.
  • Raspberry Pi deployment: 39.69% lower mean wall latency followed increasing pruning from 0.5 to 0.9 on Raspberry Pi, while EfficientNet-B3 averaged 16.38 s versus 21.39 s for ResNet-50.The fastest configuration was DeepLabV3+–EfficientNet-B3 at ratio 0.9, with 11.5 s wall latency and 8.3 s CPU latency.

6 Observations

Structured pruning can cause severe, task- and model-dependent damage while sometimes slowing edge inference through larger fallback artifacts and longer decoding. Compression can also preserve misleading strict accuracy despite prediction collapse, so deployment evaluation must include distortion, latency phases, parsing, skew, and balanced accuracy.

  • Pruning damage: At approximately 1% MLP-channel pruning, SQuAD F1 falls by 17.8–64.5 points across all three evaluated LLM families.The effect is termed pruning-induced forgetfulness because pruning removes structures carrying task-specific performance.
  • Neural-flow analysis: Qwen3.5’s fixed-activation pruning distortion ranks SQuAD F1 damage perfectly on GPU and CPU, with Spearman ρ=1.0.The trajectory is consistent with measured damage across four ratios but does not establish predictive power beyond pruning ratio.
  • BoolQ evaluation: Qwen3.5 at 5% pruning predicts no almost unconditionally, while Qwen2.5 at 3% pruning predicts yes on every parsed sample despite 67.9% parsed accuracy.Balanced accuracy exposes the collapses more clearly than strict accuracy, which is affected by the yes-heavy label prior.
  • Edge latency: 508 to 612 MB (+21%) in Q4_K_M weights makes structured pruning slower because every generated token re-reads the enlarged weight set during decode.Prefill pays the extra cost once, whereas decode is memory-bandwidth-bound and accumulates the slowdown across N_gen tokens.
  • Edge latency: Gemma’s slowdown is almost entirely verbosity (+169% generated tokens against +6% per token), whereas TinyLlama’s is mostly per-token cost (+25–26% per token against +11% tokens).Qwen3.5 mixes token-count and per-token effects, showing that latency changes have model-specific causes.

7 Deployment Recommendations

Compression effectiveness depends on the downstream task, hardware, deployment format, and acceptable accuracy–latency–memory–energy trade-offs, so deployment decisions should not rely on model size alone. Models should be evaluated and re-ranked against task-specific metrics and real-device constraints at each deployment stage.

  • General deployment guidance: Compression effectiveness depends on the task, hardware, deployment format, and acceptable accuracy, latency, memory, and energy trade-offs.Deployment decisions should evaluate candidate models against task-specific constraints rather than model size alone.
  • General deployment guidance: Task-specific evaluation should use extractive-answer quality, parse reliability, mIoU, classification accuracy, and wall-clock latency for real-time edge inference.These metrics correspond to question answering, binary reasoning, segmentation, image recognition, and real-time edge deployment, respectively.
  • Task-specific model selection: LoRA recovery at 1% pruning raises BoolQ accuracy from 57 to 72 while dropping SQuAD F1 from 64.56 to 51.27, whereas DepGraph pruning raises CPU latency.The opposing outcomes show that model family, compression level, and recovery method must be selected according to the downstream task profile.
  • Segmentation deployment: Segmentation models should be re-ranked after full-size selection, compression, and real-device deployment because accuracy, latency, and thermal rankings can change at each stage.The recommendations summarize this stage-wise filtering process for selecting compressed models under edge constraints.

8 Conclusion and Future Work

The study connects model compression with practical AIoT deployment through deployment-oriented review, multi-task experiments across GPU, CPU, and Raspberry Pi platforms, and recommendations tied to use cases and constraints. Its findings show that effectiveness depends on task, model, and hardware, motivating future work on prefill–decode optimization, robust quantization, and broader domains.

  • Contributions: The study reviewed pruning, quantization, tensor decomposition, and knowledge distillation, then evaluated them across SQuAD, BoolQ, Natural Questions, and image segmentation on GPU, CPU, and Raspberry Pi.The resulting recommendations map downstream tasks to use cases, constraints, and empirical evidence.
  • Contributions: Compression effectiveness is strongly task-, model-, and hardware-dependent: smaller checkpoints can increase deployed latency, and pre-compression leaders may change after compression or deployment.The conclusion emphasizes that pruning, quantization, recovery, and real-device deployment can alter model rankings.
  • Future directions: A future optimization framework should profile prefill and decode separately, estimate latency–memory–accuracy trade-offs, and recommend configurations for resource-constrained devices.This is especially relevant to Raspberry Pi-class and mobile AIoT platforms, where memory bandwidth and decoding latency dominate.
  • Future directions: A model-agnostic quantization framework should identify quantizable regions and unsupported operators, apply safe graph-preserving conversions, and provide operator-level deployment reports.The motivation is that stock PyTorch FX-graph-mode post-training quantization can fail on complex segmentation architectures and omit conversion details.
  • Future directions: Extending deployment-aware evaluation beyond compact LLMs and image models to ASR and other domain-specific workloads would reveal additional latency, memory, accuracy, and energy trade-offs.Candidate domains include audio event detection, multimodal sensing, time-series forecasting, medical signal analysis, and robotics.
Loading 2608.15693v1…