PaperlayerView exact revision on arXiv

Source-linked AI summary

GaLe: memory-efficient Global Approximate and Local Exact features

Alberto Ancilotto, Elisabetta Farella

arXiv:2609.02689v1cs.CV

TL;DR

Embedded devices constrain deployment of modern networks through limited memory and compute, while existing inference methods incur overhead or lose accuracy. GaLe partitions feature maps into local exact and global approximate representations, extending memory-efficient inference to global operations and attention. Across classification, detection, and generation, it preserves reported performance while substantially reducing memory and improving speed over patch-based inference.

  • Problem

    Embedded devices have limited memory and compute, while existing inference methods trade high patch-based overhead against accuracy loss from approximation.

  • Method

    GaLe partitions feature maps into local exact features for fine details and global approximate features for long-range dependencies, supporting global operations and attention.

  • Results

    GaLe demonstrates similar performance to exact inference, up to 90% memory reduction, and versatility across classification, detection, and diffusion.

  • Takeaways & Limitations

    GaLe is positioned as a foundation for efficient edge AI and future memory-aware neural architecture search.

Abstract

from arXiv · show

Embedded devices typically lack the resources of GPU-equipped machines, and existing inference methods suffer from either high computational overhead (patch-based) or accuracy loss (approximation-based). We propose GaLe, a memory-efficient technique that enables the deployment of pretrained networks on constrained devices without retraining. GaLe partitions feature maps into two components: a local exact (Le) representation that preserves fine details and a global approximate (Ga) representation that retains long-range dependencies. Unlike standard tiling, GaLe supports global operations and attention mechanisms found in hybrid CNN-transformer models. Validated on ImageNet, our method matches exact-inference performance while achieving up to 65% speedup and 90% RAM reduction on a Cortex-M33 compared to patch-based inference. We further demonstrate GaLe's versatility across classification, detection, and generation tasks, highlighting its potential as a foundation for resource-efficient architecture design.

1. INTRODUCTION

GaLe targets severe embedded-device memory and compute constraints by enabling pretrained-network inference without retraining. It decomposes feature maps into local exact and global approximate representations while supporting global operations and attention.

  • Embedded systems often have under 1MB of dynamic RAM, making modern inverted residuals and attention mechanisms difficult to deploy.
  • GaLe enables pretrained networks on resource-constrained devices without retraining, requiring only a small calibration set.
  • Feature maps are decomposed into local exact components for fine-grained details and global approximate components for long-range dependencies.
  • GaLe supports global receptive fields and attention blocks while remaining compatible with ONNX, TFLite, and STM32Cube.Ai.

2. RELATED WORKS

Prior work reduces inference cost through either trained efficient architectures or training-free runtime transformations. These approaches trade memory, resolution, accuracy, or compatibility with global operations.

  • Training-based methods reduce computational complexity through efficient architectures, hardware-aware NAS, scaling strategies, and pruning.
  • Training-free methods reduce runtime memory through input-resolution reduction or patch-based inference, but each has important trade-offs.
  • Reducing input resolution lowers RAM usage quadratically but degrades fine-grained details.
  • Full patch-based inference preserves resolution but sacrifices global context.

3. GALE FOR CONVOLUTIONAL NETWORKS

GaLe extends patch-based inference for convolutional networks by combining exact local slices with approximate global context. Calibration, adaptive slicing, learned overlap, and contiguous processing reduce memory and transfer overhead while preserving accuracy.

  • Local Exact feature maps: Partial patch-based inference processes overlapping input areas independently to reduce intermediate activation tensors.
  • Local Exact feature maps: Learned padding reduces overhead relative to padding to the full receptive field and stores NHWC data in one continuous memory area.
  • Local Exact feature maps: GaLe calibration selects overlap O_GaLe ≤ O_ppbi subject to a user-controlled reconstruction-error threshold ϵ and memory constraints.
  • Adaptive Slicing: Adaptive slicing progressively reduces patch counts as feature-map resolution decreases, minimizing overlap and weight reloading.
  • Global Approximate feature maps: Ga + Le reaches 80.66% accuracy, compared with 80.34% for Le only and 79.16% for Ga only.
  • Global Approximate feature maps: The GA branch downsamples the full feature map, then upsamples and merges its result with fine-grained LE output.

4. GALE FOR HYBRID NETWORKS

GaLe extends local-exact/global-approximate feature decomposition to hybrid CNN-transformer networks, including attention, while reducing the memory demands of global operations. Its attention formulation combines strided local terms with a downsampled global term and avoids materializing the full attention matrix.

  • Hybrid architecture support: GaLe applies feature-map slicing to hybrid architectures, including EfficientViT, FastViT, and MobileNetV4, while requiring less overhead than exact memory-efficient attention methods.The approach targets both convolutional and attention-based components in modern hybrid networks.
  • Attention memory bottleneck: Standard attention stores an O(N^2) attention map, creating a memory bottleneck for sequences of N tokens.The attention map is A = QK^T, computed from query, key, and value matrices.
  • LE/GA attention formulation: GaLe approximates each attention block with local sparse and global low-rank components, implemented as b strided local-exact heads plus one downsampled global-approximate head.The local term preserves fine-grained same-phase correlations, while the global term captures coarse context from downsampled features.
  • LE/GA attention formulation: The resulting attention computation operates on smaller subsets of size N/b rather than directly on all N tokens.The input sequences are partitioned into b interleaved subsets, with the global branch using downsampled inputs.
  • Memory complexity: Decomposing attention reduces its peak memory requirement from O(N^2) to O((N/b)^2), while sequential execution bounds RAM by one subset's requirements.The smaller attention maps are independent, so they need not be held simultaneously.
  • Hybrid architecture memory profile: Hybrid models exhibit two memory peaks: high-resolution convolutional activations and attention-map computation.Figure 4 highlights the separate contributions of convolutional blocks and attention to peak RAM usage.

5. RESULTS

Across hardware platforms and tasks, GaLe reduces memory with minimal accuracy loss and lower overhead than competing inference strategies. It also handles global dependencies and hybrid architectures that standard patch-based methods cannot support effectively.

  • Overall evaluation: GaLe consistently achieves superior memory reduction with minimal accuracy loss (< 1%) and significantly lower overhead across Raspberry Pi4 and MCU platforms.The comparison includes resolution scaling and patch-based inference methods such as FPBI and PPBI.
  • Classification results: 92% RAM reduction with 18% computational overhead improves on PPBI's 88% RAM reduction with 80% overhead for MobileNetV2.This comparison is reported for GaLe versus PPBI on the evaluated MobileNetV2 setting.
  • Global operations: GaLe handles MobileNetV3 Squeeze-and-Excitation global dependencies, where standard tiling fails structurally.The global-approximate features preserve the global information required by these blocks.
  • Hybrid models: For hybrid MobileNetV4 and FastViT models, GaLe compresses both convolutional and attention layers, unlike PPBI and ToMe used separately.The result addresses the distinct memory demands of both parts of the hybrid networks.
  • Object detection: GaLe achieves negligible mAP reduction in object detection by retaining both local detail and global context through LE/GA decomposition.Resolution scaling harms small-object performance, while FPBI harms large-object performance through spatial fragmentation.

6. CONCLUSION

GaLe partitions feature maps into local exact and global approximate components to deploy pretrained models on memory-constrained devices without retraining. Across detection and other evaluated settings, it retains performance while reducing memory and supporting global operators.

  • GaLe enables deployment of pretrained deep learning models on memory-constrained devices without retraining.
  • GaLe maintains mAP comparable to exact methods with minimal overhead for RT-DETR-L and YOLOv11n.The evaluated targets are below 2.5MB and 512KB, respectively.
  • 90% memory reduction and substantial speedups were achieved while maintaining performance similar to exact inference on ImageNet classification.
  • GaLe generalizes across classification, object detection, and diffusion tasks.

A.1. Case study: diffusion models

GaLe is applied to diffusion models to reduce attention-related memory consumption while avoiding the computational overhead of memory-efficient attention. In SD-Turbo, it reduces memory substantially, preserves generated-image fidelity, and can combine with token merging for more extreme compression.

  • GaLe replaces self-attention layers in diffusion models to reduce memory consumption without memory-efficient attention's computational overhead.
  • Token merging can compute the global approximate feature map, increasing performance and allowing lower FID.
  • GaLe achieves greater memory reduction than ToMe alone while preserving generated-image fidelity measured by FID and LPIPS.
  • Combining GaLe with ToMe enables extreme memory compression factors while maintaining a low FID score.
  • GaLe offers high performance at 25× compression for both 512×512 and 1024×1024 image generation resolutions.

A.2. Detailed platform performance

GaLe's platform results show speedups over partial-patch inference that depend on hardware, while its calibration adapts slicing to memory constraints. Additional fine-grained classification results indicate high performance with substantial RAM reductions and minimal loss.

  • Higher speedups over traditional partial-patch inference occur on simpler MCUs such as Cortex-M33.More powerful devices with tiered caches, such as Raspberry Pi4, benefit less from the memory-aware slice layout.
  • GaLe calibration pass: GaLe calibration increases slice count until target performance is met while evaluating error and layer overlap at each iteration.
  • GaLe calibration pass: Calibration initializes layer overlaps and iteratively increases them until MSE is below tolerance or memory limits are reached.
  • Generalization: GaLe maintains high fine-grained classification performance on iNaturalist with significant RAM reductions and minimal performance loss.

C. SENSITIVITY ANALYSIS OF WEIGHTING FACTOR α

The weighting factor α controls the fusion of local exact and global approximate feature maps. ImageNet accuracy peaks near α = 0.8, while relying entirely on either component reduces performance.

  • α controls the balance between fine-grained local details and global semantic context in the hybrid representation.
  • α ≈ 0.8 gives peak ImageNet Top-1 Accuracy within the α ∈ [0.7, 0.9] range.
  • At α = 0.5, excessive reliance on the downsampled GA map loses high-frequency details.
  • At α = 1.0, pure slicing reaches approximately 77.1% accuracy versus approximately 77.6% at the hybrid peak.
  • The LE component alone suffers from insufficient global context when memory constraints prevent full receptive-field overlap.

D. CALIBRATION EFFICIENCY AND SAMPLE SIZE

GaLe’s calibration is data-efficient: Top-1 Accuracy stabilizes with only 16–32 samples, while larger calibration sets provide negligible additional gains.

  • Calibration convergence: Top-1 Accuracy stabilizes significantly with as few as 16 to 32 calibration samples.The calibration process exhibits rapid convergence.
  • Calibration convergence: 256 calibration samples yield negligible performance gains compared with the smaller sample sizes.
  • Calibration efficiency: 32 calibration samples provide optimal performance stability, indicating high data efficiency and low setup overhead.

E. ADDITIONAL RESULTS ON TIMM MODELS

Additional GaLe results are reported for common timm models, with configurations varying by slicing, split blocks, calibration error, and selected feature maps.

  • Configuration: GaLe configurations are identified by starting slice number S, split-block count B, allowed calibration MSE error ϵ, and selected feature maps.
  • Feature-map selection: For small RAM reductions, local exact features alone can match full-network performance, so global approximate features are omitted to reduce overhead.
  • Additional timm results: Tables 6 and 7 report additional results achieved with GaLe on common timm models.
Loading 2609.02689v1…