Source-linked AI summary
Quantized Edge Inference of a Self-Supervised SAR-Optical Encoder on Commodity x86 Hardware
Christian Sanchez-Leon, Dongming Chen
TL;DR
The inference cost of self-supervised Earth-observation encoders is rarely characterized across the capabilities of field-available x86 hardware. This paper deploys a pretrained SAR–optical masked-autoencoder encoder across commodity x86 nodes and measures accuracy, latency, and energy. Quantization preserves accuracy while INT8 lowers CPU latency, and the preferred device shifts with CPU class.
Problem
Inference cost and post-training quantization are seldom characterized across commodity x86 hardware differences such as AVX2, VNNI, and integrated-GPU availability.
Source: §I · p. 1Method
The authors deploy a pretrained SAR–optical masked-autoencoder encoder at FP32, FP16, and INT8 across three commodity x86 nodes and measure linear-probe accuracy, latency, and energy.
Source: A. Encoder, Intermediate… · p. 1, C. Measurement Protocol · p. 2Results
Across the tested nodes, INT8 preserves accuracy while lowering CPU latency, and the Pareto-optimal device shifts from iGPU to CPU as CPU class improves.
Source: A. Quantization is Accuracy-Neutral · p. 2, B. INT8 Lowers Cost on the CPU; FP16… · p. 2, D. The Best Device Depends on the… · p. 3Takeaways & Limitations
Precision can be chosen on cost alone when accuracy is unchanged, but the preferred device depends on the node’s hardware class.
Source: §IV · p. 3The characterization covers one encoder and does not directly rule out thermal throttling; broader encoder and hardware coverage is left to future work.
Source: §IV · p. 3
Abstract
from arXiv · showhide
Self-supervised vision transformers are increasingly used as reusable encoders for Earth observation, but their inference cost on the hardware available in the field is rarely characterized. We deploy a pretrained SAR-optical masked-autoencoder encoder on three commodity x86 nodes, from a low-power CPU without AVX2 to an AVX2+VNNI reference, at FP32, FP16 and INT8 on the CPU and, where present, the integrated GPU, and report latency, energy (RAPL and wall plug) and linear-probe accuracy. Quantization is accuracy-neutral: INT8 changes micro-mAP by only +0.0006, within the seed spread, so precision becomes a pure cost decision. INT8 lowers CPU latency on every node, by 1.45x even on the CPU without AVX2, and CPU energy wherever it is measured, and the VNNI reference (1.53x) suggests a further gain beyond AVX2. The Pareto-optimal device shifts from the integrated GPU, which is 4.38x faster than the CPU on the weakest node, to the CPU as the CPU class improves.
I. INTRODUCTION
Earth-observation encoders need to run on field hardware, yet accuracy-focused evaluations rarely characterize inference cost across the capabilities of commodity x86 systems. This work measures that deployment trade-off for a self-supervised SAR–optical transformer.
I. INTRODUCTION
- Inference cost matters because Earth-observation models may run on small, passively cooled x86 systems rather than server GPUs.The cost of a single inference and its response to post-training quantization are seldom reported.Source: §I · p. 1
- Evidence is limited on how self-supervised SAR–optical transformers behave across ground-based x86 hardware when latency, energy, and accuracy are measured together.Relevant hardware differences include AVX2, VNNI, and the presence of a usable integrated GPU.Source: §I · p. 1
- The study characterizes the encoder across commodity x86 classes, finding quantization accuracy-neutral, INT8 beneficial across CPU classes, and the Pareto-optimal device dependent on CPU capability.Its reproducible energy protocol combines RAPL and wall-plug measurements.Source: §I · p. 1
A. Encoder, Intermediate Representation and Task
The paper evaluates a frozen SAR–optical masked-autoencoder encoder on reBEN and deploys its compiled representations across three commodity x86 hardware classes.
A. Encoder, Intermediate Representation and Task
- The evaluated model is a self-supervised ViT-S/16 masked autoencoder pretrained on the STRATA-S12 SAR–optical corpus.It takes six co-registered optical and SAR channels and returns a 384-D class token; deployment uses the encoder without decoder or masking.Source: A. Encoder, Intermediate… · p. 1
- FP32, FP16, and INT8 OpenVINO representations are evaluated through a linear probe on frozen features for reBEN’s 19-class multi-label task.INT8 post-training quantization uses 512 calibration patches disjoint from training and test data; the probe uses 20,000 training and 10,000 test patches across three seeds.Source: A. Encoder, Intermediate… · p. 1
B. Hardware Gradient
- The deployment spans a floor CPU without AVX2, an AVX2 step node, and an AVX2+VNNI reference; only the first two include an iGPU.The reference node is used for CPU latency comparison rather than as an edge target.Source: B. Hardware Gradient · p. 2
C. Measurement Protocol
The protocol measures batch-one latency and idle-adjusted energy for each node, device, and precision, using repeated inference runs and synchronized energy sources.
C. Measurement Protocol
- Each configuration receives 20 discarded warm-ups, then five repetitions of 200 forward passes at batch size one.Latency is repetition wall-clock time divided by 200, reported as a mean with a 95% Student’s t confidence interval; batched throughput is out of scope.Source: C. Measurement Protocol · p. 2
- Energy per inference is measured from synchronized RAPL CPU-package and wall-plug readings, with a 60-second idle baseline subtracted.Sampling is nominally every 0.5 seconds; the CPU governor is set to performance.Source: C. Measurement Protocol · p. 2
- Table I orders the commodity x86 nodes by ISA class and establishes the role names used in the paper’s tables and figures.Source: Table I · p. 2
A. Quantization is Accuracy-Neutral
Quantization preserves linear-probe accuracy within the reported seed spread, while CPU INT8 reduces latency across the tested nodes. FP16 does not provide a CPU speed advantage.
A. Quantization is Accuracy-Neutral
- INT8 changes micro-mAP by only +0.0006, within the ±0.0022 seed spread, while FP16 matches FP32 at micro-mAP 0.5494.The authors conclude that precision is a pure cost decision.Source: A. Quantization is Accuracy-Neutral · p. 2, Table II · p. 2
B. INT8 Lowers Cost on the CPU; FP16 Does Not
- CPU INT8 is faster than FP32 on every node: 1.45× on the floor, 1.27× on the step, and 1.53× on the reference.The latency figure compares node, device, and precision; it identifies the iGPU as fastest on the floor node.Source: B. INT8 Lowers Cost on the CPU; FP16… · p. 2, Figure 1 · p. 2
- CPU FP16 is essentially tied with FP32 and is slower on the step node (35.74 vs 33.80 ms); INT8 is the most energy-efficient CPU precision where energy was measured.Energy tracks latency at both the CPU package and wall plug.Source: B. INT8 Lowers Cost on the CPU; FP16… · p. 2
C. INT8 Pays Off Without AVX2; VNNI Adds More
INT8 improves CPU latency even without AVX2, while the reference-node comparison points to an additional gain from VNNI.
C. INT8 Pays Off Without AVX2; VNNI Adds More
- 1.45× INT8 speedup on the floor node exceeds the step node’s 1.27×, despite the floor CPU lacking AVX2 and VNNI.At batch size one, the result is consistent with a memory-bound forward pass in which fourfold smaller INT8 weights help without wide integer instructions.Source: C. INT8 Pays Off Without AVX2; VNNI… · p. 2
- 26.56 ms INT8 latency on the step node exceeds the reference node’s 20.75 ms despite similar FP32 latency and AVX2 on both.With runtime and IR held identical, the additional reference-node acceleration points to VNNI rather than AVX2.Source: C. INT8 Pays Off Without AVX2; VNNI… · p. 2
- The floor node’s INT8 gain is consistent with memory-bound inference at batch size one, where fourfold smaller weights help without wide integer instructions.Source: C. INT8 Pays Off Without AVX2; VNNI… · p. 2
D. The Best Device Depends on the Hardware Class
The best device changes with CPU class: the iGPU leads on the floor node, while INT8 CPU inference leads on the step node.
D. The Best Device Depends on the Hardware Class
- 4.38× faster and 5.46× more efficient at the package, the floor-node iGPU is the best option at FP32 or FP16.At the wall plug, it uses 0.544 J per inference versus 3.014 J for the CPU.Source: D. The Best Device Depends on the… · p. 2, Table III · p. 3
- 26.56 ms makes step-node CPU INT8 the fastest configuration; its package energy matches iGPU FP16 at 0.859 vs 0.861 J.At the wall plug, CPU INT8 is slightly lower at 1.230 vs 1.263 J.Source: D. The Best Device Depends on the… · p. 2, Table III · p. 3
- The Pareto-optimal deployment shifts from the iGPU to the CPU as CPU class improves.On the step node the iGPU barely helps at FP32 (1.04×), and INT8 is the slowest iGPU precision because the integrated GPUs run FP16 natively and gain nothing from integer weights.Source: D. The Best Device Depends on the… · pp. 2–3
E. Implementation and Reproducibility
Reproducibility is supported by artifact provenance checks and a pinned runtime chosen after testing iGPU compatibility.
E. Implementation and Reproducibility
- A SHA-256 manifest records each checkpoint-to-ONNX-to-IR artifact, and a provenance gate rechecks the chain before every run.This links each result to the exact inputs that produced it.Source: E. Implementation and Reproducibility · p. 3
- OpenVINO 2024.6 is pinned after a floor-node sweep found newer releases regressed an iGPU normalization kernel beyond the devices’ OpenCL C 1.2 support.The tested Gen9/11 iGPUs could not build the newer OpenCL 2.0 work-group functions.Source: E. Implementation and Reproducibility · p. 3
- Runtime-version checks found CPU latency changes of +1.0% and +0.1% for FP32 and FP16 on the floor node, but +6.8% for INT8.On the reference node, FP32, FP16, and INT8 changed by −5.1%, −8.2%, and +12.2%, respectively; all nodes were measured on the pinned release.Source: E. Implementation and Reproducibility · p. 3
IV. DISCUSSION AND CONCLUSION
The preferred precision and device depend on the node, with INT8 improving CPU speed and energy where measured. The findings are bounded by a single encoder and a limited VNNI comparison.
IV. DISCUSSION AND CONCLUSION
- INT8 is faster on every CPU node, and VNNI appears to add a further gain beyond AVX2.The reference is the only VNNI node, so that observation compares two non-identical machines with similar FP32 latency.Source: §IV · p. 3
- At batch size one, the floor iGPU processes about 71,600 patches per hour at 0.544 J each, while step-node CPU INT8 processes about 135,500 at 1.230 J each.Source: §IV · p. 3
- The study evaluates one ViT-S/16 encoder; accuracy neutrality and speedup magnitude may differ for other architectures or sizes.Device temperature was not recorded, so thermal throttling cannot be ruled out directly; extending the study to more encoders and newer iGPUs and low-power VNNI CPUs is future work.Source: §IV · p. 3