PaperlayerView exact revision on arXiv

Source-linked AI summary

SEAOTTER: Sensor Embedded Autoencoding with One-Time Transcode for Efficient Reconstruction

Dan Jacobellis, Neeraja J. Yadwadkar

arXiv:2606.03940v1eess.IVcs.CVcs.LGcs.RO

TL;DR

Robotics can capture high-resolution visual data, but bandwidth and compute constraints limit conventional transmission, while asymmetric autoencoders add decoding and compatibility costs. SEAOTTER combines a sensor-embedded autoencoder with a learned JPEG-compatible one-time transcode, and reports improved perception accuracy alongside efficient encoding and decoding against AVIF.

  • Problem

    High-resolution robotic sensing exceeds practical bandwidth and on-device compute budgets, while asymmetric autoencoders add costly decoding and bespoke formats incompatible with JPEG infrastructure.

  • Method

    SEAOTTER uses a frozen sensor-side FRAPPE encoder and a cloud-side learned transcode that fine-tunes the decoder and JPEG color and quantization parameters.

  • Results

    SEAOTTER-FT improves ImageNet top-1 by +7.87 pp over AVIF and +8.00 pp over AVIF-max-speed, while also leading or tying baselines across the evaluated perception tasks.

  • Takeaways & Limitations

    The one-time transcode improves downstream accuracy over the same DNN-based autoencoder while producing standard JPEG files usable by broad data-consumer infrastructure.

  • Takeaways & Limitations

    The study tests only RGB and leaves depth, infrared, multispectral, and hyperspectral modalities uncharacterized.

Abstract

from arXiv · show

In robotics systems, vast amounts of visual data are easily captured at high resolution using low-cost, low-power hardware. Yet, limited bandwidth and on-device compute resources prevent full utilization when transmitted via conventional codecs like JPEG/MPEG. Newer codecs, like AV1/AVIF, improve the rate-distortion trade-off, but demand far more resources for encoding, impractical without custom ASICs. Recent asymmetric autoencoders deliver high quality under extreme power and bandwidth constraints, but add prohibitive decoding cost and use bespoke formats that ignore decades of infrastructure built around standards like JPEG. To address these limitations, we introduce a compression framework for cloud robotics based on a Sensor Embedded Autoencoder paired with a One-Time Transcode for Efficient Reconstruction (SEAOTTER). Because the sensor, cloud, and consumer stages face very different power and bandwidth budgets, SEAOTTER combines the compactness of a learned latent with the broad usability of a standard JPEG file. Since naive transcoding degrades performance, we propose a learnable JPEG color and quantization transform that enables increased accuracy for global, dense, and vision-language-based perception. Using SEAOTTER, we train both general-purpose and task-aware transcoding pipelines for a pre-trained, frozen encoder. At a compression ratio of 200:1 and compared to AVIF, we observe 7 times faster encoding, 3.5 times faster decoding, and +8% ImageNet top-1 accuracy, while retaining compatibility with JPEG infrastructure. Our code is available at https://github.com/UT-SysML/seaotter .

1 Introduction

SEAOTTER addresses the mismatch between resource-constrained sensors, cloud processing, and consumer-side decoding by combining a sensor-embedded autoencoder with a one-time transcode into standard JPEG. Its learned color and quantization transforms preserve JPEG compatibility while adapting reconstruction to downstream perception tasks.

  • 1 Introduction: Asymmetric autoencoders impose costly decoding and bespoke formats that are incompatible with JPEG infrastructure, especially when files are decoded repeatedly.Training workloads may reread each file once per epoch, multiplying per-decode overhead.
  • 1 Introduction: SEAOTTER combines a sensor-embedded autoencoder with a one-time transcode to reconcile sensor, cloud, and consumer resource asymmetries.The framework targets high-throughput sensor encoding, end-to-end task adaptability, and universal consumer-side compatibility.
  • 1 Introduction: SEAOTTER learns JPEG-compatible color transforms and quantization parameters specialized to the camera, environment, and downstream model.The FRAPPE encoder remains frozen while the decoder and JPEG parameters are fine-tuned.
  • 1 Introduction: The cloud-side transcode emits standard-compliant JPEG files with embedded custom quantization metadata for broad infrastructure compatibility.Machine-vision decoders can skip the inverse color transform, while fixed sRGB models require only a single approximately 81 MACs/pixel post-filter.
  • 1 Introduction: The learned codec is evaluated across global, dense, and vision-language tasks, where one-time transcoding increases downstream accuracy over the underlying autoencoder.The contribution claims improved downstream accuracy while retaining standard JPEG artifacts.

2 Proposed method: design and implementation

SEAOTTER combines a frozen, resource-efficient sensor encoder with cloud-side synthesis and a learned JPEG “sandwich” that produces standard JPEG files. Its implementation jointly learns color transforms, rate-specific quantization, and a calibrated rate proxy while supporting optional consumer-side inverse transforms.

  • Pipeline overview: SEAOTTER’s pipeline transmits a losslessly compressed latent, reconstructs an intermediate image in the cloud, and pays the JPEG transcode once per frame.The resulting on-disk artifact is a standard JPEG decoded by downstream consumers.
  • Sensor-side encoding: 10–100 MAC/pixel: the frozen FRAPPE encoder supports variable-rate sensing by transmitting prefixes of coarse-to-fine latent channels.Supported operating points use channel prefixes rather than re-encoding, and int8 latents are losslessly compressed after encoding.
  • Cloud-side synthesis: The fine-tuned FRAPPE decoder adapts cloud-side reconstructions to downstream task losses while the encoder remains frozen.Different decoder snapshots can produce task-specific RGB outputs from the same transmitted latent, while deliberately trading pixel PSNR for post-transcode accuracy.
  • Learned JPEG sandwich: The JPEG sandwich applies a learned color transform, standard JPEG encoding with a learned 3×8×8 quantization matrix, and an inverse transform at the consumer.Its lightweight transform uses a 3×3 convolution and companding, with a shared transform pair across rate points and independent quantization matrices.
  • End-to-end training: The learned transform and quantization matrices are trained jointly across rate points using a rate–distortion objective with a calibrated JPEG bits-per-pixel proxy.The proxy models zigzag AC run-length coding and adds a per-block Huffman-overhead constant calibrated to real JPEG bitrates.
  • Consumer-side decoding: The consumer can apply the inverse color transform for displayable RGB or skip it when a downstream model operates directly on JPEG-decoded coefficients.Skipping the inverse convolution is absorbed into the first downstream layer without loss of expressivity.

3 Performance evaluation

SEAOTTER is evaluated across rate, distortion, complexity, deployment suitability, and global, dense, and vision-language perception tasks. Its learned transcode improves downstream accuracy while retaining fast sensor encoding, efficient consumer decoding, and JPEG-compatible artifacts.

  • Evaluation setup: SEAOTTER evaluation measures transmission and storage rate, reconstruction distortion, downstream accuracy, and sensor-side encoding throughput against AVIF, WaLLoC, and FRAPPE.Task evaluation covers ImageNet classification, ADE20K segmentation, and SigLIP-2 zero-shot prediction.
  • Downstream accuracy: +12.80 pp ImageNet top-1: SEAOTTER-FT reaches 69.02% versus 56.22% for FRAPPE at matched transmit-bpp 0.109.At n=6, the improvement increases to +19.85 pp; SEAOTTER-ZS gains +4.03 pp at n=12.
  • Downstream accuracy: +7.87 pp ImageNet top-1 over AVIF and +8.00 pp over AVIF-max-speed: SEAOTTER-FT leads at matched rate despite both baselines using more bits per pixel.It also leads SigLIP-2 zero-shot top-1 by +5.63 pp and +4.03 pp, respectively.
  • Storage rate: 13.7% smaller storage artifact and +8.19 pp ImageNet top-1: SEAOTTER-FT outperforms a same-transmit-bpp FRAPPE-plus-vanilla-JPEG reference at n=12.The comparison uses an architecturally fair reference with a vanilla ITU T.81 transcode.
  • Complexity and deployment: More than 250 MPx/s sensor encoding for n≤9: the shared frozen FRAPPE encoder is over an order of magnitude faster than default-speed AVIF and 5–8× faster than AVIF max-speed.The SEAOTTER sandwich adds no encode-time overhead and supports 1080p 30 fps over Wi-Fi after accounting for sensor-side concurrency.
  • Complexity and deployment: ∼3.4× faster consumer decoding than AVIF and 100× faster than FRAPPE without the transcode: SEAOTTER uses vanilla JPEG decoding followed by the synthesis transform.Consumers may skip the synthesis transform altogether when operating directly in the learned color space.
  • Complexity and deployment: SEAOTTER-FT clears BLE, 5G, and Wi-Fi deployment tiers at n∈{3,6,9}, while AVIF clears none of the evaluated tiers.At n=12, SEAOTTER-FT clears 5G and Wi-Fi but misses the BLE compression-ratio threshold by a thin margin.

4 Conclusion

SEAOTTER pairs a sensor-embedded autoencoder with a one-time cloud-side transcode into a standards-compliant JPEG. Across several perception settings, the transcode improves accuracy while the paper identifies modality, component, domain-variation, and human-perception gaps for future work.

  • Conclusion: SEAOTTER pairs a sensor-embedded autoencoder with a one-time cloud-side transcode into a standards-compliant JPEG file.The resulting on-disk artifact is designed for broad consumer use.
  • Conclusion: Across global, dense, and zero-shot perception, transcoding increases downstream accuracy over the same DNN-based autoencoder without it.The artifacts remain usable by virtually any data consumer.
  • Limitations: The study tests only RGB and does not characterize depth, IR, multispectral, or hyperspectral signals.These modalities are identified as natural extensions but remain unevaluated.
  • Limitations: The study does not isolate the contributions of softsign companding, DCT-domain Q(k) matrices, or the 3×3 wrapper filter.The individual component effects therefore remain uncharacterized.
  • Limitations: Human perception of SEAOTTER-JPEG artifacts versus standard JPEG or AVIF at matched storage rate remains unevaluated.This is noted as important because SEAOTTER uses a nonstandard color space.

A.1 Multi-axis performance summary

Figure 5 provides a consolidated comparison of SEAOTTER variants with conventional and neural codecs. The overview spans sensor, cloud, and consumer costs together with reconstruction quality and downstream accuracy.

  • Multi-axis overview: Figure 5 compares SEAOTTER variants with conventional and neural codec baselines across multiple performance axes.It condenses the per-axis results into a single view.
  • Multi-axis overview: The overview includes sensor-, cloud-, and consumer-side cost axes, reconstruction quality, and downstream accuracy.The figure complements rate–accuracy–throughput and reconstruction-quality views elsewhere in the evaluation.

A.2 Learned quantization matrices

The learned DCT-domain quantization matrices are compared with matched-bpp ITU T.81 tables and reveal how SEAOTTER allocates quality across its learned color space.

  • Matrix comparison: Figure 6 visualizes three learned DCT-domain quantization matrices alongside matched-bpp ITU T.81 4:4:4 quantization tables.The comparison exposes differences in per-channel frequency weighting.
  • Learned color space: The learned color space is essentially YCgCo up to per-channel sign, with lower-bpp settings crushing mid-frequency chroma and higher-bpp settings preserving it.Per-channel colormap hues derive from the learned RGB-mixing kernel.

A.3 Standalone learned JPEG vs ITU T.81 on Kodak

The standalone learned JPEG sandwich is evaluated against ITU T.81 on native-resolution Kodak images using matched-rate operating points. The comparison tests whether learned color and quantization transforms improve reconstruction quality without the FRAPPE upstream.

  • The evaluation uses 24 native-resolution Kodak validation images without resizing or cropping.The comparison includes the trained learned JPEG bundle and ITU T.81 baselines with and without chroma subsampling.
  • SEAOTTER operating points are matched against ITU T.81 baselines using a quality ladder anchored to the learned codec’s rates.The ITU baselines use the smallest integer JPEG-sub=0 quality that satisfies the stated Kodak PSNR and bitrate comparison criterion.
  • The experiment isolates the learned JPEG sandwich by omitting FRAPPE-side encoding from the evaluation.The evaluated bundle consists of F, F−1, and the three learned quantization matrices Q(0), Q(1), and Q(2).

A.4 Standalone learned JPEG on ImageNet

The standalone learned JPEG codec compares learned per-rate transforms with matched-bitrate ITU T.81 tables on ImageNet and Kodak. Its learned quantization and color-space choices are evaluated through reconstruction quality and downstream classification.

  • The ImageNet re-evaluation reports top-1 accuracy for the same 17 standalone-codec cells used in the codec comparison.It uses 50,000 ImageNet validation images with squash-3842 preprocessing and the same teacher as the main classification task.
  • Learned DCT quantization matrices vary by rate, while the learned color space is essentially YCgCo up to per-channel sign.The lowest-bpp matrix can suppress mid-frequency chroma, whereas the highest-bpp matrix preserves it.
  • +0.27 / +1.40 / +1.27 dB in PSNR are achieved by SEAOTTER’s three operating points over matched-bpp ITU T.81 4:4:4 on Kodak.The Kodak evaluation uses native-resolution validation images and compares the learned codec with ITU T.81 variants.

A.5 Per-task rate-distortion details

The per-task evaluation reports transmit and storage rates alongside task accuracy and reconstruction PSNR across classification, segmentation, and vision-language classification. It also distinguishes downstream accuracy optimization from pixel-fidelity optimization.

  • Tables 4–6 report transmit bpp, storage bpp, task accuracy, and reconstruction PSNR for classification, segmentation, and vision-language tasks.Transmit bpp measures sensor-uplink rate, while storage bpp measures the on-disk JPEG size after cloud-side transcoding.
  • SEAOTTER-FT intentionally has low reconstruction PSNR because fine-tuning trades pixel fidelity for downstream accuracy.PSNR is reported for transparency rather than treated as the target quality measure.
  • The standalone ImageNet and Kodak evaluations provide separate codec-level reference tables for top-1 accuracy, PSNR, and bitrate.The ImageNet table uses 50,000 validation images, while the Kodak table covers 24 native-resolution images without FRAPPE upstream.

A.6 Storage-rate trade-offs

Storage-rate analyses compare downstream accuracy with on-disk JPEG compression and contrast consumer storage compression with sensor transmit compression. Deployment-tier results and timing protocols contextualize these trade-offs.

  • Figure 8 relates downstream accuracy to storage compression ratio and contrasts sensor-side transmit compression with downstream consumer compression.The top row uses on-disk JPEG size after transcode; the bottom row includes a y=x reference line.
  • Consumer decode timings exclude the one-time cloud transcode, which is paid once per image rather than during steady-state consumption.Measurements use batch size 1 on an AMD EPYC 9354 CPU and omit the downstream teacher forward.
  • Conventional-codec timings use Pillow 12.2 with libjpeg-turbo for JPEG and libavif for AVIF, without GPU or hardware-codec acceleration.AVIF encoding uses libaom and decoding uses dav1d; JPEG and AVIF use the stated Pillow quality configurations.
  • SEAOTTER and FRAPPE clear all three deployment tiers at low-bitrate operating points, whereas AVIF clears none at evaluated qualities.A tier requires the pipeline-and-operation cell to satisfy the corresponding deployment thresholds.
Loading 2606.03940v1…