Source-linked AI summary
LiVeAction: a Lightweight, Versatile, and Asymmetric Neural Codec Design for Real-time Operation
Dan Jacobellis, Neeraja J. Yadwadkar
TL;DR
Resource-constrained sensors need compression that handles machine-perception and non-traditional signals without the cost and modality-specificity of existing codecs. LiVeAction addresses this with a lightweight asymmetric encoder and variance-based rate penalty, achieving strong cross-modality compression and throughput results while supporting practical low-power deployment.
Problem
Existing codecs either target human perception, miss signal redundancies, or require over-parameterized, data-hungry, modality-specific neural designs unsuitable for resource-constrained settings.
Method
LiVeAction combines a lightweight asymmetric encoder using FFT-like structured operations with a simplified variance-based rate penalty instead of perceptual or adversarial losses.
Results
Across modalities, LiVeAction establishes a superior rate-distortion-complexity frontier, including 35.6× faster spatial-audio encoding and 9.95 MPix/s RGB encoding on a low-power mobile CPU.
Takeaways & Limitations
LiVeAction increases accessibility of learned compression for diverse signals and sensors while enabling practical deployment on low-power mobile and remote-sensing devices.
Abstract
from arXiv · showhide
Modern sensors generate rich, high-fidelity data, yet applications operating on wearable or remote sensing devices remain constrained by bandwidth and power budgets. Standardized codecs such as JPEG and MPEG achieve efficient trade-offs between bitrate and perceptual quality but are designed for human perception, limiting their applicability to machine-perception tasks and non-traditional modalities such as spatial audio arrays, hyperspectral images, and 3D medical images. General-purpose compression schemes based on scalar quantization or resolution reduction are broadly applicable but fail to exploit inherent signal redundancies, resulting in suboptimal rate-distortion performance. Recent generative neural codecs, or tokenizers, model complex signal dependencies but are often over-parameterized, data-hungry, and modality-specific, making them impractical for resource-constrained environments. We introduce a Lightweight, Versatile, and Asymmetric neural codec architecture (LiVeAction), that addresses these limitations through two key ideas. (1) To reduce the complexity of the encoder to meet the resource constraints of the execution environments, we impose an FFT-like structure and reduce the overall size and depth of the neural-network-based analysis transform. (2) To allow arbitrary signal modalities and simplify training, we replace adversarial and perceptual losses with a variance-based rate penalty. Our design produces codecs that deliver superior rate-distortion performance compared to state-of-the-art generative tokenizers, while remaining practical for deployment on low-power sensors. We release our code, experiments, and python library at https://github.com/UT-SysML/liveaction .
I. INTRODUCTION
LiVeAction targets compression for resource-constrained sensing by combining a lightweight asymmetric encoder with a modality-general rate–distortion objective. It reports improved rate-distortion-complexity trade-offs across diverse signals, including a 34% BD-rate improvement and more than 10× faster encoding than Cosmos.
- Wearable, medical, satellite, and autonomous-sensing deployments require compression under strict power and bandwidth constraints, while JPEG and MPEG target human perception.
- General-purpose scalar quantization and resolution reduction support arbitrary signals but fail to exploit inherent redundancies, producing poor rate–distortion performance.
- Generative neural codecs capture complex dependencies but remain impractical because of costly encoders, modality-specific architectures, and human-perception losses.
- LiVeAction uses a lightweight asymmetric encoder with FFT-inspired block-diagonal operations to reduce computation while preserving representational quality.
- LiVeAction replaces perceptual and adversarial objectives with a variance-based rate penalty, simplifying training and supporting diverse signal types with thousands rather than millions of examples.
- 34% BD-rate improvement and more than 10× faster encoding are reported against Cosmos, while LiVeAction supports spatial audio, hyperspectral, medical, audio, image, and video signals.
II. BACKGROUND AND RELATED WORK
Prior work spans training-free compression, learned autoencoders, generative codecs, and efficient neural layers. LiVeAction builds on these directions with structured operations in its lightweight analysis transform.
- Training-free lossy codecs offer high throughput and bounded error for scientific data, while transform-based standards balance rate, distortion, and computational cost.
- End-to-end autoencoder compression has surpassed traditional audio, image, and video codecs in rate-distortion performance, although design and runtime complexity initially limited adoption.
- ShuffleNet and Monarch replace standard convolutional and MLP layers with FFT-like structured matrix operations for efficient representation learning and compression.
- LiVeAction’s proposed analysis transform uses a lightweight DNN with block-diagonal structured operations.
III. PROPOSED METHOD: DESIGN AND IMPLEMENTATION
LiVeAction combines an asymmetric codec with an FFT-like lightweight analysis transform, flexible synthesis, and simplified quantization and rate-penalty procedures for efficient, versatile compression.
- LiVeAction targets extreme encoding efficiency, competitive rate-distortion performance, and flexibility across arbitrary signal modalities.
- The codec places an FSQ-based autoencoder between WPT and IWPT while modifying the learned transforms and training procedures asymmetrically.
- WPT and IWPT trade spatiotemporal resolution for frequency resolution before and after the learned analysis and synthesis transforms.
- Lightweight analysis transform: Factorized grouped convolutions form an FFT-like block-diagonal analysis structure, with GELU nonlinearities and eight-group normalization.
- Quantization and rate penalty: The codec normalizes latents with a learned per-channel Laplacian CDF, constraining them to a range strictly smaller than 8 bits.
- Quantization and rate penalty: FSQ uses additive uniform noise during the first 70% of training, then freezes the encoder and switches to rounding for decoder training.
- Versatile synthesis: The synthesis transform uses EfficientViT linear-attention blocks generalized to 1D, 2D, and 3D signals.
- Lightweight analysis transform: A structured matrix pair replaces dense projection, substantially reducing parameters and computation while achieving encoding throughput competitive with WaLLoC’s fully connected projection.
IV. EVALUATION
LiVeAction is evaluated across audio, spatial audio, images, hyperspectral data, 3D medical volumes, and video using rate-distortion, complexity, and downstream-acceleration metrics.
- Datasets and modalities: Experiments cover stereo audio, seven-channel spatial audio, natural images, 224-band hyperspectral images, 3D medical volumes, and video.
- Datasets and modalities: Spatial-audio evaluation measures PSNR, SSDR, and SRDR to separate spatial distortion from other impairments.
- Datasets and modalities: Image evaluation includes rate-distortion performance and top-1 ImageNet classification accuracy at 224 × 224, plus evaluation on Kodak.
- Metrics and baselines: The benchmark reports compression ratio, PSNR, per-sample throughput, and dimensionality reduction as a proxy for downstream acceleration.
- Metrics and baselines: Comparisons include JPEG2000, Stable Audio, EnCodec, and Cosmos alongside other conventional and neural compression systems.
A. Results and Discussion.
LiVeAction delivers strong rate–distortion–complexity performance across modalities, with particularly large gains in encoding efficiency and dimensionality reduction. It remains competitive with established and neural codecs while supporting machine-perception evaluation and low-power deployment.
- LiVeAction establishes a superior rate–distortion–complexity frontier across modalities, despite simpler objectives, smaller datasets, and fewer GPU hours than prior generative tokenizers.The evaluation covers spatial audio, music, RGB images, hyperspectral data, 3D medical images, and video.
- Spatial audio: 35.6× faster encoding and 12.8× greater dimensionality reduction than EnCodec are achieved for spatial audio, alongside +6.09 dB SSDR and +13.55 dB SRDR.
- RGB image: LiVeAction matches Cosmos’ downstream ImageNet top-1 classification accuracy while using 48% lower bitrate, despite not being trained on ImageNet.The figure evaluates machine perceptual quality using ImageNet classification accuracy on decoded images.
- LiVeAction provides ∼70× higher throughput than CPU-only JPEG 2000 and over 2× faster encoding on the same CPU while reducing latent dimensionality by 64×.The reduced latent dimensionality is reported to accelerate downstream models while slightly improving rate–distortion performance.
- 3D medical images: 64× dimensionality reduction and 2.1× higher compression than JPEG 2000 are achieved on MEDMNIST 3D, with 2.7dB higher PSNR.
- Video: > 1.7× higher compression ratio and > 3.8× higher GPU throughput than Cosmos are achieved for comparable-quality 1080p video.LiVeAction encodes full-length video in one pass on an RTX 4090, while Cosmos requires memory-intensive chunking.
B. Additional experiments.
Additional experiments show that the simplified rate loss reduces bitrate with minor quality impact, while a separate generative model can enhance perceptual quality after decoding.
- 22 percent bitrate reduction is achieved with minor quality impact using the approximate rate model.The ablation retrained the RGB codec with an explicit rate term using CompressAI's EntropyBottleneck and an auxiliary optimizer.
- Table IV reports actual post-entropy-coding bitrate rather than a rate estimated from the distribution.
- A score-based generative model enhances perceptual quality post-decoding because LiVeAction omits adversarial and perceptual losses.The enhancement uses FLUX ControlNet conditioned on the decoder output; neither model was trained on codec outputs.
V. CONCLUSION AND FUTURE WORK
The conclusion presents LiVeAction as a neural codec that broadens learned compression to new signal and sensor types while reducing power and bandwidth demands. Future work targets variable-rate training and joint optimization with downstream machine-learning tasks.
- Conclusion: LiVeAction increases accessibility of learned compression for new types of signals and sensors.
- Conclusion: Improved signal-ingestion efficiency lowers power and bandwidth demands while maintaining quality for mobile and remote sensing applications.
- Future work: Future work will explore variable-rate training and joint optimization with downstream ML tasks to better align compression with inference accuracy.
- Additional comparison: Figure 5 compares Cosmos, LiVeAction, and a generative-model-enhanced LiVeAction at 0.15 bpp after rescaling the image for Cosmos.
- Additional results: Table V reports additional RGB-image results and baselines, including ImageNet top-1 accuracy and an uncompressed accuracy of 0.8979.