Source-linked AI summary
Deep Residual Learning in the JPEG Transform Domain
Max Ehrlich, Larry Davis
TL;DR
Deep networks are computationally and memory intensive, motivating a way to process JPEG inputs without decompression. The paper reformulates residual-network operations in the JPEG transform domain using linear maps and approximated ReLu. The resulting method speeds image processing with little impact on classification accuracy.
Problem
Deep networks require substantial data, computation, and memory, while conventional JPEG-based learning begins by decompressing images.
Method
The method expresses residual-network operations directly on JPEG representations, using linear-map formulations and Approximated Spatial Masking for ReLu.
Results
The method provides a notable processing-time benefit with little impact on classification accuracy.
Takeaways & Limitations
JPEG-domain residual learning can process compressed images directly while retaining performant classification results.
Takeaways & Limitations
Inference efficiency improves substantially, but training efficiency is limited by the more complex update of the JPEG-domain operation.
Abstract
from arXiv · showhide
We introduce a general method of performing Residual Network inference and learning in the JPEG transform domain that allows the network to consume compressed images as input. Our formulation leverages the linearity of the JPEG transform to redefine convolution and batch normalization with a tune-able numerical approximation for ReLu. The result is mathematically equivalent to the spatial domain network up to the ReLu approximation accuracy. A formulation for image classification and a model conversion algorithm for spatial domain networks are given as examples of the method. We show that the sparsity of the JPEG format allows for faster processing of images with little to no penalty in the network accuracy.
1. Introduction
The paper targets the computational burden of deep networks by reformulating ResNet operations to run directly on JPEG images. It presents JPEG-domain convolutional learning, residual classification, model conversion, and approximated spatial masking, with faster testing and marginally faster training.
- The method reformulates ResNet operations on compressed images while targeting mathematical equivalence with spatial-domain networks.JPEG and network operations are combined through linear maps, while ReLu requires approximation.
- The work introduces a general method for expressing convolutional networks in the JPEG domain.
- It provides a residual-block formulation for image classification and a conversion algorithm for pretrained spatial-domain networks.
- Approximated Spatial Masking applies piecewise linear functions in the transform domain.
- Skipping JPEG decompression increases testing speed and provides a marginal training-speed improvement.
2. Prior Work
Prior work spans compressed-domain operations, compressed-domain machine learning, and deep learning on compressed representations. Earlier methods address linear image processing, retrieval, feature extraction, and specialized DCT- or video-based networks.
- Prior work is organized into compressed-domain operations, machine learning in the compressed domain, and deep learning in the compressed domain.
- Earlier compressed-domain methods formulate JPEG-compatible scalar, pixelwise, blockwise, and arbitrary linear operations.
- Compressed-domain machine learning includes image processing, JPEG retrieval, hypothesis testing, and DCT-domain SIFT extraction.
- Deep-learning studies use DCT components for convergence or separate networks for compressed video representations.
3. Background
The background defines JPEG compression and its transform-domain representation, emphasizing linear steps that can be fused with image operations. This enables JPEG-domain maps that are cheaper than separate decompression, processing, and recompression while remaining mathematically equivalent.
- The JPEG pipeline applies DCT, zigzag linearization, quantization, rounding, and entropy coding to image blocks.
- JPEG sparsity arises because quantization pushes coefficients toward zero, increasing the likelihood that rounding drops them.
- Theorem 1 states that reconstruction using the m lowest spatial frequencies is optimal among sets of m spatial frequencies.
- JPEG steps 1–4 are linear and can be composed into one map on the compressed representation; rounding is irreversible and entropy coding is nonlinear.
- The JPEG transform maps spatial image tensors into block-position and within-block coefficient indices.
- The fused map Ξ performs decompression, an arbitrary pixel manipulation, and recompression with fewer operations and exact equivalence to the separate process.
4. JPEG Domain Residual Networks
The paper reformulates ResNet operations directly in the JPEG transform domain, using linear mappings for convolution and batch normalization and Approximated Spatial Masking for nonlinear functions such as ReLU.
- 4.1. Convolution: JPEG-domain ResNet operations represent convolution directly as a linear map over batch, channel, height, and width dimensions, avoiding explicit spatial-domain intermediate operations.The convolution map can be constructed by reshaping the JPEG decompression tensor and applying batch convolution.
- 4.1. Convolution: Inference convolution maps can be precomputed, while training requires more complex gradient updates but remains faster than the equivalent spatial-domain model.The paper distinguishes improved inference efficiency from the more involved training update.
- 4.2. ReLu: Approximated Spatial Masking applies piecewise-linear functions by selecting low-frequency approximations, constructing masks for function intervals, and mixing masked transform-domain results.The method preserves pixel values when the mask selects the correct linear piece; errors arise from mask mistakes.
- 4.2. ReLu: ASM computes ReLU more accurately from fewer spatial frequencies than direct approximation because it needs the approximation to preserve signs rather than exact pixel values.In the illustrated comparison, ASM preserves positive pixel values while direct approximation does not.
- 4.2. ReLu: The JPEG-domain extension applies the remaining JPEG transform tensors while skipping the blocking tensor because blocks are processed independently without interactions.This extends the DCT-domain construction to JPEG representations.
5. Experiments
Experiments evaluate equivalence, ReLu approximation, and throughput for JPEG-domain residual networks. The method matches spatial-domain accuracy closely, ASM outperforms the naive approximation, and inference throughput improves substantially despite limited training gains.
- 5.2. Model Conversion: Spatial and JPEG testing accuracies are the same to within floating point error after model conversion with exact ReLu.The evaluation uses losslessly JPEG-compressed inputs across MNIST, CIFAR-10, and CIFAR-100.
- 5.3. ReLu Approximation Accuracy: ASM again outperforms APX when models are trained in the JPEG domain, while learned weights allow high accuracy with fewer spatial frequencies.The training experiment indicates that network weights learn to compensate for the ReLu approximation.
- 5.3. ReLu Approximation Accuracy: ASM consistently gives lower ReLu-block error and outperforms APX in model conversion across the tested spatial-frequency range.The raw-block comparison reports lower RMSE for ASM, while the conversion experiment evaluates 1–15 spatial frequencies.
- 5.4. Efficiency of Training and Testing: JPEG models outperform spatial models in throughput across all tested datasets, with training gains limited but inference performance greatly improved.The training limitation is attributed to the more complex gradient created by JPEG-domain convolution and ReLu operations.
6. Conclusion and Future Work
The paper formulates residual learning directly in the JPEG transform domain and reports processing-time benefits with little impact on classification accuracy. It identifies representation efficiency and missing library support as important remaining constraints.
- 6. Conclusion and Future Work: JPEG-domain residual learning provides a notable processing-time benefit with little impact on classification accuracy.The method expresses convolutions as linear maps and introduces an approximation technique for ReLu.
- 6. Conclusion and Future Work: Linear maps require more space than spatial-domain convolutions, making large-image datasets difficult to scale.This is identified as a future efficiency-of-representation concern.
- 6. Conclusion and Future Work: Missing sparse-tensor support and suboptimal einsum implementations in commodity deep-learning libraries constrain practical performance.The paper notes that coordinate-list sparse tensors are highly non-performant and that einsum is slower than optimized convolution libraries.
1. Proof of the DCT Least Squares Approximation Theorem
The proof shows that truncating a signal to its first m DCT coefficients gives the least-squares approximation. It relies on the DCT’s linearity and orthonormality.
- 2. Proof: The proof rewrites the DCT as a linear map and solves the resulting least-squares problem with the normal equations.The matrix formulation identifies the truncated-coefficient reconstruction being optimized.
- 2. Proof: The least-squares approximation uses the first m DCT coefficients.Because the DCT is orthonormal, the rows selected from its matrix are orthogonal, yielding the stated solution.
2. Proof of the DCT Mean-Variance Theorem
The proof rewrites variance through the DCT representation of a zero-mean signal. Orthogonality then simplifies the coefficient-based expression.
- 2. Proof: For a zero-mean signal, the proof rewrites variance using the linear relation X = DY between samples and DCT coefficients.The DCT representation substitutes coefficient space for the original signal space.
- 2. Proof: Orthogonality of the DCT matrix simplifies the resulting variance expression.The simplification follows after expressing the signal through the orthogonal transformation.
3. Algorithms
The paper specifies three transform-domain operations: convolution explosion, approximated spatial masking for ReLu, and batch normalization. Their pseudocode defines the required inputs, transformations, and outputs for implementation.
- 3. Algorithms: The algorithms cover convolution explosion, an ASM ReLu approximation, and batch normalization.The paper presents pseudocode for all three layer operations.
- 3. Algorithms: Convolution explosion reshapes JPEG-domain constants, applies a strided convolution with the initial filter, and constructs output dimensions from the stride and channel parameters.The operation uses constants J and eJ with JPEG-block dimensions and takes filter, channel, image-size, and stride arguments.
- 3. Algorithms: The ASM ReLu approximation computes a spatial mask from a DCT block, frequency limit, and block size, then applies that mask to the block.The mask is initialized over an N × N domain and assigns ones where the relevant index condition is positive.
- 3. Algorithms: Batch normalization operates on JPEG blocks using inverse quantization, momentum, training or testing mode, and externally stored scale and shift parameters.The procedure distinguishes batch statistics from running statistics while updating and normalizing transform-domain features.