PaperlayerView exact revision on arXiv

Source-linked AI summary

A Technical Overview of AV1

Jingning Han, Bohan Li, Debargha Mukherjee, Ching-Han Chiang, Adrian Grange, Cheng Chen, Hui Su, Sarah Parker, Sai Deng, Urvang Joshi, Yue Chen, Yunqing Wang, Paul Wilkins, Yaowu Xu, James Bankoski

arXiv:2008.06091v2eess.IV

TL;DR

The paper examines how AV1 improves video compression while remaining feasible for hardware-constrained devices. It provides a technical overview of AV1’s design theories, coding tools, and encoder strategies, reporting substantial compression-oriented complexity trade-offs. AV1 decoding is about three times as complex as VP9, while encoding is roughly 34.6–39.6 times as complex at high compression performance.

  • Problem

    Growing video demand requires higher compression efficiency, but mobile-device power constraints and limited CPU-frequency growth constrain the complexity of decodable codecs.

  • Method

    The paper surveys AV1’s compression techniques and hardware-feasibility considerations, including coding-tool design and encoder strategies for practical complexity.

  • Results

    AV1 decoding complexity is about 3 times VP9’s, while encoding complexity is roughly 34.6–39.6 times VP9’s at high compression performance.

  • Takeaways & Limitations

    AV1’s codec design balances compression performance with hardware feasibility through explicit consideration of latency, silicon area, and practical encoder complexity.

Abstract

from arXiv · show

The AV1 video compression format is developed by the Alliance for Open Media consortium. It achieves more than 30% reduction in bit-rate compared to its predecessor VP9 for the same decoded video quality. This paper provides a technical overview of the AV1 codec design that enables the compression performance gains with considerations for hardware feasibility.

I. INTRODUCTION

AV1 responds to growing video demand with improved compression while addressing the computational and hardware constraints of mobile devices and TVs. The paper surveys its coding tools, syntax, decoder feasibility, and practical encoder-complexity trade-offs.

  • I. INTRODUCTION: Web-based video applications increased bandwidth and storage demands, motivating compression techniques with better efficiency.
  • I. INTRODUCTION: AV1, released in 2018 by the Alliance for Open Media, enabled about 30% compression gains over VP9 and has been adopted by major platforms and services.
  • I. INTRODUCTION: Mobile devices and TVs require decodability under power constraints, while improved compression generally increases computational complexity.
  • I. INTRODUCTION: AV1 coding tools were reviewed for latency and silicon area, producing a design balanced between compression performance and hardware feasibility.
  • I. INTRODUCTION: The paper explains AV1 compression theories and hardware-decoder considerations, including encoder strategies for practical coding complexity.
  • I. INTRODUCTION: AV1 uses predictive-transform coding with intra prediction, inter-frame motion compensation, residual transforms, and quantization across supported pixel formats and bit depths.
  • I. INTRODUCTION: The overview proceeds from frame-level design through block operations, entropy coding, and filtering, evaluating coding performance with the libaom encoder.
  • II. HIGH LEVEL SYNTAX: Tile Lists allow decoding and display of selected frame regions without fully decoding all tiles, supporting light-field applications.

IV. SUPERBLOCK AND TILE

AV1 organizes coding around recursively partitionable superblocks and independently processable tiles, while constraining decoder operations to preserve hardware feasibility. Its block-level tools add flexible partition searches and prediction modes for compression efficiency.

  • A. Superblock: Superblocks are AV1’s largest coding units, sized 128 × 128 or 64 × 64 luma samples, and can be recursively partitioned into smaller blocks.Each block has its own prediction and transform modes, while superblock coding depends only on above and left neighboring superblocks.
  • B. Tile: Tiles are rectangular superblock arrays with references and probability updates confined within boundaries, enabling independent coding and multi-threading.AV1 supports uniform tiles and non-uniform lattices; non-uniform sizing assigns smaller tiles to more complex regions to balance thread workloads and reduce latency.
  • 1) Block Size Dependent Constraints:: 128 × 128 blocks are decoded as a series of 64 × 64 units, retaining the same silicon area without a compression-performance penalty.This rearranges entropy-coding order for luma and chroma components so hardware can process the larger block in smaller units.
  • 1) Block Size Dependent Constraints:: AV1 constrains chroma coding for 4 × 4 luma blocks in YUV420 to address worst-case latency from neighbor-dependent intra reconstruction.Chroma uses 2 × 2 prediction when all corresponding luma blocks are inter-coded, and 4 × 4 prediction following the bottom-right luma block’s mode when any is intra-coded.
  • 2) Two-Stage Block Partitioning Search:: A two-stage partition search first narrows the likely block-size range using square partitions, then performs extensive rate-distortion optimization over all 10 partition options.Changing the search range derived from the first pass trades compression performance against encoding speed.
  • 2) Non-directional Smooth Intra Prediction:: AV1 expands block-level prediction choices with finer directional angles and distance-weighted smooth modes, while using constrained options for blocks smaller than 8 × 8.Directional modes extend VP9’s eight bases by up to three 3° steps in either direction; smooth modes use boundary references for interpolation.

3) Recursive Intra Prediction:

AV1 models intra-pixel correlation with recursive linear prediction, then expands the recursion over 4 × 2 patches to eliminate dependencies and enable parallel hardware processing. It also includes luma-derived chroma prediction, IntraBC, and palette coding for specialized content.

  • Recursive linear prediction: A 2-D first-order Markov field models inter-pixel correlation, with each prediction combining above, left, and upper-left reconstructed or predicted pixels.AV1 defines five coefficient sets {α, β, γ}, each representing a different spatial correlation pattern.
  • Recursive linear prediction: AV1 expands recursive pixel prediction over a 4 × 2 patch, deriving patch coefficients from {α, β, γ} and adjacent reference pixels.The expansion produces direct expressions for pixels such as x0, x1, and x2 from boundary references.
  • Hardware-oriented implementation: Expanding the recursion avoids inter-pixel dependencies within each 4 × 2 patch, allowing hardware decoders to process predictions in parallel.Figure 5 illustrates reference pixels p0–p6 predicting one patch, whose outputs become references for the next patch.
  • Related intra tools: Chroma-from-luma prediction forms chroma pixels from reconstructed luma pixels by adding chroma DC prediction to a scaled luma AC contribution.The scaling factor is explicitly signaled in the bitstream.
  • Related intra tools: IntraBC uses a full-pixel motion vector to reference previously coded pixels within the same frame, with bilinear interpolation for chroma sub-pixel displacement.It is available only for intra coding frames and can be enabled or disabled in the frame header.
  • Related intra tools: IntraBC creates a reconstruction mismatch because encoders optimize against pre-filter pixels while hardware decoders retrieve post-processing-filter pixels.When IntraBC is allowed in an intra-only frame, AV1 disables all post-processing filters; the mode is most useful for text or repeated patterns.
  • Related intra tools: Palette mode assigns pixels to 2 to 8 predictively coded base colors and is particularly suitable for blocks with limited pixel variation.Luma and chroma planes can choose palette mode independently, while color indexes are coded using a probability model conditioned on previous indexes.

C. Inter Frame Prediction

AV1 expands inter-frame prediction with adaptive filtering, affine motion compensation, and compound prediction tools, while accounting for decoder feasibility. Its interpolation design captures directional signal differences and reuses intermediate results to limit computation.

  • AV1 supports adaptive filtering, affine motion compensation, and flexible compound prediction to exploit temporal correlation.
  • 1) Translational Motion Compensation:: 1/8 pixel motion-vector accuracy is implemented with separable horizontal and vertical interpolation filters.Horizontal filtering produces intermediate pixels that can be reused across multiple final sub-pixels.
  • 1) Translational Motion Compensation:: Independent vertical and horizontal filter selection captures directional statistical differences and improves prediction quality.Each direction selects among SMOOTH, REGULAR, and SHARP FIR filters.
  • 1) Translational Motion Compensation:: SMOOTH and REGULAR use 6-tap FIR filters, while SHARP uses an 8-tap design to mitigate ripple near the cutoff frequency.Additional 4-tap filters serve blocks with dimensions of 4 or less.
  • 2) Affine Model Parameters:: Affine motion compensation projects current pixels into a reference frame using scaling, rotation, and translational parameters.The translational pair is conventionally coded, while the other parameters describe scaling and rotation.
  • 2) Affine Model Parameters:: Local affine parameters are estimated from neighboring blocks sharing the current block’s reference frame, subject to relevance and sample-count constraints.Up to eight candidates are scanned; blocks with motion-vector differences above 8 pixels are discarded, and fewer than two valid references disable the model.

3) Affine Motion Compensation:

AV1 constructs affine predictions through blockwise, two-stage interpolation and supports several compound modes for combining reference predictions. These tools improve flexibility but increase computational and encoder-search complexity.

  • 3) Affine Motion Compensation:: Affine prediction is allowed for blocks at least 8 × 8 and decomposes each prediction block into 8 × 8 units.Each unit is centered using the translational motion vector before scaling and rotation are applied.
  • 3) Affine Motion Compensation:: Two-stage shear decomposition reduces affine interpolation to horizontal then vertical filtering over a 15×15 reference region.The process generates a 15 × 8 intermediate array before producing the final prediction block.
  • 3) Affine Motion Compensation:: 1472 multiplications generate an 8 × 8 affine prediction, matching the translational case despite higher actual cost from changing filter coefficients.
  • 3) Affine Motion Compensation:: AV1 constrains affine offsets so the reference region remains within a 15×15 pixel array, improving cache performance.The horizontal and vertical offsets must stay within 1 pixel of the corresponding spatial displacement.
  • 4) Compound Predictions:: Compound prediction linearly combines motion-compensated blocks from two reference frames using distance-, difference-, or wedge-based weighting.Wedge mode uses preset coefficient arrays to divide a block into sections filled from different references.
  • 4) Compound Predictions:: Compound modes increase encoding complexity because optimizing two motion vectors expands the search space, making efficient mode selection challenging.A joint search can reduce motion-vector search points by iteratively fixing one vector while searching the other.
  • 4) Compound Predictions:: Overlapped block motion compensation and combined inter-intra prediction blend multiple reference blocks while using a single reference frame and one motion vector.

D. Dynamic Motion Vector Referencing Scheme

AV1 dynamically derives motion-vector candidates from spatial and temporal neighbors, ranking them to improve predictive motion-vector coding efficiency.

  • AV1 obtains motion-vector candidates from spatial and temporal neighbors and ranks them for entropy coding.This targets the sizable bit-rate contribution of motion-vector coding and its dependence on prediction accuracy.

1) Spatial Motion Vector Reference:

AV1 uses spatial and temporal motion information to construct motion-vector references, while adapting storage and trajectory modeling to hardware and motion-prediction constraints.

  • 1) Spatial Motion Vector Reference:: Spatial reference search scans neighboring 8 × 8 units sharing the current block’s reference frame and records up to eight distinct motion vectors.The search covers three block rows above and three block columns to the left, beginning with the nearest neighbors.
  • 1) Spatial Motion Vector Reference:: AV1 reduces line-buffer storage by representing an 8×8 region coded with 4×4 blocks using the bottom-right 4×4 block’s information.This halves the space needed for motion data in the line buffer.
  • 1) Spatial Motion Vector Reference:: Temporal motion-vector prediction uses a motion field because collocated-block methods are limited in capturing low-velocity motion trajectories.Motion trajectories are built through linear interpolation or extrapolation and associated with 8 × 8 current-frame blocks.
  • 1) Spatial Motion Vector Reference:: When both trajectory types are available, AV1 discards the extrapolated trajectory because interpolation typically provides better estimation accuracy.

3) Hardware Constraints:

AV1 constrains motion-field construction and organizes transform processing to balance compression efficiency with hardware feasibility. Its design localizes memory access and supports flexible block structures while limiting decoder complexity.

  • Motion-field hardware: 64×64 decoder processing makes baseline hardware cost invariant to frame size, whereas unrestricted motion-field construction would scale with resolution.
  • Motion-field hardware: Maximum displacement constraints localize motion-field references to a 64 × (64 + 2 × 64) region, enabling DRAM-to-SRAM loading per 64×64 block.
  • Motion-vector coding: AV1 ranks spatial and temporal motion vectors, prioritizing immediate above, left, and top-right neighbors because they tend to correlate more strongly with the current block.
  • Motion-vector coding: The first four ranked motion vectors become candidate predictors, and the encoder signals the index of the candidate closest to the desired vector.
  • Transform blocks: AV1 extends transform blocks to 64×64, supports rectangular sizes, and permits recursive inter-block partitioning of up to two luma levels.
  • Transform blocks: Intra blocks use uniform transform sizes, while chroma uses the largest available transform because its statistics vary less.

2) Transform Kernels:

AV1 expands transform flexibility through independently selected kernels, larger and rectangular blocks, and quantization controls. These choices are paired with implementation-aware restrictions and arithmetic-coding support.

  • Kernel design: Each AV1 transform block independently selects its transform kernel, unlike VP9 coding blocks, producing 16 combinations from four 1-D kernels.
  • Kernel design: DCT, ADST, FLIPADST, and IDTX accommodate different correlation statistics and boundary conditions, with ADST variants used for larger blocks to support butterfly implementations.
  • Hardware considerations: Inverse-transform multiplication complexity falls from O(N^2) to O(NlogN) with butterfly structure, motivating its use for large transform blocks.
  • Kernel design: Only DCT and IDTX are used at dimensions 32 × 32 and above because sinusoidal transform coding gains largely converge for large blocks.
  • Quantization: AV1 quantization uses QP values from 0 to 255; QP 0 provides lossless coding, while DC coefficients receive smaller steps than AC coefficients.
  • Quantization: QP offsets can be applied at superblock and coding-block levels, with frames classifying blocks into up to eight segments, each having its own offset.

A. Probability Model

AV1 adapts probability models as symbols are coded and represents them as integer-scaled CDFs. A dual-model arithmetic-coding design preserves precision while reducing multiplication width for hardware throughput.

  • Probability adaptation: AV1 models an M-ary symbol distribution as probabilities and a cumulative distribution function, with each coded outcome updating the model.
  • Probability adaptation: The update uses an indicator vector for the observed symbol and an adaptation rate α, adjusting probability mass toward the latest outcome.
  • CDF updates: CDF updates reduce entries below the observed symbol and increase entries at or above it according to the update rate.
  • Hardware implementation: AV1 scales CDF data by 2^15 and maintains it at 15-bit precision, while the arithmetic coder consumes only the most significant 9 bits.
  • Probability adaptation: Probability models inherit from a signaled reference frame and adapt faster at the beginning of each frame.
  • Hardware implementation: The arithmetic decoder scales its interval length by 1/256 before multiplication so the product fits into 16 bits.

1) Scan Order:

AV1 scans transform coefficients according to transform dimensionality, decomposes magnitudes into progressively coded ranges, and uses context models tailored to transform kernels. Its filtering stages then address reconstruction artifacts and reference quality.

  • Scan order: One-dimensional transform coefficients use directional scans, while two-dimensional transforms use zig-zag order to reflect correlation along the transform structure.
  • Coefficient coding: AV1 codes the last non-zero coefficient first, then processes coefficients in reverse scan order.
  • Coefficient coding: Coefficient magnitudes are decomposed into sign, base-range, low-range, and high-range symbols, with BR and LR using four outcomes and HR covering the remaining range.
  • Context modeling: BR contexts use up to five previously processed coefficients for 2-D transforms and three along the transform direction for 1-D transforms.
  • Context modeling: LR contexts use three nearest coefficients for 2-D transforms and three preceding coefficients along the transform direction for 1-D transforms.
  • Coefficient coding: AC sign bits are packed as raw bits to bypass entropy coding, whereas the DC sign bit remains entropy coded.
  • Filtering: Three optional in-loop filters—deblocking, CDEF, and loop restoration—produce reference frames, while film grain synthesis is applied out of loop before display.
  • Filtering: Deblocking filter length is determined by the minimum transform-block sizes on both sides of a boundary and is further limited by a flatness metric.

2) Boundary Conditions:

AV1’s in-loop filtering combines boundary-aware deblocking, directional deringing, and unit-level restoration. These filters use signal conditions, configurable strengths, and multiple filter choices to improve reconstructed pixels while limiting artifacts.

  • Deblocking: AV1 disables deblocking at high-variance transform boundaries to avoid blurring actual image edges.Thresholds can be set independently for vertical and horizontal luma edges and for each chroma plane.
  • Deblocking: Long deblocking filters require both sides of a boundary to be flat; otherwise AV1 reverts to a shorter filter to limit ringing.The flatness test covers samples through k=3 for 8-tap filters and k=6 for 14-tap filters.
  • CDEF: CDEF selects one of 8 preset directions, then applies primary and secondary filters along the selected direction and 45° off it.The directions are generated from templates through rotation and reflection, and the decoder selects the prevalent direction from reconstructed pixels.
  • CDEF: CDEF’s piece-wise linear function suppresses reference pixels whose values differ substantially from the filtered pixel.Primary and secondary strengths and the damping factor control the filter operation, with coefficients specified by direction and strength index.
  • Loop restoration: Loop restoration independently chooses bypass, Wiener, or self-guided filtering for each 64×64, 128×128, or 256×256 restoration unit.The 7×7 separable Wiener filter signals three coefficients per horizontal or vertical filter because of symmetry and normalization constraints.

2) Self-Guided Filter:

AV1’s self-guided restoration estimates correction directions from denoised versions of reconstructed pixels, while frame super-resolution restores lower-resolution inputs through upsampling and loop restoration.

  • Self-Guided Filter: Self-guided filtering creates two denoised versions of reconstructed pixels and projects the source–reconstruction gap onto their difference sub-space.The encoder derives least-square regression parameters and signals them to the decoder.
  • Self-Guided Filter: The denoised versions are generated from local means and variances in square windows, with radius and noise variance controlling filter strength.Two parameter pairs, (r1, e1) and (r2, e2), produce the two denoised pixel vectors.
  • Frame Super-Resolution: Frame super-resolution upsamples a reconstructed frame to its original dimension, combining linear upsampling with a loop restoration filter.The process is used when the source was downscaled before compression.
  • Frame Super-Resolution: Upsampling operates horizontally after CDEF, interpolating the downsampled row onto the original-width sampling positions with an 8-tap FIR filter.The downsampled and upscaled rows contain D and W pixels, respectively, and offsets are rounded to 1/16-pixel interpolation positions.
  • Frame Super-Resolution: Offset adjustment equalizes rounding-error magnitude at the row’s left and right edges and brings the middle upscaled sample close to zero error.The adjustment addresses spatial bias caused by increasing rounding error across the row.
  • Frame Super-Resolution: Encoder-optimized loop restoration can substantially improve the objective quality of the upsampled frame by recovering high-frequency components.The restoration filter is applied after the upsampling stage.

E. Film Grain Synthesis

AV1 synthesizes film grain at the decoder from transmitted model parameters, allowing grain removal before compression and recreation before display. Its AR model uses neighboring grain samples and randomized noise, with separate scaling for luma and chroma.

  • Film Grain Model: Film grain synthesis removes grain before compression and adds synthetic grain to the decoded picture before display.This addresses the difficulty of compressing the random grain signal with conventional correlation-based coding tools.
  • Film Grain Model: AV1 generates luma grain in raster order from a neighborhood above and to the left containing 2L(L + 1) reference samples.The neighborhood uses a (2L + 1) × L block above and an L × 1 block to the left, with L ∈ {0, 1, 2, 3}.
  • Film Grain Model: The AR model combines reference-region samples with pseudo-random zero-mean, unit-variance Gaussian noise to generate grain patterns.Model parameters for each plane are transmitted through the bitstream, and chroma generation additionally uses the collocated luma grain sample.
  • Film Grain Model: Grain is generated as a 64×64 template, then 32×32 patches are selected at pseudo-random positions and applied to the reconstructed signal.This template-and-patch process supplies grain samples for the decoded video.
  • Film Grain Model: Luma grain amplitude is scaled according to decoded pixel intensity, while chroma grain depends on both chroma intensity and collocated luma values.The scaling function and chroma parameters are signaled in the bitstream for decoder reconstruction.
  • Signaling: Film-grain model parameters are selected per frame, and a frame may reuse the previous frame’s parameter set instead of signaling a new one.The parameters are carried in the frame header when newly transmitted.
Loading 2008.06091v2…