High-Quality, Low-Delay Music Coding in the Opus Codec

Jean-Marc Valin; Gregory Maxwell; Timothy B. Terriberry; Koen Vos

Source-linked AI summary

High-Quality, Low-Delay Music Coding in the Opus Codec

Jean-Marc Valin, Gregory Maxwell, Timothy B. Terriberry, Koen Vos

arXiv:1602.04845v1Feb 15, 2016cs.MMcs.SD

TL;DR

Real-time audio coding must combine low delay with broad speech and music support while preserving quality. The paper examines Opus’s CELT transform coder and its embedded psychoacoustic design, reporting 5 ms operation and higher music quality than existing non-real-time codecs. It also identifies limitations in the implicit allocation scheme and single-band reduction.

Problem
Opus targets real-time Internet applications that require speech and music support, changing bitrates, and low algorithmic delay, while competing with established codecs.
Method
The paper describes CELT’s psychoacoustically tuned band-energy coding, fine quantization, and artifact-reduction improvements within Opus.
Results
5 ms delay and higher music quality than existing non-real-time codecs, including under cascading, are reported for Opus.
Takeaways & Limitations
Embedding psychoacoustic knowledge in the format minimizes side information and coding-artifact impact while supporting high music quality.
Takeaways & Limitations
The implicit allocation does not cover two theoretical phenomena, and CELT cannot reduce allocation for a single band because the signaling cost is not worthwhile.

Abstract

from arXiv · show

The IETF recently standardized the Opus codec as RFC6716. Opus targets a wide range of real-time Internet applications by combining a linear prediction coder with a transform coder. We describe the transform coder, with particular attention to the psychoacoustic knowledge built into the format. The result out-performs existing audio codecs that do not operate under real-time constraints.

1. INTRODUCTION

Opus combines broad real-time flexibility with a CELT mode tuned for music, targeting low delay while remaining competitive with non-real-time codecs.

1. INTRODUCTION: 5 ms delay and competitive quality against Vorbis and HE/LC AAC distinguish Opus from existing high-delay codecs.The paper reports that Opus operates below AAC-ELD’s 15 ms delay and remains competitive with storage and streaming codecs.
1. INTRODUCTION: Opus supports bitrates from 6 kb/s to 510 kb/s, five bandwidths, 2.5–60 ms frames, speech and music, and mono or stereo.In-band signaling can change these properties dynamically without switching artifacts.
1. INTRODUCTION: The codec combines Skype’s linear-prediction SILK with Xiph.Org’s transform-based CELT technology.The supplied passage introduces these as Opus’s two core technologies.
1. INTRODUCTION: Opus provides CBR, constrained VBR, and true VBR without a bit reservoir imposing additional buffering delay.CBR produces exactly the requested packet size; constrained VBR permits fluctuations up to one packet’s average size.
1. INTRODUCTION: CELT is the Opus mode primarily used for music and incorporates psychoacoustic tuning plus artifact-concealment tools.The paper focuses on CELT’s psychoacoustic bit allocation and quantization and its additional artifact-concealment tools.

2. OVERVIEW OF OPUS

Opus unifies SILK and CELT through selectable operating modes and supports configuration changes designed for interactive use without discontinuities.

2. OVERVIEW OF OPUS: Opus operates in SILK, CELT, or hybrid mode, covering speech, music, and high-bitrate speech use cases.Hybrid mode runs SILK and CELT simultaneously for super-wideband and fullband speech.
2. OVERVIEW OF OPUS: CELT uses 48 kHz, while SILK supports 8, 12, or 16 kHz; hybrid mode crosses at 8 kHz.In hybrid mode, SILK operates at 16 kHz and CELT discards frequencies below the 8 kHz Nyquist rate.
2. OVERVIEW OF OPUS: CELT-only operation reduces total look-ahead to 2.5 ms by omitting the encoder’s added 4 ms delay.CELT look-ahead is 2.5 ms; SILK adds 5 ms plus 1.5 ms resampling delay.
2. OVERVIEW OF OPUS: The TOC byte signals mode, frame size, bandwidth, and channel count in-band, while internal framing can pack multiple frames per packet.The supplied passage states that internal framing supports up to 120 ms of combined frame duration.
2. OVERVIEW OF OPUS: Redundant 5 ms CELT frames can bridge discontinuities when switching between modes that use SILK.Two redundant CELT frames, one on each side, support smooth transitions between SILK sampling-rate modes.

3. CONSTRAINED-ENERGY LAPPED TRANSFORM (CELT)

CELT uses MDCT bands aligned with auditory critical bands to preserve spectral energy, while low-overlap windows reduce delay and require transient and tonal-signal mitigation.

3. CONSTRAINED-ENERGY LAPPED TRANSFORM (CELT): CELT is based on the MDCT and exposes its complete processing structure through a block diagram.The supplied text introduces the MDCT basis and identifies later sections as describing the block components.
3. CONSTRAINED-ENERGY LAPPED TRANSFORM (CELT): CELT explicitly codes band energies approximating auditory critical bands to preserve the signal’s spectral envelope.The format embeds psychoacoustic knowledge, reducing artifacts and avoiding some parameter coding.
3. CONSTRAINED-ENERGY LAPPED TRANSFORM (CELT): Fixed 2.5 ms-overlap, flat-top MDCT windows lower algorithmic delay and simplify transient handling compared with full-overlap windows.The overlap is fixed regardless of frame size and uses the Vorbis power-complementary window in its overlapping part.
3. CONSTRAINED-ENERGY LAPPED TRANSFORM (CELT): Low overlap increases spectral leakage, especially for highly tonal signals, so CELT uses pre-emphasis, inverse deemphasis, and a pitch-enhancing filter pair.The supplied passages identify increased leakage as the main drawback of low overlap and describe the mitigation approach.
3. CONSTRAINED-ENERGY LAPPED TRANSFORM (CELT): When detecting a transient, the encoder computes multiple short MDCTs and interleaves their coefficients.For 20 ms frames, the design uses eight full-overlap MDCTs with 5 ms windows and constrains band sizes accordingly.

4. QUANTIZATION AND ENCODING

CELT quantizes band energy, allocates bits implicitly, and encodes symbols through a range coder supplemented by raw bits. Its bitstream organization supports efficient decoding and robustness to corruption in raw-bit regions.

The decoder automatically makes many allocation decisions from the remaining bits, requiring the encoder to determine the final rate early.
CELT uses a range coder for most symbols and packs approximately uniform power-of-two symbols as raw bits at the packet’s opposite end.
Raw-bit packing lets the decoder switch rapidly between coding modes and limits bit-error corruption from desynchronizing the range-coded stream.
Band energy is the most important bitstream information and is quantized with a fixed 6 dB coarse resolution, inter-band prediction, and optional inter-frame prediction.

4.2. Bit Allocation

CELT uses implicit, psychoacoustically informed bit allocation that adapts to signal characteristics while avoiding explicit masking-curve signaling. It supplements the baseline allocation with tilt and band-boost mechanisms, though some phenomena and per-band reductions remain uncovered.

4.2. Bit Allocation: CELT skips bands receiving no or very few bits, reconstructing them through spectral folding and redistributing their spectrum bits to other bands.
4.2. Bit Allocation: Actual per-band usage can differ slightly from allocation, with the difference propagated to later bands so the final rate matches the target.
4.2. Bit Allocation: Implicit allocation approximates the masking curve without signaling and produces nearly constant bandwise signal-to-noise ratio with higher LF than HF resolution.
4.2. Bit Allocation: The implicit allocation does not cover tonality or inter-band masking, so tonal frames receive increased total rates and neighboring-band masking is not modeled.
4.2. Bit Allocation: The allocation tilt changes the slope by up to ±5/64 bit/sample/band, while band boosts target transient leakage and unusually energetic bands.
4.2.2. Band Boost: CELT cannot reduce the allocation of a single band because the signaling cost would not be worthwhile.

4.3. Fine Energy Quantization (Q2)

Fine energy quantization refines each band after coarse allocation, then normalizes band coefficients for pyramid vector quantization. CELT uses tuned offsets, integer rounding, and range-coded codebook indices to fit the available bits.

4.3. Fine Energy Quantization (Q2): Fine energy allocation is derived from a band’s total allocation and degrees of freedom, then rounded to an integer and coded as raw refinement bits.
4.3. Fine Energy Quantization (Q2): Bands with NDoF = 2 receive slightly more fine-energy bits, with an upward bias for the first and second fine bits.
4.3. Fine Energy Quantization (Q2): Unused frame-end bits can add one energy-refinement bit per channel to bands whose allocation was rounded down.
4.4. Pyramid Vector Quantization (Q3): CELT normalizes each band’s MDCT coefficients by unquantized energy, producing a unit vector encoded with a pyramid vector quantizer.
4.4. Pyramid Vector Quantization (Q3): The PVQ codebook is indexed by the pulse count K, and range coding handles non-power-of-two codebook sizes while raw bits encode renormalized least-significant bits.
4.4. Pyramid Vector Quantization (Q3): When a band receives no bits, spectral folding reconstructs its spectrum from lower-frequency coefficients while preserving temporal, tonal, and envelope characteristics.

4.5. Stereo

CELT supports three stereo coupling modes and selects among them according to channel correlation, energy representation, and bitrate efficiency. Mid-side coding preserves stereo width while allocating bits between normalized mid and side signals.

4.5. Stereo: CELT offers dual stereo, mid-side stereo, and intensity stereo, with a coded band index determining where intensity stereo begins.Bands below the index use either mid-side or dual stereo, selected by a frame-level flag.
4.5. Stereo: Mid-side coupling is applied independently per band after normalization, so coding channel energies separately avoids cross-talk even when dual stereo is more efficient.The normalized mid and side signals are derived from the left and right bands before quantization.
4.5. Stereo: Explicitly coding the stereo angle preserves stereo width and reduces stereo unmasking risk by preserving the difference-signal energy.The angle is quantized uniformly, which achieves optimal mean-squared error.
4.5. Stereo: Bits for mid and side are allocated according to the band size, total available bits, and quantized stereo angle, coding the larger signal first.When N = 2, one channel can be represented using a single sign bit because of mid-side orthogonality.

4.6. Band splitting

CELT splits high-bit bands recursively to keep pulse-vector calculations manageable while preserving adaptive energy allocation across sub-vectors.

4.6. Band splitting: Bands requiring more than 32 bits are split recursively, with the stereo-style split using the angle to represent energy distribution between halves.The recursion stops after four levels, corresponding to one-sixteenth of the original band size.

5. PSYCHOACOUSTIC IMPROVEMENTS

CELT combines psychoacoustically motivated tools to reduce artifacts caused by tonal signals, mixed transients, sparse high-frequency coefficients, and low-bitrate dropouts. These tools include pitch enhancement, adaptive time-frequency resolution, spreading rotations, and hole filling.

5. PSYCHOACOUSTIC IMPROVEMENTS: Four psychoacoustically motivated improvements make coding artifacts less audible beyond the quality obtained from CELT’s core algorithms.The improvements target tonal leakage, mixed time-frequency content, tonal noise, and transient dropouts.
5.1. Prefilter and Postfilter: The pitch-enhancing prefilter and matching decoder postfilter mitigate increased leakage and quantization noise from the low-overlap window on tonal signals.The paired filters provide perfect reconstruction without quantization, allowing postfilter use at high bitrates.
5.2. Variable Time-Frequency Resolution: Hadamard-based time-frequency modification adapts each band’s resolution, combining frequency resolution for low-frequency tones with time resolution for high-frequency transients.The transform can increase frequency resolution across short MDCTs or increase time localization in a long MDCT, with more ringing than equivalent short-MDCT basis functions.
5.3. Spreading Rotations: Spreading rotations reduce tonal noise by rotating normalized coefficients before quantization and applying inverse rotations during decoding.The rotations use successive 2D Givens rotations, with spreading controlled by a frame-level constant or disabled entirely.
5.4. Collapse Prevention: For transient low-bitrate holes, the decoder detects bands with no pulses and fills them with pseudo-random noise at the minimum band-energy level.Applying spreading separately to short MDCTs avoids pre-echo in transient frames.

6. EVALUATION AND RESULTS

The evaluation measures CELT quality on fullband stereo music at 64 kb/s VBR and compares it with HE-AAC and Vorbis. Opus is rated better than the other tested codecs with greater than 99.9% confidence, and it also performs better under repeated encoding.

6.1. Subjective Quality: Opus is better than the other tested codecs with greater than 99.9% confidence in the 64 kb/s fullband stereo music evaluation.The study used 13 listeners, 30 samples, headphones, and the ITU-R BS.1116-1 methodology; Apple AAC-LC at 48 kb/s was the low anchor.
6.1. Subjective Quality: Apple’s HE-AAC is better than Nero’s HE-AAC and Vorbis with greater than 99.9% confidence, while Nero’s HE-AAC and Vorbis are statistically tied.A simple ANOVA produces the same ordering and tie.
6.2. Cascading Quality: Under cascaded encoding, Opus performs better than MP3 and Vorbis, with 64 kb/s Opus outperforming 128 kb/s MP3.The comparison evaluates quality across bitrate and the number of repeated encodings using PQevalAudio and the PEAQ basic model.

7. CONCLUSION AND FUTURE WORK

Opus builds psychoacoustic knowledge into its format to reduce side information and coding artifacts, achieving higher music quality than existing non-real-time codecs even under cascading. The conclusion also points to continuing encoder improvements, including look-ahead and automatic frame-size switching for non-real-time encoding.

Opus achieves higher music quality than existing non-real-time codecs, even under cascading.The paper attributes this result to psychoacoustic knowledge built into the format, which reduces side information and coding artifacts.
Psychoacoustic knowledge minimizes transmitted side information and the impact of coding artifacts.
Future encoder work includes experimenting with look-ahead and automatic frame-size switching for non-real-time encoding.
Figure 12 examines quality degradation as cascading count and bitrate change.The left panel uses 128 kb/s while varying the number of cascadings; the right uses five cascadings while varying bitrate.