MatAnyone-E: Sparse Memory Propagation for Efficient High-Resolution Video Matting

1 INTRODUCTION

• High-resolution matting requires detailed recovery of hair, thin structures, and soft transparency alongside temporal consistency, yet rising computational cost limits practical use of high-quality methods.Memory-based methods retrieve information from earlier frames but incur substantial matching and decoding costs as resolution grows.Source: §1 · p. 2
• Dense matching is costly because both current-frame and stored-memory feature sets grow with resolution, while the need for memory updates varies by location and is difficult to identify before matching.The central challenge is preserving memory-based matting quality without exhaustive high-resolution matching and costly decoding.Source: §1 · p. 2
• MatAnyone-E uses an inexpensive low-resolution propagation pass to identify locations for closer inspection and relevant memory candidates for native-resolution matching.It reuses upsampled low-resolution matching results elsewhere, reducing both the number of locations and the amount of memory searched.Source: §1 · p. 2
• A lightweight decoder guided by measured runtime bottlenecks and trained through knowledge distillation reduces the cost of producing high-resolution mattes while aiming to preserve matting quality.The decoder complements selective memory matching, addressing the two main sources of high-resolution inference cost.Source: §1 · p. 2
• At 4K, MatAnyone-E achieves an estimated 7.4× inference speedup and reduces peak GPU memory usage to 7.6% of MatAnyone 2 while maintaining comparable matting quality.The method also provides a stronger balance between matting quality and inference efficiency at high resolutions.Source: §1 · p. 2

2 RELATED WORK

• Efficiency-oriented matting methods use recurrent modeling or selective refinement, while other approaches leverage external guidance, pretrained priors, or generative priors (RVM; SparseMat; VideoMaMa; SAM2Matting).The cited approaches span video matting, segmentation-guided matting, and methods combining transformer attention with sparse convolution.Source: §2 · pp. 2–3
• MatAnyone uses region-adaptive memory fusion, and MatAnyone 2 adds learned quality evaluation, real-world data curation, and reference-frame training; MatAnyone-E targets their high-resolution inference cost.Prior selective matte-refinement methods such as SparseMat and µMatting do not address the underlying temporal memory-matching cost.Source: §2 · p. 3
• Memory-based video object segmentation improves temporal propagation through shared correspondences, multiple memory timescales, object-level queries, memory compression, and selective retrieval (STCN; XMem; Cutie; Efficient-SAM2).MatAnyone-E applies the efficiency perspective to matting by using low-resolution uncertainty to guide native-resolution matching and query-dependent memory selection.Source: §2 · p. 3

3 METHODOLOGY

• MatAnyone-E retains MatAnyone 2’s main architecture but modifies memory propagation and decoding, the components whose costs grow most rapidly with resolution.The decoder is trained with feature distillation from the MatAnyone 2 decoder.Source: §3 · p. 3

3.1 PRELIMINARIES

• Dense memory matching remains costly because every query is compared with the full memory bank, even when top-k affinities limit each readout.The query and memory grids both grow with spatial resolution.Source: §3.1 · p. 4

3.2 SPARSE MEMORY PROPAGATION

• Sparse Memory Propagation uses one low-resolution pass to identify where high-resolution matching is useful and which memory locations are relevant.It matches only the selected query–memory pairs and reuses the upsampled low-resolution readout elsewhere; the approximate total cost is NqNm/s^4 + nqnm.Source: §3.2 · p. 5

3.3 LIGHTWEIGHT DECODER

• The lightweight decoder follows measured stage latency: it uses depthwise-separable blocks at lower resolutions and narrower dense convolutions at higher resolutions.Feature distillation from the original decoder is used to preserve its intermediate representations.Source: §3.3 · pp. 5–6

4.1 QUANTITATIVE EVALUATIONS

• Across three datasets, MAD differs from MatAnyone 2 by at most 0.04 and dtSSD by at most 0.07, while MSE is lower on all three.MSE is 0.25 vs. 0.28 on VideoMatte, 0.45 vs. 0.50 on YouTubeMatte, and 1.85 vs. 2.00 on CRGNN.Source: §4.1 · p. 7
• At 4K, MatAnyone-E achieves an estimated 7.4× speedup and 13.1× memory reduction over MatAnyone 2; its 4K baseline values are extrapolated.MatAnyone-E reaches 8.1 FPS with 6.08 GB peak memory; compilation-enabled inference reaches 13.6 FPS.Source: Table 2 · p. 7, §4.1 · p. 8

4.2 QUALITATIVE EVALUATIONS

• MatAnyone-E recovers cleaner, more complete hair and soft-boundary details and better preserves challenging foreground structures, with the advantage clearer at 4K.Figure 3 highlights thin hair structures and local boundaries, particularly in the 4K example.Source: §4.2 · p. 8, Figure 3 · p. 8
• Low-resolution uncertainty retains the main spatial distribution of dense uncertainty, supporting its use to select queries for high-resolution matching.Selected locations include object boundaries, fine structures, and other uncertain regions.Source: Figure 4 · p. 8, §4.2 · p. 9

4.3 ABLATION STUDIES

• On CRGNN at 1920 × 1080, combining SMP and LD gives 37.59 ms latency, 26.6 FPS, and 1.66 GB peak memory.SMP alone reduces latency from 73.53 ms to 44.18 ms and peak memory from 5.47 GB to 2.78 GB; LD alone yields 66.61 ms and 4.35 GB.Source: §4.3 · p. 9, Table 3 · p. 9
• At a similar HR query budget, uncertainty-guided selection achieves better matting quality than uniform sampling and alpha-boundary selection while maintaining comparable latency.The comparison uses the MatAnyone 2 checkpoint on CRGNN.Source: §4.3 · p. 9, Table 4 · p. 9

5 CONCLUSION

• At 4K, MatAnyone-E preserves MatAnyone 2’s matting quality while achieving an estimated 7.4× inference speedup and 13.1× lower peak GPU memory.The conclusion frames selective high-resolution computation as a practical direction for scaling memory-based video matting.Source: §5 · p. 9
• Sparse Memory Propagation restricts native-resolution matching to selected query–memory pairs and reuses the upsampled low-resolution readout elsewhere.A profiling-guided lightweight decoder further reduces decoding cost through feature distillation.Source: §5 · p. 9

APPENDIX

• The appendix lists additional efficiency analyses, memory-pruning and lightweight-decoder analyses, GPU-architecture results, and qualitative results.It also includes multi-level Sparse Memory Propagation.Source: APPENDIX · p. 12

A MULTI-LEVEL SPARSE MEMORY PROPAGATION

• The multi-level extension repeatedly downsamples until the coarsest memory grid has at most M locations, then refines only queries above an uncertainty threshold.At each finer level, it retains the top ρ fraction of memory candidates ranked by maximum affinity to selected query regions; for the reported 4K experiments, the schedule is 4 → 2 → 1 with M = 4096.Source: A MULTI-LEVEL SPARSE MEMORY PROPAGATION · pp. 12–13

B.1 RUNTIME BOTTLENECKS OF MATANYONE 2

• At 4K, increasing the memory bank from two to five frames raises memory-propagation cost from 314.58 ms to an estimated 779.80 ms and total latency from 445.1 ms to approximately 911 ms.The five-frame result is estimated; the full configuration exceeds GPU memory.Source: §B.1 · p. 13, Table 5 · p. 13
• Memory propagation accounts for 46% of runtime at 1080p and 65% at 1440p, and is identified as MatAnyone 2’s primary high-resolution runtime bottleneck.The decoder is the second-largest high-resolution component, but its cost is largely independent of memory-bank size.Source: §B.1 · p. 13, Table 5 · p. 13

B.2 COMPONENT-WISE EFFICIENCY ANALYSIS

• At 4K, query pruning reduces latency from 909.09 ms to 157.67 ms and peak memory from 79.69 GB to 10.10 GB.Source: §B.2 · p. 13, Table 6 · p. 14
• At 8K, adding memory pruning on top of query pruning reduces latency from 1174.04 ms to 578.00 ms and peak memory from 45.79 GB to 39.87 GB.Source: Table 6 · p. 14, §B.2 · p. 14
• At 4K, the lightweight decoder reduces latency from 143.55 ms to 123.46 ms, peak memory from 10.18 GB to 6.08 GB, and computation from 3.77 to 2.72 TFLOPs per frame; compilation cuts latency further to 73.53 ms.Source: Table 6 · p. 14, §B.2 · p. 14
• Dense memory matching accounts for the main efficiency gains; memory pruning becomes more beneficial at very high resolutions, while lightweight decoding provides complementary savings.Source: §B.2 · p. 14

B.3 RESOLUTION DEPENDENCE OF SPARSE MEMORY PROPAGATION

• Sparse Memory Propagation gains grow with resolution, but can be limited when a low-resolution feature grid is too coarse to locate uncertain regions or relevant memory candidates reliably.Source: §B.3 · pp. 14–15

• Keeping 10% of the memory bank recovers approximately 95% of dense-matching candidates and weights, while preserving the original top-1 match for 98.8% of queries.Source: Table 7 · p. 15, C ANALYSIS OF MEMORY PRUNING · p. 15
• At the default 15% keep ratio, the retained set preserves 98.1% of candidate count and affinity weight, with a 99.6% top-1 hit rate.Source: Table 7 · p. 15, C ANALYSIS OF MEMORY PRUNING · p. 15
• Recall count measures retained dense-matching candidates, recall weight measures retained affinity mass, and Top-1 hit measures queries whose highest-affinity candidate is preserved.Source: C ANALYSIS OF MEMORY PRUNING · p. 15

• Lower-resolution decoder stages have relatively high arithmetic intensity and are compute-bound, whereas final high-resolution stages have lower arithmetic intensity and are memory-bound.Source: D ANALYSIS OF THE LIGHTWEIGHT DECODER · p. 15, Table 8 · p. 15
• For compute-bound lower-resolution decoder stages, the design uses depthwise-separable convolutions to reduce arithmetic cost.Source: D ANALYSIS OF THE LIGHTWEIGHT DECODER · pp. 15–16
• For memory-bound high-resolution stages, the design reduces channel width while retaining dense convolutions to account for increasing memory traffic.Source: D ANALYSIS OF THE LIGHTWEIGHT DECODER · pp. 15–16

• Compilation improves throughput and reduces peak memory across the tested GPU devices.Source: E EFFICIENCY ACROSS GPU ARCHITECTURES · p. 16
• Across the five tested GPUs, MatAnyone-E exhibits consistent scaling trends as input resolution increases.Source: E EFFICIENCY ACROSS GPU ARCHITECTURES · p. 16
• At 4K with compilation, MatAnyone-E reaches 9.0 FPS on an RTX 3090, 19.1 FPS on an RTX 5090, 9.4 FPS on an RTX A6000, 13.6 FPS on an A800-SXM4-80GB, and 28.6 FPS on an H200 NVL.Source: E EFFICIENCY ACROSS GPU ARCHITECTURES · p. 16

Comparison with MatAnyone 2

• At 4K, MatAnyone-E processes native-resolution inputs, while MatAnyone 2 is evaluated at 2K because its full 4K configuration exceeds available GPU memory.The figure reports that MatAnyone 2’s direct upsampling produces blurry details and can cause foreground-background semantic mistakes.Source: Figure 6 · p. 17, F ADDITIONAL QUALITATIVE RESULTS · p. 17
• At 2K, MatAnyone-E achieves detail accuracy comparable to MatAnyone 2, while the accompanying comparison text reports finer hair structures.The figure caption describes the detail accuracy as comparable; the prose highlights finer hair structures.Source: Figure 6 · p. 17, F ADDITIONAL QUALITATIVE RESULTS · p. 17

Additional High-Resolution Results

• Results also cover multi-person 4K scenes and AI-generated high-resolution content, with the paper reporting detailed foreground boundaries and temporally consistent mattes across varied sequences.The multi-person examples use 3840×2160 inputs; the AI-generated sequence is 1440×2144.Source: F ADDITIONAL QUALITATIVE RESULTS · p. 17, Figure 8 · p. 19, Figure 9 · p. 20
• On a challenging real-world 4K sequence, MatAnyone-E preserves thin strands and irregular curly-hair boundaries across frames.The examples show input frames, predicted alpha mattes, and foregrounds composited on green.Source: F ADDITIONAL QUALITATIVE RESULTS · p. 17, Figure 7 · p. 18