Source-linked AI summary
Learning-based Video Motion Magnification
Tae-Hyun Oh, Ronnachai Jaroensri, Changil Kim, Mohamed Elgharib, Frédo Durand, William T. Freeman, Wojciech Matusik
TL;DR
Video motion magnification is challenged by noise, blurring, and potentially suboptimal hand-designed filters when revealing very small motions. This paper learns filters with a deep CNN using carefully designed synthetic data and two-frame training, achieving high-quality real-video magnification with fewer artifacts and better noise characteristics. The learned representations also support temporal filtering up to moderate magnification, while higher-magnification temporal filtering remains limited.
Problem
Existing motion-magnification methods can produce noisy, blurry results and rely on hand-designed decomposition filters that may not be optimal.
Method
The paper learns decomposition filters with a deep CNN trained on carefully designed synthetic small-motion data using two-frame inputs and magnified-difference targets.
Results
The learned filters produce high-quality results on real videos with less ringing or edge artifacts and better noise characteristics than prior methods.
Takeaways & Limitations
The learned representations can be combined with linear temporal filters up to moderate magnification, enabling frequency-based motion selection.
Takeaways & Limitations
With temporal filters, performance degrades at high magnification and small motion, causing prominent blurring, color artifacts, or patchy magnification in some real videos.
Abstract
from arXiv · showhide
Video motion magnification techniques allow us to see small motions previously invisible to the naked eyes, such as those of vibrating airplane wings, or swaying buildings under the influence of the wind. Because the motion is small, the magnification results are prone to noise or excessive blurring. The state of the art relies on hand-designed filters to extract representations that may not be optimal. In this paper, we seek to learn the filters directly from examples using deep convolutional neural networks. To make training tractable, we carefully design a synthetic dataset that captures small motion well, and use two-frame input for training. We show that the learned filters achieve high-quality results on real videos, with less ringing artifacts and better noise characteristics than previous methods. While our model is not trained with temporal filters, we found that the temporal filters can be used with our extracted representations up to a moderate magnification, enabling a frequency-based motion selection. Finally, we analyze the learned filters and show that they behave similarly to the derivative filters used in previous works. Our code, trained model, and datasets will be available online.
1 Introduction
The paper learns video motion-magnification filters from examples with deep CNNs and carefully designed synthetic data. Despite simplified two-frame training, it reports high-quality real-video results with fewer artifacts and analyzes learned filters as similar to prior derivative filters.
- Motivation: Small-motion magnification supports applications such as structural-health assessment and vital-sign measurement but is difficult because motion can be indistinguishable from noise.Existing methods can produce noisy outputs and excessive blurring, particularly at large magnification factors.
- Learning-based approach: Hand-designed decomposition filters may not be optimal, motivating direct filter learning from examples with deep convolutional neural networks.
- Learning-based approach: The method uses synthetic training pairs that realistically simulate small motion, with careful pixel interpolation and explicit quantization modeling to preserve subpixel effects.These design choices support training a network intended to generalize to real videos.
- Architecture and training: Training uses two-frame inputs and magnified differences as targets, simplifying temporal modeling while producing better noise performance and fewer edge artifacts.The network comprises spatial decomposition filters, a representation manipulator, and reconstruction filters.
- Architecture and training: The learned representation supports linear temporal filtering up to moderate magnification, enabling motion selection by frequency bands despite not being trained with temporal filters.
- Filter analysis: A linear approximation of the learned nonlinear filters resembles derivative filters underlying prior decomposition methods.
2 Related Work
Prior video motion-magnification methods use either explicit motion tracking or Eulerian representations, commonly with temporal filtering. The paper positions its approach as achieving comparable quality without temporal filtering while also addressing manipulation settings beyond future-frame prediction.
- Motion magnification: Lagrangian methods explicitly extract optical flow and move pixels, whereas Eulerian methods manipulate frame representations without explicit motion tracking.
- Motion magnification: Eulerian motion magnification typically decomposes frames, manipulates the representation, and reconstructs the resulting video.
- Temporal processing: Multi-frame temporal filtering isolates motions of interest and helps prevent noise amplification, but standard filters can degrade with large motions or input drifts.
- Temporal processing: The paper reports comparable quality without temporal filtering, while related work addresses large-motion limitations through affine transformation or second-order-derivative-equivalent temporal processing.
- Deep representations: Unlike many frame-interpolation methods that require retraining when the manipulation factor changes, the paper’s representation can be configured for different factors without retraining.
- Deep representations: Frame-extrapolation methods synthesize future RGB frames and often produce blurry results, whereas this work magnifies motion within an existing video.
3 Learning-based Motion Magnification
The paper learns motion-magnification representations with a convolutional network trained on synthetic two-frame examples. Its encoder, manipulator, and decoder separate shape and texture information, while temporal filtering remains usable for moderate magnification.
- Learning setup: The learning setup simplifies training to two input frames with small displacement and a magnified output target.This avoids fully specifying temporal motion during training.
- Network architecture: The network has an encoder for spatial decomposition, a manipulator for motion amplification, and a decoder for reconstructing magnified frames.The encoder extracts shape and texture representations, while the decoder reconstructs the modified representation.
- Representation design: The encoder separates shape from intensity information to reduce undesired intensity magnification and noise in the final output.The shape representation is manipulated, while the texture representation preserves intensity information.
- Representation manipulation: A nonlinear manipulator improves result quality by reducing edge blurring and noise compared with a linear manipulator.The nonlinear design uses convolution, ReLU, and a residual block around the magnified representation difference.
- Training: The model is trained end-to-end with an L1 reconstruction loss and regularization that separates texture and shape representations.Training adds intensity perturbations and Poisson noise to improve robustness.
- Temporal operation: Although trained only with two frames, the learned shape representation can be combined with pixel-wise temporal filters across video frames.This enables temporal motion selection, but undesired motion and noise become problematic as magnification increases without temporal filtering.
4 Results and Evaluations
The learned network outperforms phase-based and steerable-pyramid methods across qualitative, quantitative, and physical-accuracy evaluations, while temporal filtering improves artifact suppression and enables frequency-selective motion extraction. Its main limitations emerge at high magnification with small motion, where temporal filtering can cause blur, color artifacts, or missed motion.
- Qualitative comparison: The network preserves edges and produces fewer ringing artifacts and less blurring than phase-based and steerable-pyramid methods in qualitative comparisons.These advantages appear near edges, occlusion boundaries, and in two-frame dynamic-mode results.
- Temporal filtering: The learned representation is compatible with temporal filters that correctly select guitar strings vibrating at different frequencies.The evaluation uses 25× magnification and different temporal frequency bands.
- Temporal filtering: Temporal filtering reduces blurring and prevents unwanted motion from being magnified, while the network preserves edges even without temporal filtering.The phase-based method still shows significant ringing artifacts with temporal filtering.
- Quantitative analysis: 0.01 pixels: the network remains effective in subpixel tests and outperforms the phase-based method, including at a 1,000× magnification factor.Its performance is described as more limited by required output motion than by the magnification factor used during training.
- Physical accuracy: The method’s magnified motion is similar in magnitude and phase to the phase-based result and matches accelerometer-derived motion in the hammer sequence.The hammer comparison uses a 10× magnified result without a temporal filter.
- Limitations: At high magnification with small motion, temporal filtering causes prominent blurring and strong color artifacts, and may miss motions that require temporal filtering.In real eye videos, this can produce patchy magnification and limits cases where temporal filtering is essential.
5 Conclusion
The paper concludes that learning motion-magnification filters from carefully designed synthetic data enables strong real-video performance despite simplified two-frame training. The method has fewer edge artifacts and better noise characteristics, while temporal-filter compatibility remains limited at higher magnification.
- The network learns motion-magnification filters directly from data instead of relying on hand-designed filters.
- Two-frame input makes training tractable, while carefully designed synthetic data captures important aspects of small motion.
- The method performs well with fewer edge artifacts and better noise characteristics than state-of-the-art methods.
- The learned representation is compatible with temporal filters and produces good results up to moderate magnification factors.
- Improving temporal-filter compatibility at higher magnification is identified as an important direction for future work.
Supplementary Material: Learning-based Video Motion Magnification
The supplementary material identifies the paper’s authors and their institutional affiliations.
- The listed authors are Frédo Durand, William T. Freeman, and Wojciech Matusik.
- MIT CSAIL in Cambridge, Massachusetts, is affiliation 1.
- HBKU QCRI in Doha and Google Research are affiliations 2 and 3, respectively.
Summary of Contents
The supplementary material adds experiments, parameter setups, dataset details, analyses, applications, and video comparisons omitted from the main paper.
- The supplement contains additional experiment results, descriptions, and parameter setups excluded from the main paper due to space limits.
- The supplementary video compares other methods and baselines and includes self-evaluations and other applications.
A.1 Parameters for Example Videos
The supplement specifies experimental parameters for the main-paper and supplementary-video examples, including magnification factors and temporal filters.
- Table A.1 specifies magnification factors and temporal-filter parameters used in the main-paper experiments.
- For FIR filters, the number of taps equals the number of video frames in each sequence.
- FIR filters are applied in the frequency domain, whereas all other filters are applied in the time domain.
A.2 Additional Experiments
The supplementary material provides dataset examples, additional experiments, parameter summaries, and video resources extending the main paper.
- A.2 Additional Experiments: The supplementary material covers dataset details, additional experiments, parameter setups, and supplementary video content.It includes sections on dataset characteristics, frequency magnification, qualitative and quantitative comparisons, visualization analyses, and applications.
- A.2 Additional Experiments: The supplementary video is provided as an additional resource for qualitative comparisons and analyses.
- A.2 Additional Experiments: Table A.1 lists parameters used for the reported results.
- A.2 Additional Experiments: Figure A.1 illustrates synthetic training examples built from foreground objects and backgrounds.
A.2.1 Detail dataset Information.
The dataset combines MS COCO backgrounds with PASCAL VOC segmentations and includes controlled background motion and blur conditions.
- A.2.1 Detail dataset Information.: Synthetic training data combines MS COCO images with PASCAL VOC segmentation data.The supplementary text also records the applicable dataset license conditions.
- A.2.1 Detail dataset Information.: Real comparison videos come from prior work or from the authors’ own captured data.
A.2.2 Comparison of temporal operations
The learned representation supports both broad-band subtraction modes and, to a moderate extent, multi-frame temporal filtering for frequency-selective magnification.
- A.2.2 Comparison of temporal operations: The network is trained with two frames, while multi-frame linear temporal filtering remains compatible to some extent at test time.
- A.2.2 Comparison of temporal operations: Subtraction modes cover broader frequency ranges than the compared band-pass filters.Consequently, equal magnification factors are not directly comparable across modes because their overall energy increases differ.
- A.2.2 Comparison of temporal operations: Temporal filtering provides frequency-band selectivity and can suppress noise-frequency bands, whereas subtraction has different trade-offs.
- A.2.2 Comparison of temporal operations: The method is compared with phase representations from complex steerable filters and Riesz transformations, including tests using temporal filtering.
- A.2.2 Comparison of temporal operations: On the guitar sequence, the method magnifies three strings with different frequencies, estimating 137.14, 109.10, and 80 Hz versus ideal values of 144, 108, and 80 Hz.
A.2.3 Additional qualitative Comparisons.
Additional analyses compare visual quality, noise behavior, learned representations, and applications without retraining the network for each manipulation.
- A.2.3 Additional qualitative Comparisons.: The supplementary quantitative analysis evaluates real-data noise performance and input-motion ranges using parameter settings summarized in Table A.2.
- A.2.3 Additional qualitative Comparisons.: Under dynamic-mode comparison, only the proposed method shows clear boundaries with properly magnified motion despite large translational motion.The other methods exhibit blurry textures attributed to artifacts.
- A.2.3 Additional qualitative Comparisons.: In extreme noise tests with noise factor 9, the method preserves detail and produces less noise than the phase-based method.
- A.2.5 Additional Network Visualization for Analysis: Texture representations tend to be direction-invariant and color-sensitive, while shape representations tend to be color-invariant and direction-sensitive.
- A.2.5 Additional Network Visualization for Analysis: During training, directional-invariant texture detectors change from 22/32 (68.8%) to 15/32 (46.9%), while color-invariant shape detectors change from 26/64 (40.6%) to 37/64 (57.8%).The authors suggest this learned behavior may support directional steerable filters for shape representation.
- A.2.5 Additional Network Visualization for Analysis: Texture encoder kernel approximations resemble blur kernels or delta functions and differ from shape-representation kernels.
- A.2.5 Additional Network Visualization for Analysis: Increasing magnification scales and spatially propagates compensation activations, while fusion layers synthesize discrepancies rather than explicitly moving pixels.These observations associate the learned representation with an Eulerian rather than Lagrangian motion representation.
- A.2.6 Other applications: The same network is used for view interpolation, frame interpolation, and dynamic cinemagraph effects without retraining or fine-tuning.Magnification-factor control enables intermediate-frame generation and adjustable cinemagraph dynamism.