Source-linked AI summary
On the Choice of Tensor Estimation for Corner Detection, Optical Flow and Denoising
Freddie Åström, Michael Felsberg
TL;DR
The paper addresses the computational cost of structure-tensor methods used in image processing, especially for real-time applications. It evaluates the gradient energy tensor as an alternative across corner detection, optical flow, and GPU-based image enhancement, reporting higher frame rates without compromising image-quality measures.
Problem
High computation requirements in structure-tensor image-processing methods limit their applicability to real-time applications, including high-resolution colour-image processing.
Method
The paper replaces the structure tensor with the gradient energy tensor and uses it in corner detection, optical flow, and a GPU-based tensor PDE for image enhancement.
Results
The gradient energy tensor significantly improves frames per second over the structure tensor without compromising PSNR and SSIM error values, reaching up to 40% greater computational efficiency for the largest image size.
Takeaways & Limitations
The gradient energy tensor supports efficient GPU implementation of tensor-based image enhancement and enables real-time image denoising within the reported experiments.
Takeaways & Limitations
The optical-flow formulation does not provide state-of-the-art results and is presented as an illustration that the gradient energy tensor is a possible alternative.
Abstract
from arXiv · showhide
Many image processing methods such as corner detection, optical flow and iterative enhancement make use of image tensors. Generally, these tensors are estimated using the structure tensor. In this work we show that the gradient energy tensor can be used as an alternative to the structure tensor in several cases. We apply the gradient energy tensor to common image problem applications such as corner detection, optical flow and image enhancement. Our experimental results suggest that the gradient energy tensor enables real-time tensor-based image enhancement using the graphical processing unit (GPU) and we obtain 40% increase of frame rate without loss of image quality.
1 Introduction
The paper proposes the gradient energy tensor as an alternative to the structure tensor for image-processing applications, emphasizing reduced convolution requirements and GPU suitability. It applies this alternative to corner detection, optical flow, and tensor-based image enhancement, including real-time denoising objectives.
- The structure tensor is widely used in corner detection, optical flow, and tensor-based image denoising, but its computation contributes to high processing requirements.
- The gradient energy tensor is proposed as an alternative to the structure tensor for corner detection, optical flow, and image enhancement.
- Unlike the structure tensor, the gradient energy tensor does not necessarily require post-convolution of its tensor components to form a rank-2 tensor.
- The gradient energy tensor is applied to corner detection and dense optical flow, where the two tensors produce very similar visual results.
- The paper presents a GPU-based tensor PDE for image enhancement, targeting real-time processing of high-resolution colour images.
2 Estimating directional information
The paper compares the commonly used structure tensor with the gradient energy tensor for estimating local image orientation. The gradient energy tensor can form a rank-2 tensor without necessarily requiring post-convolution, while using higher-order derivatives to capture neighbourhood orientation.
- Structure tensor: The structure tensor estimates local orientation from outer products of image gradients followed by neighbourhood averaging.This averaging supports a rank-2 tensor and provides local energy or variance information.
- Gradient energy tensor: The gradient energy tensor uses the Hessian and higher-order derivatives to capture orientation in a neighbourhood.Its formulation is based on Hu and ∇∆u rather than only first-order image gradients.
- Comparison: Unlike the structure tensor, the gradient energy tensor does not necessarily require convolution to form a rank-2 tensor.This reduces the smoothing operations needed to construct the tensor, although the tensor can be more sensitive to noise.
- Eigenstructure: The gradient energy tensor is not necessarily positive semi-definite, but positive eigenvalues can be defined through eigendecomposition when required.The eigenvectors encode local orientation, while the eigenvalues encode directional-change magnitude.
- Illustrations: The tensors produce similar visual responses in the illustrated tensor fields, despite the structure tensor using post-smoothing and the gradient energy tensor not using it.The ellipse sizes in the illustration are scaled for visualization.
- Eigenstructure: Eigendecomposition makes directional change explicit: eigenvectors describe neighbourhood orientation and eigenvalues describe its magnitude.The presented applications primarily require the eigenvalues rather than an explicit eigendecomposition.
3 GET corner detection and optical flow
The paper applies the gradient energy tensor to Good Features to Track corner detection and Lucas–Kanade optical flow. Corner repeatability is similar between tensors, while optical-flow estimates are comparable on most sequences but diverge for Wooden at the finest scale.
- Corner detection: Good Features to Track detects corners by testing whether both tensor eigenvalues exceed a threshold.The experiment uses threshold µ = 0.01 and compares the 128 strongest corners from each tensor.
- Corner detection: The repeatability measure is similar for the structure tensor and gradient energy tensor at 40% overlap across viewpoint angles from 20–60 degrees.The structure tensor uses Gaussian post-smoothing with standard deviation 1, whereas the gradient energy tensor does not use component post-smoothing in this example.
- Optical flow: The gradient energy tensor can replace the structure tensor in the Lucas–Kanade formulation for computing dense motion fields.The comparison solves the normal equations explicitly with a pseudo-inverse across multiple scales.
- Optical flow: The Wooden sequence diverges for the gradient energy tensor at the finest scale, whereas other Middlebury sequences yield comparable final displacement estimates by MSE.Figure 5 compares the MSE after warping one image with the estimated motion field.
- Optical flow: Large image displacements require post-convolution of the positive gradient energy tensor components to capture motion.The tensor eigenvalues are also scaled across four Gaussian-filter scales for the optical-flow experiment.
- Limitation: The optical-flow formulation does not achieve state-of-the-art results, limiting the result to demonstrating the gradient energy tensor as a possible alternative.The authors present further results as an expected direction rather than an established outcome.
4 Iterative tensor-based PDE denoising
The proposed gradient-energy-tensor diffusion scheme replaces the structure tensor in PDE-based denoising and avoids post-convolution of tensor components. GPU implementation preserves comparable image quality while substantially increasing processing speed.
- Motivation: PDE-based image enhancement is computationally expensive because iterative updates and structure-tensor post-convolution dominate computation.The structure-tensor approach uses several convolutions of tensor components, limiting real-time processing of high-resolution colour images.
- Results: 60 fps was achieved for a three-channel 1280×720 image, compared with 30 fps for the structure tensor, with similar PSNR and SSIM values.The implementation uses CUDA and OpenGL on a GTX 670 GPU.
- Proposed filtering scheme: The proposed diffusion scheme uses the gradient energy tensor with positive eigenvalues to control image-structure orientation.The PDE is solved with a forward Euler iterative scheme and finite differences for image derivatives.
- GPU implementation: Three filtering stages are pre-filtering, orientation estimation, and filter update; unlike the structure tensor, the gradient energy tensor requires no tensor-component post-convolution.The structure-tensor implementation requires three full separable convolutions of the image data, while the energy-tensor implementation does not.
- Results: PSNR and SSIM were comparable between methods, although their best values occurred at different iteration numbers because of parameter tuning.After four iterations, the gradient-energy-tensor filter stabilized at 60 fps while the structure-tensor scheme reached 30 fps for the smallest resolution.
- Results: At the largest image size, the gradient energy tensor was up to 40% more computationally efficient, with a marginal PSNR loss.The SSIM difference was less than 10^-3, while PSNR differed by 3.2% in favour of the structure tensor.
5 Conclusion
The paper presents the gradient energy tensor as an alternative to the structure tensor across three image-processing applications. Its lack of post-convolution supports GPU implementation with higher frame rates without compromising PSNR and SSIM error values.
- The gradient energy tensor is evaluated for corner detection, optical flow, and adaptive image enhancement.
- Its absence of post-convolution makes the gradient energy tensor suitable for efficient GPU implementation.
- The gradient energy tensor yields significant improvement in frames per second compared with the structure tensor.
- The higher frame rate is achieved without compromising PSNR and SSIM error values.