Source-linked AI summary
Faster gaze prediction with dense networks and Fisher pruning
Lucas Theis, Iryna Korshunova, Alykhan Tejani, Ferenc Huszár
TL;DR
Deep gaze-prediction networks improve fixation prediction but remain overparameterized and costly to evaluate. The paper combines knowledge distillation with Fisher pruning and reports much faster architectures, while also addressing how to tune the performance–complexity trade-off.
Problem
Deep networks using pretrained object-recognition representations improve gaze prediction but remain highly overparameterized and slow for applications requiring efficient predictions.
Method
The paper combines knowledge distillation with a gradient-based Fisher pruning method that greedily removes parameters or feature maps according to estimated loss increase and computational cost.
Results
10x speedup is achieved for the same AUC performance as a state-of-the-art network on the CAT2000 dataset.
Takeaways & Limitations
Pruning can drastically reduce saliency-model computational complexity while maintaining similar performance, and the trade-off signal can reduce reliance on costly hyperparameter searches.
Takeaways & Limitations
The alternative automatically tuned trade-off signal may perform worse when many feature maps are pruned, and one proposed tuning option is not explored further.
Abstract
from arXiv · showhide
Predicting human fixations from images has recently seen large improvements by leveraging deep representations which were pretrained for object recognition. However, as we show in this paper, these networks are highly overparameterized for the task of fixation prediction. We first present a simple yet principled greedy pruning method which we call Fisher pruning. Through a combination of knowledge distillation and Fisher pruning, we obtain much more runtime-efficient architectures for saliency prediction, achieving a 10x speedup for the same AUC performance as a state of the art network on the CAT2000 dataset. Speeding up single-image gaze prediction is important for many real-world applications, but it is also a crucial step in the development of video saliency models, where the amount of data to be processed is substantially larger.
1 Introduction
The paper addresses the tension between accurate human-gaze prediction and the computational cost of deep networks. It combines knowledge distillation, principled pruning, and automated trade-off tuning to obtain faster architectures while preserving performance.
- Gaze prediction supports applications including image cropping, video compression, and user-interface optimization, but deep networks are slow to evaluate.
- 10x speedup is achieved at the same generalization performance in terms of AUC through knowledge distillation and pruning.
- The paper proposes automatically tuning a weighted combination of performance and computational-complexity losses to reduce expensive hyperparameter searches.
2 Fast Gaze Prediction Models
The paper develops efficient gaze-prediction models by combining pretrained representations, architectural modifications, knowledge distillation, and principled pruning. Fisher pruning greedily removes parameters or feature maps using estimated loss increases while explicitly accounting for computational cost.
- 2 Fast Gaze Prediction Models: DeepGaze II uses VGG-19 feature maps, a readout network, Gaussian blurring, and a prior to produce fixation probabilities.Feature maps are extracted, upsampled, concatenated, transformed by the readout network, blurred, and combined with the image-dependent prior.
- 2 Fast Gaze Prediction Models: Efficiency modifications apply the readout before upsampling, use separable Gaussian filters, and upsample and crop before softmax.These changes reduce intermediate computation while preserving the required saliency-map dimensions.
- 2 Fast Gaze Prediction Models: VGG-11 and DenseNet-121 provide alternative feature extractors with different efficiency–performance trade-offs.Fine-tuning feature representations can compensate for much of the performance lost by using the smaller VGG-11 network.
- 2 Fast Gaze Prediction Models: Even parameter-efficient pretrained models remain over-parameterized for gaze prediction, motivating greedy removal of redundant parameters or feature maps.The proposed pruning method is called Fisher pruning.
- 2.1 Fisher Pruning: Fisher pruning estimates the loss increase from removing parameters using a second-order approximation and an empirical Fisher-information-based diagonal Hessian.The approximation assumes the current parameters are at a local optimum; when model and data distributions match, it becomes exact under twice differentiability.
- 2.1 Fisher Pruning: The method greedily removes parameters or feature maps with the smallest estimated loss increase, using gradients accumulated over data points as the pruning signal.For convolutional networks, entire feature maps are masked and evaluated because sparse kernels may not yield practical speedups.
- 2.2 Computational Cost: Computational complexity is optimized alongside loss and measured in floating-point operations, including convolutional costs based on spatial extent, channels, and filter size.The pruning objective can therefore target runtime-related cost rather than parameter count alone.
- 2.3 Automatically Tuning β: The proposed β strategy chooses the smallest weight that achieves a desired reduction in features or computational cost after removing features with nonpositive pruning signals.Its equivalent β_i signal favors feature maps with small loss increases relative to computational-cost reductions.
3 Experiments
Experiments show that Fisher pruning, especially when computational cost is regularized, produces efficient saliency architectures that retain strong performance across datasets and pruning levels.
- Fisher pruning: Fisher pruning performs well, while incorporating computational cost improves pruning results over simpler signals.The FLOP-regularized method outperforms unregularized alternatives, and automatically tuning β works best when few features are pruned.
- Model comparison: FastGaze and DenseGaze generally outperform DeepGaze II on log-likelihood, NSS, and SIM when evaluated on CAT2000.For AUC, DeepGaze II slightly exceeds FastGaze, while DenseGaze performs better than DeepGaze II.
- Speed-performance trade-off: 10x speedup is achieved with DenseGaze at the same AUC, while the most heavily pruned model improves log-likelihood with more than 75x speedup.FastGaze runs faster because of its less complex architecture, whereas DenseGaze achieves better AUC performance.
- Speed-performance trade-off: Up to 2x additional FastGaze speedup is obtained at the same AUC by explicitly regularizing computational complexity.The gain depends on the amount of pruning and compares regularized with non-regularized models.
- Architecture analysis: Pruned architectures tend to alternate convolutions with large and small numbers of feature maps while retaining fixation-prediction performance.This structure appears at lower computational complexity despite the need to detect varied object types.
- External benchmark: On MIT300, FastGaze and DenseGaze perform slightly below state of the art but remain comparable to other recent deep saliency models.The models were selected for CAT2000 performance and may have lost information about MIT-dataset subtleties.
4 Conclusion
The conclusion presents gradient-based Fisher pruning combined with knowledge distillation as a practical way to reduce saliency-model complexity while preserving similar performance. The resulting efficiency is relevant to resource-constrained and high-volume applications.
- Conclusion: Fisher pruning requires only gradients and is efficient and easy to implement.The method explicitly optimizes a chosen trade-off between prediction performance and computational complexity.
- Conclusion: Knowledge distillation and reduced complexity enable end-to-end training with good generalization performance.The approach combines distillation with pruning rather than relying on pruning alone.
- Conclusion: A hyperparameter-free pruning signal can account for computational complexity when extensive trade-off tuning is infeasible.The signal can be combined with pruning methods that estimate feature-map or parameter importance.
- Implications: Less resource-intensive models matter for applications processing large amounts of data or running on resource-constrained devices.Faster gaze prediction may also accelerate video-saliency-model development by reducing iteration costs.
- Qualitative evidence: Pruned FastGaze models can reduce runtime substantially while producing visually similar saliency maps.Figure 3 reports runtimes from 1.39s to 91ms, despite a 15x speed difference.
- Qualitative evidence: Heavily pruned models retain responses to faces, people, objects, signs, and text at a 39x speedup compared with DeepGaze II.The maps become somewhat blurrier at this speedup but generally detect the same structures.
S1 Details of Fisher pruning
The Fisher-pruning derivation uses the Hessian diagonal of the cross-entropy loss under stated regularity and convergence assumptions.
- Fisher pruning derivation: Under mild regularity conditions, the cross-entropy Hessian diagonal is derived as the pruning-related curvature quantity.The derivation uses the quotient rule for one step and assumes the model distribution is trained to convergence and close to the target distribution.
S2 Alternative derivation of Fisher pruning
Fisher pruning estimates the output change caused by removing features, using a local second-order approximation whose Hessian is related to Fisher information. This yields a greedy single-feature pruning cost based on diagonal curvature.
- Binary masks determine which activation features remain: mk = 0 removes a feature, while mk = 1 retains it.
- Pruning cost is defined by the change in the model output, measured with KL divergence.
- The KL divergence is locally approximated by the Fisher-Rao quadratic distance around the unpruned model.
- Under mild conditions, the Hessian at the unpruned model equals the Fisher information matrix and can be estimated with empirical Fisher information.
- The diagonal pruning terms use empirical gradient information, with gnk defined as in the main text.