Source-linked AI summary
STAViS: Spatio-Temporal AudioVisual Saliency Network
Antigoni Tsiami, Petros Koutras, Petros Maragos
TL;DR
Video saliency research has often focused on visual information despite the role of audiovisual stimuli in human attention. STAViS combines spatio-temporal visual and auditory information in one network, and it outperforms visual-only and other state-of-the-art models in most cases across six databases.
Problem
The paper targets video saliency estimation that models bottom-up attention while integrating auditory information, an area the authors describe as relatively neglected compared with visual-only modeling.
Source: §1 · p. 1Method
STAViS is a single multimodal network that localizes sound sources and fuses auditory and spatio-temporal visual saliencies into a final saliency map.
Source: §1 · p. 2Results
Across six databases, STAViS outperforms its visual-only variant and eight state-of-the-art models in the majority of cases.
Source: §1 · p. 2Takeaways & Limitations
Its consistently good performance across varied videos indicates suitability for saliency estimation in-the-wild without prior knowledge of video content.
Source: §1 · p. 2
Abstract
from arXiv · showhide
We introduce STAViS, a spatio-temporal audiovisual saliency network that combines spatio-temporal visual and auditory information in order to efficiently address the problem of saliency estimation in videos. Our approach employs a single network that combines visual saliency and auditory features and learns to appropriately localize sound sources and to fuse the two saliencies in order to obtain a final saliency map. The network has been designed, trained end-to-end, and evaluated on six different databases that contain audiovisual eye-tracking data of a large variety of videos. We compare our method against 8 different state-of-the-art visual saliency models. Evaluation results across databases indicate that our STAViS model outperforms our visual only variant as well as the other state-of-the-art models in the majority of cases. Also, the consistently good performance it achieves for all databases indicates that it is appropriate for estimating saliency "in-the-wild". The code is available at https://github.com/atsiami/STAViS.
1. Introduction
STAViS addresses video saliency estimation by integrating auditory information with spatio-temporal visual saliency. Across six audiovisual eye-tracking databases, it outperforms visual-only and other state-of-the-art models in most cases.
1. Introduction
- Audiovisual stimuli can influence attention and perception differently from isolated auditory or visual stimuli; incongruent and congruent examples can alter or enhance attention, respectively.The introduction connects this motivation to applications including video summarization, robotics, and virtual reality.Source: §1 · p. 1
- The paper addresses the limited use of audio in attention modeling by predicting bottom-up saliency in videos without prior knowledge of their content.The introduction illustrates the motivation with a bell-tolling scene in which the audiovisual map better captures human attention than the visual-only map.Source: §1 · p. 1, Figure 1 · p. 1
- STAViS uses a single multimodal network to localize sound sources and learn how to fuse auditory and visual saliencies into a final map.The approach combines spatio-temporal visual information with auditory features and integrates the modalities at multiple stages.Source: §1 · p. 2
- Across six databases, STAViS outperforms its visual-only variant and eight state-of-the-art models in the majority of cases.The databases include a wide variety of videos, supporting the authors’ claim that the model is suitable for saliency estimation in-the-wild.Source: §1 · p. 2
2. Related Work
Prior work spans extensive visual-saliency methods and a smaller set of audiovisual approaches. Earlier audiovisual saliency systems include application-specific methods and approaches that combine audio and visual cues for fixation prediction.
2. Related Work
- Visual-saliency research includes CNN-based methods, end-to-end metric-optimized networks, GANs, skip connections, LSTMs, and multi-stream or temporal models.These approaches address visual saliency across images and videos, including pathways for optical flow, depth, motion, objectness, or gaze.Source: §2 · p. 2
- Early audiovisual saliency approaches were often application-specific and used traditional signal processing for visual saliency and audio localization.Later work validated fixation predictions with humans or localized sound-generating objects before combining auditory and visual saliency.Source: §2 · p. 2
- Other prior approaches fused audio and video for fixation prediction with signal-processing techniques, while deep audiovisual work addressed tasks such as correspondence learning and event localization.The surveyed tasks also include audiovisual scene analysis and locating image regions that produce sound.Source: §2 · p. 3
3. Spatio-Temporal AudioVisual Saliency Network
STAViS combines spatio-temporal visual features and audio-derived features to localize sound sources and fuse auditory and visual saliency into a final map. Its network uses deep supervision for visual representations, learns audiovisual correspondences, and trains with eye-tracking-based losses.
3. Spatio-Temporal AudioVisual Saliency Network
- The proposed network combines spatio-temporal visual saliency, audio features, sound-source localization, audiovisual fusion, and saliency losses.Its components are trained as a unified architecture.Source: §3 · p. 3
3.1. Spatio-Temporal Visual Network
- DSAM enhances visual feature representations and provides multi-level saliency maps through attention applied across visual-network feature scales.The module uses temporal averaging, spatial convolutions, upsampling, and a spatial softmax to form attention maps.Source: §3.1 · p. 3
3.2. Audio Representation Network
- The audio network processes 16-frame-matched waveforms with a Hanning window and SoundNet-based layers, then temporally max-pools them into a fixed-dimension sequence representation.It handles variable-length audio without resampling for video sampling-rate variation.Source: §3.2 · p. 4
3.3. Sound Source Localization in Videos
- Sound localization matches audio features with spatial visual features, using a bilinear transformation that generalizes cosine similarity and weighted inner products.The method projects both modalities into a shared hidden dimension before producing localization maps.Source: §3.3 · p. 4
3.4. Audiovisual Saliency Estimation
- Audiovisual fusion combines visual-only and audio-related saliency because some attention-attracting video content is unrelated to audio events.The main approach concatenates multimodal feature maps and applies a convolution to produce one saliency map.Source: §3.4 · pp. 4–5
3.5. Saliency Losses
- Training losses compare predicted maps with eye-tracking ground truth using cross-entropy, correlation coefficient, and normalized scanpath saliency measures.The visual stream is trained first; audiovisual training starts from its weights and omits DSAM skip connections.Source: §3.5 · p. 5
3.6. Implementation
- Implementation uses a pretrained 3D ResNet-50 visual backbone and pretrained SoundNet audio layers, training visual-only first and then the audiovisual network.Inputs are 16-frame, 112×112 RGB clips with corresponding audio; testing uses a 16-frame sliding window with step 1.Source: §3.6 · pp. 5–6
4. Experiments
Experiments evaluate STAViS on six audiovisual eye-tracking databases spanning varied video types, using cross-validation where applicable and comparisons with visual saliency models. Across databases, STAViS performs best in most cases, while ablations favor bilinear localization and feature-map concatenation.
4.1. Datasets
- The six datasets are DIEM, AVAD, Coutrot1, Coutrot2, SumMe, and ETMD, covering structured clips, documentaries, Hollywood movies, and unstructured user-made videos.DIEM is mostly documentary-like, where audio and visual information may not correspond to the same event.Source: §4.1 · pp. 6–7
4.2. Experimental Results
- STAViS outperforms state-of-the-art methods in most cases across DIEM, Coutrot1, Coutrot2, AVAD, SumMe, and ETMD, with consistently good performance across varied videos.The tables report results over five evaluation metrics; the discussion describes performance as best for almost all databases and metrics.Source: Table 2 · p. 7, Table 3 · p. 7, §4.2 · p. 8
- For five databases, evaluation uses three-fold cross-validation with non-overlapping videos and averages performance across splits; DIEM uses its standard literature split.The same procedure is used for audiovisual and visual variants to ensure a fair comparison.Source: §4.2 · p. 7
- Bilinear localization L3 performs better than the other localization methods, and concatenating multiple auditory maps with visual maps achieves the best performance in almost all metrics.Late fusion is also competitive; cosine localization L1 performs worst in all cases.Source: Table 1 · p. 6, §4.2 · p. 8
- Sample maps from ETMD, AVAD, DIEM, and SumMe show STAViS predictions closer to ground truth than the displayed comparison models.Figure 6 compares eye-tracking data and ground truth with STAViS, ACLNet, DeepVS, and TASED maps.Source: §4.2 · p. 8, Figure 6 · p. 8
5. Conclusions
STAViS combines spatio-temporal visual saliency with audio representation, sound localization, and fusion to predict audiovisual saliency maps. Evaluation across six databases and five metrics against eight state-of-the-art methods supports the approach for saliency modeling.
5. Conclusions
- STAViS combines an extended visual saliency network and an audio representation network with sound localization and fusion modules to produce an audiovisual saliency map.The components are designed and trained end-to-end.Source: §5 · p. 8
- The approach is evaluated across varied videos in six databases, using five metrics and comparisons with eight state-of-the-art methods.Source: §5 · p. 8
- Results indicate the audiovisual approach is appropriate and efficient for saliency modeling.Source: §5 · p. 8
6. Supplementary material
The supplementary material documents STAViS’s network configurations, training and evaluation splits, database-specific ablation results, and demonstration videos.
6.1. Configuration and parameters of the employed CNN architectures
- The visual and audio subnetworks use 3D ResNet-50 and SoundNet, respectively, and Figure 7 depicts the 3D bottleneck block.In the figure, F is the number of convolutional-filter feature maps and BN denotes batch normalization.Source: §6.1 · p. 11, Figure 7 · p. 11
6.2.1 Data splits
- For five databases, three non-overlapping-video cross-validation splits ensure every video appears once in a test set; DIEM uses its standard split.Final performance is averaged across the three splits, and the same procedure is used for audiovisual and visual variants.Source: §6.2.1 · p. 11, Table 6 · p. 14
6.2.2 Training
- Training first initializes the visual-only network from Kinetics 400, then trains the audiovisual network end-to-end using pretrained visual and Flickr audio paths.Training runs for up to 100 epochs or stops after 10 epochs without loss improvement, usually around 60; a multi-step learning rate is used.Source: §6.2.2 · pp. 11–12
6.3. Ablation study per database
- Audiovisual combinations usually outperform visual-only methods, with especially strong gains in Coutrot2; in SumMe, an audio-only method leads on 3 of 5 metrics.The supplementary tables compile ablation and state-of-the-art results by database.Source: §6.3 · p. 12, Table 7 · p. 14, Table 8 · p. 15
6.4. Video demos of the proposed STAViS Network
- Four supplementary demo videos correspond to figures in the main paper and show the approach on videos from the employed databases.The listed split contents include videos from AVAD, Coutrot1, Coutrot2, and DIEM.Source: §6.4 · pp. 12–13