PaperlayerView exact revision on arXiv

Source-linked AI summary

A Lightweight Dual-Branch Visual Front-End for Efficient Audio-Visual Target Speaker Extraction

Zhan Jin, Peijun Yang, Juan Liu, Ming Li

arXiv:2610.36834v1eess.AS

TL;DR

AVTSE uses face-related visual cues to extract a target speaker, but visual encoding can remain computationally significant with lightweight separation backbones. This paper splits visual modeling between a compact lip ROI and facial landmarks, then evaluates the front-end in IIANet and SpatialNet. On LRS3-2mix under a unified reduced training protocol, it reports a performance–computation trade-off close to the evaluated lightweight image-based front-end while reducing module-level visual encoding computation.

  • Problem

    As separation backbones become lighter, visual-front-end computation may take a larger share of total computation and limit low-computing-power deployment.

    Source: §1 · pp. 2–3

  • Method

    The front-end combines a compact lip ROI branch for local appearance with a landmark branch for low-dimensional facial motion, integrating both into IIANet and SpatialNet.

    Source: §1 · p. 3, §3.4 · p. 8

  • Results

    Experiments on LRS3-2mix under a unified reduced training protocol show a performance–computation trade-off close to the evaluated lightweight image-based visual front-end while reducing module-level visual encoding computation.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The method is a trade-off design for low-computation AVTSE visual encoding, combining compact local image modeling with low-dimensional geometric modeling.

    Source: §5 · p. 13

    The evaluation lacks multi-seed assessment, and broader generality across separation backbones is not established.

    Source: §4.4 · p. 12, §4.5 · p. 13

Abstract

from arXiv · show

Audio-Visual Target Speaker Extraction (AVTSE) uses face-related visual cues to extract a target speaker from a speech mixture. In many AVTSE systems, visual features are obtained from convolutional image encoders, and their computation may become a non-negligible part of the overall system when lightweight separation backbones are used. This work proposes a dual-branch visual front-end that reorganizes visual information according to its spatial characteristics. A compact lip region of interest is used to model local articulation cues through convolutional processing, while facial landmark coordinates are used to represent global geometric motion with a lightweight temporal convolutional network. The two representations are integrated into IIANet and SpatialNet with limited fusion overhead. Experiments on LRS3-2mix under a unified reduced training protocol show that the proposed method achieves a performance-computation trade-off close to the lightweight image-based baseline. Compared with the ResNet18-based visual encoder, it reduces the module-level visual encoding computation by more than 90% across both evaluated separation backbones, with a moderate performance gap. This comparison excludes face detection and landmark estimation and therefore applies to systems where landmark coordinates are already available from preprocessing. Demos are available at https://anonymous.4open.science/r/Lightweight-Visual-Frontend-E46E.

1 Introduction

AVTSE uses face-related visual cues to extract a target speaker from a mixture, but visual encoding remains a computational concern as separation backbones become lighter. This work reallocates visual modeling between a compact lip ROI and facial landmarks to reduce visual-front-end computation.

1 Introduction

  • AVTSE uses face-related visual cues to help deep-learning networks extract a target speaker’s speech from a mixture.The introduction situates the task in multimodal perception and notes applications including meetings, hearing assistance, and robotic audition.Source: §1 · pp. 1–2
  • Prior AVTSE systems include time-domain separation, multi-task learning, hierarchical fusion, interference identification, and end-to-end multimodal processing.These approaches report separation or generalization benefits, while practical deployment may remain constrained by computation and storage costs.Source: §1 · p. 2
  • As separation backbones become lighter, visual-front-end computation may account for a larger share of total computation and limit low-power deployment.Prior work has also explored lower-computation front-ends, but the introduction says this cost still needs further control.Source: §1 · pp. 2–3
  • The proposed front-end pairs a small lip ROI for local appearance and articulation with landmark sequences for global facial dynamics.The design responds to visualizations suggesting concentrated inner-lip responses alongside more spatially dispersed auxiliary cues; its objective is lower visual-front-end computation, not a higher separation-performance upper bound.Source: §1 · p. 3

2 Related Work

Related work addresses visual-front-end cost with lighter image encoders and alternative facial representations. This work explicitly allocates local lip appearance to image modeling and broader facial motion to landmark geometry.

2 Related Work

  • A typical pretrained lip-reading visual front-end combines a 3D convolutional layer and ResNet18, whose non-negligible computation can weigh more heavily as separation backbones become lighter.The resulting visual-front-end share can constrain deployment on resource-limited platforms.Source: §2 · pp. 3–4
  • BlazeNet64 and Dolphin reduce image-encoding cost through efficient convolution and compact discrete lip-motion tokens, respectively.The passage notes that visual front-ends can still contribute considerably to total computation in lightweight AVTSE systems.Source: §2 · p. 4
  • Unlike approaches relying on a single image-based representation, this work separates local lip appearance modeling from global geometric-motion modeling with landmark coordinates.The passage positions this representation allocation as a way to reduce visual-front-end computation within AVTSE systems.Source: §2 · p. 4

3 Methods

The methods allocate visual processing by spatial scale: image-based modeling handles compact lip motion, while landmarks encode broader facial geometry and motion. The two branches are integrated into IIANet and SpatialNet with branch-specific feature fusion.

3.1 Preliminary Observation and Motivation

  • Attention visualizations show concentrated inner-lip responses alongside more dispersed responses around the outer lip, subnasal region, and jaw.This qualitative observation motivates assigning dense local articulation cues to image modeling and distributed facial motion to geometric coordinates; it does not establish causal importance of individual regions.Source: §3.1 · pp. 4–5, Figure 1 · p. 5

3.2 Lip Motion Modeling Branch

  • The lip branch crops a compact 32 × 32 ROI from a local window centered on the lip-contour landmarks, using random crops for training and center crops for evaluation.The crop is intended to retain main local lip motion while reducing computation over peripheral regions; a wider 32 × 64 crop is also evaluated because 32 × 32 may miss horizontally extended mouth shapes.Source: §3.2 · pp. 5–6
  • The Lip Encoder processes the patch with an initial 3D convolution, a ResNet-style 2D convolutional stack, spatial aggregation, and a temporal projection to 64-channel visual features.Its computation reduction relative to conventional 88 × 88 ROI encoders comes mainly from limiting the input spatial range, rather than changing the basic convolutional modeling principle.Source: §3.2 · p. 6, Figure 2 · p. 6

3.3 Facial Landmark Modeling Branch

  • The landmark branch adds 16 points to the standard 68-point template, yielding 84 points, and processes their framewise coordinates with V-TCN blocks to model facial dynamics.The added points densify sampling near the upper-lip/subnasal and lip-to-jaw regions without implying those areas are causally more important.Source: §3.3 · pp. 6–7
  • Landmark normalization uses training-set statistics computed separately for each landmark and coordinate dimension, which are reused for validation and testing.Horizontal flipping is applied consistently to lip images and landmarks before normalization, including exchanging symmetric landmark indices.Source: §3.3 · pp. 7–8

3.4 Integration with Separation Backbones

  • The front-end replaces IIANet’s visual front-end while retaining its remaining architecture and fusion strategy; for SpatialNet it supplies conditional visual information without modifying the backbone.These backbones are used to examine integration across different separation frameworks under a unified evaluation setting.Source: Figure 3 · p. 7, §3.4 · p. 8

3.5 Visual Feature Fusion

  • For IIANet, the two 64-channel branch features are concatenated and projected back to 64 dimensions; for SpatialNet, they are aligned to 129 channels and concatenated with the complex spectrogram.The fusion adapts the visual features to each backbone’s input while leaving its main separation architecture unchanged.Source: §3.5 · p. 8

4 Experiments

Experiments compare visual front-ends on LRS3-2mix under a unified reduced training protocol. The dual-branch design offers a visual-encoding computation and separation-performance trade-off across IIANet and SpatialNet, while ablations examine the roles of compact lip imagery and landmarks.

4.1 Datasets

  • LRS3-2mix contains 50,000 training, 5,000 validation, and 3,000 evaluation mixtures, formed from two-second speech signals sampled at 16 kHz with SNRs from -5 dB to 5 dB.Videos are 25 FPS at 224 × 224 pixels; preprocessing produces a 96 × 96 mouth patch.Source: §4.1 · p. 9

4.2 Training Configuration

  • A reduced training protocol selects one clean signal per mixture as target and uses schedules 3/10 and 5/15; results are intended for relative visual-front-end comparison, not benchmark refreshing.Methods within each comparison group share training conditions.Source: §4.2 · p. 10

4.3 Evaluation Metrics

  • Evaluation uses SI-SNR improvement, PESQ, and GMACs; the reported computation covers module-level visual encoding and excludes video decoding, face detection, ROI extraction, and landmark estimation.GMACs are profiled with THOP.Source: §4.3 · p. 10

4.4 Experiment Results

  • The proposed method reduces visual encoding computation from 2.4 to 0.3 GMACs while remaining close to BlazeNet64; at tolerance = 5 and early_stop = 15, PESQ matches and SI-SNRi is slightly higher.The small SI-SNRi difference was not evaluated across multiple seeds and is not a statistically verified improvement.Source: §4.4 · pp. 10–11
  • Compared with ResNet18, the proposed method reduces visual encoding computation from 8.0 to 0.3 GMACs under IIANet, with a separation-performance gap.The comparison represents a trade-off rather than an improvement in the upper bound of performance.Source: §4.4 · p. 11
  • Across IIANet and SpatialNet, the proposed front-end uses less module-level visual encoding computation while achieving separation performance comparable to lightweight image-based front-ends under the evaluated setting.The results support a relative comparison under the unified protocol, not a refreshed benchmark claim or broad architectural generality.Source: §4.4 · pp. 10–12

4.5 Ablation Study

  • With a 32 × 32 ROI, adding lmk84 improves SI-SNRi from 14.9 dB to 16.0 dB and PESQ from 2.38 to 2.59, consistent with complementary geometric information at low computational cost.A 32 × 64 ROI-only variant raises SI-SNRi from 14.9 to 15.0 dB but remains below the dual-branch model and doubles image-branch computation.Source: §4.5 · p. 12
  • The lmk84-only model achieves 15.3 dB SI-SNRi and 2.59 PESQ versus 14.9 dB and 2.38 for the 32 × 32 ROI-only model; adding the lip branch raises SI-SNRi to 16.0 dB while PESQ remains 2.59.The ablation uses a limited training protocol without multi-seed evaluation, so its conclusion is restricted to empirical trends in this setting.Source: §4.5 · pp. 12–13

5 Discussion

The discussion frames the dual-branch front-end as a trade-off design that combines compact local image modeling with low-dimensional geometric modeling. Results support this direction on LRS3-2mix, while broader robustness and end-to-end efficiency remain open.

5 Discussion

  • The proposed method separates local lip appearance from global facial geometry, reducing dense image convolution while retaining a performance–computation trade-off.The inner lip uses a compact ROI; landmark coordinates represent broader facial motion.Source: §5 · p. 13
  • The 32 × 32 ROI plus lmk84 combination outperforms either branch alone under the evaluated setting.Fine-grained lip texture and articulation use the small image branch; landmarks compactly represent peripheral geometric motion.Source: §5 · p. 13
  • Experiments with IIANet and SpatialNet indicate compatibility with two backbones, but do not establish broad generality across AVTSE systems.Further evaluation on additional architectures and datasets is needed for a stronger conclusion.Source: §5 · p. 13
  • Reported GMACs cover module-level visual encoding, not the complete video-processing pipeline; newly required landmark estimation belongs in end-to-end cost analysis.Landmark localization quality, noise, occlusion, pose variation, and alignment failures were not explicitly evaluated, and THOP GMACs are not hardware-specific latency.Source: §5 · p. 13
  • The results support this low-computation trade-off only under the unified LRS3-2mix evaluation setting; broader robustness and end-to-end efficiency require further study.The visual front-end can become a larger share of computation when paired with lightweight separation backbones.Source: §5 · pp. 13–14

6 Conclusion

The conclusion presents a dual-branch visual front-end that combines compact lip-ROI appearance modeling with landmark-based facial-motion modeling. LRS3-2mix experiments support a close performance–computation trade-off, with broader robustness and end-to-end efficiency left for future study.

6 Conclusion

  • The front-end combines a compact lip ROI branch for local appearance with a landmark branch for low-dimensional facial motion modeling.It is evaluated on LRS3-2mix under a unified reduced training protocol.Source: §6 · p. 14
  • The ablation results suggest that lip appearance and landmark-based geometric motion provide complementary cues.Source: §6 · p. 14
  • The evaluation does not cover multi-seed training, cross-dataset testing, real-world scenarios, modality-missing conditions, or full end-to-end latency including landmark detection.These aspects remain to be investigated to better assess robustness, generalizability, and deployment efficiency.Source: §6 · p. 14
Loading 2610.36834v1…