Source-linked AI summary
BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs
Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, Matthias Grundmann
TL;DR
Mobile-camera pipelines need face detection that is substantially faster than standard real-time processing. BlazeFace addresses this with a lightweight mobile-GPU detector using a compact feature extractor, GPU-friendly anchors, and blending-based tie resolution; it reports reduced jitter and improved accuracy, while using separate models for front- and rear-facing cameras.
Problem
Mobile video-processing pipelines require object detection to run as fast as possible before downstream tasks such as segmentation, tracking, or geometry inference.
Method
BlazeFace adapts SSD with a compact feature extractor, a GPU-friendly anchor scheme, and a blending-based alternative to non-maximum suppression for mobile-GPU face detection.
Results
BlazeFace's tie-resolution modification increased face-detection accuracy by 10% and reduced jitter by 40% on frontal-camera data and 30% on rear-facing data.
Takeaways & Limitations
BlazeFace can provide facial regions and keypoints for downstream AR tasks, including rotated, centered, scale-normalized face crops.
Takeaways & Limitations
The paper focuses on mobile phone camera viewfinders and uses separate models for front- and rear-facing cameras because focal lengths and typical object sizes differ.
Abstract
from arXiv · showhide
We present BlazeFace, a lightweight and well-performing face detector tailored for mobile GPU inference. It runs at a speed of 200-1000+ FPS on flagship devices. This super-realtime performance enables it to be applied to any augmented reality pipeline that requires an accurate facial region of interest as an input for task-specific models, such as 2D/3D facial keypoint or geometry estimation, facial features or expression classification, and face region segmentation. Our contributions include a lightweight feature extraction network inspired by, but distinct from MobileNetV1/V2, a GPU-friendly anchor scheme modified from Single Shot MultiBox Detector (SSD), and an improved tie resolution strategy alternative to non-maximum suppression.
1. Introduction
BlazeFace is a mobile-GPU face detection framework adapted from SSD, designed for lightweight inference and improved prediction stability. Its contributions target both inference speed and prediction quality.
- The framework introduces a compact feature extractor related in structure to MobileNetV1/V2 for lightweight object detection.
- Its GPU-friendly anchor scheme modifies SSD to improve GPU utilization and determine prediction granularity.
- A tie-resolution strategy alternative to non-maximum suppression provides stabler, smoother resolution between overlapping predictions.
2. Face detection for AR pipelines
BlazeFace targets face detection in mobile-camera viewfinders as the first stage of augmented-reality processing pipelines. It also predicts facial keypoints to support rotated face crops for downstream models.
- The paper focuses on detecting faces in mobile phone camera viewfinders and builds separate front- and rear-facing models.The models account for different focal lengths and typical captured object sizes.
- BlazeFace predicts six facial keypoint coordinates in addition to axis-aligned face rectangles.The keypoints cover eye centers, ear tragions, mouth center, and nose tip.
- The keypoints allow face rotation estimation and support rotated face rectangles for later task-specific pipeline stages.This alleviates the requirement for substantial translation and rotation invariance in subsequent processing.
3. Model architecture and design
The architecture is tailored to mobile GPU costs through a compact feature extractor, modified anchor layout, and enlarged receptive fields. Its blending-based post-processing improves accuracy and reduces prediction jitter.
- Enlarging the receptive field sizes: BlazeFace bottlenecks swap expansion and projection stages so residual connections operate at expanded channel resolution, with double BlazeBlocks used in higher-abstraction layers.An additional depthwise layer accelerates receptive-field progression.
- Feature extractor: The front-facing feature extractor takes 128×128 RGB input, uses five single and six double BlazeBlocks, reaches 96 channels, and stops at 8×8 spatial resolution.SSD instead reduces spatial resolution to 1×1.
- Anchor scheme: SSD-like anchors provide fixed base boxes whose predicted regression parameters adjust their positions into tight bounding rectangles.
- Model architecture and design: GPU layer-dispatch overhead makes deep low-resolution computation relatively costly on mobile GPUs.In one experiment, only 3.9 ms of 4.9 ms MobileNetV1 inference time was spent in GPU shader computation.
- Anchor scheme: BlazeFace stops feature-map downsampling at 8×8 and replaces lower-resolution anchors with six 8×8 anchors per pixel.The scheme uses only 1:1 aspect-ratio anchors, which was sufficient for face detection because human face aspect ratios have limited variance.
- Post-processing: Non-maximum suppression can cause temporal jitter when overlapping anchors alternate across video frames.
- Post-processing: The replacement blending strategy estimates bounding-box regression parameters as weighted means of overlapping predictions at virtually no additional NMS cost.
4. Experiments
The experiments evaluate BlazeFace on geographically diverse face images, measuring detection performance, inference speed, and regression quality against a MobileNetV2-based detector. They also examine how the smaller model affects regression predictions and overall AR-pipeline quality.
- Evaluation setup: 2K geographically diverse evaluation images were used, with face-area thresholds of 20% for the frontal model and 5% for the rear-facing model.The model was trained on 66K images; evaluation used a private dataset.
- Detection and speed: Table 1 compares average precision at a 0.5 intersection-over-union threshold and mobile GPU inference time against MobileNetV2-SSD.Inference time was evaluated with TensorFlow Lite GPU in 16-bit floating point mode.
- Detection and speed: Table 2 examines GPU inference speed for BlazeFace and MobileNetV2-SSD across multiple flagship mobile devices.The comparison provides a device-level perspective on inference speed.
- Regression quality: The experiments quantify degradation in regression-parameter prediction quality associated with the smaller model.Table 3 reports regression-parameter prediction quality.
- Pipeline quality: A smaller model does not necessarily cause proportional degradation of the complete augmented-reality pipeline.The paper examines this relationship in the following application-focused section.
5. Applications
BlazeFace supplies face boxes and keypoints that can be converted into normalized, rotated crops for downstream facial models. A contour-estimation example shows how refined outputs can also support tracking and detector reruns when needed.
- Pipeline integration: BlazeFace can provide the first stage for facial keypoint, geometry, feature, expression, and segmentation applications.Its output defines a facial crop for subsequent task-specific models.
- Pipeline integration: The six predicted keypoints allow downstream crops to be centered, scale-normalized, and rotated toward a near-zero roll angle.This reduces the translation and rotation invariance required from task-specific models.
- Contour estimation: In the contour-estimation example, BlazeFace outputs a box and six keypoints, while a more complex model refines the face contour on an expanded crop.The refined keypoints yield a finer bounding-box estimate.
- Contour estimation: The refined contour box can be reused for tracking in the next frame without rerunning the face detector.If the contour model detects that the face is absent, BlazeFace is run again on the whole frame.
- Deployment: The described technology drives major AR self-expression applications and AR developer APIs on mobile phones.
Appendix A. Feature extraction network architecture
The appendix identifies the BlazeFace feature-extraction network architecture as a dedicated component of the model.
- Feature extractor: Table 4 presents the architecture of the BlazeFace feature-extraction network.