Source-linked AI summary
Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques
Narinder Singh Punn, Sanjay Kumar Sonbhadra, Sonali Agarwal, Gaurav Rai
TL;DR
The paper addresses the need to monitor social distancing during COVID-19 when vaccines and effective treatments were unavailable. It proposes a surveillance-video framework combining YOLO v3 detection with Deepsort tracking and reports balanced real-time performance, with violation indices generated for detected groups.
Problem
The paper addresses the need for automated social-distancing monitoring during COVID-19 in the absence of vaccines and effective medicines.
Method
The framework uses YOLO v3 to detect people, Deepsort to track them with bounding boxes and IDs, and vectorized pairwise L2 distances to identify close groups.
Results
YOLO v3 with Deepsort achieved balanced FPS and mAP performance, while the output computed violation indices for detected social groups.
Takeaways & Limitations
The framework automates real-time social-distancing monitoring and records detected violations with timestamps for later analysis.
Abstract
from arXiv · showhide
The rampant coronavirus disease 2019 (COVID-19) has brought global crisis with its deadly spread to more than 180 countries, and about 3,519,901 confirmed cases along with 247,630 deaths globally as on May 4, 2020. The absence of any active therapeutic agents and the lack of immunity against COVID-19 increases the vulnerability of the population. Since there are no vaccines available, social distancing is the only feasible approach to fight against this pandemic. Motivated by this notion, this article proposes a deep learning based framework for automating the task of monitoring social distancing using surveillance video. The proposed framework utilizes the YOLO v3 object detection model to segregate humans from the background and Deepsort approach to track the identified people with the help of bounding boxes and assigned IDs. The results of the YOLO v3 model are further compared with other popular state-of-the-art models, e.g. faster region-based CNN (convolution neural network) and single shot detector (SSD) in terms of mean average precision (mAP), frames per second (FPS) and loss values defined by object classification and localization. Later, the pairwise vectorized L2 norm is computed based on the three-dimensional feature space obtained by using the centroid coordinates and dimensions of the bounding box. The violation index term is proposed to quantize the non adoption of social distancing protocol. From the experimental analysis, it is observed that the YOLO v3 with Deepsort tracking scheme displayed best results with balanced mAP and FPS score to monitor the social distancing in real-time.
I. INTRODUCTION
The introduction frames social distancing as a necessary response to COVID-19 and motivates automated surveillance-based monitoring. The paper proposes comparing object detection and tracking schemes to support this task.
- Motivation: COVID-19 created an urgent need for alternatives to vaccines and medicines, with social distancing presented as the primary available containment approach.The paper cites millions of reported cases and deaths worldwide by May 4, 2020.
- Research aim: The paper proposes detecting social distancing among people gathered in public places while minimizing losses to economic activity.
- Motivation: Social distancing aims to reduce physical contact and is prescribed by WHO as maintaining at least 6 feet between people.The paper links reduced infectious contact to lower infection rates and better alignment between epidemic peaks and healthcare capacity.
- Research aim: The study compares popular object detection and tracking schemes for monitoring social distancing.
- Research aim: The proposed framework is deep-learning based and automates social-distancing monitoring from surveillance video.
II. BACKGROUND STUDY AND RELATED WORK
The related-work section reviews epidemiological, surveillance, detection, tracking, and dataset research relevant to automated monitoring of people and social distancing. It emphasizes practical challenges including occlusion, visual variability, and speed–accuracy trade-offs.
- Epidemiological and technology context: Prior COVID-19 studies used epidemic models to examine social-distancing effects and warned that sudden lifting could produce an earlier secondary peak.
- Epidemiological and technology context: Technology-based containment approaches include GPS tracking, but manual intervention is challenging for the workforce and can raise public threats.
- Surveillance challenges: Human detection in surveillance is constrained by low resolution, pose, clothing, lighting, background complexity, and limited machine-vision capabilities.
- Detection and tracking research: Object detection research spans background subtraction, optical flow, CNNs, region-based CNNs, and faster region-based CNNs for generating object boxes.
- Detection and tracking research: Crowd and tracking studies address counting, occlusion, trajectory estimation, and partially occluded people through methods such as homographies and tracklet-based detectors.
- Datasets and evaluation: Fig. 2 surveys object-detection model performance on PASCAL-VOC and MS-COCO, while the literature also highlights unified annotations for multiple vision tasks.
III. OBJECT DETECTION AND TRACKING MODELS
The paper presents object detection models as systems that balance detection speed and accuracy across benchmark datasets. Performance depends on architectural and implementation choices.
- Model comparison: RCNN, fast RCNN, faster RCNN, SSD, and YOLO variants exhibit a speed–accuracy trade-off on PASCAL-VOC and MS-COCO.
- Model comparison: The trade-off depends on backbone architecture, input size, model depth, and software and hardware environment.
- Model components: Feature extractors encode inputs into representations for learning object patterns, while anchor boxes support detection of objects at varying scales.
A. Anchor boxes
Anchor boxes provide predefined spatial templates for detecting multiple objects, with model-specific scales and aspect ratios. Training predicts class membership and box offsets, while NMS resolves overlapping assignments.
- Anchor boxes are overlaid across image locations with varying sizes and aspect ratios to detect multiple objects.
- For image dimensions b × h, boxes use scale p and aspect ratio r, producing dimensions bp√r × hp√r.Here p ∈ (0, 1] and r > 0.
- Models predict each anchor’s class and an offset that adjusts it toward the ground-truth object box.
- Non-max suppression uses intersection over union to limit multiple anchor associations with the same object.
- Training labels anchors positive or negative and combines regression and classification losses across predictions.
B. Faster RCNN
Faster RCNN replaces external selective-search proposals with a convolutional Region Proposal Network and combines it with a fast RCNN detector. Its multitask objective includes classification and bounding-box regression losses.
- Faster RCNN uses a Region Proposal Network to generate object proposals instead of external selective search.The RPN uses CNN models such as VGGNet and ResNet for localization.
- The architecture combines two modules: the RPN and the fast RCNN detector.
- Loss Function: The overall multitask loss consists of classification loss and bounding-box regression loss.The component functions are defined separately in the paper’s equations.
- Loss Function: The regression formulation uses a piecewise function with a quadratic term for |q| < 1 and a linear term otherwise.
- Loss Function: The formulation distinguishes predicted box corrections, true and predicted classes, and actual classes.
C. Single Shot Detector (SSD)
SSD detects people in real-time surveillance through a single feed-forward process using multiscale features and default boxes. It predicts fixed-size boxes and class scores, then applies NMS to produce final detections.
- SSD is used to identify people in real-time video surveillance as an alternative to faster RCNN.
- SSD uses multiscale features and default boxes in a single process to improve processing speed and accuracy.The paper contrasts this with faster RCNN’s region-proposal approach and slower FPS.
- The feed-forward network generates fixed-size bounding boxes with scores indicating object-class presence.
- SSD extracts feature maps, applies cascaded convolutional filters across multiscale layers, and removes overlaps through NMS.
- Its loss is the sum of multiclass classification loss and bounding-box localization loss.
- Default boxes are matched to ground-truth boxes when IoU exceeds 0.5.
D. YOLO
YOLO formulates object detection as regression, allowing one convolutional network to predict object classes and locations in a single image pass. YOLO v3 uses localization, classification, and confidence losses.
- YOLO predicts an object’s type and location by examining the image once and treating detection as regression.
- A single convolutional network simultaneously predicts multiple bounding boxes and class probabilities.
- YOLO has three major versions: v1, v2, and v3.YOLO v1 uses a GoogleNet-inspired architecture, while YOLO v2 targets improved accuracy and speed.
- Loss Function: YOLO v3’s overall loss combines localization loss, cross-entropy classification loss, and confidence loss.
- Loss Function: The loss weights coordinate error through λcoord and accounts for grids, bounding boxes, and object presence indicators.
E. Deepsort
Deepsort tracks detected individuals by combining motion prediction, appearance information, and data-association procedures to maintain identities across video frames.
- Tracking mechanism: Deepsort combines learned object patterns with temporal information to predict trajectories and assign unique identifiers to tracked individuals.The method is intended to support tracking under occlusion, multiple viewpoints, non-stationary cameras, and training-data annotation.
- Data association: The Kalman filter predicts future positions and recursively supports association, while the Hungarian algorithm performs association and ID attribution.An object is matched across frames when its current detection is identified as the same target as in the previous frame.
- State representation: The tracked object state uses bounding-box centroid, aspect ratio, height, and their respective velocities.The bounding coordinates are treated as direct observations in a constant-velocity Kalman-filter model.
- Track management: Tracks are removed when they exceed a predefined maximum age without successful measurement association.The frame counter increments after positive prediction and resets when the track is associated with a measurement.
- Association metrics: Mahalanobis-distance gating excludes unlikely associations, while appearance-based cosine distance provides an additional assignment metric.The two metrics are combined through a weighted sum, with the influence of each controlled by hyperparameter λ.
IV. PROPOSED APPROACH
The proposed framework combines YOLO v3 detection with Deepsort tracking to monitor social distancing in surveillance video and identify nearby groups.
- Proposed framework: The framework uses YOLO v3 and Deepsort to detect and track people while balancing processing speed and accuracy.Detected individuals are enclosed by bounding boxes for subsequent analysis.
- Distance analysis: Pairwise L2 norms computed from the detected bounding boxes identify clusters of people who do not follow social-distancing requirements.The representation is described as computationally efficient and vectorized.
- Visualization and quantification: Bounding boxes are color-coded by group membership, and each surveillance frame includes counts of social groups and a violation index.The violation index is defined as the ratio of the number of people to the number of groups, and estimated violations can be computed from it.
A. Workflow
The workflow detects and tracks people, represents each person in a three-dimensional feature space, computes pairwise distances, and forms color-coded proximity groups for social-distancing monitoring.
- Workflow: The workflow fine-tunes an object-detection model, then produces bounding boxes and an ID for each person in surveillance footage.The detection models are fine-tuned for binary person-versus-not-person classification using Inception v2 on an Nvidia GTX 1060 GPU.
- Feature construction: Each detected person is represented by three features: bounding-box centroid coordinates x and y, plus depth d estimated from the camera view.The depth formula uses the bounding-box width w and height h.
- Proximity grouping: The workflow computes pairwise L2 norms over the three-dimensional features and uses the resulting dense matrix to assign nearby neighbors.The closeness threshold is dynamically updated according to spatial location, ranging from 90 to 170 pixels.
- Violation measurement: Individuals satisfying the closeness property are grouped and shown with distinct color coding, with group formation indicating a social-distancing violation.The violation index is defined as vi = np/ng, where np is the number of people in close proximity and ng is the number of groups.
- Reported evaluation materials: The paper presents figures for true and false Person samples, training losses, and sample surveillance outputs, alongside a performance-comparison table.The captions identify these materials as dataset examples, object-detection training losses, framework outputs, and model comparisons.
- Experimental setup: The experimental data include 800 manually filtered true Person samples, divided into training and testing sets in an 8:2 ratio.The testing set also includes surveillance-footage frames from Oxford town center.
VI. OUTPUT
The framework processes surveillance video to identify and track people, group them by spatial closeness, and quantify social-distancing violations. YOLO v3 with Deepsort achieved balanced real-time detection performance, while deployment remains sensitive to camera location and raises accuracy, privacy, and individual-rights concerns.
- Violation index is computed as the ratio of people to social groups, and detected violations are recorded with timestamps for later analysis.Example displayed violation-index values are 3, 2, 2, and 2.33.
- High accuracy and precision are desired because false positives may cause discomfort and panic in observed working environments.
- Deployment raises privacy and individual-rights concerns that may require consent, identity hiding, and transparency about fair use.
- The framework identifies people with bounding boxes, tracks them in real time, and groups individuals according to spatial closeness.Pairwise vectorized computation supports identifying clusters or groups of people.
- YOLO v3, Faster RCNN, and SSD were evaluated, with YOLO v3 showing efficient performance through balanced FPS and mAP scores.
- The approach is highly sensitive to camera spatial location and can be fine-tuned for the corresponding field of view.