Source-linked AI summary
Video-to-Text Pedestrian Monitoring (VTPM): Leveraging Computer Vision and Large Language Models for Privacy-Preserve Pedestrian Activity Monitoring at Intersections
Ahmed S. Abdelrahman, Mohamed Abdel-Aty, Dongdong Wang
TL;DR
Pedestrian video monitoring raises privacy concerns even as intersection safety remains an important concern. The paper proposes Video-to-Text Pedestrian Monitoring (VTPM), and reports reduced memory usage alongside resolution of privacy concerns.
Problem
Intersection monitoring must address pedestrian safety, while video-based systems can fail to preserve privacy.
Source: §1 · pp. 3–4Method
The paper proposes Video-to-Text Pedestrian Monitoring (VTPM) for pedestrian activity monitoring.
Source: §1 · p. 4Results
Up to 253 million percent reduction in memory usage is reported, alongside resolution of privacy concerns.
Source: §5 · p. 17Takeaways & Limitations
The conclusion presents VTPM as a pedestrian activity monitoring and reporting system.
Source: §5 · p. 17The conclusion identifies integration of VTPM with cameras at research organizations as future work.
Source: §5 · p. 17
Abstract
from arXiv · showhide
Computer vision has advanced research methodologies, enhancing system services across various fields. It is a core component in traffic monitoring systems for improving road safety; however, these monitoring systems don't preserve the privacy of pedestrians who appear in the videos, potentially revealing their identities. Addressing this issue, our paper introduces Video-to-Text Pedestrian Monitoring (VTPM), which monitors pedestrian movements at intersections and generates real-time textual reports, including traffic signal and weather information. VTPM uses computer vision models for pedestrian detection and tracking, achieving a latency of 0.05 seconds per video frame. Additionally, it detects crossing violations with 90.2% accuracy by incorporating traffic signal data. The proposed framework is equipped with Phi-3 mini-4k to generate real-time textual reports of pedestrian activity while stating safety concerns like crossing violations, conflicts, and the impact of weather on their behavior with latency of 0.33 seconds. To enhance comprehensive analysis of the generated textual reports, Phi-3 medium is fine-tuned for historical analysis of these generated textual reports. This fine-tuning enables more reliable analysis about the pedestrian safety at intersections, effectively detecting patterns and safety critical events. The proposed VTPM offers a more efficient alternative to video footage by using textual reports reducing memory usage, saving up to 253 million percent, eliminating privacy issues, and enabling comprehensive interactive historical analysis.
1. Introduction
Pedestrian monitoring can support intersection safety, but video surveillance raises privacy concerns because footage can reveal pedestrians’ identities. VTPM shifts monitoring from video to privacy-preserving text reports that can be stored and analyzed historically.
1. Introduction
- The United States recorded 6,516 pedestrian deaths and approximately 54,769 pedestrian injuries, while pedestrian fatalities represented 17% of all traffic fatalities, underscoring the safety stakes.The deaths marked a 3.9% rise from the 6,272 fatalities recorded in 2019.Source: §1 · p. 3
- Video surveillance can extract pedestrian counts, volume, and safety information, but footage may reveal identities and raises privacy concerns.The introduction notes that identifying individuals from footage may expose them to risks such as data breaches and misuse of personal information.Source: §1 · p. 4
- VTPM combines camera-based computer vision with LLM text analysis, converting intersection activity into reports that do not reveal road users’ identities.The approach is presented as preserving computer-vision functionality while shifting analysis from video to text.Source: §1 · pp. 4–5
- Textual records can be stored and analyzed over years while reducing memory requirements for richer analysis at locations of interest.The proposed historical archive consists of textual data rather than retained video footage.Source: §1 · p. 5
2. Related Work
Prior work protects pedestrian privacy through visual obfuscation and local processing, but the reviewed approaches retain limitations in identity protection, computational demand, or deployment scope.
2. Related Work
- Other visual approaches include head inpainting and methods that preserve pose, shape, and background context while anonymizing video.The review also describes Generative Adversarial Privacy as obfuscating video without sensitive labels while enabling visual analysis.Source: §2 · p. 5
- Face blurring may not fully obscure identity because pedestrians can still be recognized from personal appearance, including clothing.The review notes that blurring faces alone may leave identifying information exposed.Source: §2 · p. 5
- Generative Adversarial Privacy obfuscates video without sensitive labels, while a multi-camera tracking approach uses path-image inpainting to maintain anonymity and tracking.The review describes these as separate privacy-preserving visual-analysis approaches.Source: §2 · p. 6
- Avatar-based methods can be computationally expensive, often handle only one person, and may reveal identity when mask allocation is inaccurate.The review notes that allocation errors can expose pedestrians over several video frames.Source: §2 · pp. 5–6
- Edge-processing studies use local computation to reduce latency and transmission risks while applying privacy mechanisms to pedestrian data.One reviewed system uses differential privacy on edge servers; another processes data locally on solar-powered Jetson devices and uses overhead fisheye cameras.Source: §2 · p. 6
- The reviewed edge-camera approach uses YOLO-V4, DetectNet-V2, or Faster-RCNN models optimized for edge deployment.Its overhead fisheye cameras capture images without identifying individual faces.Source: §2 · p. 6
- The review identifies a broader trade-off: privacy methods must protect identifiable information while retaining system functionality, despite edge devices’ limited compute and memory.It also notes that avatar methods can be computationally demanding and blurring may not fully preserve privacy.Source: §2 · p. 6
3. Methodology
The methodology describes VTPM’s pipeline for converting intersection video activity into privacy-preserving real-time text reports and supporting historical pedestrian-safety analysis, combining computer-vision monitoring with signal and weather context and language models. It also explains the selection of Phi-3 medium for historical analysis and the QLoRA fine-tuning of Phi-3 models for traffic-safety analysis.
3. Methodology
- VTPM is designed to preserve road-user privacy, minimize storage requirements, generate real-time reports, and support historical analysis using textual data.The paper presents textual reports as a way to retain computer-vision and traffic-safety functionality without overly complicating video de-identification.Source: §3 · pp. 6–7
3.1 VTPM Pipeline
- The pipeline monitors pedestrian activity and vehicle interactions, generates reports from activity data plus weather, date, time, and location, and supports interactive historical analysis of anomalies and safety issues.The analyzer uses Phi-3 medium with 14 billion parameters on accumulated reports to examine interactions over time and identify long-term trends.Source: §3.1 · pp. 7–8
3.2 Monitor: Pedestrian Activity Monitoring
- RT-DETR detection and ByteTrack tracking extract trajectories for pedestrians and right-turning vehicles, enabling estimates of pedestrian speed and heading.The monitoring setup uses detection and tracking to follow pedestrians and capture their interactions with right-turning vehicles.Source: §3.2 · p. 9
3.2.1 Conflict Analysis using Surrogate Safety Measure
- TTC estimates time remaining before a collision if trajectories continue unchanged; conflicts are serious below 1.5 seconds, slight from 1.5 to 3 seconds, and absent above 3 seconds.The method focuses on TTC to limit computational demands and support real-time monitoring and report generation.Source: §3.2.1 · p. 9
3.2.2 Crossing Violation Detection
- Crossing violations are identified by comparing pedestrian crossing times with signal-phase start and end times imported through FDOT’s ATSPM API.Crossing during the designated pedestrian phase is safe; crossing outside it is classified as a violation.Source: §3.2.2 · p. 10
3.4 Analyzer: Historical Analysis
- Comprehensive historical analysis requires an LLM with a long context window and strong reasoning to process many generated pedestrian-activity reports.Source: §3.4 · p. 11
- Phi-3 medium scores highly on MMLU reasoning relative to most tested LLMs and has the longest tested context window, making it suitable for historical analysis.These capabilities support analyzing extensive databases and extracting insights and patterns.Source: Figure 4 · p. 11, §3.4 · p. 11
3.5 LLM Fine Tuning
- Fine-tuning targets traffic and pedestrian safety: the authors assembled a custom dataset from information and analyses in over 200 intersection-safety studies.Source: §3.5 · p. 12
- Full tuning is computationally intensive for Phi-3; LoRA adapters offer an alternative reported to match or outperform it and prevent forgetting (Hu et al., 2021; Biderman et al., 2024).Source: §3.5 · p. 12
- QLoRA combines LoRA with 4-bit NF4 quantization and double quantization to reduce memory use without sacrificing performance (Dettmers et al., 2024).Source: §3.5 · p. 12
- QLoRA leaves base-model parameters unchanged, updates only adapters, and merges trained adapters back into the base model for deployment.Source: §3.5 · p. 12, Figure 5 · p. 12
4. Results
The intersection monitoring results report real-time processing and identify substantial differences in crossing violations by crosswalk, time, and weather. A human evaluation found strengths in the fine-tuned Analyzer’s analytical summaries and traffic-safety knowledge, while Llama3 70B achieved the highest reasoning score.
4.1 Experiment Results of the Pedestrian Activity Monitoring and Reporting
- Reports revealed numerous violations and a significant pedestrian–right-turning-vehicle incident involving a pedestrian crossing before the signal activated.The Phi-3-medium coding capability generated charts from the reports to highlight safety-critical events.Source: §4.1 · p. 13
- 72.2% of cases at crosswalk B were violations, compared with 18.2% at crosswalk A, despite higher pedestrian traffic at A.The authors attribute B’s high violation rate primarily to an hour of rainfall.Source: §4.1 · p. 13, Figure 7 · p. 14
- 76% of violations occurred at night versus 24% during daytime, and the authors report that rain influenced signal violations.Fig. 8 visualizes comparisons by time of day and weather.Source: §4.1 · p. 13, Figure 8 · p. 14
- Processing took 0.05 seconds per frame (20 FPS), report generation took 0.33 seconds, and violation detection achieved 90.2% accuracy, 91.96% precision, and 93.2% recall.Generating each chart took about 5–7 seconds.Source: §4.1 · p. 14
4.2 Human Evaluation of Analytical Text Generation by LLM
- The analyzer’s textual analysis summarized 15 reports and highlighted safety-critical events associated with pedestrian activity at the intersection.Source: §4.2 · p. 16, Figure 9 · p. 16
- Researchers preferred the fine-tuned Phi-3-medium Analyzer for analytical summarization and traffic-safety knowledge, while Llama3 70B achieved the highest reasoning score.The questionnaire involved 16 traffic-safety researchers and evaluated reports covering 15 hours of activity.Source: §4.2 · pp. 15–16
5. Conclusions and Future Recommendations
The conclusion presents VTPM as a privacy-preserving system that converts video into textual reports for real-time pedestrian monitoring and historical analysis. In the reported results, it processes frames and generates reports quickly, while the analyzed intersection data show elevated violations at night and at crosswalk B.
5. Conclusions and Future Recommendations
- Processing took 0.05 seconds per video frame and report generation took 0.33 seconds in the reported system.The conclusion describes this as real-time analysis and swift report generation.Source: §5 · p. 17
- Violations were especially common at night (76%); crosswalk B had a 72.2% violation rate, compared with 18.2% at crosswalk A.The authors indicate that these results point to a need for targeted safety interventions.Source: §5 · p. 17
- The fine-tuned Phi-3-medium model showed improved performance against other models and facilitated historical analysis extendable to years of saved textual data.The conclusion also reports qualitative evaluation by 16 doctoral transportation researchers.Source: §5 · p. 17
- Textual reports reduce memory usage by up to 253 million percent, address privacy concerns, and support interactive historical analysis.The authors identify transportation departments and research organizations as users of this analysis.Source: §5 · p. 17
- The authors recommend adding cameras at all intersection corners to provide more complete pedestrian coverage and capture more events.They state that broader coverage would allow more detailed analysis of pedestrian behavior and safety.Source: §5 · p. 17