PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniEye: Efficient Multimodal Forensic Video Intelligence for Law-Enforcement Body-Worn Cameras

Mamadou K. Keita, Angela Srbinovska, Anita Srbinovska, Nishka Desai, Isabella Zicari, P. Kwaku Sanaah-Faried, Sanjay Charitesh Makam, Wyatt Auten, Vivek Senthil, Hannah Desnick, Jonathan Bateman, Adrian Martin, Christopher Homan, John McCluskey, Ernest Fokoué

arXiv:2609.09460v1cs.ET

TL;DR

Body-worn camera footage is too extensive for review teams to watch, while human review and report searches are slow or incomplete and cloud processing is constrained by privacy and custody obligations. OmniEye processes footage locally with joint multimodal perception, searchable records, and question-focused re-perception before citation. It agrees with annotator A more than annotator B does, but its categories are unsafe as a filter, so it uses them to order work and confirms candidates through re-perception.

  • Problem

    Review teams cannot watch the volume of body-camera footage, and current human review or keyword searches of reports are slow or incomplete at scale.

    Source: §1 · p. 1

  • Method

    OmniEye runs on agency-controlled hardware, perceives footage windows, preserves their records in a tamper-evident structure, and requires question-focused re-perception before an agent may cite a window.

    Source: §1 · p. 1, §2 · p. 2, §4 · p. 6

  • Results

    OmniEye agrees with annotator A more than B does (κ = 0.290 versus 0.171), but category filtering would lose 147 of A’s 278 worth-reviewing windows.

    Source: §3.1 · pp. 5–6

  • Takeaways & Limitations

    Because the category is unsafe as a filter, OmniEye uses it to order work and confirms candidates by re-perception.

    Source: §3.1 · p. 6

    The authors disclose no data details beyond the sampling procedure and reported results, citing strict collaboration disclosure guidelines.

    Source: §3 · p. 5

Abstract

from arXiv · show

We introduce OmniEye, a multimodal video intelligence system for law-enforcement training and review (source code available on request to verified law-enforcement and public-safety agencies). OmniEye ingests body-worn camera footage and perceives every 30-second window jointly across video and audio with one multimodal foundation model. It then stores the model's structured output in an embedded SQLite database with BM25 full-text search. Officers can question the footage through an agent that writes structured queries, retrieves candidate windows, and re-perceives them with the model before it may cite them. The whole system runs on one 16 GB GPU with a 4-bit quantization-aware-trained model, and it also scales to full bf16 precision on a multi-GPU cluster.

1 Introduction

Body-worn camera footage is produced at a scale that overwhelms review teams, while human review and report searches are slow or incomplete. OmniEye combines local multimodal perception with searchable storage and question-answering designed for agency-controlled hardware.

Deployment constraints

  • Agency-controlled deployment is required because footage contains personally identifiable information and may be barred from third-party infrastructure; processing scale also demands hardware-efficient, resumable runs.The system targets workloads of about a thousand image frames plus raw audio per minute across archives of tens of thousands of recordings, from one 16 GB GPU to a shared two-GPU partition under a 64 GB host-memory cap.Source: §1 · p. 1

OmniEye's workflow

  • OmniEye indexes every recording window, then retrieves and re-perceives candidates for question-focused answers with citations; officers can also correct fields non-destructively.The ingest pass supplies recall, while re-perception against raw pixels and audio supplies precision before a window may be cited.Source: §1 · p. 1

Contributions

  • OmniEye combines 4-bit quantization-aware-trained weights, data-parallel replicas, lossless speculative decoding, and adaptive batching with hash-linked records, attributed corrections, and re-perception-grounded answers.Its citation mechanism permits citing a window only after the model has re-run on that window’s pixels and audio in the current turn.Source: §1 · pp. 1–2

Evaluation

  • The evaluation compares system categories and ranking with human annotators on a year-stratified sample of 1,000 recordings drawn from a population above 90,000.The experiment was conducted in collaboration with the Rochester Police Department.Source: §1 · p. 2

2 OmniEye

OmniEye’s design joins local video-and-audio perception, searchable records, integrity controls, and question-answering that re-perceives evidence before citation. Its pipeline and runtime techniques target constrained agency hardware while keeping records portable, auditable, and recoverable.

2 OmniEye

  • The query path retrieves candidate windows from the ingest index, re-perceives them with the reviewer’s question, and cites only windows that pass grounding; corrections append to an attributed audit log.The ingest path commits structured window records to the store, while the correction path lets reviewers edit fields directly.Source: §2 · p. 2

2.1 Perception Pipeline

  • Each recording is split into 30-second windows with 5-second overlap; frames and raw audio are jointly perceived, and categories, scores, and redundant event signals support review.The categories are Boring, Could be interesting, and Must absolutely watch; the score vector lets reviewers threshold by confidence.Source: §2.1 · p. 2

2.2 Perception Engine

  • Local 4-bit quantization-aware inference fits a 12B multimodal model in roughly 6 to 8 GB; speculative decoding yields a 1.48× window speedup, while cluster batching and adaptive controls address GPU and host-memory pressure.Full bf16 is used on cluster GPUs; data-parallel replicas provide the multi-GPU ingest lever, and AIMD controllers adapt to changing resource availability.Source: §2.2 · p. 3

2.3 Storage and Search

  • A workspace stores video, window, document, correction, audit, and report data in one SQLite database; BM25 full-text ranking combines with structured filters, and policy documents are indexed for citation alongside footage.The workspace can be exported as a single archive that resumes unfinished jobs without reprocessing.Source: §2.3 · pp. 3–4

2.4 Integrity and Non-Repudiation

  • Per-video hash chains detect and localize window reordering, insertion, deletion, or modification, while a global attributed log records mutations and corrections remain append-only and replayable.The hash chain binds the source-file hash, raw model output, ingest timestamp, and previous window hash into each window record.Source: §2.4 · p. 4

2.5 Re-Perception-Grounded Agent

  • The agent retrieves candidate windows, re-perceives them with the question in focus, and filters citations and unsupported claims before answering; its re-perceptions, answers, and citations are attributed in the audit log.The re-perception loop is bounded to at most 8 steps and 6 re-perceptions per question.Source: §2.5 · pp. 4–5

3 Evaluation and Discussion

OmniEye was evaluated on a year-stratified sample of recordings, with human annotators providing a comparison despite substantial disagreement. Its anomaly-score ranking surfaced high-priority content above the base rate, while the category labels were treated as an ordering aid rather than a safe filter.

3 Evaluation and Discussion

  • The evaluation sampled 1,000 recordings from a population exceeding 90,000, then processed 993 after seven zero-duration files were skipped; two annotators blindly labelled 500 windows.The 500 windows were drawn from the processed recordings, and annotators used four review-priority levels.Source: §3 · p. 5

3.1 Results and Discussion

  • 0.60 versus a 0.180 base rate: the top 25 windows ranked by anomaly_score contained high-priority content at a 3.3× lift.The ranking outcome is reported for the 500 annotated windows.Source: Table 2 · p. 5, §3.1 · p. 5
  • Annotators agreed on 55.0% of binary labels (κ = 0.171); their different thresholds largely preserved a shared ordering, with A marking 4.7 times as many windows worth reviewing as B.B used high_priority_review once in 500 windows, contributing to the four-category κ of 0.107.Source: §3.1 · p. 5
  • OmniEye agreed more with annotator A than annotator B did (κ = 0.290 versus 0.171), although agreement remained low and one-directional.The video-resampled bootstrap difference was 0.119, with a 95% interval of [0.030, 0.200].Source: §3.1 · p. 5
  • OmniEye’s categories were used to order work, not filter it: skipping Boring would lose 147 of A’s 278 review-worthy windows, including 42 high-priority windows.The agent confirms retrieved candidates by re-perception rather than category alone.Source: §3.1 · p. 6
  • Departmental reviewers’ tactical and procedural perspectives grouped the same recordings differently; on two recordings, all 16 answers to an eight-question review form had timestamped window citations.The disclosed evaluation details are limited to the sampling procedure and reported results under department guidelines.Source: §3 · p. 5, §3.1 · p. 6

4 Conclusion

OmniEye combines open-weights multimodal perception with local storage, search, dialogue, and integrity controls, and its agent re-perceives windows before citing them. In evaluation, anomaly-score ranking surfaced high-priority content at 3.3 times the base rate in the top 25 windows; deployment at Rochester is staged.

4 Conclusion

  • OmniEye runs on agency hardware and binds window records into per-video hash chains, while corrections are attributed and replayable.The agent may cite a window only after re-perceiving it in the same turn.Source: §4 · p. 6
  • 3.3 times the base rate: high-priority content appeared in the top 25 windows ranked by anomaly score, despite disagreement between annotators.The evaluation found no single ground truth because review-worthiness depends on reviewer perspective.Source: §4 · p. 6
  • Deployment at the Rochester Police Department is staged, beginning with archived recordings and reviewer annotation before staff training and workflow integration.The final planned stage is making the tool available to other law-enforcement agencies.Source: §4 · p. 6

B Limitations

The evaluation is constrained by limited annotator access, an informal expert session, and restrictions on permitted models, compute, and footage handling. These limits narrow how broadly its agreement findings and system comparisons can be interpreted.

B Limitations

  • Only two annotators were available, and their κ = 0.171 leaves the reported agreement pattern uncertain beyond this pair.The Must absolutely watch category contains 15 windows; an equally allocated, population-reweighted stratified design would sharpen its estimate.Source: B Limitations · p. 6
  • The expert session covered two recordings with few participants and no formal protocol, so it explains rather than measures.Source: B Limitations · p. 6
  • The study lacks multiple baselines because restrictions on permitted models and available compute constrained comparisons; footage also could not leave agency infrastructure without jurisdictional approval.Source: B Limitations · p. 6

C Annotator Disagreement

Annotators disagreed substantially, while their differing readings of the same recordings suggest that review judgments can depend on whether reviewers attend to tactics or procedure and liability.

C Annotator Disagreement

  • 45.0% binary disagreement and 57.0% four-category disagreement were reported between annotators.Table 3 counts annotator-window mentions by reason category, with shared tagged windows counted twice.Source: Table 3 · p. 6
  • Both annotators assigned unable_to_assess to obstructed, off-target, or severely low-light windows: 34 for one annotator and 23 for the other.Source: C Annotator Disagreement · p. 6
  • Reviewers read two recordings differently by perspective: tactics-focused reviewers saw no commonality, while procedure-and-liability reviewers grouped them as similar traffic stops.Both interpretations were described as right.Source: C Annotator Disagreement · p. 7

D Category Agreement

Category agreement is reported under alternative treatments of unable_to_assess, while the sampling and windowing passages describe population-proportional year allocation and intervals that can overlap.

D Category Agreement

  • κ = 0.171 when unable_to_assess counts as not worth reviewing; κ = 0.174 when 57 such windows are dropped, and κ = 0.191 when they count as worth reviewing.Source: D Category Agreement · p. 7
  • Table 4 compares the two annotators on all 500 windows, while Table 5 compares annotator categories against OmniEye and excludes unable_to_assess windows.Table 5 reports row-normalized values, so row totals differ between annotators.Source: Table 5 · p. 7, Table 4 · p. 7

E Windowing and Sampling

  • Year-stratified sampling allocates each year a share of the sample proportional to its share of the population, but the year allocation cannot be verified from the exported workspace alone.Source: E Windowing and Sampling · p. 7
  • Each recording is divided into indexed windows covering intervals that advance by window length minus overlap, with the final interval capped at recording duration.Source: E Windowing and Sampling · p. 7

F Detection Signals

OmniEye combines detection signals and produces bounded, two-level video narratives. Its perception engine addresses decoding, throughput, preemption, and memory constraints, while search ranking uses BM25.

F Detection Signals

  • The anomaly score captures unusualness beyond the category and drives sorting and spike detection; open-vocabulary cues, structured booleans, and cross-modal tension add filterable signals.Joint perception flags modality disagreement, such as calm imagery paired with shouted audio, which separate detectors may leave unflagged.Source: F Detection Signals · p. 7

G Summarization

  • After window perception, OmniEye groups ten consecutive windows into roughly five-minute segments, then composes a whole-video overview from segment narratives and category distribution.Source: G Summarization · p. 7

H Perception Engine

  • The temporal speculative decoder uses accepted drafted tokens to improve decoding speed, and requires no draft model, extra weights, or additional memory.Its speedup depends on drafting and verification overhead.Source: H Perception Engine · p. 7
  • The batch controller targets B*=8 and changes batches after three clean batches.Source: H Perception Engine · p. 7
  • When chat is waiting and no replica is free, ingestion aborts the current window mid-decode, releases the replica, and redoes that window afterward.Source: H Perception Engine · pp. 7–8
  • A 64 GB host-memory cap can kill naive thousand-video runs; OmniEye reclaims memory and halves the batch above 82% of the limit, then grows it when memory frees.Source: H Perception Engine · p. 8

I Search Ranking

  • BM25 ranks query-window matches using term frequency, document length, and collection frequency, with k1 = 1.2 and b = 0.75.Source: I Search Ranking · p. 8

J Audit Log

OmniEye links agent actions and human changes to an attributed, tamper-evident audit trail. Together with per-video hash chains, this supports verification and a continuous custody record.

J Audit Log

  • Per-video records hash each window’s raw model output, timestamp, index, source-file hash, and previous hash.Source: Figure 2 · p. 8, J Audit Log · p. 8
  • Each global-log entry records the actor, action, and timestamp in canonical form, alongside the acting officer’s identity.Source: J Audit Log · p. 8
  • The chat implementation reflects system constraints; the authors note that an unconstrained setting could use a more stable approach.Source: J Audit Log · p. 8
  • Corrections record the action, changed field, and old and new values; a window can also be exported as an evidence packet with integrity information.Source: J Audit Log · p. 8
  • Hash-linked, attributed records make edits detectable and localizable, helping defend findings against claims of fabrication or quiet rewriting.The combined source-file hash, per-window hashes, and audit trail provide a continuous, verifiable custody record from ingest through human review.Source: J Audit Log · p. 8

K Window Record

The window record stores perception outputs alongside classification, evidence, and integrity fields; videos and segments carry related aggregate records. Temporal speculative decoding reuses output from the preceding window.

K Window Record

  • A window row combines descriptions, transcript, classification, evidence flags, anomaly scores, timestamps, and hash-chain fields; several column names are historical.Here, “cluster” denotes the classification label, not the output of a clustering procedure.Source: K Window Record · p. 8
  • Video and segment records aggregate fields such as category distributions, peak-anomaly windows, critical-window lists, overviews, and training-value labels.Source: K Window Record · p. 8

N Speculative Decoding

  • High token acceptance in speculative decoding comes from five-second overlap between consecutive windows, which share scene details and repeated JSON scaffolding.The first window in each stream uses plain greedy decoding because no previous output is available as a reference.Source: N Speculative Decoding · p. 9

O Detectors

Detectors translate event queries into candidate searches and confirm candidates through focused re-perception before citation. The system also provides diarized transcripts and review-oriented reports.

O Detectors

  • In a supervisory review test on two recordings, all sixteen answers included window-level citations with timestamps.Source: O Detectors · p. 9
  • In a worked firearm query, search returns candidate windows, focused re-perception assigns yes, no, or unsure verdicts, and only re-perceived windows remain citable.The example removes a drafted citation to a window that search surfaced but the model did not re-perceive.Source: O Detectors · p. 10
  • A find-event query combines structured prefilters, lexical terms, and an anomaly fallback, then re-perceives candidates before reporting or citing them.Detector aliases map to high-precision prefilters, high-recall terms, and a preferred modality; unknown queries use a generic detector based on the user’s terms.Source: O Detectors · p. 10

P Transcription

  • Standalone transcription produces speaker-labeled, timestamped, confidence-tagged entries and carries speaker identities across windows; an optional pass checks transcript faithfulness.The stated rules prohibit invented dialogue, mark unclear audio as [INAUDIBLE], and preserve spoken identifiers.Source: P Transcription · p. 10

Q Reports

  • Review views include an incident timeline, safety summary, policy check, redaction worksheet, and reproducible debrief reports grounded in critical windows.The redaction worksheet intentionally over-flags likely personal information for public-records release.Source: Q Reports · p. 10

R Transcription, Analysis, and Operations

OmniEye’s operations combine staged deployment, transcription and review tools, resumable workflows, command-line controls, and verification of video and audit records.

R Transcription, Analysis, and Operations

  • OmniEye is not yet in daily production; rollout begins with archived recordings on department infrastructure and reviewer annotation before later workflow integration.The planned stages continue with staff training to identify training gaps and eventual availability to other agencies.Source: R Transcription, Analysis, and… · p. 10
  • Transcription maintains speaker identity across a recording, while review views include timelines, safety summaries, policy checks, redaction worksheets, and reproducible debrief reports.Source: R Transcription, Analysis, and… · p. 10
  • Ingestion logs and skips bad or oversized files, commits incrementally, and can resume missing work after interruption or move a paused job to another machine.Source: R Transcription, Analysis, and… · p. 10

S Commands

  • Headless commands support ingestion, job management, exports, verification, evidence packets, sampling, and precision selection; commands are safe to rerun.Source: S Commands · p. 11

T Verification

  • Verification checks per-video predecessor hashes and global-log links and content hashes, reporting mismatches by window index or sequence number.Audit entries bind actions to officer identity and, for agent actions, to the re-perceived window set and answer hash.Source: T Verification · p. 11

Related papers

Loading 2609.09460v1…