AiSearch: Interactive Multi-Modal Search with VLMs

• Static automated video analysis cannot adapt to users’ evolving, subjective criteria for open-ended exploratory searches.Source: §1 · p. 2
• AiSearch is a modular multimodal retrieval framework for flexible searches across image and video collections.Source: §1 · p. 2
• Ingestion builds searchable databases from videos and images, while Search handles multimodal user queries.Source: Figure 1 · p. 2

2 System Design

• AiSearch supports image, text, and hybrid queries over large-scale visual collections, integrating instruct and embedding VLMs through a modular architecture.Instruct models generate natural-language responses, while embedding models map multimodal inputs into high-dimensional embedding spaces.Source: §2 · pp. 2–3

2.1 Ingestion Process

• Ingestion samples video frames, computes pre-trained VLM embeddings, and stores frames and embeddings; image collections are processed similarly.Visual and text representations are jointly processed through transformer layers and pooled into fixed-size embeddings. The system uses a generic visual-question prompt that can be updated for different use cases.Source: §2.1 · p. 3
• Textual prompts alongside visual inputs support context-aware, task-adaptive representations, and users can compare VLM retrieval results to identify models suited to their domain.Unlike CLIP, which computes text and image embeddings independently, the VLMs used here combine textual prompts with visual inputs.Source: §2.1 · p. 3

2.2 Search Process

• Search encodes each text, image, or hybrid query with the ingestion VLM, ranks stored items by cosine similarity, and retrieves the top-k using FAISS.Source: §2.2 · p. 3
• Positive feedback combines the original text prompt, when present, with selected results to form an updated query; negative feedback removes selected and highly similar items before replenishing results.Items above a default cosine-similarity threshold of 0.95 to a thumbed-down result are removed.Source: §2.2 · pp. 3–4

3 Benchmarking

• The benchmark evaluates frame-level AUC for anomaly detection in UCF-Crime surveillance videos, using the standard test split of 290 videos across 13 anomaly categories.Anomaly scores contrast mean similarity to 84 abnormal keywords against 58 normal keywords; the pools were generated using GPT-5.Source: §3 · p. 4

3.1 Results

• AUC ranged from 47.46% to 81.54%, with Ops-MM-embedding-v1-7B best on UCF-Crime; other VLMs may perform better on other domains or datasets.Table 1 reports the frame-level AUC benchmark.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 Ablation Study

• Pool Prompting generally outperformed Single Keyword and Multi-Keyword prompting, boosting performance by up to 9.11% (4.53% on average).Ops-MM-embedding-v1-7B led with Pool Prompting at AUC 81.7 ± 0.41; the study also tested randomly sampled pools at 25%, 50%, and 75%.Source: Table 2 · p. 5, §3.2 · pp. 5–6

3.3 Qualitative Results

• Figure 2 illustrates retrieval with text, image, or combined inputs, from broad queries to fine-grained searches.Selecting a retrieved image seeks the video player to a timestamp several frames earlier; the flexible UI may differ slightly in position from the figure.Source: §3.3 · p. 6, Figure 2 · p. 6

Abnormal Keywords Pool

• The abnormal pool includes specific crime and incident categories such as abuse, arson, assault, burglary, robbery, and road accidents.It also includes related terms such as battery, break-in, mugging, and traffic collision.Source: A Keyword Pools for Anomaly Detection · p. 7
• The abnormal pool also contains broad descriptors for anomalies, unlawful acts, danger, disorder, and emergencies.Examples include “irregular,” “suspicious,” “violent,” “incident,” and “catastrophe.”Source: A Keyword Pools for Anomaly Detection · p. 7

Normal Keywords Pool

• The normal pool spans benign interactions and activities alongside descriptors of safety, legality, routine, and order.Examples include friendship, purchase, safe driving, peaceful, authorized, and typical.Source: A Keyword Pools for Anomaly Detection · p. 7