PaperlayerView exact revision on arXiv

Source-linked AI summary

Thinking with Cameras: Active Visual Reasoning via Dynamic Viewpoint Control for Surveillance Video Understanding

Xiao Zhang, Wang Zeng, Sheng Jin, Wentao Liu, Chen Qian, Shichao Kan

arXiv:2609.06475v2cs.CV

TL;DR

Fixed surveillance views can miss important evidence when targets are small, occluded, or move out of frame. CamVLM combines active viewpoint control, surveillance datasets, and reinforcement-learned camera policies; experiments report improved performance under passive and dynamic viewpoints.

  • Problem

    Surveillance understanding lacks large-scale domain-specific data and fixed viewpoints can miss evidence when targets are distant, small, occluded, or leave the camera view.

    Source: §1 · pp. 1–2

  • Method

    CamVLM uses iterative viewpoint actions and a reinforcement-learning policy, supported by CCTV-Anomaly and the object-centric CamTrack-53K trajectory dataset.

    Source: §1 · pp. 2–3, §3.1 · p. 4, §3.3 · p. 5

  • Results

    On CCTV-Anomaly, CamVLM achieves the best performance on event classification and video captioning under full-frame observation, and dynamic control improves event classification by 33.8% over the passive baseline.

    Source: §5.2 · p. 8

  • Takeaways & Limitations

    The experiments support active camera-based reasoning as a way to improve fine-grained surveillance understanding by acquiring more informative visual evidence.

    Source: §6 · p. 10

    Current viewpoint trajectories use predefined rules and fixed adjustment intervals, and the framework focuses on single-camera control.

    Source: A LIMITATIONS · p. 15

Abstract

from arXiv · show

Large vision-language models (LVLMs) have recently achieved remarkable progress in general-purpose video understanding. However, their application to real-world surveillance remains challenging due to the lack of large-scale domain-specific datasets and the limitation of passive observation from fixed viewpoints. In surveillance scenarios, critical visual evidence can be easily missed when targets are distant, small, occluded, or move beyond the current camera view. In this work, we introduce CamVLM, a new framework for Thinking with Cameras, which enables LVLMs to actively acquire visual evidence in real-world surveillance by continuously controlling camera viewpoints. We first construct CCTV-Anomaly, a large-scale surveillance video understanding dataset containing 14,133 videos across 10 anomaly categories, with detailed captions and event annotations. We further formulate viewpoint control as an active visual perception problem and build CamTrack-53K, an object-centric viewpoint trajectory dataset for learning camera actions. Moreover, we propose a reinforcement learning based viewpoint policy optimization framework, which models camera control as a sequential decision-making process and learns long-horizon observation strategies beyond supervised trajectory imitation. Extensive experiments demonstrate that CamVLM achieves state-of-the-art performance under both passive observation and dynamic viewpoint settings, validating the effectiveness of active camera-based reasoning for surveillance video understanding. Our datasets, model, and code will be available at https://github.com/xiaozhang79/CamVLM.

1 INTRODUCTION

Surveillance video understanding needs active viewpoint selection because fixed views can miss small, occluded, or departing objects. CamVLM combines dynamic camera control, reinforcement-learning policy optimization, and two datasets to acquire task-relevant visual evidence.

1 INTRODUCTION

  • Wide surveillance views can leave evidence incomplete when distant or occluded objects move beyond the camera’s current field of view.Understanding therefore requires deciding where to observe next, not only recognizing events in the captured video.Source: §1 · p. 1
  • Fixed-stream video methods can select frames or clips but cannot move the camera to recover evidence outside its current view.The introduction identifies this limitation as especially relevant when surveillance targets are small, partially occluded, or unpredictable.Source: §1 · p. 2
  • CCTV-Anomaly contains 14,133 videos across 10 anomaly categories, while CamTrack-53K provides object-centric viewpoint trajectories for camera-control learning.CCTV-Anomaly includes detailed captions and event-category annotations.Source: §1 · p. 3
  • CamVLM treats viewpoint control as sequential decision-making and uses reinforcement learning to learn long-horizon camera strategies beyond supervised trajectory imitation.The policy jointly optimizes reasoning accuracy and visual evidence acquisition.Source: §1 · pp. 2–3
  • CamVLM actively controls viewpoints to acquire fine-grained visual evidence and is reported to support more reliable surveillance understanding.The framework’s contribution is framed as enabling LVLMs to determine where to observe next.Source: §1 · p. 3

2 RELATED WORK

Prior video-reasoning approaches retrieve evidence from fixed streams, while active perception has been studied mainly in other settings. The section presents CamVLM as addressing these gaps alongside limitations in surveillance datasets.

Thinking with Images and Videos

  • “Thinking with images” and “thinking with videos” methods iteratively retrieve visual evidence, but still operate on fixed streams rather than controlling viewpoints (Fu et al., 2025; Zeng et al., 2026).The section says this leaves them insufficient when surveillance targets are distant, occluded, or leave the current field of view.Source: §2 · p. 3

Surveillance Video Datasets

  • Existing surveillance datasets are described as limited in scale, realism, or annotation richness, while UDVideoQA has limited duration and domain coverage (Vishal et al., 2026).CCTV-Anomaly is presented as offering 14,133 videos with detailed captions and event-level annotations.Source: §2 · p. 3

Active Visual Perception

  • Active-perception research has focused on robotics, object tracking, and embodied agents; LVLM-based surveillance viewpoint control remains largely unexplored (Zhang et al., 2025; Liu et al., 2026; Wang et al., 2025b).CamVLM is positioned as a paradigm for actively controlling viewpoints and learning long-horizon camera policies through reinforcement learning.Source: §2 · p. 3

3 METHOD

CamVLM treats camera movement as part of visual reasoning: it iteratively selects viewpoints, gathers evidence, and produces an answer. The method combines object-centric trajectory supervision with reinforcement learning that evaluates both answer correctness and acquired visual evidence.

3.1 OVERVIEW

  • At each step, the viewpoint policy uses the query and current viewpoint to choose an action; the transformed view adds evidence for the final response.The interaction trajectory records the query, initial viewpoint, successive actions and viewpoints, and final answer.Source: §3.1 · p. 4

3.2 DYNAMIC VIEWPOINT CONTROL

  • At one-second intervals, the policy moves or zooms from an initial central 1/9-frame viewpoint to acquire complementary views when objects are distant, partly visible, or outside view.The central crop is chosen to balance observation coverage against the need for active adjustment.Source: §3.2 · pp. 4–5

3.3 CAMTRACK-53K: VIEWPOINT TRAJECTORY GENERATION

  • CamTrack-53K uses object masks and object-level question-answer annotations from three video sources to generate task-oriented viewpoint trajectories with seven camera actions.The sources are A2D, MeViS, and Refer-YouTube-VOS in VideoRefer-700K; actions include no movement, four directions, and zoom in or out.Source: §3.3 · p. 5
  • Trajectories use object-viewpoint relations: reacquire objects outside the view, pan or zoom for a better view, and hold when an object is well observed.These generated trajectories provide an initial supervised policy prior, which reinforcement learning later refines.Source: §3.3 · p. 5

3.4 TRAINING PIPELINE

  • Mixed supervised fine-tuning trains CCTV-Anomaly event classification and captioning from full frames, and CamTrack-53K action sequences plus final reasoning responses from dynamic viewpoints.When several viewpoint adjustments are needed, the model predicts an ordered sequence of actions.Source: §3.4 · p. 5
  • RL with GRPO evaluates complete camera-control trajectories and final answers using task-correctness and visual-evidence rewards, allowing multiple informative policies rather than exact trajectory imitation.The visual-evidence reward assesses how well viewpoints cover informative target objects throughout the interaction.Source: §3.4 · pp. 5–6

4 CCTV-ANOMALY DATASET

CCTV-Anomaly is a surveillance video dataset designed to support event classification and video captioning. It combines event labels and detailed captions, with additional structured semantic annotations on the test set.

4 CCTV-ANOMALY DATASET

  • 14,133 real-world videos across 10 event categories are annotated with an event category and detailed natural-language caption for classification and captioning.The dataset addresses the stated limitation that existing surveillance anomaly datasets mainly provide coarse event labels.Source: §4 · p. 6
  • Table 1 compares CCTV-Anomaly with existing surveillance anomaly datasets, while Figure 4 shows examples of different anomalies in the dataset.Source: §4 · p. 6, Table 1 · p. 6, Figure 4 · p. 7
  • Videos come from public surveillance datasets and newly collected footage, with LLM-assisted annotations followed by human verification.Test videos also receive structured labels for location, time of day, subjects, activities, and objects.Source: §4 · p. 7
  • The split contains 13,133 training videos and 1,000 test videos, with 100 test videos per event category.The two supported benchmark tasks are event classification and video captioning.Source: §4 · p. 7

5 EXPERIMENTS

Experiments compare passive observation with dynamic viewpoint control across CCTV-Anomaly and UDVideoQA∗. CamVLM performs strongly across both settings, while ablations and qualitative examples examine when viewpoint control helps and where fine-grained reasoning remains difficult.

5.1 EXPERIMENTAL SETUP

  • Evaluation covers CCTV-Anomaly and UDVideoQA∗, using passive observation or a dynamic viewpoint initialized at the central 1/9 region and adjusted over time.For CCTV-Anomaly, the metrics are event classification accuracy and video captioning accuracy; UDVideoQA∗ contains 1,190 QA pairs.Source: §5.1 · pp. 7–8

5.2 EXPERIMENT RESULTS

  • Under full-frame observation, CamVLM achieves the best performance on CCTV-Anomaly event classification and video captioning; dynamic control improves event classification by 33.8% over the passive baseline.The reported comparisons cover both tasks under passive observation and dynamic viewpoints.Source: §5.2 · p. 8, Table 2 · p. 8
  • 15.7% overall improvement over passive observation on UDVideoQA∗ accompanies CamVLM’s best performance across all question types under dynamic viewpoints.The authors report this as generalization to unseen surveillance scenarios.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 ABLATION STUDIES

  • Dynamic viewpoints improve both the baseline and CamVLM on UDVideoQA∗, with larger gains for CamVLM; fixing its inference view to the central 1/9 region causes a significant performance drop.The authors attribute the improvement mainly to effective viewpoint control rather than video-understanding capability learned during training.Source: §5.3 · p. 9, Table 4 · p. 9
  • Across central 1/4 and 1/6 initial viewpoint sizes, CamVLM outperforms the corresponding passive and dynamic baselines.With smaller views that may initially miss the target, SFT enables zooming out to search and then precise tracking.Source: §5.3 · p. 9, Table 5 · p. 9
  • SFT alone performs strongly, RL directly on Qwen3-VL-8B-Instruct brings substantial gains, and their combination performs best.The authors describe SFT as a strong initialization for subsequent RL refinement.Source: §5.3 · p. 9, Table 6 · p. 9

5.4 QUALITATIVE ANALYSIS

  • CamVLM captures complete action sequences in simple interactions but may miss subtle motion differences or critical events in challenging videos.One example describes running as walking and overlooks a fall.Source: §5.4 · p. 9, Figure 5 · p. 9

6 CONCLUSION

The conclusion presents CamVLM as an active visual reasoning framework built around CCTV-Anomaly, CamTrack-53K, and sequential decision-making for viewpoint control. It reports improved fine-grained surveillance understanding through more informative visual evidence, while noting privacy risks and providing reproducibility resources.

6 CONCLUSION

  • CamVLM combines dynamic camera control, CCTV-Anomaly and CamTrack-53K, and sequential decision-making to learn camera policies for active evidence acquisition.The conclusion reports that experiments show active camera-based reasoning improves fine-grained surveillance understanding.Source: §6 · p. 10
  • The work focuses on event-level understanding and active visual perception rather than personal identification, while acknowledging privacy and misuse risks in surveillance data and models.The authors encourage use in accordance with applicable data licenses, privacy requirements, and ethical guidelines.Source: ETHICS STATEMENT · p. 10
  • The paper describes its architecture, training pipeline, viewpoint control, datasets, evaluation protocols, implementation settings, and prompts, and says the datasets, models, and code will be released.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS

The limitations concern the virtual-to-physical control gap and restricted trajectory and camera settings. Physical-system validation remains future work, while predefined adjustment rules limit adaptability in complex scenarios.

A LIMITATIONS

  • Camera control is simulated in a virtual panoramic frame space; translating predicted actions to physical commands requires camera calibration and hardware specifications, and direct physical-system validation remains future work.Source: A LIMITATIONS · p. 15
  • Predefined trajectory rules and fixed adjustment intervals limit adaptability in complex scenarios, and the framework focuses on single-camera control.Future work includes adaptive trajectories, dynamic viewpoint scheduling, and multi-camera coordination.Source: A LIMITATIONS · p. 15

B EVALUATION DETAILS ON UDVIDEOQA∗

Because the original UDVideoQA test split is unavailable, the authors create UDVideoQA∗ from two released subsets and evaluate answers with semantic matching and complexity-weighted accuracy. They also specify deterministic single-run evaluation.

B EVALUATION DETAILS ON UDVIDEOQA∗

  • UDVideoQA∗ uses Sets 03 and 20, selected for question diversity, and contains 1,190 QA pairs across five question types.Each type contains 238 pairs: Basic Understanding, Attribution, Event Reasoning, Reverse Reasoning, and Counterfactual Inference.Source: B EVALUATION DETAILS ON UDVIDEOQA∗ · p. 15
  • GPT-5.5 semantically scores answers as binary correct or incorrect, with human annotators independently verifying each judgment.This follows the UDVideoQA evaluation protocol.Source: B EVALUATION DETAILS ON UDVIDEOQA∗ · p. 15
  • The weighted overall score gives more weight to cognitively challenging question types, especially Counterfactual Inference, and is reported alongside per-type accuracy.Weights are 1.0 for BU, 1.2 for Atr, 1.3 for ER and RR, and 1.5 for CI.Source: B EVALUATION DETAILS ON UDVIDEOQA∗ · p. 15
  • All evaluations use temperature 0 and greedy decoding, with reported results from one run of the final model checkpoint.Source: B EVALUATION DETAILS ON UDVIDEOQA∗ · p. 15

C DETAILS OF CCTV-ANOMALY CONSTRUCTION

CCTV-Anomaly combines public surveillance anomaly datasets with newly collected real-world videos. The authors filter the data before constructing the dataset.

C.1 VIDEO COLLECTION

  • The dataset combines publicly available surveillance anomaly data with newly collected real-world surveillance videos.Source: §C.1 · p. 15
  • Filtering removes duplicate videos, manually edited clips, severely occluded scenes, and samples with ambiguous event descriptions.Source: §C.1 · p. 15
  • The filtered collection contains 5,832 public videos and 8,301 newly collected videos from varied environments, viewpoints, and event scenarios.The public videos come from XD-Violence, CCTV-Fights, GTA-Crime, and UBnormal.Source: §C.1 · p. 16

C.2 MULTI-LEVEL VIDEO ANNOTATION

CCTV-Anomaly uses an LLM-assisted annotation pipeline with human verification to support event labeling, surveillance-focused captions, and fine-grained evaluation.

C.2 MULTI-LEVEL VIDEO ANNOTATION

  • The annotation pipeline combines event categories, detailed captions, and structured evaluation annotations, using LLM assistance with human verification.Doubao-Seed-2.0-Pro is used for automatic annotation.Source: §C.2 · p. 16
  • Videos receive one category from a predefined taxonomy, with inherited public labels retained when semantically consistent and other labels human-verified.Source: §C.2 · p. 16
  • Captions describe subjects, actions, and outcomes relevant to surveillance reasoning, while 1,000 test videos receive reviewed annotations for location, time of day, subjects, activities, and objects.Captions are generated with LLM assistance and inspected by humans; structured annotations are manually reviewed for consistency and completeness.Source: §C.2 · p. 16

D.1 DATA CURATION

This section describes CamTrack-53K curation, viewpoint representation and construction, representative trajectories, and associated prompt templates. The constructed viewpoints adapt their scale and position to cover relevant objects while retaining context.

D.1 DATA CURATION

  • CamTrack-53K contains 53,059 samples drawn from A2D, MeViS, and Refer-YouTube-VOS subsets of VideoRefer-700K.The respective counts are 26,221, 8,459, and 18,379 samples.Source: §D.1 · p. 16

D.2 VIEWPOINT REPRESENTATION

  • A viewpoint is represented by a normalized rectangular region with top-left and bottom-right coordinates in [0, 1], initialized at the frame center over 1/9 of the frame area.Ablation variants use initial viewpoint area ratios of 1/6 and 1/4.Source: §D.2 · p. 16

D.3 VIEWPOINT CONSTRUCTION

  • For multiple task-relevant objects, their bounding boxes are merged into the minimum enclosing box that defines the target region at each step.Source: §D.3 · p. 17
  • The target viewpoint scale adapts to target-box size, with ρ = 0.2 setting target occupancy, κ = 1.3 adding context padding, and smin = 0.2 and smax = 0.8 bounding scale.Smaller targets receive more compact viewpoints, while larger targets require broader ones.Source: §D.3 · p. 17
  • Target-centered views shift at frame boundaries and remain unchanged when the target is already adequately covered, centered, and scaled.Source: §D.3 · p. 17

D.4 VIEWPOINT TRAJECTORY VISUALIZATION

  • Representative CamTrack-53K trajectories continuously cover moving task-relevant objects while adjusting viewpoint position and extent to object scale and spatial distribution.The trajectories suppress excessive background while preserving surrounding context and interaction cues; Figure 6 uses yellow boxes for objects and blue boxes for current viewpoints.Source: §D.4 · p. 17, Figure 6 · p. 19

E PROMPT TEMPLATE

  • The appendix provides prompt templates for CCTV-Anomaly training and passive and dynamic evaluation, caption evaluation, and UDVideoQA∗ evaluation.The UDVideoQA∗ materials include training and dynamic and passive viewpoint evaluation templates, plus an LLM-based semantic evaluation template.Source: Figure 7 · p. 20, Figure 8 · p. 21, Figure 9 · p. 22
Loading 2609.06475v2…