OpenHome: From Perception to Insight in Smart-Home Camera Videos

1 Introduction

• Smart-home cameras increasingly aim to help homeowners understand what happened and whether it matters, while general video benchmarks show progress in broader video understanding.The introduction situates this need in examples such as package theft, approaching wildlife and a waiting visitor.Source: §1 · p. 1
• OPENHOME assesses visible content and changes, spatial, temporal and causal relationships, and subject intent, evidence and appropriate homeowner responses across three separately scored levels.Its 10 tasks are grouped into Perception, Reasoning and Insight.Source: §1 · p. 2, Figure 1 · p. 2
• 2,254 questions on 1,118 real-world clips cover practical conditions including restricted visibility, challenging lighting and incomplete event recordings.The clips include people, vehicles, animals and natural phenomena.Source: §1 · p. 2
• Across 32 evaluated VLMs, counting and temporal ordering remain challenging for the strongest model, and the closed-versus-open-weight gap is largest on Insight.The findings show capability differences that a single aggregate score would obscure.Source: §1 · p. 2

2 Related Work

• General video QA and VLM benchmarks broadened coverage of description, temporal and causal questions, task types and video durations, but predominantly use edited, well-lit, often narrated content.OPENHOME instead targets short, unedited fixed-camera clips with sparse visual signals.Source: §2 · pp. 2–3
• Diagnostic benchmarks isolate perceptual, temporal or causal skills, while OPENHOME applies this diagnostic approach to questions homeowners would actually ask.The cited examples include Perception Test, TempCompass and synthetic-scene CLEVRER.Source: §2 · p. 3
• Unlike surveillance anomaly detection, scripted indoor activity datasets or smart-home anomaly benchmarks, OPENHOME covers visible content, event relationships, intent, visual evidence and homeowner responses.Its response assessment includes cases where no action is warranted.Source: §2 · p. 3

3.1 Design principles

• The design separates a hierarchical capability profile, grounds questions in visual cues across the clip, judges behavior rather than identity or demographics, and treats probes as standalone questions.Homeowner-centred answers reflect what a homeowner can reasonably conclude, including that nothing needs to be done.Source: §3.1 · p. 3
• Questions are intended to rely on visible actions and cues such as movement, gaze, posture and handled objects, without audio.The design also bars answers based on appearance, demographic attributes or identity.Source: §3.1 · p. 3

3.2 Task taxonomy

• The 10 tasks span Perception (P1–P4) for what appears, Reasoning (R1–R3) for causal, spatial and temporal relations, and Insight (I1–I3) for intent and homeowner responses.The taxonomy includes attribute and recognition questions, illustrated with timestamped frames, questions and correct answers.Source: §3.2 · pp. 3–4, Figure 2 · p. 4

3.3 Data collection

• Clips from the Wyze user forum were selected to cover day and night, color and infrared, predominantly outdoor views, partial events, and varied subjects and phenomena.The footage is licensed CC BY-NC 4.0.Source: §3.3 · p. 5

3.4 Question construction and annotation

• The 2,254 questions across 1,118 clips combine model-generated drafts guided by forum descriptions with visual grounding, text-only answerability screening, human verification and review.Near-miss distractors differ from the correct answer in one detail; questions still answerable without video after refinement are discarded.Source: §3.4 · pp. 5–6

3.5 Statistics

• The benchmark totals 5.97 hours across short clips averaging 19.2 seconds, with day and night questions nearly evenly represented and infrared accounting for 22%.Animals predominate in Perception and Reasoning, while Insight centers more on people; indoor cameras contribute fewer than one question in ten.Source: §3.5 · p. 6, Table 2 · p. 7

4 Evaluation Protocol

• Models receive one clip and one four-option question, using frames sampled at 1 fps up to 64 per clip and resized to a 200,000-pixel budget, without audio or metadata.For clips longer than 64 seconds, frames are sampled uniformly across the full clip.Source: §4 · p. 7
• Evaluation uses the minimum available reasoning setting and zero or lowest permitted temperature, so results reflect those specified decoding and reasoning settings.Explicit reasoning is disabled for locally served models when the mode is switchable.Source: §4 · p. 7
• Accuracy is reported for each task, each capability level and all 2,254 questions, with questions scored independently and level and overall scores weighted by question counts.A constant-answer baseline scores 26.9% overall against 25% chance, checking that answers are not concentrated in one option letter.Source: §4 · p. 7

5.1 Models

• The evaluation covers 32 models from the GPT, Gemini, Grok, Gemma, Qwen, MiMo, and Meta Muse families, with complete results across all three capability levels.Table 4 lists model input configurations and evaluation settings.Source: §5.1 · p. 7

5.2 Model performance analysis

• The best closed-source model leads the best open-weight model by 16.3 points on Perception, 22.4 on Reasoning, and 37.4 on Insight.The strongest open-weight model, Muse-Glimmer-30B, ranks ninth overall.Source: §5.2 · pp. 7, 9
• Insight task spreads are 54.4–58.3 points, exceeding every Perception or Reasoning task; Spatial averages 45.5% and Temporal 40.0%.Across models, Spatial ranges from 27.0% to 71.7% and Temporal from 25.4% to 65.7%; the three Insight probes have the widest spreads.Source: §5.2 · p. 9, Figure 4 · p. 10
• A roughly 44× increase in Qwen3.5 model size yields only about 4 percentage points, while successive evaluated Gemini Flash releases improve overall accuracy by 11.4 and 8.1 points.Evaluated MiMo-V2.6 MOPD and reinforcement-learning variants score within one percentage point at both model scales.Source: §5.2 · p. 9
• Four models lead at every capability level, but 14 of 15 open-weight models score lower on Insight than Perception, by 8.4 points on average.The pattern is not universal: GPT-4o, Gemini 2.5 Flash, and Qwen3-VL-8B Instruct rank higher on Insight than on Perception or Reasoning.Source: §5.2 · p. 10

• OPENHOME is a benchmark comprising 2,254 questions on 1,118 real smart-home camera clips, with its stated scope defined by those question and clip counts.Source: §6 · p. 10
• Its 10 tasks span Perception of visible content and changes, Reasoning about spatial, temporal, and causal relationships, and Insight into intent, evidence, and homeowner responses.Source: §6 · p. 10
• The best model achieves 81.2% overall accuracy, while the best closed-source model leads the best open-weight model by 37.4 percentage points on Insight.Counting and temporal ordering remain challenging even for the strongest model.Source: §6 · p. 10

• Table 4 reports full results and settings for OPENHOME, sorted by overall accuracy.It defines Perception, Reasoning, Insight, and Overall scores, with Overall calculated across all 2,254 questions.Source: Table 4 · p. 13
• Table 4 includes model input configurations and settings; API frame inputs and locally served video follow the Section 4 sampling protocol.Unspecified settings use provider or serving-framework defaults.Source: Table 4 · p. 13
• Figure 5 plots capability-level accuracy for all 32 models, using spokes ordered clockwise by overall accuracy and a radial scale from 20% to 90%.Exact scores are reported in Table 4.Source: Figure 5 · p. 14