PaperlayerView exact revision on arXiv

Source-linked AI summary

VideoJev: System One Model for Video Understanding and Interaction

Yuhan Zhu, Xiangyu Zeng, Qingyi Si, Chenxu Yang, Yifei Li, Luozheng Qin, Junhao Zhou, Dingyu Yao, Jun Zhang, Chuanyu Qin, Zhiqiu Zhang, Yuandong Yang, Yuqi Xu, Tianyu Zhang, Hongcheng Guo, Jiaqi Wang, Nan Duan, Limin Wang

arXiv:2610.18615v1cs.CV

TL;DR

Video agents need timely decisions over video, but MLLMs typically generate timestamps or answer text even for compact structured choices. VideoJev directly scores intervals and options and reuses visual computation, improving benchmark performance while accelerating inference.

  • Problem

    Video MLLMs typically generate timestamps or answer text even when tasks require compact structured decisions, motivating efficient direct prediction with MLLM semantic flexibility.

    Source: §1 · p. 2

  • Method

    VideoJev unifies temporal grounding and multiple-choice video QA through direct candidate scoring, with Cached and Joint inference reusing visual computation.

    Source: §6 · p. 12

  • Results

    Across seven benchmarks, both model sizes improve over Instruct backbones; grounding gains are 6.7 and 10.8 mean mIoU points, with over 100× QA speedups.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VideoJev offers a foundation for visual agents to compose structured judgments into complex, open-ended workflows.

    Source: §6 · p. 12

    Without calibration-specific training, VideoJev probabilities tend to overestimate correctness, with four-benchmark mean ECEs of 8.65 and 8.13 points for 4B and 35B-A3B.

    Source: §3.5 · p. 11

Abstract

from arXiv · show

Visual agents must turn video observations into timely decisions to track progress and choose what to do next. Building on the emerging System One paradigm, we introduce VideoJev, a family of models that cast video understanding and interaction as structured decision tasks. We adapt pretrained multimodal large language models (MLLMs) by replacing autoregressive answer decoding with a shared head that scores temporal intervals and textual options against video evidence. These scores yield event spans or answer choices, unifying temporal grounding and multiple-choice video question answering. This combines the semantic flexibility of MLLMs with efficient decisions that agents can use directly. As an interaction unfolds, new goals and intermediate results prompt further queries about the same visual history. VideoJev therefore reuses visual computation through Cached inference for queries as they arrive and Joint inference for multiple queries in a single forward pass. Across three temporal grounding benchmarks, VideoJev-4B and VideoJev-35B-A3B improve mean mIoU over their backbones by 6.7 and 10.8 points, respectively, with measured inference speedups of $2.8$-$3.1\times$. On four video QA benchmarks, both achieve performance comparable to their backbones in Thinking mode, with measured inference speedups exceeding $100\times$. These results offer a unified, reusable decision interface for composing structured visual judgments into complex, open-ended workflows.

1 Introduction

VideoJev frames video understanding as structured decisions, replacing autoregressive outputs with shared candidate scoring while reusing visual computation for repeated queries. Across grounding and QA benchmarks, it improves on its backbones and reduces inference time.

1 Introduction

  • VideoJev combines pretrained MLLM semantics with direct interval and option scoring, replacing generated timestamps or answer text with structured decisions.Temporal scores yield event spans, while option scores select answers.Source: §1 · p. 2
  • Cached inference prefills the video once for arriving queries, while Joint inference scores grouped queries in one causal forward pass; training uses the same layout to share computation.Both inference modes use the same backbone and scorer.Source: §1 · p. 2, Figure 2 · p. 3
  • VideoJev-4B gains 6.7 mean grounding mIoU and 8.0 mean QA points over its Instruct backbone; VideoJev-35B-A3B gains 10.8 and 5.7, respectively.Grounding speedups reach 2.8–3.1×; QA remains competitive with Thinking counterparts at over 100× speedups.Source: §1 · p. 2
  • With identical training data and settings, VideoJev exceeds autoregressive fine-tuning by 1.3 mean grounding mIoU and 1.9 mean QA points.The matched-data comparison supports an architectural benefit beyond training data alone.Source: §1 · p. 2
  • The introduction presents dense 4B and MoE 35B-A3B models evaluated across seven benchmarks, including a controlled comparison with autoregressive fine-tuning.The motivation is to support timely video decisions as agents locate evidence, check progress, or assess actions using the same visual history.Source: §1 · p. 2

2 VideoJev

VideoJev uses a shared candidate-scoring head to make temporal grounding and multiple-choice video QA decisions from video evidence. Its causal video prefix supports reuse across queries, while task-specific outputs and training preserve the structure of each task.

2.1 A Unified Video Decision Framework

  • Cached inference reuses one encoded video prefix for arriving queries, while Joint inference scores multiple queries in one forward pass with a separate decision readout for each.Training also groups queries around a shared video prefix; later queries can see preceding query text and candidates, but reference or predicted answers are not appended.Source: §2.1 · p. 4

2.2 Temporal Grounding through Interval Decisions

  • Grounding assigns membership labels to frame-pair intervals, combines local BCE supervision with soft IoU event-coverage supervision, then decodes a contiguous span.Soft IoU is omitted when no interval midpoint is positive, and BCE uses only valid candidates.Source: §2.2 · p. 4

2.3 Video QA through Option Selection

  • Video QA scores textual answer options and uses softmax probabilities with cross-entropy training to select one option.Training permutes eligible options independently of the answer and updates the correct index; questions referring to option positions or earlier turns retain their order.Source: §2.3 · p. 5

2.4 Training Recipe

  • Training uses TimeLens datasets for grounding and an approximately 240K-question, 19K-video corpus for option selection, then aligns the scorer before adapting the language backbone.Stage one trains the head on 30% of videos with the backbone and marker embeddings frozen; stage two trains the backbone, marker embeddings, and head on the remaining 70%. The visual encoder and merger remain frozen throughout.Source: §2.4 · p. 5

3 Experiments

Experiments evaluate temporal grounding and multiple-choice video QA, then test design choices, query efficiency, and confidence. Long-video QA uses a fixed 256K-token context; latency measurements use H200 GPUs at batch size 1 and exclude preprocessing and data transfers.

3.2 Video Temporal Grounding

  • VideoJev-4B reaches 65.3 mean mIoU, 6.7 points above Qwen3.5-4B; VideoJev-35B-A3B reaches 67.7, 10.8 points above Qwen3.6-35B-A3B, with 2.8–3.1× speedups.Scaling from 4B adds 2.4 mean mIoU points with a 22% increase in measured latency.Source: Table 1 · p. 6, §3.2 · p. 7

3.1 Experimental Setup

  • Evaluation covers three grounding benchmarks and four QA benchmarks; QA uses video frames without subtitles or audio, and latency excludes preprocessing and data transfers.Grounding reports R1@0.3/0.5/0.7 and mIoU; QA reports accuracy or M-Avg. Long-video QA caps inputs at 2,048 frames.Source: §3.1 · p. 6, §3.3 · p. 7

3.3 Option Selection on Video QA Benchmarks

  • Across four QA benchmarks, VideoJev-4B and 35B-A3B improve mean scores over Instruct backbones by 8.0 and 5.7 points, while remaining competitive with Thinking baselines at 129× and 100× speedups.Thinking retains an edge on Video-MME and MLVU, while both VideoJev models outperform their Thinking counterparts on LVBench and match or improve on LongVideoBench.Source: §3.3 · p. 7, Table 2 · p. 8

3.4 Ablation Study

  • With matched data and schedules, VideoJev-4B improves mean grounding and QA scores over autoregressive fine-tuning by 1.3 and 1.9 points.The control retains the native final MLP and language-model head, supporting an architectural benefit from directly supervising temporal membership and option preferences.Source: §3.4 · p. 7, Table 3 · p. 8
  • Joint inference improves 13 of 14 model–benchmark comparisons, and joint grounding–QA training improves all seven benchmarks.Joint inference raises mean grounding/QA scores by 0.8/0.5 points at 4B and 0.3/0.7 at 35B-A3B; joint training raises mean grounding mIoU by 0.7 and mean QA by 1.3 points.Source: §3.4 · pp. 7–9, Table 4 · p. 8, Table 5 · p. 9
  • Soft IoU raises mean grounding mIoU from 60.7 to 65.3 (+4.6) and mean QA from 73.9 to 74.3 (+0.4); removing the final MLP leaves overall scores essentially unchanged.The MLP-free readout is adopted to avoid an additional computation layer.Source: Table 6 · p. 9, §3.4 · p. 9, Table 7 · p. 9
  • Two-stage adaptation achieves comparable performance while updating the language backbone on only 70% of videos; single-stage training leads by 0.1 mean grounding mIoU and 0.4 mean QA points.The remaining 30% use cheaper head-only training.Source: §3.4 · p. 9, Table 8 · p. 10

3.5 Analysis

  • At ten queries, Joint inference takes 0.39 s/1.64 s versus Cached inference’s 0.85 s/2.17 s for grounding/QA, with independently measured quality improving by 1.5 mIoU/0.8 accuracy points.Confidence generally accompanies accuracy, but probabilities tend to overestimate correctness; four-benchmark mean ECE is 8.65 and 8.13 percentage points for 4B and 35B-A3B.Source: Figure 3 · p. 10, §3.5 · pp. 10–11, Figure 4 · p. 11

4 Related Work

The section situates VideoJev among video MLLMs, temporal grounding methods, and System One visual decision models, emphasizing shared scoring and repeated-query video decisions.

MLLM video understanding

  • Video MLLMs extend instruction-driven video understanding through multimodal training, long-video methods, streaming interaction, and real-time response.Examples include VideoChat [22], VideoChat2 [23], VideoChat3 [25], VideoChat-Flash, VideoLLM-online [5], and JoyAI-VL-Interaction [46].Source: §4 · p. 11

Temporal grounding

  • Prior temporal grounding spans candidate matching, span and moment prediction, and clip-level outputs; VideoJev scores temporal membership and textual options, then decodes contiguous spans.Other approaches include temporal tokens in LITA and Grounded-VideoLLM [17] [39], and combined grounding and QA in GroundVQA [9] and SEAL [40].Source: §4 · pp. 11–12

System One models

  • Jev, one-pass decision readouts [7] [36], and image decision systems [48] [58] establish a System One context that VideoJev extends to repeated temporal video queries.VideoJev aligns multi-query training with inference.Source: §4 · p. 12

5 Future Work

Future work considers extending candidate-based decisions to more visual tasks and composing them into open-ended agent workflows. It also identifies streaming inputs and confidence reliability as areas for further work.

Toward general-purpose visual decisions

  • Candidate decisions could extend beyond grounding and Video QA to action recognition, event detection, scene-state classification, and visual condition verification.The proposed extension could reduce reliance on task-specific models and prediction heads.Source: §5 · p. 12

From closed-set decisions to open-ended agent workflows

  • Agents could decompose open-ended goals into visual checks, ask VideoJev to evaluate candidate events or actions, and use judgments to guide planning and execution.Ambiguous observations could prompt candidate revision or further reasoning.Source: §5 · p. 12

Toward streaming and reliable visual interaction

  • Streaming inputs require incremental visual encoding, efficient memory management, and timely decision-context updates; reliability work could address overconfident probabilities.The section suggests calibration-aware training, uncertainty estimation, and selective prediction as possible approaches.Source: §5 · p. 12

6 Conclusion

VideoJev unifies temporal grounding and multiple-choice video QA through direct candidate scoring and reusable visual computation. The conclusion presents its benchmark performance and ablations as support for structured judgments in visual-agent workflows.

6 Conclusion

  • Across seven benchmarks, dense 4B and MoE 35B-A3B models achieve strong prediction quality and inference efficiency.The conclusion reports that ablations show gains beyond training data alone and benefits from joint learning across the two tasks.Source: §6 · p. 12
  • Cached and Joint inference reuse visual computation to make repeated judgments efficient.VideoJev unifies temporal grounding and multiple-choice video QA through direct candidate scoring.Source: §6 · p. 12
  • VideoJev offers a foundation for visual agents to compose structured judgments into complex, open-ended workflows.The conclusion attributes this direction to its unified decision interface.Source: §6 · p. 12

A Detailed Evaluation Protocol

The evaluation protocol specifies baseline modes, video inputs, and frame-sampling limits for the main benchmarks.

A Detailed Evaluation Protocol

A.1 Latency

The latency section measures synchronized model forward time under separate grounding, QA, and multi-query protocols. VideoJev and generative baselines reuse visual encoding and video-prefix prefill.

A.1 Latency

  • Latency sums synchronized visual-encoding, prefill, and decoding time, divided by query count; video loading, preprocessing, transfers, and postprocessing are excluded.Truncated and invalid outputs remain included in the timing.Source: §A.1 · p. 18
  • Grounding latency covers all 1,313 Charades videos and 3,363 queries on one H200, with one untimed warm-up per video.VideoJev retains FP32 scoring heads and MoE routers; decoding limits and context settings vary by baseline.Source: §A.1 · p. 18
  • QA latency covers all 103 LVBench test videos and 1,549 questions on one H200, measured once per video after an untimed warm-up.VideoJev-35B-A3B retains FP32 routers and a FP32 decision head.Source: §A.1 · p. 18
  • Multi-query scaling tests 1–10 ordered queries per video across Charades and Video-Holmes, charging shared visual encoding and prefill once before query and decoding forwards.It reports equal-video means and pointwise 95% confidence intervals from paired bootstrap resamples.Source: §A.1 · p. 18

A.2 Performance

The performance section specifies evaluation protocols for temporal grounding, Video QA, and multi-query prediction quality. It describes input budgets, baselines, and scoring rules rather than reporting outcome values.

A.2 Performance

  • Grounding evaluates all Charades, ActivityNet, and QVHighlights test queries at 4 fps, using a 144K-token context and counting invalid intervals as failures.VideoJev uses Joint inference with τ = 0.25 and smoothing window 1 across both model sizes and all experiments.Source: §A.2 · p. 19
  • Video QA evaluations use fixed 256K-token contexts, QA sampling defaults, and reduced frame area to keep visual tokens within 80% of the budget.MLVU averages seven development tasks.Source: §A.2 · p. 19
  • Generative QA baselines answer independently without answer history, and invalid extracted answers count as incorrect.Published scores retain source settings; prediction coverage is checked before deterministic option parsing and text-only extraction when needed.Source: §A.2 · p. 19
  • Multi-query prediction quality is evaluated separately from timing using full-Charades mIoU and full-LVBench accuracy, with Direct assessed separately from the Instruct baseline.Invalid or truncated answers receive zero; scoring accepts unambiguous final options with optional answer text.Source: §A.2 · p. 19

A.3 Autoregressive Control with the Same Training Data

The section describes an autoregressive control trained on VideoJev’s data and inference modes for comparing answer generation with direct scoring. It also documents a sensitivity to empty answer histories in Direct inference.

A.3 Autoregressive Control with the Same Training Data

  • The autoregressive control trains native Qwen3.5-4B and single-stage VideoJev-4B on the same 404,037 questions from 58,424 videos.It uses one epoch of answer-token cross-entropy training with the video encoder and merger frozen; targets are interval lists or QA option letters, without Thinking.Source: §A.3 · p. 19
  • The autoregressive control evaluation branches queries independently from an immutable video prefix, without answer history, and uses greedy decoding limits of 256 tokens for grounding and 16 for QA.Grounding scores the first interval, with invalid or truncated outputs counted as failures.Source: §A.3 · p. 19

A.4 Inference Modes and Direct Baseline

  • Cached inference processes queries sequentially from independent copies of one prefixed video cache, while Joint inference concatenates queries after one prefix and reads predictions in one causal forward.Joint queries can attend to earlier queries, but neither reference nor predicted answers are inserted.Source: §A.4 · p. 19
  • VideoJev supports direct candidate scores in both modes, whereas autoregressive Joint inference requires single-token answers; multi-token outputs use Cached inference.Direct option prediction reads one option token without reasoning and masks logits to valid option tokens before sampling filters.Source: §A.4 · pp. 19–20
  • 1,418 of 1,446 later LVBench readouts produced an end-of-turn token rather than a valid option under unconstrained empty answer histories.Reported Direct results instead use complete turns, uniform official non-thinking prefixes, no inserted answers, and option-token masking.Source: §A.4 · p. 20

B Qualitative Benchmark Cases

The qualitative cases compare Joint inference outputs across seven benchmarks, with ground truth and model predictions shown together. They include close localization, correct option selections, a disagreement, and a shared counting error.

B Qualitative Benchmark Cases

  • The qualitative figures use green for ground truth and brown and red for model outputs; QA bars show uncalibrated scores with selected options outlined.Grounding times are video-relative.Source: B Qualitative Benchmark Cases · p. 20
  • Both models match a one-second transition in six Charades queries, while VideoJev-35B-A3B locates adjacent actions more precisely.The compound jump-rope interval on ActivityNet is also more closely matched by VideoJev-35B-A3B; both models match the QVHighlights preparation interval.Source: Figure 6 · p. 20, Figure 7 · p. 21, Figure 8 · p. 21
  • On a 3,665.5-second LVBench video, Joint inference answers 18 questions in one forward, including caption-retrieval and weather-recognition readouts.The caption detail is enlarged for display.Source: Figure 9 · p. 21
  • Both models select the annotated answer for an event-relation question on LongVideoBench and a weather question on LVBench.On MLVU, both select the annotated answer when counting five recurring scenes, though their option scores differ.Source: B Qualitative Benchmark Cases · p. 21, Figure 10 · p. 22, Figure 11 · p. 22
  • For one LVBench intent question, VideoJev-4B selects courtesy while VideoJev-35B-A3B selects the annotated arrest intent.Source: Figure 12 · p. 22

06 Counting recurring scenes across a video MLVU / 500.1s count_108

  • Both models miss the annotated count of four books in a Video-MME small-object counting case.The book detail is enlarged for display.Source: Figure 13 · p. 23

Related papers

Loading 2610.18615v1…