Source-linked AI summary
DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton
Jintao Cheng, Weibin Li
TL;DR
VLMs remain limited in reasoning about coordinated, context-dependent multi-agent actions, a gap that conventional action-focused evaluations rarely expose. The paper introduces DoublesEval, a key-moment diagnostic benchmark using professional doubles badminton, and TacticCheck, a consistency-based test-time reranker. Across four open-source VLMs and approximately 9,600 instances, TacticCheck consistently improves zero-shot results, although the curated data trade breadth of player and style coverage for annotation depth.
Problem
Existing video evaluations mainly assess individual actions or global events, providing limited evidence about role, spatial, and cross-event reasoning in interactive scenes.
Method
DoublesEval evaluates zero-shot tactical reasoning from key rally moments across four diagnostic dimensions, while TacticCheck reranks answers using consistency with lower-level predictions.
Results
TacticCheck delivers consistent zero-shot improvements across evaluated open-source VLMs without parameter updates or ground-truth labels during inference.
Takeaways & Limitations
The findings support structured, interaction-aware evaluation that localizes tactical reasoning failures rather than relying only on final-answer correctness.
Takeaways & Limitations
Using four source matches yields comparatively limited coverage of distinct player pairs and tactical styles because annotation depth is prioritized.
Abstract
from arXiv · showhide
Visual Language Models (VLMs) excel at describing visible scene content but struggle to reason about dynamic multi-agent interactions, where action semantics depend on coordinated roles and spatial-temporal dependencies. We formalize this capability as \textbf{multi-agent tactical reasoning} and introduce \textbf{DoublesEval}, a diagnostic evaluation framework that leverages professional doubles badminton as a structurally tractable testbed. DoublesEval employs a key-moment-based protocol that decomposes rallies into tactically salient instants and probes models across four interpretable dimensions: atomic recognition, intra-segment composite understanding, cross-segment causal reasoning, and high-level tactical abstraction. This design isolates \emph{where} reasoning fails, rather than merely measuring answer correctness. To address observed failure modes, we propose \textbf{TacticCheck}, a lightweight constraint-guided test-time consistency checker that reranks candidate answers using the model's own lower-level tactical predictions, requiring no parameter updates or ground-truth labels at inference time. Evaluating four representative open-source VLMs on 60 curated rallies (yielding $\sim$9.6K structured instances) via a zero-shot protocol, we find that models remain weak across all diagnostic levels, with especially clear bottlenecks in spatial state, interaction binding, and terminal evidence. TacticCheck delivers consistent gains across all evaluated models, while still leaving a substantial gap to robust tactical reasoning. These results highlight the need for structured, interaction-aware evaluation paradigms for next-generation VLMs. The source code is available in \href{https://github.com/Chengjt1999/DoublesEval}{\textcolor{blue}{our GitHub repository}}.
1 Introduction
Existing VLM evaluations emphasize visible actions, while DoublesEval targets multi-agent tactical coordination through structured, key-moment diagnostics. The framework also introduces TacticCheck and evaluates representative open-source VLMs on curated professional doubles rallies.
- Current VLM benchmarks mainly assess perception-level descriptions, leaving dynamic roles, spatial coverage, and causal chains underexamined.
- Professional doubles badminton provides a bounded, four-player environment with rapid role transitions and dense shot-to-shot causal dependencies.
- DoublesEval decomposes rallies into tactically salient moments and probes atomic recognition, composite understanding, causal reasoning, and tactical abstraction.
- TacticCheck reranks higher-level answers by checking consistency with the model’s own lower-level tactical predictions, without parameter updates or inference-time labels.
- DoublesEval uses 60 curated rallies and approximately 9,600 structured instances to evaluate four representative open-source VLMs under zero-shot prompting.
Related Work
Diagnostic VLM research has exposed weaknesses that aggregate benchmarks can obscure, while badminton analysis has progressed from low-level perception toward structured rally and tactical modeling.
- Aggregate VLM benchmarks can obscure systematic reasoning failures, motivating diagnostic evaluations of specific cognitive bottlenecks.
- Early diagnostic studies reported deficits in attribute binding and relational composition, including bag-of-words-like representations in contrastive VLMs.
- Badminton video research progressed from shuttlecock and player tracking to stroke classification and structured rally representations.
3.1 Task Formulation: Multi-Agent Tactical Reasoning
DoublesEval maps queries over tactically salient rally moments to predictions covering roles, spatial occupancy, temporal causality, and strategic patterns. Four ordered diagnostic dimensions separate basic grounding from higher-order tactical reasoning.
- A professional doubles rally is segmented into ordered key moments, and each query conditions prediction on the video, a moment, and a reasoning target.
- The target space includes agent role assignment, spatial occupancy, temporal causality, and strategic pattern recognition, with failures analyzed per query.
- L1: Single-Field Atomic Recognition: L1 tests isolated player, court-zone, and stroke recognition at a single moment.
- L2: Intra-Segment Composite Understanding: L2 tests simultaneous binding of spatial relation, agent role, and action state within a localized segment.
- L3: Cross-Segment Causal Reasoning: L3 links an earlier tactical decision to later repositioning or stroke selection across sequential moments.
- L4: High-Level Tactical Semantic Abstraction: L4 summarizes recurrent rotation, coverage, and transition patterns over extended rally phases.
3.2 DoublesEval: A Diagnostic Suite for Tactical Reasoning
DoublesEval builds a structured diagnostic suite from curated professional doubles badminton rallies, using tactically salient segments and paired closed- and open-ended queries. Expert annotation anchors labels to scoring windows and supports both controlled evaluation and free-form tactical probing.
- Video Source and Rally Curation: 60 curated rallies from four professional matches are partitioned into diagnostic segments covering attacking, defending, rotation, and net-play situations.Selection excludes short rallies, camera cuts, replays, and substantial occlusion while prioritizing tactical diversity and visual clarity.
- Paired Query Architecture: The benchmark yields approximately 9,600 paired instances: 4,800 single-choice questions and 4,800 short-answer questions across four diagnostic dimensions.SCQs provide one correct schema-derived option; SAQs remove options and request concise tactical explanations or summaries.
- Query Construction: SCQ distractors encode swapped roles, inverted court zones, and temporally misaligned causal links to isolate perceptual and relational bottlenecks.SAQs extend the same contexts to test causal articulation, multi-cue binding, and strategic abstraction without option scaffolding.
- Expert-Verified Scoring-Window Annotation: Each rally uses a scoring window containing the final scoring stroke and 3–5 preceding shots as the unified annotation target.The full rally supplies temporal context, while L1/L2 probe the immediate window and D3/D4 connect earlier phases to it.
- Annotation and Quality Control: Two analysts annotate all rallies, while certified National Level-1 Athletes audit a stratified 30% subsample and adjudicate tactical discrepancies.Labels are created in Label Studio and populate closed answer spaces and semantic targets for short-answer mapping; player references use relative descriptors.
3.3 Two-Stage Evaluation Protocol
The two-stage protocol first elicits zero-shot answers from four open-source VLMs and then uses GPT-5 to map free-form responses into standardized expert-defined labels. Human verification measures agreement between the judge and domain experts.
- Stage 1: Zero-Shot Generation: Four open-source VLMs process full rally videos with dimension-conditioned prompts targeting the scoring-window state under deterministic zero-shot decoding.The evaluated models are Qwen2.5VL, Qwen3-VL, VideoLLaMA3, and Molmo2.
- Stage 2: Label Mapping: GPT-5 receives each response and the dimension-specific label space, then performs semantic alignment into a strict JSON schema rather than subjective scoring.This standardizes lexical variation against the annotation vocabulary without generative phrasing bias.
- Judge Validation: On 480 randomly sampled instances, Cohen’s κ is 0.86 between experts and 0.81 between the GPT-5 judge and experts.Disagreements are resolved through majority voting or expert arbitration.
3.4 TacticCheck: Constraint-Guided Test-Time Consistency Checking
TacticCheck uses a VLM’s own lower-level tactical predictions to test higher-level answers for schema-level consistency at inference time. It reranks candidates conservatively without retraining, additional data, or ground-truth labels.
- Checklist Extraction: The checker extracts a compact tactical checklist from the model’s zero-shot L1 predictions, covering spatial states, interaction cues, outcome attribution, and terminal evidence.No expert annotation is used during checklist extraction.
- Constraint-Guided Reranking: Each candidate answer receives a tactical signature, and incompatible spatial, interaction, attribution, or terminal-evidence fields incur weighted penalties.Examples include far-team versus near-team interaction contradictions and score explanations conflicting with predicted terminal evidence.
- Conservative Selection: The final answer is reranked by combining the model’s original preference with its consistency penalty, replacing the default only when another option is clearly more consistent.This conservative rule reduces unnecessary answer changes.
- Scope and Limitations: TacticCheck tests whether explicit consistency helps, but its signal remains bounded by the accuracy of the model-generated checklist.Incorrect L1 predictions can produce incorrect consistency judgments, so the method is not presented as a complete solution.
3.5 Evaluation Metrics
DoublesEval treats each mapped prompt as an evaluation unit and reports dimension-wise and macro accuracy across paired SCQ and SAQ formats. TacticCheck gains are reported separately on the identical SCQ subset because the reranker requires explicit options.
- Primary Metrics: Dimension-wise accuracy averages SCQ and SAQ accuracy within each diagnostic level, while the overall macro score is the unweighted mean across four levels.SAQ responses are first mapped into the same discrete label space as SCQs.
- TacticCheck Accounting: TacticCheck comparisons use SCQ accuracy before and after reranking on the identical SCQ subset, leaving SAQ accuracy unchanged and excluded from deltas.The reranker requires an explicit option set, so mixed Base and TacticCheck tables use SCQ-only accounting for fair comparison.
4 Experiments
Zero-shot evaluation shows broadly weak VLM performance across DoublesEval’s diagnostic levels, with failures concentrated in spatial, interaction, and terminal evidence. TacticCheck consistently improves accuracy, but remains error-prone and below robust tactical reasoning.
- 4.1 Overall Zero-Shot Performance: L1 is easier for most models, but atomic recognition remains far from saturated.Level-wise results aggregate single-choice and label-mapped short-answer questions.
- 4.1 Overall Zero-Shot Performance: 28.65% to 32.35%: overall performance remains weak across models, with non-trivial errors at every diagnostic level.The narrow range indicates a shared weakness across architectures rather than an isolated model-family failure.
- 4.2 Diagnostic Analysis: Temporal phase recognition is strongest, while spatial state, interaction binding, and score detail are major capability bottlenecks.These categories require binding players, tactical roles, spatial configuration, and outcome across the rally.
- 4.2 Diagnostic Analysis: Within L1, phase and scoring side are comparatively reliable, whereas near/far interactions and stroke pattern remain difficult.Stroke pattern is the hardest L1 field for all models and supports later causal questions.
- 4.3 Improved Performance with Test-Time Consistency Checking: TacticCheck improves overall accuracy by 6.55–7.83 percentage points across four models, without parameter updates or inference-time ground-truth labels.It reranks higher-level candidates using the model’s own L1 tactical checklist and penalizes schema-level contradictions.
- 4.3 Improved Performance with Test-Time Consistency Checking: TacticCheck accuracy remains below 40% for all models and can turn originally correct answers into incorrect ones.When the model’s checklist is wrong, consistency reranking can propagate that error.
- 4.3 Improved Performance with Test-Time Consistency Checking: The benchmark’s central weakness is binding visible evidence into a coherent multi-agent tactical chain, not merely recognizing badminton actions.A lightweight consistency reranker recovers part of the gap, while remaining errors point to insufficient interaction modeling.
5 Conclusion
DoublesEval exposes persistent multi-agent tactical reasoning limitations in VLMs through structured professional doubles badminton evaluation. Across four diagnostic levels, experiments reveal consistent bottlenecks in spatial state recognition, interaction binding, and causal reasoning.
- DoublesEval exposes VLM limitations in multi-agent tactical reasoning through structured evaluation on professional doubles badminton.
- Experiments reveal consistent bottlenecks in spatial state recognition, interaction binding, and causal reasoning across four diagnostic levels.