PaperlayerView exact revision on arXiv

Source-linked AI summary

SynDORBench: Evaluating LVLM Perceptual Robustness Under Physically Constrained Visibility Conditions

Jeremy Stephen Gabriel Yee, Zhengkui Wang, Zhiyuan Zhang, Avinash Anand, Timothy Liu, Benedict Chan, Aik Beng Ng, Simon See

arXiv:2609.31823v1cs.CVcs.AI

TL;DR

Existing evaluations offer limited evidence about LVLM reliability when camera distance, illumination, viewpoint, and pixel density constrain visual information. SynDORBench builds a DORI-calibrated synthetic benchmark and evaluates models across degraded visibility conditions. Results show that physical imaging constraints shape perceptual reliability, while several compact open-source models match or exceed larger commercial systems and LVLMs outperform detector-based pipelines in long-range, low-visibility settings.

  • Problem

    Existing evaluations predominantly use idealized visual inputs and provide limited understanding of how physical imaging constraints affect semantic recoverability and LVLM reliability.

    Source: §1 · p. 2

  • Method

    SynDORBench uses a controllable synthetic pipeline to evaluate LVLMs under DORI-calibrated imaging conditions, with single-image human-presence classification and action-recognition tasks.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 5

  • Results

    Modern LVLMs substantially outperform detector-based pipelines under long-range and low-visibility conditions, while several compact open-source models achieve robustness comparable to larger commercial systems.

    Source: §5 · p. 9

  • Takeaways & Limitations

    SynDORBench establishes physically grounded multimodal evaluation as an underexplored dimension of LVLM assessment and supports study of reliability under real-world perceptual constraints.

    Source: §5 · p. 9

    The current benchmark focuses on single-human, frame-level scenarios under systematically calibrated visibility conditions.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large vision-language models (LVLMs) have demonstrated remarkable performance on multimodal reasoning benchmarks, yet their perceptual reliability under physically constrained imaging conditions remains poorly understood. Existing evaluations predominantly assume ideal visual inputs and therefore fail to characterize how camera distance, illumination, viewpoint, and pixel density fundamentally affect semantic recoverability. We introduce SynDORBench, the first physically grounded benchmark for evaluating LVLM perceptual robustness under DORI-calibrated conditions aligned with human visual capability standards. SynDORBench comprises over 54k question--answer pairs generated through a controllable synthetic pipeline that systematically varies viewing distance, lighting, camera geometry, and action pose according to physically interpretable pixel-density regimes. To support scalable low-visibility supervision, we further propose a discernibility annotation framework that propagates human perceptual labels using mask-conditioned statistical features and ensemble learning. We evaluate 16 open-source LVLMs, a commercial LVLM baseline, and YOLO11x across human-presence classification and action recognition tasks under progressively degraded visibility conditions. Our results reveal that perceptual failure in LVLMs is strongly governed by pixel density and physical imaging constraints rather than model scale alone. Surprisingly, several compact open-source LVLMs outperform larger commercial baselines and substantially exceed YOLO11x robustness under long-range and low-light conditions. SynDORBench establishes a new benchmark paradigm for physically grounded multimodal evaluation, enabling systematic analysis of LVLM reliability under real-world perceptual constraints and direct comparison against human visibility thresholds.

1 Introduction

SynDORBench addresses the gap between idealized semantic evaluations and the physically constrained conditions that shape whether visual evidence is recoverable. Its DORI-calibrated synthetic benchmark varies imaging factors and evaluates LVLM perceptual robustness, with results showing that reliability tracks pixel density and physical constraints more than model scale alone.

1 Introduction

  • Existing evaluations emphasize semantic competence under idealized visual inputs, leaving reliability under physically constrained imaging conditions poorly characterized.Distance, illumination, sensor characteristics, viewpoint, and environmental interference affect pixel density and semantic recoverability.Source: §1 · p. 2
  • SynDORBench is a DORI-calibrated benchmark with over 54k question–answer pairs for human-presence classification and action recognition across degraded visibility conditions.It explicitly models camera distance, illumination, viewpoint, and pixel-density regimes.Source: §1 · p. 2
  • A discernibility annotation framework propagates human perceptual labels using mask-conditioned statistical features and ensemble learning for low-light and night-vision evaluation.The framework supports scalable low-visibility supervision.Source: §1 · p. 2
  • Experiments across 16 open-source LVLMs, a commercial LVLM baseline, and YOLO11x find that perceptual failure is governed more by pixel density and imaging constraints than model scale alone.Source: §1 · p. 2
  • Modern LVLMs outperform traditional detection pipelines under long-range and low-visibility conditions, while several compact open-source models match or exceed larger commercial baselines.These comparisons indicate that perceptual robustness does not scale monotonically with parameter count.Source: §1 · pp. 2–3
  • SynDORBench connects camera physics, human perceptual standards, and multimodal reasoning evaluation to study LVLM reliability under real-world perceptual constraints.The benchmark enables systematic study of physically grounded multimodal evaluation.Source: §1 · p. 3

2 Related Work

Prior synthetic and multimodal benchmarks support controlled visual research and broad semantic evaluation, but generally do not tie degradation to measurable imaging properties. SynDORBench addresses this gap with physically calibrated evaluation across controlled visibility conditions.

2 Related Work

  • Prior synthetic benchmarks emphasize semantic diversity, photorealism, or downstream performance, while degradation is usually not tied to measurable imaging properties such as viewing distance or pixel density.This makes benchmark difficulty dataset-relative rather than physically interpretable.Source: §2 · p. 3
  • Standard multimodal benchmarks largely assess reasoning and semantic understanding under curated, high-visibility conditions, leaving physical factors such as distance, illumination, viewpoint, and pixel density underexplored.Video benchmarks include long-video reasoning and action understanding, but their degradation factors are observational rather than systematically parameterized by imaging constraints.Source: §2 · pp. 3–4
  • SynDORBench evaluates LVLM capability across DORI-calibrated visibility regimes with controlled variation in distance, lighting, and viewpoint, linking multimodal evaluation to camera physics and human perceptual standards.The related-work comparison is summarized in Table 1.Source: §2 · p. 4

3 SynDORBench

SynDORBench uses controlled physical degradation and DORI-calibrated visibility regimes to study when LVLM perception breaks down. Its synthetic scenes, discernibility labels, task interface, and evaluation metrics support factorized analysis of perceptual robustness.

3 SynDORBench

  • SynDORBench is a physically grounded framework that studies LVLM perceptual capability as imaging conditions become progressively constrained.It models camera distance, illumination, viewpoint, and pixel density, anchoring synthetic image generation to DORI-calibrated visibility ranges.Source: §3 · pp. 4–5

3.1 Synthetic Image Creation

  • The synthetic pipeline controls scene geometry, camera configuration, illumination, viewpoint, and subject appearance; its single-human design isolates visibility effects from occlusion and crowd-related ambiguity.The single-human setup is intended to support analysis of how distance, lighting, and viewpoint independently affect perception.Source: §3.1 · p. 5

3.2 Discernible Labels

  • Discernibility labels distinguish a person's physical presence from whether a human annotator can reliably perceive the subject under rendered conditions.The annotation framework uses mask-conditioned perceptual features and an RF+XGBoost ensemble to propagate labels across night and night-vision subsets.Source: §3.2 · p. 6

3.3 Task Definition and Question Types

  • Each benchmark task is a bounded perceptual objective solved from a single image, using a question-template interface for human-presence classification and action recognition.These tasks represent core long-range visual understanding capabilities relevant to robotics, assistive technologies, and smart infrastructure.Source: §3.3 · pp. 6–7

3.4 Evaluation Metrics

  • Human-presence classification is reported with accuracy and macro-F1 across DORI-calibrated visibility regimes, with 80% accuracy defined as the operational-effectiveness threshold.Performance below that threshold is treated as operationally unreliable under the corresponding imaging conditions.Source: §3.4 · p. 7

4 Evaluation Results

Across DORI-calibrated presence-classification evaluations, LVLM performance declines with distance, while LVLMs outperform detector-based systems overall. Model scale is informative but does not fully determine robustness.

4 Evaluation Results

  • The evaluation covers 16 open-source LVLMs, one commercial LVLM baseline, and YOLO11x, with models restricted to 8B parameters or fewer.The authors frame this range as practically relevant to low-latency visual understanding systems.Source: §4 · p. 8
  • Accuracy and macro-F1 consistently decrease for all evaluated models as viewing distance increases from 5.4 m to 23.0 m.The evaluation averages over illumination and viewpoint to isolate distance degradation.Source: §4 · p. 8
  • MVLMs achieve the strongest overall group performance across distances, but gemma-3-4b-it and gemma-3n-E2B-it match or exceed larger MVLMs.The reported pattern is not monotonic with parameter count.Source: §4 · p. 8
  • YOLO11x is weaker than LVLMs, particularly at longer distances, and falls below the 80% reliability threshold at 5.4 m.Its degradation is also associated with heterogeneous illumination, contrast variation, and perceptual ambiguity.Source: §4 · p. 8
  • LVLMs substantially outperform traditional detector-based systems on DORI-calibrated human-presence classification and remain competitive with commercial LVLMs as visibility degrades.YOLO11x falls below the 80% operational reliability threshold even at 5.4 m.Source: §4 · p. 8, Table 3 · p. 9
  • LVLM robustness does not scale monotonically with parameter count; vision encoder design, visual alignment, and feature integration may also matter.The discussion contrasts physically grounded perception with properties optimized for conventional semantic reasoning benchmarks.Source: §4 · p. 8

5 Conclusion

SynDORBench evaluates LVLM perceptual robustness under DORI-calibrated imaging conditions, focusing on how perception changes as visual evidence degrades. The evaluation finds that imaging constraints matter and that LVLMs outperform detector-based pipelines in the reported settings.

5 Conclusion

  • SynDORBench is a physically grounded benchmark for evaluating LVLM perceptual robustness under DORI-calibrated imaging conditions.It studies perception as visual evidence deteriorates under controlled physical constraints.Source: §5 · p. 9
  • Pixel density and imaging conditions play a central role in multimodal perceptual reliability across the evaluated open-source, commercial, and detector-based systems.The experiments also report that LVLMs outperform detector-based pipelines under long-range and low-visibility conditions.Source: §5 · p. 9
  • Several compact open-source models achieve robustness comparable to larger commercial systems, while modern LVLMs outperform traditional detector-based pipelines in the reported conditions.Source: §5 · p. 9
  • The dataset and evaluation framework are planned for non-commercial academic release with rendered images, metadata, and evaluation code.The stated release respects upstream-resource licensing terms.Source: §5 · p. 9
  • The benchmark is limited to single-human, frame-level scenarios under systematically calibrated visibility conditions.Source: §5 · p. 9

A.1.1 Camera Specification Survey

The camera survey informs the simulated hardware configuration by identifying a common fixed-camera profile among major manufacturers.

A.1.1 Camera Specification Survey

  • The simulated hardware uses the dominant surveyed profile: a 1/2.8-inch sensor paired with a fixed 2.8 mm lens.The profile is reported across Axis, Hikvision, and Dahua systems.Source: §A.1.1 · p. 14
  • The survey’s stated purpose is to guide the hardware used to capture the Blender images.Source: §A.1.1 · p. 14
  • The surveyed camera specifications cover representative commercial models commonly used in fixed-camera systems.Source: Table 4 · p. 14

A.1.2 Action Pose Filtering Pipeline

The action-pose pipeline filters BABEL frame labels for visually relevant actions, then retargets and animates selected poses in Blender.

A.1.2 Action Pose Filtering Pipeline

  • BABEL action candidates are sampled, ranked for relevance using E5-large-v2 embeddings and Gemma-3-13b-it reranking, then manually checked for semantic consistency and still-frame perceptibility.Source: §A.1.2 · p. 14
  • The selected subset prioritizes visually distinguishable static actions for single-frame perception evaluation.Source: §A.1.2 · p. 14
  • The final action set includes walking, kicking, sneaking, choking, dodging, covering the face with hands, stepping over something, and guard up.Selected poses are retargeted and animated within Blender.Source: §A.1.2 · p. 14

A.1.3 Discernible Label Model Ablation

Table 5 reports validation performance for discernibility annotation models using 5-fold cross-validation.

A.1.3 Discernible Label Model Ablation

  • Table 5 reports validation performance for discernibility annotation models using 5-fold cross-validation.Source: Table 5 · p. 14

A.1.4 Modeling Distance by Pixel Density

The appendix evaluates action recognition and human-presence classification across DORI distances and lighting conditions, alongside the modeling, task, metric, and simulation procedures. Performance degrades with increasing distance and poor illumination, while model size alone does not determine robustness.

A.1.4 Modeling Distance by Pixel Density

  • The distance model derives vertical field of view, scene coverage, and pixel density from camera geometry, then expresses viewing distance using image height and a target DORI pixel-density threshold.Source: §A.1.4 · pp. 14–15

A.1.5 Night-Vision Simulation

  • The night-vision simulation applies a vendor-neutral ISP pipeline of sRGB linearization, gain and color normalization, BM3D denoising, CLAHE enhancement, and tone mapping.The resulting imagery is described as physically plausible while retaining the underlying DORI-calibrated visibility conditions.Source: §A.1.5 · p. 15

A.2 Additional Task Definition and Question Type

  • Action recognition asks models to name the subject’s activity, testing recognition of pose, motion cues, and context under degraded visibility.Source: §A.2 · p. 15

A.3 Additional Metrics

  • E5Score measures semantic alignment for short open-ended action responses, avoiding unstable lexical-overlap scoring and vision-encoder scoring bias.The evaluation also reports distance-based z-scores and a human baseline from one non-expert annotator on 300 randomly sampled instances.Source: §A.3 · pp. 15–16

A.4 Computational Resources

  • Rendering used about 535.7 GPU hours, night-vision image creation about 19 CPU hours, and evaluation across 18 models about 72 GPU hours.Source: §A.4 · p. 16

A.5.1 Action Recognition Robustness Across Distances

  • Action recognition declines monotonically with distance; MLVLMs have higher group means than SVLMs, but Gemma-3n-E2B-it leads at all radii and all LVLMs trail the non-expert human.The mean MLVLM performs similarly to GPT-4o mini at the considered DORI ranges.Source: Table 6 · p. 16, §A.5.1 · p. 16

A.5.2 HPC Performance Over Lightings

  • The additional evaluations stratify performance by distance, lighting, and pixel-density regimes to examine robustness beyond aggregate metrics.Source: §A.5.2 · p. 17
  • Night lighting leaves every model below F1-score 0.8, while night-vision enhancement yields modest, consistent gains for several models.Across lighting and distance conditions, Gemma3 and Gemma3n lead overall, and smaller variants tend to outperform larger counterparts within the families.Source: Table 7 · p. 18, Figure 3 · p. 19, §A.5.2 · p. 20

Related papers

Loading 2609.31823v1…