PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboStressBench: Benchmarking VLM Robustness to Physical Visual Stress in Embodied Scenes

Leyi Wu, Yifan Zhao, Jinjie Zhang, Suzeyu Chen, Wosong Chen, Zhifei Chen, Tianshuo Xu, Qingchun He, Hongxin Hu, Haojian Huang, Yangkai Wei, Wenqian Li, Yinchuan Li, Ying-Cong Chen

arXiv:2606.00828v1cs.CV

TL;DR

Existing benchmarks do not systematically capture or diagnose physically grounded visual stress in embodied scenes. RoboStressBench organizes stress by material, viewpoint, lighting, and geometry, revealing task-specific weaknesses across VLMs and motivating stress-aware evaluation and intervention.

  • Problem

    Existing benchmarks under-characterize physically grounded visual stress and lack a principled way to diagnose how scene factors affect VLM reliability.

  • Method

    RoboStressBench evaluates embodied-scene VLM robustness by organizing physical visual stress into Material, Viewpoint, Lighting, and Geometry dimensions.

  • Results

    Geometry stress strongly degrades localization and spatial reasoning, while material and lighting stress more often affect recognition and state understanding across 16 VLMs.

  • Takeaways & Limitations

    Stress-aware evaluation and explicit stress diagnosis with targeted visual rectification improve the assessment and robustness of VLM perception under challenging physical conditions.

  • Takeaways & Limitations

    The four-factor taxonomy is not exhaustive, and physical stress axes can remain entangled, making fine-grained attribution challenging in real scenes.

Abstract

from arXiv · show

Vision-Language Models (VLMs) have shown strong visual understanding and are increasingly deployed in embodied AI systems, where reliable perception under real conditions is essential. However, existing benchmarks assess VLMs using clean images or isolated perturbations rather than stresses caused by physical scene formation. This design has two limitations: it covers only a narrow subset of everyday visual stresses, and some perturbations rarely appear in realistic embodied scenes. This gap raises a fundamental question: how can we define visual stress in a principled way that captures the diverse factors encountered in physical environments? To address this question, we formulate visual perception from an inverse graphics perspective and introduce RoboStressBench, a benchmark for evaluating VLM robustness to physical visual stress in embodied scenes. Inspired by the physical rendering equation, RoboStressBench decomposes visual stress into four physically grounded dimensions: Material (M), Viewpoint (V), Lighting (L), and Geometry (G). This design enables RoboStressBench to cover a broad range of visual stresses in real-world environments, while allowing controlled analysis of their effects on VLM capabilities such as visual recognition, reasoning, and planning. Through comprehensive evaluations of state-of-the-art VLMs, we identify stress-specific failure modes and reveal that different physical factors degrade different embodied capabilities, which are often obscured by aggregate accuracy. We further introduce a stress-aware agentic solver that detects visual stressors and invokes visual-editing skills before reasoning, improving robustness in high-stress scenarios. Overall, RoboStressBench provides a principled evaluation framework for diagnosing and improving VLM perception under real-world physical stress, supporting the development of more reliable embodied AI systems.

1 Introduction

RoboStressBench defines physical visual stress as degradation from physically plausible scene changes and organizes it into Material, Viewpoint, Lighting, and Geometry dimensions. Evaluations expose stress-specific task failures, while StressDART uses stress diagnosis and targeted visual rectification to improve robustness without fine-tuning.

  • Motivation: Physical visual stress is visual degradation caused by physically plausible changes in scene appearance that embodied agents must robustly handle.The paper distinguishes these challenges from generic visual difficulty in real-world embodied operation.
  • Limitations of Existing Benchmarks: Existing benchmarks under-characterize physical visual stress by omitting explicit stress-factor annotations or relying on artificial ImageNet-C-style corruptions.Examples include noise, pixelation, and algorithmic blur.
  • RoboStressBench: RoboStressBench models image formation as I = F(M, V, L, G) and diagnoses stress through Material, Viewpoint, Lighting, and Geometry dimensions.The dimensions indicate whether failures arise from surface appearance, camera pose, illumination, or spatial structure.
  • Evaluation Findings: 16 state-of-the-art VLMs across five model families show uneven degradation: geometry harms localization and spatial reasoning, while material and lighting affect recognition and state understanding.These task-stress interactions reveal failure modes hidden by aggregate accuracy.
  • StressDART: StressDART detects the dominant stress factor, applies targeted visual rectification, and reasons over original and rectified images, yielding modest robustness gains without fine-tuning.The intervention is modular and highlights the need for content-preserving rectification.
  • Dataset: The benchmark contains approximately 7.2K visual-stress examples built through human-annotated filtering, controlled synthesis, and real-world data collection.Its construction balances realism, diversity, and controllability.

2 Related Work

Prior robustness benchmarks progressed from adversarial perturbations and image corruptions toward natural, physically plausible, and controllable distribution shifts. Embodied VLM benchmarks increasingly evaluate situated understanding, grounding, spatial reasoning, action, memory, planning, and agent-level capabilities, while RoboStressBench diagnoses failures along physical image-formation axes.

  • From Visual Corruption to Physical Stress: Robustness research progressed from worst-case perturbations to controllable corruptions, natural distribution shifts, and physically plausible or high-level controllable stresses.ImageNet-C/P organized non-adversarial corruptions into controllable families, while later benchmarks covered real-world shifts, hard natural images, and physically plausible stress.
  • Robustness Evaluation for Multimodal Understanding: Multimodal robustness evaluation extended classification protocols to visual question answering, alongside robustness benchmarks for object detection and semantic segmentation.ImageNet-C/P established a standard classification protocol that was later extended to detection and segmentation; newer multimodal benchmarks evaluated VQA robustness.
  • Embodied Benchmarks for Vision-Language Models: Embodied VLM benchmarks evolved from situated question answering toward grounding, spatial reasoning, referring, action-relevant outputs, temporal understanding, memory, planning, and agent-level evaluation.OpenEQA and RoboVQA test scene understanding and experience, while RoboRefIt, RoboSpatial, RefSpatial-Bench, and later benchmarks target manipulation, spatial reasoning, action, and broader agent capabilities.
  • Embodied Benchmarks for Vision-Language Models: Task-level scores often conflate perception, reasoning, and planning errors, motivating diagnosis along physical image-formation axes.RoboStressBench complements embodied evaluations by attributing failures to physical visual stress factors rather than relying only on aggregate task scores.

3 Preliminaries

This section grounds visual stress in physically based rendering and an inverse-graphics view of image formation. It decomposes stress into four interpretable factors—Material, Viewpoint, Lighting, and Geometry—that can reduce access to task-relevant visual evidence.

  • Image Formation and Visual Stress: Physically based rendering identifies material reflectance, illumination, viewing direction, and surface geometry as factors shaping image appearance.The rendering equation uses BRDF fr, incident radiance Li, and surface normal n to characterize these influences.
  • Image Formation and Visual Stress: RoboStressBench abstracts image formation as four factors: Material (M), Viewpoint (V), Lighting (L), and Geometry (G).M corresponds to reflectance such as fr; L to incident illumination such as Li; V to viewing direction ωo; and G to spatial structure such as position and normal (x, n).
  • Image Formation and Visual Stress: Physical visual stress consists of physically plausible states of M, V, L, and G that make task-relevant visual evidence less accessible.This definition follows an inverse-graphics perspective on image formation.

4 RoboStressBench: Benchmarking Physical Visual Stress in Embodied Scenes

RoboStressBench defines physical visual stress through four image-formation dimensions—Material, Viewpoint, Lighting, and Geometry—and refines each into fine-grained categories for controlled VLM diagnosis. It combines natural, synthesized, and additional real-world examples to support VQA and grounding under realistic, diverse, and controllable stress conditions.

  • 4.1 Stress Taxonomy: RoboStressBench organizes visual stress as I = F(M, V, L, G), covering Material, Viewpoint, Lighting, and Geometry dimensions.Each dimension is further divided into fine-grained categories for controlled diagnosis of VLM perception and reasoning.
  • 4.1 Stress Taxonomy: Material stress obscures object identity or semantic cues through dark absorption, low-contrast blending, complex texture, and transparency.These stresses arise from surface appearance properties that reduce visible detail or interfere with recognition.
  • 4.1 Stress Taxonomy: Viewpoint and Lighting stress alter visual evidence through unusual views, partial framing, small scale, overexposure, underexposure, glare, and shadows.Viewpoint stress affects canonical appearance, while Lighting stress suppresses, saturates, or unevenly distorts scene information.
  • 4.1 Stress Taxonomy: The taxonomy supports dimension-level and category-level analysis, identifying both whether a model fails and which physical factor or fine-grained pattern is associated with failure.This enables diagnosis across the four dimensions and within each dimension’s stress categories.
  • 4.2 Dataset Curation: RoboStressBench combines naturally occurring cases, synthesized stress variants, and Internet-sourced or self-captured images to balance realism, diversity, and controllability.It supports VQA and grounding, retaining verified annotations when available and creating or relabeling task-specific annotations otherwise.

5 StressDART: Test-Time Stress Detection and Rectification for Robust Visual Reasoning

StressDART is a test-time solver that detects the dominant physical visual stress, applies a targeted category-specific rectification, and reasons over both original and rectified evidence. By separating diagnosis, rectification, and reasoning, it provides an interpretable framework for improving VLM robustness under physical visual stress.

  • Stress Detection: StressDART diagnoses identifiable scene factors, such as poor illumination, specular surfaces, occlusion, and unusual viewpoints, before targeted visual processing.The detector predicts a coarse stress dimension s ∈ {M, V, L, G} and a fine-grained category c.
  • Design Rationale: Separating diagnosis, rectification, and reasoning makes StressDART an interpretable test-time framework for improving VLM robustness under physical visual stress.The pipeline first detects stress, then rectifies task-relevant evidence, and finally reasons over the resulting visual information.
  • Stress Detection: The Stress Detector predicts the task-relevant stress condition, conditioning subsequent processing on why the image is difficult.Fine-grained categories include transparent, global underexposure, occlusion, and small scale.
  • Stress Rectification: The Stress Rectifier applies a category-specific visual operation, including illumination enhancement, highlight recovery, cropping, or zooming.When correction is unsafe, it preserves the original image or uses only conservative transformations.
  • Joint Reasoning: StressDART supplies both original and rectified images to the reasoner, preserving task context while enabling exploitation of recovered visual cues.The final answer is produced by reasoning over both visual inputs.

6 Experiments

RoboStressBench evaluates VLM robustness across physical stress dimensions, task formats, and model families, then tests whether StressDART improves robustness through test-time stress diagnosis and visual rectification. Results show substantial unsolved challenges, task-dependent failure patterns, and generally beneficial scaling and rectification effects.

  • Evaluation setup: The evaluation covers 16 open-source and commercial VLMs across five model families using multiple-choice and grounding tasks.Grounding combines point-based accuracy and box-based IoU@0.95 in the reported scores.
  • Overall robustness: 58.1% accuracy is the best overall result, achieved by Qwen3.5-35B-A3B, while Gemini-3.1 and GPT-5.5 reach 44.8% and 46.2%.These results indicate that current VLMs remain far from saturated on physically degraded visual evidence.
  • Scaling effects: Scaling generally improves average performance but does not consistently remove stress-specific failures.Qwen3.5 rises from 49.8% with 4B to 58.1% with 27B, an 8.3% gain; Qwen3VL rises from 43.2% with 4B to 55.9% with 30B-A3B, while InternVL3.5-14B underperforms InternVL3.5-4B overall.
  • Task-dependent sensitivity: Geometry stress is especially harmful for placement grounding, target grounding, and spatial multiple-choice questions, which generally reach their lowest accuracies under Geometry.The findings attribute these failures to occlusion, clutter, and ambiguous spatial structure weakening localization and spatial-relation reasoning.
  • StressDART: StressDART improves over Qwen3-VL-4B in both rectified-only and original-plus-rectified settings without updating model parameters.The rectified-only setting provides most of the gain, indicating that targeted visual rectification is the main improvement source.

7 Conclusion · Appendix · A RoboStressBench Details

RoboStressBench is a physically grounded benchmark for evaluating VLM robustness under visual stress in embodied scenes. It organizes stress by four image-formation factors to enable interpretable diagnosis of model failures.

  • 7 Conclusion: RoboStressBench evaluates VLM robustness under visual stress in embodied scenes.The benchmark is explicitly designed for embodied-scene perception under stress.
  • 7 Conclusion: Its stress taxonomy comprises Material, Viewpoint, Lighting, and Geometry.These four dimensions are defined as image-formation factors.
  • 7 Conclusion: The four-factor organization makes model-failure diagnosis more interpretable than treating degradation as arbitrary image corruption.The comparison concerns interpretability of failure analysis, not benchmark accuracy.
  • 7 Conclusion: The benchmark is constructed using human-annotated filtering.Human annotation is one component of the benchmark-construction process.
  • 7 Conclusion: It also uses controlled stress synthesis to construct evaluation data.Controlled synthesis complements the benchmark’s filtering and collection procedures.
  • 7 Conclusion: Real-world data collection is the third stated component of benchmark construction.The passage identifies this alongside human-annotated filtering and controlled stress synthesis.

A.1 Data Sources … A.4 Controlled Stress Synthesis

RoboStressBench combines public, Internet-sourced, and self-collected images with expert annotation and normalized grounding labels. It supplements naturally occurring stress with controlled edits to Material, Viewpoint, Lighting, or Geometry while preserving task-relevant content and verifying annotation validity.

  • A.1 Data Sources: RoboStressBench draws on public benchmarks, Internet-sourced real-world images, and self-collected images under each source’s license and usage terms.
  • A.2 Annotation Protocol: Six trained annotators with embodied-AI and VLM expertise assign coarse dimensions and fine-grained stress tags, while verifying or revising task questions and answers.
  • A.2 Annotation Protocol: For Internet-sourced and self-collected images, a vocabulary-driven pipeline uses GroundingDINO and SAM to propose object annotations before manual inspection, stress labeling, and task-specific QA writing.
  • A.3 Grounding Annotation Normalization: Grounding point and bounding-box annotations are normalized to [0, 1000], and models are prompted to output coordinates in that shared range.
  • A.4 Controlled Stress Synthesis: Controlled synthesis edits one intended physical factor—Material, Viewpoint, Lighting, or Geometry—of a nominal image while preserving task-relevant scene content.
  • A.4 Controlled Stress Synthesis: Each edit profile specifies the nominal image and annotation, target stress category, editing instruction, preservation constraints, and annotation-reuse policy.
  • A.4 Controlled Stress Synthesis: The synthesis pipeline uses region-guided preservation, language-only spatial editing, and appearance-factor editing, with grounding labels reused only after alignment verification or replaced through re-annotation.

A.5 Dataset Statistics · B Additional Experimental Details

RoboStressBench comprises 7,183 examples assembled from filtered existing data, controlled stress synthesis, and additional real-world sources. It spans four physical stress dimensions and five embodied evaluation tasks targeting complementary capabilities.

  • A.5 Dataset Statistics: 7,183 total examples combine 2,927 filtered unconstrained-dataset examples, 2,596 controlled stress-synthesis examples, and 1,660 additional real-world examples.The real-world sources include Internet-sourced images and images captured by the authors, enabling naturally occurring and controlled high-stress cases.
  • A.5 Dataset Statistics: The benchmark contains 2,785 Material, 1,292 Viewpoint, 1,753 Lighting, and 3,327 Geometry stress examples.Material tags include dark absorptive, low-contrast blend, complex texture, transparent, and specular-confusion cases; Viewpoint tags include extreme-viewpoint, truncated-out-of-frame, and small-scale cases.
  • A.5 Dataset Statistics: Grounding evaluation reports point-based grounding accuracy and box-based IoU@0.50, IoU@0.95, and mean accuracy over IoU thresholds from 0.50 to 0.95.The mean uses a step of 0.05 and follows the standard COCO-style protocol.
  • A.5 Dataset Statistics: Per-tag counts are reported independently, so their sum may exceed the total number of examples.This reflects overlapping stress tags rather than an inconsistency in the dataset total.
  • A.5 Dataset Statistics: The benchmark includes 949 placement-grounding, 3,411 target-grounding, 1,369 spatial-reasoning, 633 state-understanding, and 821 planning examples.The latter three are multiple-choice tasks, while the first two evaluate grounding capabilities.
  • A.5 Dataset Statistics: These tasks evaluate object localization, target grounding, spatial relation reasoning, object-state understanding, and high-level planning.Together, the task types are designed to assess complementary embodied capabilities.

B.1 Detailed Grounding Results

RoboStressBench evaluates grounding through point-based and box-based tasks, using mask inclusion for point correctness and COCO-style IoU metrics for boxes. Its overall grounding score averages point accuracy with box-based IoU@0.95.

  • Grounding Evaluation: Point-based grounding is correct when the predicted point falls inside the ground-truth mask.Box-based grounding reports IoU@0.50, IoU@0.95, and mean accuracy over thresholds from 0.50 to 0.95 in 0.05 increments.
  • Grounding Evaluation: The grounding score averages point-based grounding accuracy and box-based IoU@0.95.

B.2 Compute Resources

RoboStressBench experiments use inference-only evaluation with fixed decoding settings and official implementations or APIs. Open-source evaluation requires substantial GPU time, with additional cost for StressDART’s image rectification.

  • Evaluation setup: All experiments run inference-only, without model fine-tuning or parameter updates, using official implementations for open-source VLMs on 8 NVIDIA H100 GPUs.Each H100 provides 80 GB memory.
  • Evaluation setup: Open-source models use deterministic greedy decoding with a 64-token maximum, temperature = 0.0, top-p = 1.0, and approximately 48 GPU-hours total.Sampling is disabled.
  • StressDART and closed-source models: StressDART combines Qwen3-VL-4B with Qwen-Image-Edit, adding approximately 150 GPU-hours for the evaluated subset, while closed-source models use official APIs without local GPU resources.The image-editing rectification step creates the additional test-time cost.

C Limitations … F Licenses

RoboStressBench is presented as a transparent, diagnostic benchmark whose taxonomy, data construction, task coverage, and StressDART intervention retain important limitations. The paper outlines broader-impact risks, mitigation guidance, and source-license requirements for responsible use and release.

  • C Limitations: The Material–Viewpoint–Lighting–Geometry taxonomy is physically grounded and interpretable but does not exhaust real-world visual difficulty, and its stress axes are not perfectly orthogonal.RoboStressBench supports multi-label stress annotation, but real scenes can combine multiple factors.
  • C Limitations: Human filtering, controlled synthesis, and real-world collection balance realism, diversity, and controllability but may introduce source bias and generative-editing artifacts.Manual verification and re-annotation reduce these risks, but synthetic examples cannot fully replace naturally occurring phenomena.
  • C Limitations: The benchmark evaluates image-based VQA and grounding rather than fully capturing closed-loop behavior, long-horizon interaction, or temporal robustness in dynamic scenes.The paper proposes extending evaluation to video, multi-view interaction, and real robot execution.
  • C Limitations: StressDART is an initial test-time intervention, and negative flips can occur when editing changes task-relevant cues or diagnosis mismatches the true failure mode.Future directions include more reliable stress detectors, content-preserving rectification, and reasoning improvements.
  • D Broader Impacts: RoboStressBench can help diagnose unreliable VLM perception in challenging environments such as low illumination, occlusion, reflective materials, unusual viewpoints, and clutter.The benchmark exposes failures under physically plausible stress through stress annotations and task-level evaluation.
  • D Broader Impacts: VLM hallucinations, object mislocalization, and overconfidence under severe ambiguity may cause unsafe robotic manipulation, navigation, or planning decisions.StressDART may improve robustness through rectification, but editing can also alter task-relevant visual cues.
  • E Impact Mitigation Measures: The release is intended for diagnosing VLM failures under realistic physical stress, not surveillance, biometric identification, or high-stakes automated decisions.StressDART is described as an exploratory test-time strategy rather than a complete safety mechanism.
  • F Licenses: RoboStressBench combines existing public benchmarks, Pexels-sourced images, and controlled synthesis while retaining each source’s license and usage terms.Annotations, metadata, and construction code may use a research license, whereas images and derived assets remain subject to their corresponding source terms; listed sources include CC BY 4.0 and Apache 2.0 datasets.
Loading 2606.00828v1…