PaperlayerView exact revision on arXiv

Source-linked AI summary

LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios

Hanjing Zhou, Mingze Yin, Ying Lian, Jun Ma, Chang-Yu Hsieh, Yanbing Zhou

arXiv:2609.09790v1cs.CVcs.AIcs.CL

TL;DR

Limited evaluation of LMMs in industrial scenarios leaves their hazard-oriented capabilities insufficiently established. LogiScope-VQA benchmarks logistics safety across perception, understanding, and risk reasoning; evaluations find models remain far below logistics experts, with fine-grained perception a shared bottleneck.

  • Problem

    Evaluation of LMMs in industrial scenarios remains limited, despite the distinctive safety-oriented requirements and dense target distributions of these environments.

    Source: §1 · p. 1

  • Method

    The paper introduces LogiScope-VQA, a logistics-safety benchmark assessing industrial-element perception, warehouse-knowledge understanding, and potential-risk reasoning.

    Source: §1 · p. 3, §6 · p. 10

  • Results

    Models remain far below logistics experts, with fine-grained perception a shared bottleneck and conservative risk over-reporting prevalent.

    Source: §3.2.1 · p. 7, §3.2.3 · p. 8

  • Takeaways & Limitations

    LogiScope-VQA provides actionable guidance for assessing the real-world readiness of LMMs developed for large-scale industrial deployment.

    Source: §1 · p. 3

    Models struggle with fine-grained perception in low-resolution, wide-angle, heavily occluded, and densely cluttered surveillance footage.

    Source: §4 · p. 9

Abstract

from arXiv · show

Large Multimodal Models (LMMs) large-scale deployment in industrial warehouse settings specifically necessitates that models exhibit human-expert-level hazard-oriented perception, understanding, and reasoning capabilities. However, the scarcity of real industrial data, tightly coupled to commercial terms, significantly hampers further advancement. To bridge this gap, we curate LogiScope-VQA to investigate the practical applicability of mainstream LMMs in real-world logistics operations. LogiScope-VQA comprises 2,476 images and 2,918 videos primarily sourced from real-world logistics parks, along with 10,274 VQAs meticulously curated and validated by human annotators. Grounded in 18 core objects and 20 risk types, we devise 39 subtasks aligned with three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Furthermore, we incorporate dynamic thinking-budget configurations and dual-dimensional risk bias analyses to elucidate the properties of LMMs. Extensive experiments unveil that even powerful proprietary models, including GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7, exhibit a significant gap relative to human performance. The unique challenge of jointly integrating perception, understanding, and reasoning for hazard identification poses substantial headroom for further improvement on LogiScope-VQA. We additionally reveal the pervasive security bias issue that impedes LLMs' practical deployment in real-world settings. The industrial dataset is publicly available under the CC BY-NC-SA 4.0 license.

1. Introduction

LogiScope-VQA addresses the limited evaluation of LMMs in real-world logistics, where safety-critical tasks and dense scenes challenge practical deployment. It benchmarks perception, warehouse knowledge understanding, and risk reasoning, and finds substantial gaps and response biases in current models.

Motivation

  • Industrial logistics evaluation remains constrained by simulated data, narrow single-format task coverage, and limited assessment of the safety-critical breadth required for deployment.The introduction notes that IndustryEQA uses virtual simulations, while ARMBench and iSafetyBench assess specific tasks under a single question format.Source: §1 · pp. 1–2

Benchmark contribution

  • LogiScope-VQA comprises 2,476 images, 2,918 videos, 1,0274 VQAs, 20 risk factors, and 18 logistics objects across three competency dimensions.Its dimensions are industrial element perception, warehouse knowledge understanding, and potential risk reasoning.Source: §1 · p. 2, Figure 1 · p. 2
  • The benchmark combines 3.5 million Cainiao surveillance clips, logistics-expert annotations developed over 52 person-days, and open-ended risk questions evaluated with an LLM-as-a-Judge protocol.The expert annotations cover spatial locations, object attributes, and hazardous actions; the reasoning protocol assesses reliability and fairness.Source: §1 · p. 3

Key findings

  • Qwen3.5-Plus and other open-source LMMs closely track proprietary performance, including parity with GPT-5.5 and Claude-Opus-4.7, yet remain markedly below human experts.The authors report a pronounced performance disparity relative to human experts, indicating substantial room for improvement.Source: §1 · p. 3
  • Open-source models average 0.39 accuracy on FG-S and 0.36 on FG-C, while perception and warehouse-knowledge tasks rely mainly on visual cues and risk reasoning requires sophisticated thinking.LMMs consistently underperform expectations on fine-grained industrial element perception tasks.Source: §1 · p. 3
  • Only 25% of LMMs pass the fairness audit, with proprietary models tending toward conservative risk predictions and open-source models toward more permissive, candid reports.The authors characterize this pattern as pervasive risk-averse bias in responses to risk-prediction queries.Source: §1 · p. 3

2. LogiScope-VQA

LogiScope-VQA is a real-world logistics-safety VQA benchmark designed to assess LMM perception, warehouse knowledge, and risk reasoning. Its coverage and hierarchical curation distinguish it from narrower or simulated industrial benchmarks.

2.1. Overview of Logistics Benchmark

  • LogiScope-VQA contains 5,394 visual samples and 10,274 VQA pairs, covering 18 core objects, 20 risk types, and perception, warehouse knowledge, and risk reasoning.Its tasks also challenge dense-object perception, logistics-specific knowledge, multimodal reasoning, and anticipation of potential safety risks.Source: §2.1 · p. 4

2.2. Distinctive Positioning of LogiScope-VQA

  • Unlike IndustryEQA (Li et al., 2026), which uses simulated industrial data, and iSafetyBench (Abdullah et al., 2025), which focuses on hazardous actions, LogiScope-VQA uses real warehouses and broader multi-dimensional evaluation.The paper describes LogiScope-VQA as nearly tenfold the size of the previous dataset.Source: §2.2 · pp. 4–5

2.3. Dataset Curation Pipeline

  • A five-step pipeline distilled 3.5 million Cainiao warehouse clips into 5,394 visual samples and 10,274 VQA pairs through targeted editing, template-generated questions, expert review, and category balancing.Basic annotation took 12 person-days; specialist answer annotation and three-round cross-validation took 40 person-days, and approximately 90% of the data was retained as high-quality VQA pairs.Source: §2.3 · p. 5

3. Experiment

LogiScope-VQA exposes a substantial gap between leading models and logistics experts, especially in fine-grained perception; thinking-mode benefits vary by task, and safety judgments show pervasive conservative bias.

3.1. Evaluation Setups

  • The evaluation compares proprietary and open-source LMMs with chance-level, prior-based, novice-human, and expert-human references using standardized answer matching and triplicate averaged micro-accuracy.Ten students without logistics expertise represented novice performance, while a logistics specialist with over four years of warehouse safety experience represented expert performance. Open-ended answers were matched through an LLM-as-a-Judge workflow; the Frequent Guess baseline used each task category’s most frequent canonical answer.Source: §3.1 · pp. 5–7

3.2.1. Main Performance Analysis

  • Proprietary models retain an advantage, but the best open-source model ties the strongest proprietary systems; leading models exceed novice students yet trail the expert, with fine-grained perception a shared bottleneck.Models also outperform untrained humans on knowledge-intensive tasks such as AR and EOC, while wide-angle, low-resolution, cluttered surveillance imagery makes perception especially challenging.Source: Table 1 · p. 6, §3.2.1 · p. 7

3.2.2. In-depth Thinking Analysis

  • Thinking mode has task-dependent effects: it benefits Potential Risk Reasoning most, while Industrial Element Perception is least responsive and reasoning-task gains saturate near the standard budget.Warehouse Knowledge Understanding shows modest gains; low budgets severely degrade performance through output truncation, while gains beyond the standard budget are marginal, motivating selective use and a budget near saturation.Source: §3.2.2 · pp. 7–8, Table 2 · p. 8, Figure 4 · p. 8

3.2.3. Safety Risk Bias Analysis

  • Most models exhibit conservative risk bias, while top proprietary models cluster near neutrality and open-source models span the risk-bias range.In paired hazard and risk-free samples, most models exceed 60% risk recall but false-alarm rates vary substantially; some models overreact, others miss threats, and even the optimal trade-off remains far from expert performance.Source: §3.2.3 · pp. 8–9, Figure 5 · p. 9

4. Challenges and Future Directions

LogiScope-VQA exposes two limitations in current models: weak fine-grained perception in difficult surveillance footage and a tendency to over-report safety risks. Future work should improve visual capacity and address bias at its source.

4. Challenges and Future Directions

  • Models struggle with low-resolution, wide-angle, heavily occluded, and densely cluttered surveillance footage, motivating stronger visual pre-training and use of temporal information across video frames.The passage proposes optimizing pre-training for dense and low-resolution targets or exploiting information across video frames.Source: §4 · p. 9
  • Most models are heavily inclined to over-report risks, so industrial applications need fairness as well as accuracy and bias prevention through balanced data curation or bias-aware training objectives.The passage identifies safety risk bias as a significant but overlooked issue.Source: §4 · p. 9

5. Related Work

LogiScope-VQA addresses the scarcity of real-world logistics safety evaluations with an expert-annotated benchmark for a domain that requires integrated perception, spatial reasoning, and hazard identification. It is positioned as the first dedicated evaluation suite for logistics scenarios.

5.1. LMM Industrial Assessment

  • Industrial safety assessment requires integrated object recognition, spatial reasoning, and hazard identification, while IndustryEQA (Li et al., 2026) has only about 1.3 thousand physics-simulated warehouse samples.LogiScope-VQA provides 10,274 VQA instances annotated by logistics experts as a practical industrial testbed.Source: §5.1 · p. 9

5.2. Practical Application Benchmarks

  • Logistics safety evaluation examines workflows, equipment states, human behaviors, and regulations as embodied risk-oriented QA in real industrial settings.LogiScope-VQA is presented as the first dedicated evaluation suite for logistics scenarios, filling a gap among targeted benchmarks for other domains.Source: §5.2 · p. 10

6. Conclusion · A. Large Language Model Usage · B. Ethics Statement

LogiScope-VQA evaluates LMMs’ industrial element perception, warehouse knowledge understanding, and risk reasoning, finding shortcomings in industrial visual perception, the need for sophisticated reasoning, and pervasive security-related bias. The authors also describe their use of LLMs and privacy safeguards for dataset release.

6. Conclusion

  • LMMs fall short in industrial visual perception, effective hazard identification depends on sophisticated reasoning, and current models exhibit pervasive security-related bias.The authors say these findings inform foundation-model construction and practical deployment.Source: §6 · p. 10
  • LogiScope-VQA evaluates industrial element perception, warehouse knowledge understanding, and potential risk reasoning in logistics hazard identification.Source: §6 · p. 10
  • Future work will enrich the visual question-answering corpus and extend the benchmark to agentic industrial operations.Source: §6 · p. 10

A. Large Language Model Usage

  • Human authors independently conceived the study and wrote the manuscript, using LLMs only at the final stage to polish academic terminology.Source: A. Large Language Model Usage · p. 13
  • About 20.5% of the dataset was generated with Nano Banana 2 to diversify risk factors; experiments used vendor-provided official API endpoints under applicable terms and licenses.Dataset construction primarily relied on proprietary real-world data.Source: A. Large Language Model Usage · p. 13

B. Ethics Statement

  • Privacy-sensitive content in private LogiScope-VQA data, including faces, warehouse names, and commercial identifiers, was blurred or masked through automated detection and manual verification.Source: B. Ethics Statement · p. 13
  • Synthetic data without real-world identities and publicly available open-source data were released without additional anonymization, and the dataset’s construction and release comply with relevant ethical and data-protection guidelines.Source: B. Ethics Statement · p. 13

C. Data Usage and Licensing … D.2. Detailed Risk Definitions

The curation pipeline transforms surveillance footage and logistics safety knowledge into verified, balanced VQA data through structured annotation and question generation. The dataset is restricted to non-commercial use, and the appendix identifies 20 risks used in potential-risk reasoning.

C. Data Usage and Licensing

  • LogiScope-VQA is licensed under CC BY-NC-SA 4.0 for academic and non-commercial research only; commercial use is prohibited, and derivatives must use the same license.Accessing the dataset signifies agreement to these terms.Source: C. Data Usage and Licensing · p. 13

D.1. Hierarchical Curation Pipeline

  • From 3.5 million Cainiao warehouse clips collected between April 2025 and April 2026, cleaning and deduplication produced over 7,000 unique visual samples; no U.S. surveillance data was included.The pipeline segmented videos into clips or keyframes, grouped them by camera zone, filtered frames using object detection, and deduplicated the results.Source: §D.1 · p. 13
  • Logistics specifications were encoded as hierarchical JSON tuples of object, attribute, and predefined value, covering 18 core warehouse-safety entities.Attributes capture object states or interactions, while predefined values constrain the attribute space to domain regulations.Source: §D.1 · p. 15
  • Annotators localized objects before labeling attributes, using the most salient risk keyframe for videos; person tags were extended, and a third-party audit corrected remaining inaccuracies.Labels were constrained to predefined knowledge-base values, and person annotations included gender and upper-garment color.Source: §D.1 · p. 15
  • Fifteen templates generated questions: 12 multiple-choice templates assessed scene, object, and interaction understanding, while three open-ended templates targeted risk perception and response capabilities.Rule-driven refinement added natural object descriptors and made minimal grammatical adjustments while preserving most query structures.Source: §D.1 · pp. 15, 18
  • Four annotators spent 40 person-days on answer labeling and verification, discarding ambiguous or malformed pairs and retaining 5,394 visual samples before final checks and dataset rebalancing.A senior expert re-annotated every pair, adjudicated disagreements, and conducted a final check; VQA pairs were rebalanced across hallucination proportions, object types, and hazard categories.Source: §D.1 · p. 18

D.2. Detailed Risk Definitions

  • The paper provides precise definitions for 20 specific risks involved in potential-risk reasoning, and Figure 8 is captioned as definitions of 20 risks in potential risk reasoning.Source: Figure 8 · p. 16, §D.2 · p. 18

D.3. Full Task Taxonomy · D.4. Prompt for Targeted Image Editing

The appendix presents the full task taxonomy and describes the targeted image-editing prompt template, including how its placeholders support diverse prompts.

D.3. Full Task Taxonomy

D.4. Prompt for Targeted Image Editing

  • The targeted image-editing prompt template uses multiple placeholders and includes a concrete usage example.Source: §D.4 · p. 18
  • The range of resulting prompts reflects the allowable values for each placeholder, which Fig. 11 comprehensively enumerates.Source: §D.4 · p. 18, Figure 11 · p. 19
  • Fig. 12 presents prompts for solving and evaluating questions.Source: Figure 12 · p. 20

E. Evaluation Details · E.1. More Decoding Settings

The evaluation uses distinct prompts for answering and judging, with a strict all-key-points criterion for short answers; documentation and inference settings are provided for reproducibility.

E.1. More Decoding Settings

  • Short-answer responses count as correct only when they contain every essential ground-truth point; omitting even one makes them incorrect.Additional elaboration is allowed, and the omission rule prioritizes high recall in hazard detection because missed industrial risks can have severe consequences.Source: §E.1 · p. 21
  • Fig. 12 shows separate prompt templates for answering multiple-choice and open-ended questions and for third-party LMM judging.The judge outputs “correct” or “incorrect.”Source: §E.1 · p. 21
  • For short-answer judging, the ground truth typically consists of several core key points, and extra response elaboration does not invalidate a correct answer.Source: §E.1 · p. 21
  • The omission rule is intended to prioritize high recall in hazard detection, where missed risks can have severe consequences.Source: §E.1 · p. 21
  • The authors describe the rigorous judging criterion as a stringent test of models’ capability and reliability in logistics safety surveillance.Source: §E.1 · p. 21
  • Tab. 4 provides official documentation or checkpoint links for evaluated LMMs, detailed inference configurations, and the evaluation timeframe.These details are included to improve transparency and reproducibility.Source: §E.1 · p. 21

E.2. Human Evaluation Protocol

Human performance was evaluated by contrasting generalist undergraduate volunteers with an experienced logistics safety expert.

E.2. Human Evaluation Protocol

  • The protocol compared 10 novice students with one logistics expert, whose results are labeled “Novice Student” and “Logistics Expert” in Tab. 1.Students collectively answered approximately 1,000 questions each over two weeks for $15 per person; the expert, with at least four years of hands-on experience, independently answered the full benchmark over two weeks at a total cost of $700.Source: §E.2 · p. 21

E.3. Reliability of LLM-as-Judge Evaluation · F. Visualization

The LLM judge showed strong repeatability and agreement with human experts on a 100-question sample. Visualization examples span real and synthetic warehouse scenes, while supplementary passages describe model inference configurations.

E.3. Reliability of LLM-as-Judge Evaluation

  • The reliability check randomly sampled 100 open-ended VQA questions to assess stability and alignment with expert judgments.The two dimensions were intra-judge repeatability and agreement between the LLM judge and human experts.Source: §E.3 · p. 21
  • The same LLM judge evaluated each question three times at temperature 0.7 to measure intra-judge agreement.Agreement was reported across the three runs.Source: §E.3 · p. 21
  • Human experts independently judged the same subset, and agreement with the LLM judge measured inter-judge alignment.The paper concludes that the judge is stable and aligned with human judgment.Source: §E.3 · pp. 21–22
  • Intra-judge repeatability exceeded 93%, and agreement with human experts exceeded 87% on 100 VQA pairs.Table 3 reports stability and human alignment; the same judge rated each question three times at temperature 0.7, while experts independently judged the same subset.Source: Table 3 · p. 21, §E.3 · pp. 21–22

F. Visualization

  • Figure 13 provides raw-image examples of real and synthetic visual data across daytime and nighttime scenes and varied warehouse zones.The examples are presented to facilitate a more intuitive understanding.Source: F. Visualization · p. 22, Figure 13 · p. 23
  • Inference configurations covered proprietary and open-source LMMs; listed Qwen models used a 32k maximum length, batch size 1, and PyTorch backend.The listed configurations also specify temperature 0.2, maximum new tokens of 4096/256 for think/non-think, and an evaluation date of June 25–July 22, 2026.Source: Table 4 · p. 24, F. Visualization · p. 24

Related papers

Loading 2609.09790v1…