PaperlayerView exact revision on arXiv

Source-linked AI summary

FloodReasonBench: Benchmarking VLM Reasoning Segmentation for Embodied Flood Response at the Edge

Rajat Bhattacharjya, Yoomee Jung, Minwoo Kim, Sing-Yao Wu, Eli Bozorgzadeh, Nalini Venkatasubramanian, Nikil Dutt

arXiv:2608.15410v1cs.DCcs.AIcs.CVcs.ROeess.SY

TL;DR

Flood-response VLM benchmarks overlook domain-specific scenes and edge-resource constraints, limiting evaluation of reasoning segmentation for embodied response. FloodReasonBench addresses this with a flood-specific dataset and system-level assessment, finding that adaptation improves accuracy while reducing partition-dependent variation and exposing deployment tradeoffs.

  • Problem

    Existing reasoning-segmentation benchmarks focus largely on generic scenes and omit flood-response conditions and constrained edge deployment requirements.

  • Method

    FloodReasonBench combines the FloodResponseSeg dataset with lightweight visual encoding, hierarchical split inference, compressed features, and embedded evaluation.

  • Results

    Flood-specific adaptation raises gIoU from 0.7223 to 0.8423 and cIoU from 0.7385 to 0.9013, while adapted partition accuracy remains comparatively compact.

  • Takeaways & Limitations

    The benchmark supports quality-constrained selection of practical edge operating points for embodied flood-response perception.

Abstract

from arXiv · show

Reasoning segmentation enables vision-language models (VLMs) to translate mission-relevant language requests into pixel-level visual grounding, offering a natural perception interface for embodied agents. However, existing benchmarks largely focus on generic visual scenes and overlook the domain and resource constraints encountered in flood-response platforms. We present FloodReasonBench, a benchmark for VLM reasoning segmentation for embodied flood response at the edge. At its core, FloodReasonBench introduces FloodResponseSeg, a flood-specific reasoning-segmentation dataset constructed from real-world scenes and response-relevant targets. Beyond task accuracy, the benchmark characterizes reasoning-segmentation pipelines under lightweight visual encoding, hierarchical split inference, and compressed intermediate representations. We observe strong partition-dependent accuracy variation in the generic pre-adaptation setting, while the flood-adapted target-workload design space exhibits a substantially more compact accuracy range across partitions. Evaluation on an NVIDIA Jetson AGX Xavier further exposes the tradeoffs among reasoning-segmentation accuracy, edge-side latency, energy, and communication footprint, enabling quality-constrained selection of edge operating points. Together, these results provide a task- and system-level characterization of reasoning segmentation for resource-constrained embodied flood response at the edge.

I. INTRODUCTION

FloodReasonBench addresses the gap between generic reasoning-segmentation benchmarks and flood-response perception by introducing a flood-specific workload and evaluating its deployment under edge-resource constraints. It jointly characterizes domain adaptation, partitioned inference, and quality-constrained operating points for embodied flood-response platforms.

  • Motivation: Flood-response platforms require associating mission-relevant language requests with corresponding physical regions, beyond recognizing objects in images.This perception interface supports autonomous platforms operating where human responders may face unsafe or difficult access.
  • Problem: 0.7223 gIoU and 0.7385 cIoU are achieved by off-the-shelf LISA on the flood-response workload without flood-specific fine-tuning.Existing benchmarks largely use generic visual scenes, creating a consequential domain gap for flood response.
  • Contributions: FloodReasonBench introduces FloodResponseSeg, a flood-specific reasoning-segmentation dataset built through semi-automated curation, grounding, segmentation, and quality control.The benchmark couples this workload with systematic evaluation of lightweight and partitioned visual inference under embedded-resource constraints.
  • Contributions: Hierarchical split reasoning segmentation reveals strong partition sensitivity on generic ReasonSeg but a substantially more compact partition-quality range for the flood-adapted target workload.The evaluation uses MobileSAM and learned intermediate-feature compression.
  • Contributions: Evaluation on an NVIDIA Jetson AGX Xavier jointly considers task quality, edge-side latency, energy, and transmitted feature size to identify quality-constrained operating points.Multiple platform power modes represent differing compute and power availability when perception shares the platform with other onboard functions.

II. FL O O DRE S P O NSESEG: FLOOD-RESPONSE REASONING SEGMENTATION DATASET

FloodResponseSeg is the flood-specific reasoning-segmentation dataset underlying FloodReasonBench. It targets language-conditioned visual grounding in flood scenes, complementing generic reasoning-segmentation and conventional flood-vision datasets.

  • Dataset motivation: FloodResponseSeg provides the flood-specific reasoning-segmentation dataset underlying FloodReasonBench.It is designed for the benchmark’s language-conditioned visual grounding task.
  • Dataset motivation: The dataset addresses the gap between generic reasoning-segmentation datasets and flood-vision datasets focused on classification, detection, or conventional segmentation.ReasonSeg represents generic visual scenes, while existing flood-vision datasets largely use non-language-conditioned vision tasks.
  • Dataset scope: The current release covers three response-relevant target categories: people, buildings, and vehicles.These categories are listed as the current release’s scope in the supplied passage.
  • Curation pipeline: FloodResponseSeg curation combines foundation-model-assisted filtering, localization, and mask generation with manual inspection and natural-language query authoring.The pipeline is semi-automated, with human inspection and query authoring integrated into dataset construction.

A. Dataset Curation · B. Dataset Composition and Augmentation · III. EDGE REASONING-SEGMENTATION BENCHMARK DESIGN

FloodResponseSeg is curated from real-world flood imagery through semantic filtering, manual review, localization, mask generation, and annotation quality control. FloodReasonBench combines this workload with augmented training data and a hierarchical edge pipeline spanning MobileSAM, compressed split execution, multimodal reasoning, and mask decoding.

  • A. Dataset Curation: The curation pipeline collects real-world flood imagery and applies CLIP-based semantic filtering to identify images containing target categories.Candidate images are manually inspected before target localization with Grounding DINO and mask generation with SAM2.
  • A. Dataset Curation: Manual visual quality inspection rejects unsuitable generated annotations before accepted image–mask pairs enter the dataset.Quality control follows localization and SAM2 mask generation.
  • B. Dataset Composition and Augmentation: 2,128 training instances comprise 532 original annotated samples and three photometrically augmented variants per sample.Augmentations include color, brightness, contrast, blur, noise, and sharpening while preserving target geometry and annotations.
  • B. Dataset Composition and Augmentation: The evaluation set contains 100 non-augmented samples, with Table I summarizing the classwise composition.The training and evaluation sets therefore use distinct augmentation policies.
  • III. EDGE REASONING-SEGMENTATION BENCHMARK DESIGN: FloodReasonBench evaluates how reasoning-segmentation pipelines can be structured for resource-constrained edge execution, beginning with an embodied platform acquiring a flood image.The benchmark design is summarized in Fig. 3.
  • III. EDGE REASONING-SEGMENTATION BENCHMARK DESIGN: MobileSAM’s TinyViT backbone supports selectable candidate partition blocks, with a learned latent encoder compressing intermediate features for transmission and remote reconstruction.The remaining backbone and projection process the reconstructed representation.
  • III. EDGE REASONING-SEGMENTATION BENCHMARK DESIGN: The MobileSAM visual pathway combines with CLIP representations and responder queries in a multimodal reasoning pathway whose <SEG> output drives the mask decoder.The decoder generates the requested target region.
  • III. EDGE REASONING-SEGMENTATION BENCHMARK DESIGN: Relative to the original SAM-based LISA configuration, the benchmark considers replacing the heavyweight SAM image encoder with MobileSAM and then partitioning it.These variants increase the edge orientation of the system design.

A. Lightweight LISA with MobileSAM

This section presents MobileSAM-LISA as a lightweight alternative to LISA’s computationally expensive SAM ViT-H encoder for embedded flood-response execution. Its hierarchical TinyViT backbone enables structurally diverse split-execution designs beyond the fixed early partition used by AVERY.

  • Lightweight encoder: MobileSAM replaces LISA’s computationally expensive SAM ViT-H image encoder with a substantially smaller TinyViT backbone while retaining the SAM segmentation pathway.This replacement targets embedded execution constraints without removing compatibility with LISA’s pixel-level segmentation process.
  • Hierarchical backbone: TinyViT’s hierarchical backbone contains 12 blocks across four stages: two MBConv blocks followed by ten transformer blocks.The two MBConv blocks are denoted MB1 and MB2, while the transformer blocks are indexed Blocks 0–9.
  • Hierarchical backbone: The TinyViT hierarchy changes computation, intermediate-representation dimensions, and representation characteristics across depths for split execution.These depth-dependent differences expand the available design choices for partitioning the model.
  • Baseline and split design: After adapting the pretrained TinyViT feature interface and flood-specific fine-tuning, MobileSAM-LISA becomes FloodReasonBench’s lightweight, non-split baseline.The configuration also exposes structurally distinct candidate cuts throughout the hierarchical backbone, unlike AVERY’s fixed early SAM ViT partition.

B. Hierarchical Split Inference and Feature Compression · C. Split-Aware Flood Adaptation

The framework evaluates hierarchical MobileSAM split points with partition-specific feature compression, then contrasts generic pre-adaptation behavior with split-aware flood adaptation. This design characterizes partition sensitivity for both standard ReasonSeg and the FloodResponseSeg target workload.

  • B. Hierarchical Split Inference and Feature Compression: The edge executes the TinyViT prefix, compresses its intermediate feature, transmits it, and remotely reconstructs it before completing the backbone and projection.The split pathway places a latent encoder at the selected partition and a corresponding decoder across the edge–remote boundary.
  • B. Hierarchical Split Inference and Feature Compression: Candidate cuts are evaluated after all 12 TinyViT blocks: MB1, MB2, and transformer Blocks 0–9.These partitions span the hierarchical TinyViT backbone.
  • B. Hierarchical Split Inference and Feature Compression: Hierarchical stages expose features with different spatial resolutions, channel dimensions, and semantic depths, so deeper partitions do not necessarily transmit smaller representations.Different stages can also respond differently to the same compression ratio.
  • B. Hierarchical Split Inference and Feature Compression: A separate autoencoder is trained for each candidate partition to match its intermediate feature’s dimensions and characteristics.This partition-specific design addresses variation in representation structure across TinyViT stages.
  • B. Hierarchical Split Inference and Feature Compression: Each autoencoder forms an intermediate-feature bottleneck while the TinyViT backbone remains fixed and is trained independently through feature reconstruction.The compression module is kept separate from subsequent task adaptation to measure partition sensitivity and construct split-aware flood-adapted operating points.
  • C. Split-Aware Flood Adaptation: Compression-induced reconstruction error can vary by partition because stages preserve different task-relevant information and feature characteristics.Accordingly, every MobileSAM partition is evaluated under pre-adaptation and split-aware flood adaptation.
  • C. Split-Aware Flood Adaptation: In pre-adaptation, each trained autoencoder is inserted without flood-specific split adaptation and evaluated on the standard ReasonSeg validation set.This setting characterizes generic partition sensitivity under intermediate-feature compression.
  • C. Split-Aware Flood Adaptation: In split-aware flood adaptation, each autoencoder remains fixed while reasoning-segmentation components are fine-tuned with compressed reconstructed features on FloodResponseSeg.The resulting configurations are evaluated on FloodResponseSeg to characterize the partition design space for flood response.

IV. EXPERIMENTAL METHODOLOGY · A. Model and Training Setup

The methodology evaluates FloodReasonBench at both task and system levels, covering flood-domain transfer, lightweight encoding, hierarchical split execution, and embedded computation and communication footprints. It uses LISA-7B with original SAM ViT-H and MobileSAM TinyViT configurations, plus flood-specific fine-tuning and partition-specific autoencoders.

  • IV. EXPERIMENTAL METHODOLOGY: The evaluation spans task-level transfer and execution sensitivity alongside system-level computation and communication measurements on an NVIDIA Jetson AGX Xavier.Task-level analysis covers flood-domain transfer, lightweight visual encoding, and hierarchical split execution; system-level analysis measures each MobileSAM partition.
  • A. Model and Training Setup: LISA-7B serves as the base reasoning-segmentation model, using SAM ViT-H in the original configuration and MobileSAM TinyViT in the lightweight configuration.The lightweight configuration replaces the original SAM ViT-H image encoder with MobileSAM TinyViT.
  • A. Model and Training Setup: The original LISA-SAM checkpoint is evaluated directly on FloodResponseSeg without flood-specific fine-tuning to quantify the flood-domain gap.This evaluation uses the FloodResponseSeg evaluation set.
  • A. Model and Training Setup: Flood-specific adaptation fine-tunes LISA-SAM and LISA-MobileSAM on the 2,128-instance FloodResponseSeg training set with a learning rate of 1e-4 and effective batch size four.The two configurations use a common fine-tuning setup; the passage also states that the TinyViT backbone remains frozen during MobileSAM-LI.
  • A. Model and Training Setup: A separate autoencoder is trained for each candidate TinyViT partition to support split execution.The autoencoders follow AVERY’s feature-compression setup and keep the TinyViT backbone and feature-alignment components fixed.
  • A. Model and Training Setup: The autoencoders minimize intermediate-feature reconstruction error on ADE20K rather than the downstream segmentation objective.ADE20K validation data monitor reconstruction error, while the passage also references ReasonSeg validation data.

B. Accuracy Evaluation · C. Embedded Platform and System Metrics

FloodReasonBench evaluates reasoning segmentation with gIoU as its primary quality metric and cIoU as a complementary measure, progressing from flood-domain adaptation to split-aware edge design. Embedded profiling on an NVIDIA Jetson AGX Xavier jointly characterizes latency, energy, and compressed representation size across partition and resource configurations.

  • B. Accuracy Evaluation: gIoU averages IoU across samples and serves as the primary task-quality metric, while cIoU aggregates intersections and unions as a complementary measure.cIoU is more influenced by large-area targets.
  • B. Accuracy Evaluation: Accuracy evaluation compares original LISA-SAM before and after flood-specific adaptation, then compares adapted unsplit LISA-SAM with adapted unsplit LISA-MobileSAM.
  • B. Accuracy Evaluation: The benchmark next examines generic MobileSAM partition sensitivity on ReasonSeg and the final split-aware flood-adapted partition landscape on FloodResponseSeg.
  • B. Accuracy Evaluation: Quality-constrained exploration uses a user-defined minimum gIoU requirement to expose candidate operating points.The flood-adapted gIoU and cIoU results are jointly analyzed with embedded latency, energy, and compressed representation size.
  • C. Embedded Platform and System Metrics: Embedded profiling evaluates MobileSAM split configurations on an NVIDIA Jetson AGX Xavier at 1024 × 1024 resolution using FP16 inference with CR = 0.1.The sweep includes MAXN, 10 W, 15 W, and several 30 W Jetson power and CPU configurations.
  • C. Embedded Platform and System Metrics: For each candidate partition, the study measures edge-side latency, average power, energy per frame, and compressed representation size transmitted to the remote processor.Energy per frame is computed from latency and average-power measurements.
  • C. Embedded Platform and System Metrics: These metrics quantify onboard execution and compression cost, communication footprint, and task–compute–communication tradeoffs across the hierarchical partition design space.Latency and energy capture TinyViT-prefix execution and intermediate-representation compression, while compressed feature size captures communication footprint.

V. RESULTS · A. Flood-Domain Adaptation and Lightweight Reasoning Segmentation

The results evaluate FloodReasonBench from task and system perspectives, emphasizing flood-specific adaptation, lightweight visual encoding, and partition-level analysis. Flood-specific adaptation substantially improves reasoning-segmentation accuracy, while MobileSAM preserves much of that accuracy with a lighter backbone.

  • V. RESULTS: FloodReasonBench evaluates both task accuracy and system tradeoffs for reasoning segmentation in flood-response settings.The evaluation considers lightweight visual encoding, partition landscapes, edge-side latency, energy, and compressed representation size.
  • A. Flood-Domain Adaptation and Lightweight Reasoning Segmentation: 0.8423 gIoU and 0.9013 cIoU are achieved by flood-specific adaptation, versus 0.7223 gIoU and 0.7385 cIoU without fine-tuning.These correspond to gains of 12.0 and 16.28 points, respectively.
  • A. Flood-Domain Adaptation and Lightweight Reasoning Segmentation: Flood-specific adaptation increases accuracy under both per-image gIoU and cumulative cIoU metrics.The reported gains are 12.0 points for gIoU and 16.28 points for cIoU.
  • A. Flood-Domain Adaptation and Lightweight Reasoning Segmentation: 0.8202 gIoU and 0.8572 cIoU are obtained with MobileSAM after adaptation, reducing adapted LISA-SAM performance by 2.21 gIoU points and 4.41 cIoU points.MobileSAM replaces SAM ViT-H with the substantially lighter TinyViT visual backbone.
  • A. Flood-Domain Adaptation and Lightweight Reasoning Segmentation: MobileSAM preserves much of flood-response reasoning-segmentation accuracy while enabling the hierarchical split design space studied next.The comparison is made after flood-specific adaptation.
  • A. Flood-Domain Adaptation and Lightweight Reasoning Segmentation: 0.8081–0.8240 gIoU and 0.8304–0.8547 cIoU are achieved by the original SAM split configuration across CRs of 0.25, 0.10, and 0.05.This provides a SAM-based compressed split-inference reference for the subsequent MobileSAM study.

B. Partition Sensitivity and Split-Aware Flood Adaptation

The benchmark reveals pronounced partition sensitivity before adaptation but a substantially narrower accuracy range after split-aware flood adaptation. Because multiple partitions become quality-comparable on the flood workload, edge latency, energy, and compressed representation size guide operating-point selection.

  • Pre-adaptation partition sensitivity: Before adaptation, gIoU spans 0.5322–0.7146 and cIoU spans 0.6448–0.8303 across 12 MobileSAM partitions on ReasonSeg.These correspond to 18.24-point and 18.55point spreads, respectively, with pronounced non-monotonic variation.
  • Split-aware flood adaptation: After split-aware flood adaptation, FloodResponseSeg accuracy narrows to gIoU 0.7599–0.8025 and cIoU 0.7840–0.8386 across the 12 partitions.The corresponding spreads are 4.26point for gIoU and 5.46point for cIoU.
  • Deployment operating points: With comparable reasoning-segmentation accuracy across multiple candidate partitions, edge latency, energy, and compressed representation size become important selection dimensions.These dimensions distinguish quality-feasible operating points in the target deployment setting.

C. Quality-Constrained Edge Deployment Tradeoffs

Flood-adapted edge deployment selects among quality-feasible reasoning-segmentation partitions by balancing accuracy against latency, energy, and communication costs. Both TinyViT partition depth and Jetson operating mode materially affect these resource costs.

  • Resource tradeoffs: Deeper TinyViT partitions progressively increase edge-side latency and energy, while communication footprint varies non-monotonically with hierarchical feature dimensions.At CR = 0.1, compressed representation size ranges from 0.750 MiB at MB1 to 0.125 MiB across Blocks 1–6, then increases to 0.250 MiB across Blocks 7–9.
  • Quality-constrained selection: Given Qmin, partitions with gIoU ≥Qmin form the quality-feasible set, enabling selection by latency, energy, and communication costs.Fig. 6 evaluates flood-adapted gIoU alongside edge-side latency and energy under the 30 W ALL configuration.
  • Quality-constrained selection: With Qmin = 0.79, Blocks 1, 7, 8, and 9 qualify, and Block 1 achieves 0.7941 gIoU with 50.1 ms latency and 0.461 J per frame.Block 1 also has a 0.125 MiB compressed representation and is lowest-cost among these quality-feasible points.
  • Platform operating mode: At Block 1, the 10 W mode requires 142.8 ms and 0.947 J per frame, compared with 50.1 ms and 0.461 J under 30 W ALL.Thus, platform operating mode and model partition both affect the resource cost of satisfying a perception-quality requirement.

VI. CONCLUSION AND FUTURE WORK

FloodReasonBench is introduced as a task- and system-level benchmark for VLM reasoning segmentation in resource-constrained embodied flood response, combining FloodResponseSeg with systematic pipeline and embedded-execution characterization. Future work will broaden the dataset and models, platforms, and end-to-end evaluation settings.

  • Contributions: FloodReasonBench combines the flood-specific FloodResponseSeg dataset with characterization of lightweight visual encoding, hierarchical split inference, compressed intermediate representations, and embedded execution.The benchmark targets resource-constrained embodied flood response.
  • Findings: Flood-specific adaptation substantially improves task accuracy.
  • Future Work: Future work will extend FloodResponseSeg to additional response-relevant entities and scenarios.
  • Future Work: Future work will incorporate emerging reasoning-segmentation models and edge platforms, and evaluate distributed and closed-loop embodied execution under varying network conditions.
Loading 2608.15410v1…