Source-linked AI summary
Search-Based Metamorphic Testing of Vision-Language Models in Autonomous Underwater Robotic Software
Muhammad Yousaf, Aitor Arrieta, Shaukat Ali, Paolo Arcaini, Shuai Wang
TL;DR
VLM use in AUR software remains underexplored despite promising performance and difficult underwater conditions, creating a need to assess model suitability and dependability. MetaVLM uses multi-objective search-based metamorphic testing to find minimal image transformations that induce prediction failures, and it outperforms random search on BLIP and CLIP while revealing limitations and lessons for practice.
Problem
VLM use in AUR software is underexplored, despite promising performance with limited labeled data and the need to assess suitability for dependable deployment.
Method
MetaVLM applies NSGA-II to search for minimal compositions of six image transformations that maximize changes in VLM predictions, evaluating BLIP and CLIP against random search.
Results
MetaVLM is more effective than random search, selecting low-order metamorphic-relation compositions with higher violation rates and detecting more severe violations for both VLMs.
Takeaways & Limitations
Metamorphic testing offers a practical oracle for finding VLM failures without explicit ground-truth labels, while multi-objective search identifies failures under minimal transformations.
Takeaways & Limitations
Results may not generalize beyond the two VLMs and SeaClear images, and different sequential metamorphic-relation orders may produce different outcomes.
Abstract
from arXiv · showhide
Our industry partner focuses on quality assurance for industrial systems across multiple domains, including maritime systems, such as overwater vessels and autonomous underwater robots (AURs). Despite the strong performance of vision-language models (VLMs) in scene understanding, image captioning, and object recognition, their use in AUR software operating in underwater environments is underexplored. Therefore, in this context, it is important to evaluate the quality of VLMs for integration into AUR software and, so, automated software testing tools are needed to assess their suitability and improve their dependability. To this end, we propose a search-based metamorphic testing approach (MetaVLM) that identifies a minimal set of transformations on underwater images to induce incorrect model predictions, thereby revealing VLM failures. We employ NSGA-II as a multi-objective search algorithm and evaluate it over open-source VLMs, BLIP and CLIP, against a random search baseline. Results demonstrate the strengths and limitations of each VLM in the context of AUR software systems. Based on the results, we derive lessons for software engineering practitioners and researchers working on quality assurance of VLM-based software systems.
1 Introduction
AUR perception systems operate in challenging underwater environments, while VLMs offer promising performance with limited labeled data. The paper therefore develops MetaVLM to systematically test VLM robustness by searching for realistic image transformations that induce prediction failures.
- AUR perception software must process underwater images despite scarce labeled data and variations in motion, turbidity, and illumination.
- Industrial interest in deploying VLMs within AUR software creates a need for automated techniques that assess their suitability and dependability.
- MetaVLM searches for minimal transformation sequences that preserve image realism while maximizing prediction failures between source and transformed images.
- MetaVLM significantly outperforms random search in detecting VLM prediction failures, including higher violation rates from low-order metamorphic-relation compositions and more severe violations with minimal transformations.The evaluation uses NSGA-II on BLIP and CLIP, with random search as the baseline.
2 Industrial Context
Maritime systems are complex cyber-physical systems whose distributed software and emerging behaviors challenge industrial assurance. The paper situates AUR perception within this context, using underwater trash collection as a representative application requiring dependable detection.
- DNV AS provides independent classification, certification, and advisory services intended to safeguard life, property, and the environment across maritime system lifecycles.
- Maritime digitalization produces complex cyber-physical systems with software from multiple vendors, late integration, and behaviors such as interoperability that are difficult to verify.
- AURs support industrial tasks such as inspection, underwater repair, and trash collection, where reliable trash detection helps avoid operational damage to vegetation.
3 Proposed approach MetaVLM
MetaVLM searches for small, realistic compositions of underwater-image transformations that expose prediction violations in VLM-based perception. It models testing as multi-objective optimization over transformation selection and parameters, with prediction differences measured according to each VLM’s output format.
- 3.1 Approach Overview: Six equality-based metamorphic relations cover rotation, scaling, histogram equalization, downsampling, shear, and translation.The relations represent realistic underwater-image variations while preserving as much semantic information as possible.
- 3.1 Approach Overview: MR compositions apply multiple selected transformations sequentially because underwater images may experience several simultaneous factors.Examples include motion causing rotation, translation, and blur, or depth changes reducing brightness and contrast.
- 3 Proposed approach MetaVLM: MetaVLM minimizes transformation magnitude while maximizing differences between predictions on source and transformed images.These objectives seek subtle, realistic changes that produce large model-prediction differences.
- 3.2 Solution Encoding: The solution encoding uses 12 variables: six Boolean activation variables paired with six transformation-parameter variables.Selected relations are applied in a fixed sequential order to produce the final transformed image.
- Metamorphic Relation Magnitude: The transformation objective is a normalized distance from nominal parameter values, where smaller values represent subtler transformations.The formulation uses selected metamorphic relations, their nominal parameters, and maximum parameter deviations.
- Prediction Deviation: Prediction differences use object counts for BLIP and confidence scores for VLMs such as CLIP, reflecting their differing output formats.For BLIP, the objective compares predicted object counts; for confidence-based models, it compares class-confidence differences.
4 Experimental Setup
The evaluation tests MetaVLM against random search across research questions covering effectiveness, metamorphic-relation composition, and violation strength. It uses underwater images, hypervolume, violation rates, mean violation strength, and statistical significance tests.
- MetaVLM is compared with random search to assess search effectiveness in identifying VLM prediction failures.
- The benchmark contains 30 underwater images from SeaClear that BLIP classified correctly, isolating metamorphic-relation violations from pre-existing classification errors.
- Evaluation Metrics and Statistical Tests: Hypervolume evaluates the tradeoff between minimizing transformation magnitude and maximizing prediction differences, with higher values indicating better diversity and convergence.
- Evaluation Metrics and Statistical Tests: Violation rate measures the proportion of attempts producing failures for specific metamorphic-relation orders or compositions, using model-specific Boolean criteria and CLIP thresholds of 1%, 3%, or 5%.
- Evaluation Metrics and Statistical Tests: Mean violation strength quantifies severity through average object-count differences for BLIP or confidence-score differences for CLIP.
- Evaluation Metrics and Statistical Tests: Mann–Whitney U tests assess statistical differences, while Vargha–Delaney A12 reports effect sizes when p-values indicate significance below 0.05.
5 Results and analyses
MetaVLM generally outperformed random search in detecting VLM prediction failures, while its effectiveness and optimal transformations varied across models, thresholds, and metamorphic-relation compositions.
- 5.1 RQ1 – Comparison with Rand in terms of effectiveness: MetaVLM significantly outperformed Rand on 16 CLIP benchmarks, while Rand won once and the approaches did not differ elsewhere.The MetaVLM advantage had large effect sizes (Â12 > 0.71).
- 5.1 RQ1 – Comparison with Rand in terms of effectiveness: MetaVLM significantly outperformed Rand on 11 BLIP benchmarks, while Rand won on 7 and no significant difference appeared on 12.Overall, the advantage for BLIP was favorable but moderate.
- 5.2 RQ2 – Binary Violations: MetaVLM identified different high-violation compositions for each model: CLIP was especially affected by scaling and histogram equalization, whereas BLIP was most affected by scaling alone.For CLIP, MR2 and MR3 produced 21.6% and 24.8% of violations at the cited thresholds; for BLIP, composition {2} reached a 38.1% violation rate versus Rand’s 2.9%.
- 5.2 RQ2 – Binary Violations: MetaVLM concentrated on lower-order compositions and achieved higher violation rates with fewer transformations than Rand across the evaluated VLMs.For BLIP, first- and second-order compositions accounted for 74.6% and 17.6% of occurrences, with violation rates of 92.8% and 88.1%; higher orders collectively accounted for less than 3%.
- 5.3 RQ3 – Strength of Metamorphic Relation Violations: MetaVLM generally produced significantly higher violation strengths than Rand across metamorphic-relation orders for both CLIP and BLIP.For CLIP, MetaVLM’s strength increased from 0.05, 0.04, and 0.05 at order 1 to 0.14, 0.14, and 0.09 at order 6; BLIP peaked at 8.37 and 6.38 for orders 2 and 3.
6 Discussion and Lessons Learned
The discussion argues that small image transformations can expose VLM failures without ground-truth labels, and that guided search improves failure discovery over random search. It also emphasizes model-specific objective functions.
- 6 Discussion and Lessons Learned: Small numbers of transformations can cause incorrect VLM predictions, so perception modules require robustness testing beyond standard benchmarks.Metamorphic relations provide a practical oracle when underwater labels are scarce and expensive.
- 6 Discussion and Lessons Learned: MetaVLM consistently found more prediction failures than random search with minimal transformations, indicating that guided exploration of the transformation space is important for efficient testing.The discussion attributes differing violation degrees to model-specific objective functions for BLIP and CLIP.
7 Threats to Validity
The study’s validity is constrained by algorithm configurations and transformation ordering, limited VLMs and images, and the chosen violation measures.
- External validity: Results may not generalize beyond the two generic VLMs and SeaClear images, motivating evaluation with additional models, broader datasets, and other perception tasks.Different model output characteristics may produce different behavior under the same metamorphic-relation configurations.
- Internal validity: Sequential ordering of active metamorphic relations may change the final transformed images and results, requiring additional experiments.The study also used default NSGA-II and random-search parameter settings as a fairness choice.
- Construct validity: The selected violation measures constitute a construct-validity threat for evaluating the metamorphic-relation violations.
8 Related Work
Prior work has used image transformations and metamorphic testing for deep-learning and multimodal systems, but this study combines search-based and metamorphic testing for VLMs in AURs.
- Prior metamorphic testing: Earlier studies used environmental, image-level, background, object, reconstruction, and scenario transformations to test deep-learning systems.Examples include blur, rain, fog, channel permutation, scaling, background changes, and object insertion.
- Search-based metamorphic testing: Few studies combined search-based and metamorphic testing to generate test inputs and reveal faults, motivating the use of NSGA-II for expensive composition searches.The approach applies geometric transformations to generate underwater follow-up test images.
- Metamorphic testing for multimodal systems: Metamorphic testing has been applied to multimodal systems including image captioning, speech recognition, vision-language action models, visual entailment, and embodied AI.This work focuses on VLM testing in AURs using multiple transformations on input images.
9 Conclusion
The paper presents MetaVLM, a multi-objective search-based metamorphic testing approach for VLM-based trash identification in AURs. Compared with random search, MetaVLM more effectively detects failures while using few transformations, and the authors identify future extensions for maritime VLM testing.
- MetaVLM minimizes underwater-image transformations while maximizing changes in VLM predictions to detect failures in AUR trash-identification systems.The approach treats minimal image alteration and maximal prediction change as competing objectives.
- MetaVLM consistently outperforms random search by selecting low-order metamorphic-relation compositions with higher violation rates.
- The study proposes future work on specialized maritime VLMs, additional multi-objective search algorithms, and domain-specific underwater metamorphic relations.