PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Augmentation as Test Generation for Deep Learning-Based Image Retrieval Systems

Yehan De Silva, Anirudh Sridhar, Armin Lotfy, Nafiseh Kahani, Yvan Labiche, Ziyu Wang, Frank Ouyang, Clare Carty, Azalia Shamsaei

arXiv:2608.27502v1cs.SEcs.CV

TL;DR

Deep learning image retrieval systems need reliable testing under difficult image conditions, but manual labelling and oracle construction are costly. The paper reviews augmentation techniques and evaluates them as metamorphic test generators across embedding models, datasets, and four analytical dimensions. At the tested severity level, weather simulation and related techniques produced high embedding uncertainty and retrieval failure while preserving favorable realism and stability trade-offs, whereas GAN-based methods ranked low in realism.

  • Problem

    Reliable testing is difficult because image retrieval systems must handle challenging query conditions, while manual testing and labelled oracles are prohibitively expensive.

  • Method

    The paper conducts a systematic review and evaluates augmentation techniques as test generators for Amazon Titan and OpenCLIP across three datasets using similarity, uncertainty, semantic realism, and retrieval-failure metrics.

  • Results

    At the tested severity level, weather simulation and Dimension Reduction produced the highest embedding uncertainty and failure counts across both models, while GAN-based augmentation was among the lowest in realism.

  • Takeaways & Limitations

    The findings support selecting augmentation techniques by trading off fault-exposure effectiveness against input plausibility when constructing retrieval test suites.

  • Takeaways & Limitations

    Each technique was evaluated at a single hyperparameter setting, so the reported rankings are configuration-specific and may change across severity levels.

Abstract

from arXiv · show

Ensuring the reliability of deep learning-based image retrieval systems is a software engineering challenge. This paper presents a dual contribution: (1) a literature review of augmentation and generation techniques which resulted in the identification of 50 techniques which we organized into a ten-category taxonomy, and (2) a large-scale empirical study that evaluates these techniques as test generators for embedding-based image retrieval systems. Augmented images are embedded using Amazon Titan and OpenCLIP, and evaluated across four analytical dimensions: (1) embedding-space similarity, (2) embedding uncertainty measured via four estimators, (3) semantic realism scored by LLaVA, and (4) retrieval failure rate. Experiments are performed on three datasets: CIFAR-10, ImageNet-1K, and a dataset from an industrial partner (March Networks). Across all evaluated datasets and embedding models, and under the single severity level tested for each technique, weather simulation and SaSPA are the image augmentation/generation techniques that produce the highest embedding uncertainty and failure rates while maintaining a favorable balance between performance stability, visual realism, and augmentation effectiveness. The results we discuss are configuration-specific and may shift under milder or stronger perturbation settings. In contrast, GAN-based augmentation techniques are among the lowest in realism, indicating the presence of synthetic artifacts and perceptual inconsistencies that reduce their suitability to produce realistic test inputs. Overall, our findings provide practical guidelines for selecting augmentation techniques that maximize test diversity while preserving realistic image characteristics, thereby enabling the construction of comprehensive and effective test suites for image retrieval systems while reducing the cost of manual data labeling through the use of metamorphic testing.

1 INTRODUCTION

Image retrieval systems must remain reliable under difficult viewing and environmental conditions, but manual testing is prohibitively expensive. This study uses image augmentations as metamorphic test generators and evaluates their challenge, stability, and realism across multiple dimensions.

  • Motivation: Manual testing is prohibitively expensive for retrieval systems expected to return semantically correct results despite degraded or challenging query images.Relevant conditions include low light, rain, fog, glare, sensor noise, motion blur, compression artifacts, and resolution loss.
  • Metamorphic testing: Image augmentation creates scalable metamorphic tests by modifying queries while preserving category semantics, so changed-category retrievals identify faults without manual labels.Useful transformations must be strong enough to expose weaknesses but not so strong that they destroy the image’s semantic category.
  • Evaluation metrics: Retrieval failure count measures how often an augmented query retrieves an image from the wrong category, representing a violation of the metamorphic relation.Embedding similarity captures representation displacement, while embedding uncertainty characterizes instability under augmentation.
  • Research gap: The study addresses the lack of a systematic catalogue and multidimensional evaluation of augmentations for automated robustness testing of embedding-based retrieval models.Prior surveys focused on training-time accuracy improvements rather than embedding-space analysis, automated robustness testing, or semantic realism.
  • Study design: The empirical study evaluates Amazon Titan and OpenCLIP on CIFAR-10, ImageNet-1K, and March Networks data using four complementary metrics.The metrics cover embedding similarity, embedding uncertainty, semantic realism scored by LLaVA, and retrieval failures.

2 SYSTEMATIC LITERATURE REVIEW

The systematic literature review applies a reproducible, criteria-driven search and screening process to catalogue recent image augmentation research. It emphasizes quantitative evidence, methodological detail, and coverage of the modern augmentation landscape.

  • Review protocol: The review follows Kitchenham’s three-phase process: defining objectives and criteria, searching scholarly sources, and combining findings into a reported synthesis.Its objective is to identify and catalogue the landscape of image augmentation techniques, including generative methods.
  • Search strategy: The search covers Google Scholar, IEEE Xplore, ACM Digital Library, SpringerLink, ScienceDirect, Scopus, and manually reviewed relevant conference proceedings.The sources were selected for broad coverage of peer-reviewed computer-vision and image-retrieval research.
  • Search strategy: The database query combines synonyms for image augmentation, retrieval or recognition tasks, and deep-learning architectures across titles, abstracts, and keywords.The same logical structure was applied uniformly across databases for consistency and reproducibility.
  • Eligibility criteria: Only studies with quantitative augmentation evaluations and sufficient experimental detail were included, while studies lacking empirical evidence or methodological information were excluded.The criteria required measurable outcomes and details such as datasets, performance metrics, and model descriptions.
  • Scope and reporting: The review was limited to January 2020–September 2025 and followed PRISMA 2020 to improve transparency, completeness, and reproducibility.The time boundary reflects the emergence of diffusion-based synthesis and CLIP-based embedding methods.
  • Screening: The search identified 88 studies, after which 16 duplicates or conference-proceedings entries were removed before full-text screening.Three reviewers conducted the process between June and September 2025 and resolved inclusion disagreements through consensus.

3 RELATED WORKS AND BACKGROUND

The background frames image retrieval as embedding-based matching and metamorphic testing as relation checking without a known oracle. It also introduces augmentation families, image transformations, encoders, generative models, and domain translation as foundations for the study.

  • Related work: Earlier work examined augmentation effects on model training and representation quality, motivating a broader assessment of augmentation behavior across datasets and complementary measures.The study specifically emphasizes severe augmentation methods, diverse datasets, and multiple assessment criteria.
  • Image representation: Photometric augmentations modify pixel values, whereas geometric augmentations rearrange spatial positions in an image.Digital images are represented as RGB pixel matrices, with color also expressible in spaces such as HSV.
  • Image transformations: Affine transformations preserve collinearity, with flipping, rotating, shearing, and translating forming common geometric augmentation operations.Histogram equalization enhances contrast by redistributing pixel intensities across the available brightness range.
  • Deep learning architectures: CNNs build hierarchical spatial features through convolutions, while ViTs use self-attention over image patches to capture long-range dependencies before mapping images into embeddings.These architectures provide the encoding basis for learned visual representations.
  • Generative models: GANs synthesize images through generator–discriminator competition, while CLIP aligns image and text encoders so semantically similar inputs occupy nearby shared-embedding locations.OpenCLIP is an open-source CLIP reproduction trained on LAION datasets.
  • Domain translation: Domain translation changes attributes such as illumination, season, texture, or artistic style while preserving semantic content and spatial structure.Season transfer changes environmental conditions, and cycle consistency constrains round-trip translations to remain consistent with the original.
  • Retrieval and testing: Embedding-based retrieval returns the k database images closest to a query embedding, so semantically equivalent images should remain nearby despite viewpoint, lighting, or minor corruption.Metamorphic testing checks this expected relation across executions rather than relying on a known correct output.

4 AUGMENTATION TECHNIQUES

The paper organizes image augmentation techniques into ten categories according to their transformation principles and the image features they modify. It reviews techniques spanning spatial, appearance, corruption, selection, filtering, mixing, and diffusion-based operations, while noting that excessive or domain-invalid transformations can impair recognition.

  • Taxonomy: The taxonomy groups augmentation techniques into ten categories, separating GAN-based and diffusion-based methods because they use distinct generative mechanisms.The categories include geometric transforms, photometric adjustments, noise injection, selection-based techniques, filtering, self-mixing, sample mixing, GAN-based synthesis, diffusion-based generation, and adversarial perturbations.
  • Transformation families: Geometric augmentations alter spatial configuration through flipping, rotation, shearing, and translation, whereas photometric augmentations modify pixel values such as brightness, contrast, color, and illumination.Geometric methods change position, orientation, or aspect ratio; photometric methods preserve spatial structure while changing visual tone and color.
  • Validity constraints: Excessive or domain-invalid geometric transformations can distort images, move key features out of frame, or create inaccuracies when spatial orientation is semantically important.The paper emphasizes that label preservation and validity depend on the application and perturbation strength.
  • Region and corruption operations: Selection-based methods modify chosen image regions, filtering applies convolutional operations, and noise injection introduces controlled perturbations to increase variability while preserving semantic structure.Selection methods may remove or emphasize regions, while filtering includes smoothing, sharpening, and edge enhancement.
  • Generative methods: Diffusion-based methods can edit appearance using textual prompts while preserving spatial layout and class-relevant semantic content.Pix2Pix-Zero modifies an input according to a specified transformation without additional training or manually defined masks; DiffuseMix blends generated variants with the original.

5 EMPIRICAL STUDY DESIGN

The empirical study evaluates image augmentations as test generators for embedding-based retrieval, measuring retrieval failures, embedding shifts and uncertainty, and semantic realism across models and datasets. It uses Amazon Titan and OpenCLIP on CIFAR-10, ImageNet-1K, and the March Networks dataset, with statistical modeling accounting for dataset, model, and category variation.

  • Study objective: The study evaluates whether augmentation techniques preserve the original image’s semantic category in embedding-based retrieval.A robust model should retrieve images from the original category after augmentation.
  • Research questions: Four research questions examine retrieval failures, embedding-space displacement, representation stability, semantic realism, and overall test-generation effectiveness.The evaluation combines raw fault exposure with embedding similarity, diversity, uncertainty, and visual plausibility.
  • Embedding models: Amazon Titan Multimodal Embeddings G1 and OpenCLIP ViT-H/14 provide commercial and open-source embedding systems for comparison.Both models have the same output dimensionality and are evaluated without fine-tuning.
  • Datasets: Experiments use CIFAR-10, ImageNet-1K, and the March Networks industrial-partner dataset.CIFAR-10 was chosen over CIFAR-100 because its larger per-class sample count supports more stable evaluation under the study configuration.
  • Statistical analysis: Retrieval failures are modeled with fixed effects for augmentation, embedding model, and dataset and a random effect for image category.The fitted model estimates each augmentation’s effect while accounting for category-specific unpredictability.
  • Evaluation metrics: Embedding uncertainty uses centroid dispersion, original–augmented pairwise distance, Mahalanobis distance, and disagreement across KNN classifiers.These estimators operate on output embeddings because Titan and OpenCLIP do not expose model internals.

6 RESULTS

Retrieval failures depend on augmentation technique, dataset and category, with several methods consistently challenging both embedding models. Embedding shifts and uncertainty help characterize difficulty, but realism and semantic preservation determine whether perturbations expose faults reliably.

  • Retrieval failure count: MixGen, RICAP, and KeepAug substantially modify visual representations while preserving enough class-discriminative information for correct retrieval.Large visual or embedding-level changes therefore do not necessarily produce retrieval failures.
  • Retrieval failure count: Weather simulation and Dimension Reduction offer a better overall tradeoff than NST and Shuffle because they combine high failure effectiveness with lower variability and relatively greater semantic realism.NST and Shuffle achieve high failure counts at the cost of low semantic realism.
  • Retrieval failure count: Shuffle, Neural Style Transfer, Weather, Dimension Reduction, and Shuffle-related analyses identify the strongest failure-inducing techniques across evaluated datasets and embedding models.Table 9 highlights Shuffle, NST, Weather, and Dimension Reduction; broader results identify NST, Weather, Dimension Reduction, and Shuffle as consistently failure-inducing.
  • Retrieval failure count: ImageNet generally yields lower failure counts, whereas CIFAR-10 and the MN dataset make semantic category preservation more challenging.Failure variation also depends strongly on image category and dataset characteristics.
  • Embedding-space similarity and diversity: Shuffle, Weather, and Dimension Reduction produce the largest embedding shifts and high retrieval failure counts, while PCA Jitter, Auto Contrast, and Gamma Correction produce high similarity and few failures.Embedding similarity should be considered together with retrieval failure behavior when assessing augmentation effectiveness.
  • Embedding model uncertainty: Mahalanobis distance and ensemble-based uncertainty significantly associate with failure count, whereas embedding dispersion and pairwise distance show weak, nonsignificant relationships.High uncertainty alone does not guarantee retrieval failure; the nature and direction of embedding displacement also matter.
  • Embedding model uncertainty: Shuffle, NST, Pix2Pix-Zero, Weather, and IP-Mix exhibit the highest uncertainty, but Pix2Pix-Zero, IP-Mix, and DIAGen do not consistently produce retrieval failures.Their embeddings can remain within class-consistent regions despite substantial feature-space variation, unlike perturbations that damage task-relevant structure.
  • Embedding model uncertainty: High uncertainty does not necessarily imply retrieval failure because both embedding-shift magnitude and displacement direction influence whether augmented inputs leave class-consistent regions.Semantic or generative techniques may preserve object identity while introducing substantial appearance-level diversity.

7 THREATS TO VALIDITY

The study identifies threats involving realism scoring, embedding-distance proxies, metamorphic assumptions, augmentation severity, dataset resolution, statistical testing, and review reliability. These constraints limit how broadly and confidently the reported rankings and realism comparisons should be interpreted.

  • Internal validity: Rephrasing LLaVA’s realism criteria changed the mean score by 2.82 of 18 (≈16%), so absolute realism values remain prompt-dependent.Reordering criteria changed scores by only 0.21 of 18 (≈1%); the analysis therefore relies on relative technique ordering.
  • External validity: Very low-resolution CIFAR-10 images make realism values difficult to interpret and prevent direct comparison with datasets of differing resolution.CIFAR-10 images are 32×32 pixels, and Qwen2.5-VL scores CIFAR-10 at 5.5 versus ImageNet at 15.57.
  • Construct validity: Embedding-space distance is only a proxy for semantic similarity and may not perfectly capture human-perceived similarity across all image categories.The proxy is considered valid when embedding models use contrastive objectives and diverse training data, as with Titan and OpenCLIP.
  • Metamorphic relation validity: For balanced sample mixing and strong transformations, failure counts can overestimate faults because the augmentation may introduce content or alter the image’s semantic category.Weather reduces CIFAR-10 original–augmented cosine similarity to 0.288, so changed retrievals may reflect genuinely different content.
  • Severity of augmentation: Each technique was tested at a single hyperparameter setting, so rankings may change under milder or stronger perturbations.A severity sweep across mild, moderate, and severe settings remains future work.
  • Reliability: The literature review did not formally quantify inter-rater agreement, and Amazon Titan depends on a cloud API.The supplied passage identifies formal agreement measurement as future work and begins an additional API-related limitation.
  • Conclusion validity: No statistical significance testing was applied, requiring caution when interpreting differences between closely ranked techniques.The paper recommends non-parametric pairwise tests with correction for multiple comparisons.

8 CONCLUSION

The paper combines a systematic review and taxonomy with a large empirical evaluation of augmentation techniques as test generators for embedding-based image retrieval. Across models and datasets, the results distinguish fault exposure from realism and show that technique rankings depend on the evaluated configuration.

  • Contributions: The paper reviews 56 augmentation papers from 2020–2025 and organizes their techniques into a ten-category taxonomy.The taxonomy spans methods from geometric flips to text-guided diffusion models.
  • Empirical study: The empirical study evaluates Amazon Titan and OpenCLIP across CIFAR-10, ImageNet-1K, and March Networks using four complementary metrics.The metrics cover retrieval failures, embedding similarity and diversity, uncertainty through four estimators, and LLaVA-based semantic realism.
  • Findings: At the tested severity level, weather simulation and Dimension Reduction produced the highest embedding uncertainty and failure counts across both models.These techniques were the strongest test generators in raw fault-detection power.
  • Findings: Photometric techniques preserved embedding structure most strongly, producing the lowest failure counts and uncertainty and the highest embedding similarity scores.Brightness adjustment, gamma correction, and color-space manipulation are named examples.
  • Findings: GAN-based augmentation ranked among the lowest in realism, while mixing techniques varied: CutMix and IPMix shifted embeddings without reliably causing failures, whereas MixUp offered a middle ground.The result separates fault exposure from visual plausibility as sometimes conflicting objectives.

APPENDIX A ADDITIONAL EXPERIMENTAL RESULTS

The appendix contains supplementary experimental results omitted from the main body to preserve readability and structural consistency. Additional analyses are organized by research question.

  • Appendix organization: Supplementary results and supporting materials for each research question are presented in separate appendix subsections.The appendix supplements the main text rather than replacing its organization.

A.1 Hyperparameter Settings of the Evaluated Augmentation Techniques

Table 7 summarizes the hyperparameter configurations used to evaluate all augmentation techniques. These empirically selected settings produced meaningful and perceptible image modifications.

  • Configuration scope: All reported experiments and results use the hyperparameter configurations listed in Table 7.The configurations were determined empirically to create meaningful, perceptible changes to the original images.

A.2 RQ1: Supplementary Failure Count Analysis

Failure effects vary across datasets and image categories, with visually complex multi-object scenes presenting a particularly challenging retrieval setting.

  • Failure counts vary across datasets, and standard deviations indicate that augmentation effects are not uniform across categories.Table 8 reports mean failure counts and corresponding standard deviations across evaluated datasets and embedding models.
  • Multi-object categories tend to exhibit higher failure counts under different augmentation techniques.Figure 15 compares the 15 categories with the highest failure counts across March Networks, CIFAR-10, and ImageNet.
  • The 95% confidence intervals support that the observed augmentation effects are not attributable solely to sampling variability.Green intervals lie entirely above one, while red intervals lie entirely below one.
  • SaliencyMix changed an ImageNet table-lamp image into an incorrect mosque association by altering dominant semantic cues.The example illustrates how salient-region composition can substantially change the embedding model’s interpretation.

A.3 RQ2a: Supplementary Embedding-Space Similarity Results

Embedding similarity remains relatively stable for most augmentation techniques, with Titan generally more stable than OpenCLIP and some mixing methods showing particularly stable behavior.

  • Cosine similarity quantifies the relationship between each original image and its corresponding augmented image.Table 10 reports pairwise embedding similarity across datasets, models, and augmentation techniques.
  • Titan generally exhibits greater embedding-similarity stability than OpenCLIP, reflected by lower standard-deviation values.The comparison spans image categories and augmentation conditions.
  • Both embedding models show relatively stable similarity behavior for most augmentation techniques.This stability holds across the evaluated original–augmented image pairs.
  • Robust MixGen, MixUp, and MixGen are among the evaluated techniques highlighted in the embedding-similarity analysis.The supplied passage identifies these methods in the reported comparison, but does not provide their individual values.

A.4 RQ2b: Supplementary Embedding-Uncertainty Results

Embedding uncertainty is evaluated under augmentation using four complementary estimators that capture deviation, pairwise change, distributional distance, and ensemble disagreement.

  • Four estimators measure embedding uncertainty: embedding dispersion, pairwise distance, Mahalanobis distance, and ensemble-based uncertainty.The estimators are reported in Tables 11 and 12 for different augmentation techniques.
  • Embedding dispersion measures each image embedding’s deviation from the mean embedding of its corresponding class.
  • Pairwise distance quantifies the difference between an original image embedding and its augmented counterpart.

A.5 RQ3: Supplementary Semantic-Realism Results

Semantic realism is assessed with model-based scores that compare how realistically augmented images preserve visual characteristics across datasets and techniques.

  • Prompt variants test the sensitivity of the final realism score to keyword selection and ordering.Algorithms 2 and 3 compare modified prompts with the original prompt used in the study.
  • The realism evaluator scores augmented images using nine visible-content criteria.The criteria include semantic coherence, physical plausibility, perspective, lighting, depth of field, texture, and object boundaries.
  • The total realism score is constrained to 0–18 and returned without explanations or intermediate outputs.The score sums nine criterion ratings under the specified prompt variants.
  • Model-based realism scores quantify the perceptual realism of augmented images across three datasets and all evaluated techniques.Higher values indicate that augmented samples retain more realistic visual characteristics according to the evaluation model.

A.6 RQ4: Overall-Ranking Sensitivity Analysis

The sensitivity analysis evaluates whether augmentation rankings remain stable when the relative weights assigned to failure rate, realism, and stability vary. It also compares embedding similarity, dispersion, uncertainty, realism, and failure-rate effects across techniques and models.

  • Weighting sensitivity: Weight variations assess whether techniques ranked as most effective remain consistently ordered across failure rate, realism, and stability.The analysis compares alternative weighting configurations across all considered evaluation metrics.
  • Evaluation dimensions: The evaluation spans embedding similarity, embedding dispersion and pairwise distance, Mahalanobis and ensemble-based uncertainty, and LLaVA realism scores.These dimensions are represented in Tables 10–13 and the LLaVA realism table.
  • Failure-rate sensitivity: Techniques near the diagonal affect both models comparably, while upper-right techniques produce higher and more consistent failure counts.Greater distance from the diagonal indicates greater model-sensitivity differences.
Loading 2608.27502v1…