PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Difference Quantification Using Autoencoder-Based Latent Representations

Manish Sharma, Timothy Yim, Clifton Forlines

arXiv:2608.24782v1cs.CV

TL;DR

Pixel-based image metrics can miss perceptually meaningful differences, motivating a semantic latent-space alternative. The paper trains a convolutional autoencoder and compares image embeddings with cosine similarity, finding class separation and perceptual sensitivity across datasets while emphasizing computational efficiency.

  • Problem

    Traditional MSE, PSNR, and SSIM comparisons rely on pixel-level information and can fail to capture perceptually meaningful image differences.

  • Method

    A convolutional autoencoder learns latent image representations, whose cosine similarities are used to quantify semantic image differences.

  • Results

    The learned latent representations capture high-level semantic attributes and show strong within-class cohesion and between-class separability on dog–cat images.

  • Takeaways & Limitations

    The approach provides a computationally efficient and perceptually grounded alternative for large-scale similarity analysis.

  • Takeaways & Limitations

    The authors identify remaining limitations and suggest future work using disentangled or contrastive learning techniques to improve the approach.

Abstract

from arXiv · show

Traditional image similarity metrics such as Mean Squared Error (MSE), Peak Signal-to-Noise Ratio (PSNR), and the Structural Similarity Index Measure (SSIM) rely on pixel-level comparisons and often fail to capture perceptually meaningful differences between images. In contrast, latent representations learned by deep neural networks encode high-level semantic information that is more closely aligned with human visual perception. This paper proposes a convolutional autoencoder-based framework for quantifying image differences using cosine similarity in latent space. The learned compact embeddings enable robust differentiation between visually distinct images under variations in illumination, pose, and background. Extensive evaluation on dog-cat images and additional cross-domain datasets demonstrates clear class-wise clustering and strong inter-class separability in the latent space, with 98.4% of dog-cat image pairs exhibiting similarity scores below 0.5. Further validation using the TID2013 dataset shows that latent-space distance correlates positively with human Mean Opinion Scores (MOS), demonstrating sensitivity to perceptually relevant image distortions. The proposed approach provides a computationally efficient and semantically grounded alternative to conventional pixel-based similarity metrics, with potential applications in content-based retrieval, perceptual quality assessment, and semantic similarity analysis.

1. Introduction

Traditional pixel-based metrics can miss perceptually meaningful image differences, motivating latent-space representations and cosine similarity as a semantic alternative. The proposed CAE framework is evaluated across multiple datasets and supports efficient image-difference quantification.

  • Motivation: MSE, PSNR, and SSIM operate on pixel intensities or local statistics and can be sensitive to noise, illumination, and geometric changes.These limitations can cause them to miss perceptually meaningful differences between images.
  • Motivation: Deep neural representations encode visual properties including texture, shape, structure, and object identity, offering improved alignment with perception.The learned representations provide an alternative to pixel-based metrics.
  • Research gap: Cosine similarity in autoencoder latent spaces is insufficiently studied as a general-purpose image-difference metric, especially across domains and against human ratings.The identified gaps concern semantic difference quantification and perceptual validation using datasets such as TID2013.
  • Proposed framework: The proposed framework compresses images into latent vectors with a convolutional autoencoder and quantifies their semantic differences using cosine similarity.The encoder-decoder design is intended to retain salient structural and semantic information.
  • Evaluation: Evaluation spans natural images, geometric shapes, outline-shaded pairs, and TID2013 human-provided Mean Opinion Scores.The study examines class-wise separability, cross-domain behavior, and perceptual alignment.
  • Contributions: The results characterize the latent similarity measure as semantically grounded, perceptually meaningful, and computationally efficient for modern image applications.The framework is presented as suitable for large-scale or real-time use cases through linear-time similarity computation.

2. Related Work

Classical metrics offer simple distortion measures but remain limited in perceptual sensitivity. Prior deep-feature and autoencoder research motivates using CAE latent spaces and cosine similarity for semantic image-difference quantification.

  • Classical Image Similarity Metrics: MSE and PSNR provide simple closed-form distortion measures, while SSIM incorporates structural statistics but still struggles with semantic variation.SSIM also remains sensitive to illumination changes and local artifacts.
  • Learned Perceptual Metrics: LPIPS, DISTS, and PieAPP use deep convolutional features and have shown superior correlation with human judgments than traditional metrics.These methods represent a shift from pixel-based similarity assessment.
  • Autoencoder-Based Representations: Autoencoders have been used for unsupervised feature extraction, anomaly detection, image clustering, and feature compression.Earlier work established their ability to learn compact semantic representations.
  • Research Gap: Few studies specifically investigate autoencoder latent spaces for quantifying semantic differences with cosine similarity.This gap distinguishes the present focus from prior applications such as clustering and anomaly detection.
  • Latent-Space Research: Research on VAEs and generative models highlights latent spaces as representations of meaningful relationships, geometry, and disentangled features.These findings motivate extending latent-space analysis to image difference quantification.
  • Present Work: The proposed work extends prior research with a generalizable and computationally efficient CAE-based framework evaluated across multiple domains.Its procedure combines CAE-derived features with cosine similarity analysis.

3. Methodology

The methodology uses a convolutional autoencoder to learn hierarchical image representations, reconstruct inputs, and produce latent vectors for similarity analysis. Reconstruction training is used to preserve essential semantic structure.

  • Architecture: The CAE contains an encoder, latent representation layer, and decoder for unsupervised image representation learning.The architecture is designed to extract hierarchical, semantically meaningful image features.
  • Encoder: Convolutional and max-pooling layers reduce spatial resolution while increasing feature depth, supporting learning from edges to high-level semantics.The encoder hierarchy covers low-level edges, mid-level textures, and high-level semantic features.
  • Latent Representation: The final convolutional feature map is flattened and passed through a fully connected layer to produce a latent vector.Latent dimensions of 128 and 256 were explored to balance representational capacity and computational efficiency.
  • Decoder: The decoder mirrors the encoder with transposed convolutions and upsampling to reconstruct the input image from its latent representation.This reconstruction process is intended to ensure that the latent space retains essential semantic information.
  • Training Objective: The CAE is trained by minimizing reconstruction error using Mean Squared Error.The training objective encourages meaningful structure in the learned latent representations.

Training details

After training, the encoder extracts image embeddings that are compared with cosine similarity, while PCA provides qualitative visualization rather than the similarity computation itself.

  • Training: The framework uses TensorFlow/Keras and follows stated autoencoder training practices for stable convergence.The supplied training description identifies the implementation framework and convergence rationale.
  • Latent Space Extraction: The trained encoder extracts one latent vector per image, with the vector intended to encode semantic content more effectively than pixel space.The latent representation is then used for feature-space comparison.
  • Similarity Measurement: Cosine similarity compares latent vectors to quantify semantic differences between images.Values close to 1.0 indicate highly similar vectors, whereas values close to –1.0 indicate strong dissimilarity.
  • Similarity Measurement: Semantic difference is defined so that larger values correspond to greater perceptual difference.This establishes the direction used when interpreting the similarity-based difference measure.
  • Visualization: PCA projects latent vectors into one or two dimensions for qualitative analysis only, not for similarity computation.The projection is used to study latent distributions and visualize structure.

4. Experimental Setup

The evaluation uses a curated dog–cat dataset designed to test semantic separation and robustness to substantial visual variation. Its balanced classes support direct comparison of inter-class and intra-class similarity distributions.

  • The primary evaluation uses 2,000 dog and cat images, with 1,000 images per class.
  • The images vary substantially in pose, illumination, background, and breed characteristics.
  • The dataset provides clear semantic contrast between the dog and cat classes.
  • Its intra-class variability supports testing robustness across images within each class.
  • The dataset enables direct analysis of inter-class versus intra-class similarity distributions.

Cross-Domain Dataset A: Outline–Shaded Image Pairs

The first cross-domain dataset pairs outline and shaded versions of the same objects. It tests whether latent representations capture structural similarity despite changes in shading and surface detail.

  • The synthetic dataset contains paired outline and shaded versions of the same objects.
  • Filename conventions were used to group corresponding outline–shaded images.
  • The dataset contains 6,480 outline–shaded images.
  • Outline–shaded comparisons test structural similarity despite differences in shading and surface detail.

Cross-Domain Dataset B: Plain–Shape Images

The second cross-domain dataset uses geometric shapes rendered in plain and outline styles. It evaluates whether latent representations capture geometric structure independently of texture.

  • The dataset contains circles, squares, and triangles rendered in plain and outline styles.
  • Plain–outline rendering tests latent capture of geometric structure independent of texture.
  • The dataset includes 2,700 images spanning different shapes and colors.

TID2013 Perceptual Quality Dataset

The study combines TID2013 perceptual validation with autoencoder implementation and latent-space analyses. It evaluates reconstruction, clustering, semantic similarity distributions, and correlation between latent distance and human quality judgments, while using a computationally compact comparison pipeline.

  • TID2013 Perceptual Quality Dataset: TID2013 contains 3,000 distorted images derived from 25 reference images, each annotated with human Mean Opinion Scores.
  • Implementation and Analysis: The CAE uses convolutional encoder and decoder layers with a 128-dimensional or 256-dimensional latent vector.
  • Evaluation Procedures: PSNR and SSIM assess reconstruction quality, while PCA and KDE examine clustering and latent-space structure.
  • TID2013 Perceptual Quality Dataset: The evaluation correlates latent distance between distorted and reference images with MOS to assess perceptual grounding.
  • Latent-Space Analysis: The analysis extracts encoder latent vectors, computes cosine similarities for dog–cat, outline–shaded, and plain–outline pairs, and groups scores into three regions.
  • Computational Efficiency: Each latent comparison requires O(d) operations, and encoder extraction takes approximately 5 ms per image on an NVIDIA RTX-class GPU.
  • Computational Efficiency: The computational profile makes the method scalable to large datasets and suitable for near-real-time applications.

5. Results

The autoencoder produced structurally adequate reconstructions and latent representations that separated semantic classes, distinguished stylistic domains, and aligned with perceptual judgments.

  • Reconstruction Quality: 28.5 dB PSNR and 0.92 SSIM accompanied stable convergence and reconstructed images with preserved structural details.These reconstruction results supported extracting reliable semantic representations, although reconstruction itself was not the primary focus.
  • Latent Space Structure: Dog and cat latent vectors occupied clearly distinct PCA regions with negligible overlap and compact, semantically interpretable clusters.Class means and individual-sample plots reinforced the separation.
  • Class Separability: 0.79 and 0.77 were the mean within-class cosine similarities for dog and cat pairs, versus 0.32 for between-class pairs.The contrast indicates coherent within-class structure and strong inter-class separation.
  • Class Separability: 98.4 percent of dog-cat pairs had cosine similarity below 0.5, and no inter-class pair exceeded 0.7.These thresholds quantitatively reinforce the separation between dog and cat categories.
  • Perceptual Validation: Latent distance correlated positively with MOS, while cosine similarity correlated negatively, linking greater distortion with lower perceived quality.The TID2013 analyses showed sensitivity to perceptually relevant distortions.

6. Discussion

The discussion interprets the latent space as semantically and perceptually meaningful across categories, stylistic variations, and distortions, while identifying limits in generalization, disentanglement, and perceptual modeling.

  • Semantic Representation: The framework captures high-level semantic differences while suppressing irrelevant pixel-level variations.Strong contrasts between within-class and between-class cosine similarities support characteristic semantic signatures across samples.
  • Semantic Representation: Small proportions of moderately high inter-class similarities suggest sensitivity to shared global attributes such as shape or color.The discussion contrasts these shared attributes with reliance on low-level patterns.
  • Cross-Domain Flexibility: Outline-shaded evaluations indicate sensitivity to structural features despite substantial changes in shading or stylistic attributes.Similarity distributions varied with dataset complexity, while the encoder retained structural relationships across domains.
  • Cross-Domain Flexibility: Minimal surface detail produced tightly clustered embeddings, whereas shaded or textured variations produced more widely spread representations.The latent vectors therefore reflected underlying geometry and visual composition alongside appearance variation.
  • Perceptual Alignment: Negative cosine-similarity and positive latent-distance correlations with MOS indicate sensitivity to perceptually relevant distortions.The correlations were not perfect because human perception is complex, but they indicate potential for perceptual quality estimation.
  • Limitations: Robustness across highly heterogeneous real-world datasets remains unexplored because training used only the study’s datasets.Broader categories may require retraining or fine-tuning.

7. Conclusion

The study presents a convolutional autoencoder framework for image similarity that uses latent-space representations instead of pixel-level comparisons. Experiments report semantic separation across image classes and domains, perceptual correspondence with human judgments, computational efficiency, and remaining scope limitations.

  • Contribution: The framework quantifies image similarity using convolutional autoencoder latent representations and cosine similarity rather than pixel-level comparisons.This approach is presented as computationally efficient and semantically grounded.
  • Semantic representation: Latent embeddings captured high-level semantic attributes that pixel-based measures such as MSE, PSNR, and SSIM fail to represent.The conclusion links this representation to stronger semantic discrimination.
  • Evaluation: Dog–cat evaluations revealed strong within-class cohesion and clear between-class separability, with cross-domain tests extending evaluation to outline–shaded and plain–shape images.Additional analyses examined generalization across stylistic variations and diverse image domains.
  • Perceptual validation: Latent-space distance showed meaningful correlations with human Mean Opinion Scores on TID2013, supporting sensitivity to perceptually relevant image distortions.The conclusion presents this as evidence for applicability in perceptually oriented tasks.
  • Limitations: Dataset-specific training and latent dimensions that may not reflect complex visual datasets constrain the framework's robustness and scalability.The paper suggests domain-agnostic encoders, transfer learning, and dimensionality optimization as directions for future work.
  • Applications: The method is positioned for content-based retrieval, anomaly detection, perceptual quality assessment, and large-scale similarity analysis.These applications follow from the reported semantic discrimination, generalization, and computational efficiency.
Loading 2608.24782v1…