PaperlayerView exact revision on arXiv

Source-linked AI summary

Convolutional Set Transformer

Federico Chinello, Giacomo Boracchi

arXiv:2509.22889v1cs.CVcs.AIcs.LG

TL;DR

Existing set-input networks operate on vectors, requiring images to be encoded before their relationships can be modeled. CST processes image sets directly through contextualized convolutional feature extraction; the paper reports gains across tasks, explainability support, and adaptation after pre-training.

  • Problem

    Deep Sets and Set Transformer process sets of vectors rather than image tensors, so image-set modeling requires separate CNN feature extraction before contextualization.

    Source: §1 · p. 3

  • Method

    CST stacks SetConv2D blocks to jointly extract features from images and model context across an image set.

    Source: §1 · p. 3, §4.1 · p. 8

  • Results

    CST consistently outperforms Deep Sets and Set Transformer across multiple tasks, including Set Classification and Set Anomaly Detection.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The authors report that ImageNet-pretrained CST-15 can be adapted to new tasks through transfer learning and release it publicly.

    Source: §1 · p. 5, §7 · p. 25

    The described training and inference setup uses labeled image sets, with anomalous images marked for Set Anomaly Detection.

    Source: §2 · p. 6

Abstract

from arXiv · show

We introduce the Convolutional Set Transformer (CST), a novel neural architecture designed to process image sets of arbitrary cardinality that are visually heterogeneous yet share high-level semantics - such as a common category, scene, or concept. Existing set-input networks, e.g., Deep Sets and Set Transformer, are limited to vector inputs and cannot directly handle 3D image tensors. As a result, they must be cascaded with a feature extractor, typically a CNN, which encodes images into embeddings before the set-input network can model inter-image relationships. In contrast, CST operates directly on 3D image tensors, performing feature extraction and contextual modeling simultaneously, thereby enabling synergies between the two processes. This design yields superior performance in tasks such as Set Classification and Set Anomaly Detection and further provides native compatibility with CNN explainability methods such as Grad-CAM, unlike competing approaches that remain opaque. Finally, we show that CSTs can be pre-trained on large-scale datasets and subsequently adapted to new domains and tasks through standard Transfer Learning schemes. To support further research, we release CST-15, a CST backbone pre-trained on ImageNet (https://github.com/chinefed/convolutional-set-transformer).

1. Introduction

Image-set models must capture relations among images, but vector-based set networks process images only after independent feature extraction. CST integrates contextual modeling into convolutional feature extraction, with reported gains across tasks, Grad-CAM compatibility, and transfer learning from ImageNet pre-training.

1. Introduction

  • Image sets can combine complementary image information with semantic relationships, supporting applications such as medical diagnosis, hotel classification, and disaster damage assessment.The introduction gives melanoma detection, fine-grained hotel classification, and building damage assessment as examples.Source: §1 · p. 2
  • Deep Sets and Set Transformer cannot directly process image tensors: they require a CNN to encode each image before modeling relationships among images.The sequential pipeline prevents feature extraction and context modeling from developing direct synergies, and places CNN saliency maps before the set-specific operation.Source: §1 · pp. 2–3
  • CST processes image sets through stacked SetConv2D blocks, jointly extracting features and modeling context while retaining spatial activation volumes.The blocks use attention-derived dynamic biases to incorporate information from the set into convolutional processing.Source: Figure 1 · p. 2, §1 · pp. 3–4, Figure 2 · p. 3
  • CST consistently outperforms Deep Sets and Set Transformer across Set Classification and Set Anomaly Detection, and supports transfer to new domains after ImageNet pre-training.The authors also report meaningful Grad-CAM visualizations for CST in Set Anomaly Detection, where Set Transformer fails to provide reliable explanations.Source: §1 · pp. 4–5
  • The authors introduce CST-15, an ImageNet-pretrained CST encoder, and report that CST is natively compatible with CNN explainability tools such as Grad-CAM.They also introduce Contextualized Image Classification and Combinatorial Training for pre-training CSTs.Source: §1 · p. 5

2. Problem Formulation

The paper formulates image-set recognition as set-to-set or set-to-global prediction, with inputs whose size may vary and whose ordering is irrelevant. It considers anomaly detection and contextualized image classification as set-to-set tasks, alongside set-level classification.

2. Problem Formulation

  • Set-to-set tasks produce an output for each image while exploiting relationships among images; the paper considers Set Anomaly Detection and Contextualized Image Classification.In Contextualized Image Classification (CIC), all images share an unknown label, and each image's prediction is influenced by the others.Source: §2 · pp. 5–6
  • Set Anomaly Detection identifies images inconsistent with the majority, so whether an image is anomalous depends on its surrounding set.The same image may be anomalous in one set but not another.Source: §2 · p. 6
  • Set-level classification is a set-to-global task that predicts one shared label for the entire image set.The formulation also allows classification on singleton sets; training and inference sets may contain any number of images, and order is irrelevant.Source: §2 · p. 6
  • The formulation distinguishes permutation-equivariant outputs for set-to-set tasks from permutation-invariant outputs for set-to-global tasks.Permuting inputs should correspondingly permute set-to-set outputs, while set-to-global predictions remain unchanged.Source: §2 · p. 6

3. Related Work

Related work includes vector-based set networks and multi-image models that fuse image representations. The paper positions CST as a way to model semantic relations among heterogeneous images while preserving image-specific spatial representations.

3. Related Work

  • Deep Sets and Set Transformer require CNN-generated image vectors, whereas CST models context during feature extraction and preserves spatial information.The paper also contrasts CST's reported explainability support with the opacity of the vector-based set networks.Source: §3 · p. 7
  • Deep Sets for Symmetric elements assumes spatially aligned set elements, limiting its fit to restoration and denoising rather than heterogeneous, semantically related image sets.CST instead combines Global Average Pooling and Multi-Head Self-Attention to exploit high-level semantic relationships, including for event recognition from personal photo albums.Source: §3 · p. 7
  • Typical multi-image networks fuse image-specific representations into one representation, while CST retains a contextualized representation for each image.The paper notes that fusion loses image-specific information after the representations are combined.Source: §3 · pp. 7–8

4. Convolutional Set Transformer (CST)

CST stacks SetConv2D blocks to extract image features while incorporating information from other images in the set. Its contextualized spatial representations support task-specific outputs and CNN-style explanations.

4.1. SetConv2D

  • SetConv2D is an equivariant set-to-set block that returns a volume for each input and uses set context to enhance each image's representation.Its stages apply shared convolution, pooling, attention, context-aware bias addition, and a non-linear activation; preliminary tests favored GAP with MHSA.Source: §4.1 · pp. 8–9

4.2. CST Architecture

  • The CST encoder stacks SetConv2D blocks to extract features and model set relationships simultaneously, producing contextualized representations for downstream task networks.The downstream network processes individual representations for set-to-set tasks or pools them for a set-level prediction.Source: §4.2 · p. 9
  • Dynamic biases in SetConv2D adapt to set context at training and inference, with early layers modeling fine patterns and deeper layers capturing higher-level concepts.SetConv2D blocks can be interleaved with Conv2D layers, and CSTs can process singleton image sets.Source: §4.2 · pp. 9–10

4.3. Explainability

  • The paper reports demonstrated explainability support for CST, while Deep Sets and Set Transformer remain opaque by design.The authors describe explainability in set-input networks as a largely unexplored area.Source: §4.3 · p. 10
  • Grad-CAM can be applied to SetConv2D's contextualized spatial representations, exposing how other images in a set modulate features relevant to a prediction.The same image may receive different explanation maps depending on its set; in contrast, CNN explanations for Deep Sets or Set Transformer precede and do not explain their set-specific modeling.Source: §4.3 · p. 10

5. CST Pre-training and Transfer Learning

The section presents Contextualized Image Classification (CIC) and Combinatorial Training (CT) for CST pre-training, then describes transfer to new tasks, including adaptation trained only on individual images. Together, these approaches preserve contextualized image representations while allowing training on small, variable-size sets.

5. CST Pre-training and Transfer Learning

  • After large-scale pre-training, CST encoder E can transfer as a generic backbone; downstream heads may be replaced or reinitialized while E is frozen or fine-tuned.CST-15 is released as an ImageNet-pretrained set-input backbone.Source: §5 · p. 11
  • Set-free transfer learning can retain inference on arbitrarily large image sets with improved accuracy even when downstream adaptation uses only individual images.The encoder preserves contextualization learned during large-scale pre-training, avoiding the need to relearn inter-image relationships.Source: §5 · p. 11

5.1. Pre-training Objective: Contextualized Image Classification (CIC)

  • CIC pre-trains CST encoders by predicting each image’s class from that image plus shared set context, improving image-level classification accuracy.The encoder learns both image feature extraction and contextual relationships, and its classifier can be removed for downstream adaptation.Source: §5.1 · pp. 11–12
  • CIC preserves contextualized image-level representations for set-to-set tasks, while pooling can produce invariant set-level outputs; CIC also trains more stably than set-level classification.Removing pooling after set-level training is not viable because it may disrupt the network.Source: §5.1 · p. 12
  • Label granularity shapes learned representations: finer labels encourage subtle category distinctions, while coarser labels encourage higher-level semantic connections.ImageNet’s hierarchy allows labels such as husky, dog, or mammal.Source: §5.1 · p. 12

5.2. Combinatorial Training (CT)

  • CT randomly samples a set size between nmin and nmax each epoch, assembles within-class image sets, and keeps the number of sets per batch fixed.The total images per batch therefore varies from b × nmin to b × nmax.Source: §5.2 · p. 13
  • CT acts as data augmentation through random sets of varying sizes, not element permutations; setting nmin = 1 can speed convergence on large datasets.Training only on sets may otherwise be an overly strong augmentation that slows optimization.Source: §5.2 · p. 13
  • Training CST-15 on single images and pairs from the same class still supports inference on arbitrarily large sets, with accuracy increasing as set size grows.CT enables memory-efficient training on small sets while preserving larger-set inference capacity.Source: §5.2 · p. 13

6. Experiments

Experiments benchmark CSTs on classification and anomaly detection, and test whether an ImageNet-pretrained CST can transfer to new tasks. Across these settings, the experiments compare CSTs with set-learning and CNN baselines.

6.1.1. Tested Architectures

  • The tested CIC baselines pair an equivalent CNN feature extractor with Deep Sets or Set Transformer layers, while CSTs stack SetConv2D blocks.CST uses six blocks on CIFAR datasets and eight on ImageNet64x64 and Tiny ImageNet; the equivalent CNN preserves CST’s convolutional structure and receptive field.Source: §6.1.1 · pp. 14–16

6.1.2. Training Hyperparameters

  • Combinatorial Training improves Top-1 accuracy for CST, Set Transformer, and Deep Sets models on ImageNet64x64, with relative gains of +24% to +32% for sets of size 1 and +14% to +21% for size 2.The comparison is against conventional training.Source: §6.1.2 · pp. 17–18, Figure 4 · p. 17

6.1.3. Results and Analysis

  • CSTs outperform baselines in 58 of 60 classification cases across four datasets, with gains up to +20.3% and gains above +10% in 17 cases.The comparison spans CIC and SC tasks, fusion strategies, and input-set sizes.Source: §6.1.3 · p. 18, Table 2 · p. 19
  • CIFAR-10 latent embeddings become more separated by class as context grows, forming ten minimally overlapping clusters at set size 10.With no context, embeddings are scattered and substantially overlap across classes.Source: §6.1.3 · pp. 18–19, Figure 5 · p. 19

6.2. Explainable Set Anomaly Detection

  • On CelebA set anomaly detection, CSTs outperform competing models across set sizes and anomaly prevalences, and produce meaningful Grad-CAM explanations where Set Transformer explanations are not meaningful.CST models anomaly decisions through SetConv2D blocks, while Set Transformer Grad-CAM requires linearizing its contextual Set Attention Blocks.Source: §6.2 · pp. 20–21, Table 3 · p. 20

6.3. Pre-training CSTs for Transfer Learning

  • CST-15 is an ImageNet-pretrained CST backbone released for transfer learning, with a 28M-parameter model compared with VGG-19’s 144M parameters.The paper tests its adaptation to new tasks using standard transfer learning.Source: §6.3 · p. 21, §6.3.1 · p. 22

6.3.1. CST-15 Trained on ImageNet

  • On ImageNet validation, CST-15 already outperforms VGG-19 at set size 1 and gains performance as set size increases by using image-set context.CST-15 can process arbitrarily large input sets.Source: Figure 7 · p. 22, §6.3.1 · p. 23

6.3.2. Transfer Learning on the PEC dataset

  • For PEC event recognition, CST-15 and VGG-19 are adapted with frozen backbones and singleton training, but only CST-15 improves image-level accuracy as album set size grows.CST-15 also outperforms VGG-19 when both receive a single image without context.Source: Figure 8 · p. 22, §6.3.2 · pp. 23–24

7. Conclusion

CST models image-set context within convolutional blocks, supporting performance, CNN explainability, and transfer learning. The paper introduces CST-15, an ImageNet-pretrained set-input network released publicly.

  • CST models image-set relationships within convolutional blocks, jointly with feature extraction, unlike Deep Sets and Set Transformer.The conclusion presents this design as the basis for performance across diverse tasks and datasets and compatibility with CNN explainability tools.Source: §7 · p. 25
  • CST supports CNN explainability tools and is reported to deliver superior performance across diverse tasks and datasets.Source: §7 · p. 25
  • CST-15 is presented as the first set-input network pre-trained on ImageNet and is released publicly.The conclusion also reports that CST-15 can be adapted to new tasks via Transfer Learning.Source: §7 · p. 25
  • The paper identifies medical imaging, surveillance, e-commerce, and social media as domains for future exploration of CST applications.Source: §7 · p. 25

A. Ablation Study on Combinatorial Training

The ablation reports that CSTs outperform baselines in most test cases even without Combinatorial Training (CT), while CT consistently benefits tested models.

B. Additional Details About the Experiments

The appendix organizes additional experimental details into classification, explainable set anomaly detection, CST-15 pretraining, and transfer-learning sections.

B.1. Classification Experiments

The appendix specifies classification training settings, the explainable anomaly-detection experiment and set construction, and CST-15’s ImageNet training setup.

B.1. Classification Experiments

  • Classification experiments use Combinatorial Training with nmin = 2 and nmax = 5, dataset-specific batch sizes, and validation-based stopping.The learning-rate schedule warms up for five epochs; training stops when validation accuracy no longer improves for 50 epochs.Source: §B.1 · p. 31

B.2. Explainable Set Anomaly Detection

  • The anomaly-detection comparison includes CST, two Set Transformers, and Deep Sets, trained from scratch to label each CelebA image normal or anomalous.Sets are generated using two randomly chosen attributes: normal images exhibit both, while anomalous images lack both; set size is 10 and anomaly prevalence varies from 0 to 0.4.Source: §B.2 · pp. 31–33

B.3. CST-15 trained on ImageNet

  • CST-15 is trained from scratch on ImageNet at 224 × 224 resolution for 250 epochs on a single A100 GPU.Its Combinatorial Training uses nmin = 1 and nmax = 2.Source: §B.3 · p. 33

B.4. Transfer learning on the PEC dataset

The PEC transfer-learning setup adapts frozen CST-15 and VGG-19 backbones to 14 classes by training new classification layers, with CST-15 trained on individual images.

B.4. Transfer learning on the PEC dataset

  • CST-15 and VGG-19 are adapted to 14 PEC classes by replacing their ImageNet classifiers and training only the new classification layers.The backbones remain frozen.Source: §B.4 · p. 33
  • Six personal albums per event class are reserved for validation, and the official PEC test split is used for testing.Images are center-cropped and resized to 224 × 224.Source: §B.4 · p. 33
  • CST-15 is trained using individual images without sets, matching the set-free transfer-learning scheme used for VGG-19.Training uses Adam with a 1 × 10−3 learning rate and stops after validation accuracy fails to improve for 10 epochs.Source: §B.4 · p. 34

C. Explainability of CSTs

CSTs are compatible with CNN explainability tools, and CST-15 Grad-CAMs are reported as precise and focused in qualitative comparisons. For CSTs trained for CIC, explanations should be computed at the penultimate SetConv2D block because final-block maps are affected by contextual information.

C. Explainability of CSTs

  • For the shuttle-carrier image, CST-15’s map identifies the space shuttle, while the other models highlight a coarse region covering both aircraft.Source: C. Explainability of CSTs · p. 34
  • For CSTs trained for CIC, compute Grad-CAMs at the penultimate SetConv2D block; final-block maps are affected by the importance of contextual information.CSTs spread contextual information evenly across spatial dimensions.Source: C. Explainability of CSTs · p. 34, Figure C2 · p. 36
  • Figures C3–C5 show Grad-CAMs for four-image sets, with each row representing one set and maps computed at the penultimate SetConv2D block.Source: C. Explainability of CSTs · p. 34, Figure C3 · p. 37, Figure C4 · p. 38
  • CST-15 Grad-CAMs are more precise and focused than ResNet50 and VGG-19 maps, and comparable or sometimes better than ConvNeXt-XL maps.The qualitative comparison uses isolated images without context; CST-15 has 28M parameters, versus 350M for ConvNeXt-XL, 26M for ResNet50, and 144M for VGG-19.Source: C. Explainability of CSTs · p. 34, Figure C1 · p. 35

Related papers

Loading 2509.22889v1…