QS-Attn: Query-Selected Attention for Contrastive Learning in I2I Translation

• CUT selects anchors randomly, although some image locations contain little domain information and individual features have limited receptive fields.The paper argues that contrastive constraints are more meaningful on features carrying significant domain information and that relations among locations provide cues for preserving source content.Source: §1 · p. 2
• QS-Attn measures source-feature significance using attention-row entropy and selects queries with smaller entropy as anchors for contrastive learning.The introduction illustrates lower-entropy locations on horse bodies, cat faces, and label edges.Source: §1 · p. 2
• The reduced attention matrix routes value features from both source and target domains, retaining source-domain feature relations to avoid excessive modifications.Source: §1 · p. 2
• The method retains CUT's simple design without adding model parameters, and its experiments find entropy-based query selection with global cross-domain value routing robust.Source: §1 · pp. 2–3
• The proposed method achieves state-of-the-art results in most two-domain I2I tasks.Source: §1 · p. 3

• CycleGAN and DiscoGAN use two generators and cycle consistency for unpaired translation, while cycle consistency can impose an unnecessary pixel-level constraint and sometimes degrade image quality.Source: §2 · p. 3
• Single-direction approaches preserve content with pixel-distance, geometry, or perceptual losses, while foreground-mask methods add parameters to locate relevant regions.Source: §2 · p. 3
• CUT applies cross-domain self-supervised contrastive loss, while F-LSeSim adds local self-similarity but relies on VGG features and neither method intentionally selects anchor features.Source: §2 · p. 3
• Self-supervised contrastive learning is used in pretraining, image generation, and I2I, but the cited I2I approaches do not select anchors or use source-domain relations like QS-Attn.Source: §2 · p. 3

3.1. Preliminaries on CUT

• CUT’s contrastive loss uses translated-image features as anchors, same-location real-image features as positives, and randomly selected real-image features as negatives.The loss gradient updates the anchor but is detached on positives and negatives; the full objective also includes adversarial and identity losses.Source: §3.1 · p. 4

3.2. QS-Attn for Contrastive Learning

• QS-Attn adds attention-based query selection to CUT without extra model parameters or separate query, key, and value projection heads.It selects anchors for the contrastive loss rather than sampling them randomly.Source: §3.2 · p. 4, §3.2.1 · p. 5

3.2.1 Attention for query selection.

• Global attention selects the N source-feature queries with the lowest row entropy, using each feature’s similarities to all image locations to measure significance.The selected rows form AQS, which is determined by Ix rather than G(Ix).Source: §3.2.1 · p. 5
• Local attention compares each query with neighboring keys in a w×w window, selects the N lowest-entropy rows, and locates their corresponding local value features.This captures local spatial interactions while reducing computation relative to non-local attention.Source: §3.2.1 · p. 5

3.2.2 Cross domain value routing for contrastive learning.

• AQS routes source and translated value features to form real-image positives and negatives and translated-image anchors for contrastive learning.The same selected relations provide context for the features and are imposed across both domains during translation.Source: §3.2.2 · p. 5

4.1. Implementation Details

• Experiments train and evaluate on Cityscapes, Horse→Zebra, and Cat→Dog at 256×256 input and generated resolution; QS-Attn is applied to the encoder’s last two layers to limit global-attention cost.FID and SWD evaluate translated-image quality; Cityscapes additionally reports segmentation mAP and pixel-wise accuracy (pixAcc).Source: §4.1 · pp. 5–6

4.2. Results

• FID results favor QS-Attn over other methods on all three datasets, and Cityscapes results are also better on mAP, pixAcc, and classAcc.The method adds no generator or discriminator parameters and uses the same generator architecture as CycleGAN.Source: Table 1 · p. 6, §4.2 · p. 7
• Visual comparisons report accurate translation of domain-relevant features and background consistency on Horse→Zebra and Cat→Dog.Figure 4 compares QS-Attn with FSeSim, CUT, CycleGAN, and MUNIT across three benchmark datasets.Source: Figure 4 · p. 6, §4.2 · p. 7
• A user study asks 60 participants to compare Global QS-Attn with CUT, FSeSim, and CycleGAN on image quality, target-domain conformity, and domain-irrelevant consistency.The study uses Horse→Zebra images; the supplied passages describe its measures and setup but not its outcome.Source: §4.2 · p. 7, Table 2 · p. 7

4.3. Ablation Study

• Entropy-sorted query selection outperforms random selection, while omitting selected-attention value routing produces worse metrics than either configuration.The ablation interprets routing as necessary for the encoder to learn significant image features through the contrastive loss.Source: §4.3 · pp. 7–8, Table 3 · p. 8
• Self-domain value routing performs worse than the complete model; the authors report that cross-domain routing establishes closer source-target correlation.The source-selected attention matrix captures source-image relations and applies them to generated-image features.Source: §4.3 · p. 8, Table 3 · p. 8
• The max-mean query-selection strategy also performs well when its selected queries route source and target value features.Model F ranks queries by the max-mean measurement and selects the Top-N.Source: §4.3 · p. 8
• Local attention with local value routing achieves the best FID, but its visual quality is worse than configurations using global value routing.The authors suggest global value routing helps reconstruct image texture.Source: §4.3 · p. 8, Table 3 · p. 8

5. Conclusion

• QS-Attn selects lower-entropy source-domain queries as significant anchors for cross-domain contrastive learning.The selected queries form a row-reduced attention matrix.Source: §5 · p. 8
• The reduced attention matrix routes source and target features, enlarging selected features’ receptive fields and helping the output retain input relations.Source: §5 · p. 8
• The paper evaluates QS-Attn on popular domain-translation datasets and reports intensive ablation studies.Source: §5 · p. 8

A. Limitations

• CUT is the baseline because the study targets significant domain-specific features for contrastive learning.Source: A. Limitations · p. 11
• QS-Attn was not tested in bidirectional or multi-domain I2I tasks because of limited training resources.The authors think it should also work in those settings.Source: A. Limitations · p. 11
• Global attention slows training relative to CUT because of complex matrix multiplication, while local attention mitigates the problem.Source: A. Limitations · p. 11

B. Additional Ablation Study

• Model B applies QS-Attn across all layers, average-pooling first-three-layer features to 64×64 (so Ag is 4096×4096), and can compete with Model A on two tasks.Model A applies QS-Attn only to the last two layers and randomly selects points on the first three.Source: B. Additional Ablation Study · p. 11
• Even with pooling, all-layer global attention and entropy calculations consume substantial memory and time, so the final model uses QS-Attn on two layers.Model A applies QS-Attn on the last two layers and randomly selects points on the first three.Source: B. Additional Ablation Study · p. 11
• 128 selected queries outperform the original setting on Horse → Zebra, and query selection remains valid with 64, 128, or 512 queries.Table 5 is the ablation comparing query counts for QS-Attn and CUT.Source: B. Additional Ablation Study · p. 11

C. Query-Selection Algorithm

• The three proposed query-selection algorithms are Global, Local, and Local+Global.Source: C. Query-Selection Algorithm · p. 11
• Pseudocode for the three algorithms appears in Algorithms 1, 2, and 3.Source: C. Query-Selection Algorithm · p. 11
• Table 4 covers the different-layer QS-Attn ablation, while Table 5 compares query counts for QS-Attn and CUT.Source: Table 4 · p. 11, Table 5 · p. 11

D. Network Architectures

• The network-architecture section describes the generator and discriminator, with detailed architectures provided in Appendices D.1 and D.2.Source: D. Network Architectures · p. 11

D.1. Generator

• The generator has two down-sampling blocks, nine intermediate blocks, two up-sampling blocks, and two input/output convolution layers.It uses Instance Normalization and ReLU except at the output, which uses Tanh; the encoder ends before the sixth residual block, and features come from five layers.Source: §D.1 · p. 11

D.2. Discriminator

• The discriminator is a PatchGAN with three down-sampling blocks and two convolution layers.Leaky ReLU is used as its activation.Source: §D.2 · p. 11

E. Training details

• Global, Local, and Local+Global QS-Attn models train for 400 epochs with batch size 1 and a learning rate decaying from 2 × 10−4 to 0 over the final 200 epochs.All networks use Adam with β1 = 0.5 and β2 = 0.999.Source: E. Training details · p. 12

F. More Results

• Figures 7–9 compare the model with FSeSim, CUT, CycleGAN, and MUNIT on Cityscapes, Cat →Dog, and Horse →Zebra.Each figure places input source images in the leftmost column, followed by the model and the four baselines.Source: F. More Results · p. 12, Figure 7 · p. 13, Figure 8 · p. 14
• Global QS-Attn computes attention from flattened features, ranks rows by entropy, selects the N smallest-entropy queries, and applies their rows to the features.Source: F. More Results · p. 12
• Local QS-Attn forms window-based features, computes local attention, ranks rows by entropy, and retains the N smallest-entropy queries.Source: F. More Results · p. 12
• Figure 6 presents ablation results for different QS-Attn layers on two datasets.Source: Figure 6 · p. 12