PaperlayerView exact revision on arXiv

Source-linked AI summary

NearID: Identity Representation Learning via Near-identity Distractors

Aleksandar Cvejic, Rameen Abdal, Abdelrahman Eldesokey, Bernard Ghanem, Peter Wonka

arXiv:2604.01973v2cs.CV

TL;DR

Existing identity metrics can confuse object identity with background context, making personalized generation and editing evaluation unreliable. NearID removes contextual shortcuts with matched-context distractors and trains a lightweight projection head on a frozen backbone using hierarchical contrastive learning. The resulting representations substantially improve object- and part-level discrimination and align better with human judgments.

  • Problem

    Existing vision encoders and identity metrics can be fooled by visually similar distractors sharing the reference background, weakening identity-preservation evaluation.

  • Method

    NearID combines a matched-context dataset and discriminability protocol with a lightweight MAP head trained on frozen backbone features using same identity > NearID distractor > random negative.

  • Results

    99.17% SSR is achieved on NearID versus 30.74% for frozen SigLIP2, alongside improved MTG part-level discrimination and DreamBench++ human alignment.

  • Takeaways & Limitations

    NearID provides a diagnostic and identity-specific representation framework for testing whether vision embeddings capture genuine identity rather than contextual shortcuts.

  • Takeaways & Limitations

    The scope targets common rigid objects, distractor realism depends on inpainting fidelity, and stylized identity is left for future work.

Abstract

from arXiv · show

When evaluating identity-focused tasks such as personalized generation and image editing, existing vision encoders entangle object identity with background context, leading to unreliable representations and metrics. We introduce the first principled framework to address this vulnerability using Near-identity (NearID) distractors, where semantically similar but distinct instances are placed on the exact same background as a reference image, eliminating contextual shortcuts and isolating identity as the sole discriminative signal. Based on this principle, we present the NearID dataset (19K identities, 316K matched-context distractors) together with a strict margin-based evaluation protocol. Under this setting, pre-trained encoders perform poorly, achieving Sample Success Rates (SSR), a strict margin-based identity discrimination metric, as low as 30.7% and often ranking distractors above true cross-view matches. We address this by learning identity-aware representations on a frozen backbone using a two-tier contrastive objective enforcing the hierarchy: same identity > NearID distractor > random negative. This improves SSR to 99.2%, enhances part-level discrimination by 28.0%, and yields stronger alignment with human judgments on DreamBench++, a human-aligned benchmark for personalization. Project page: https://gorluxor.github.io/NearID/

1 Introduction

NearID exposes how shared backgrounds can make standard vision encoders confuse visually similar distractors with true identity matches. It addresses this with matched-context distractors, a frozen-backbone identity adapter, and hierarchical training that improves identity discrimination.

  • Motivation: Standard encoders can rank visually confusable NearID distractors above true identity views when backgrounds differ.This background entanglement undermines retrieval and identity-preservation scoring.
  • NearID framework: NearID places semantically similar but distinct instances on the exact reference background, isolating identity as the discriminative signal.The dataset and protocol are designed to remove contextual shortcuts from evaluation.
  • NearID framework: The method trains only a lightweight MAP projection head on a frozen foundation encoder using the hierarchy same identity > NearID distractor > random negative.The head comprises approximately 3.6% of total parameters and reshapes similarity geometry for identity.
  • Results: 99.17% SSR is achieved on NearID, compared with 30.74% for frozen SigLIP2.The evaluation uses matched-context distractors and positives under the NearID protocol.
  • Results: 35.0% SSR is achieved on MTG, while standard encoders reach 0.0% and VSM reaches 7.0% under the same protocol.The result indicates improved sensitivity to localized identity changes at the part level.

2 Related Work

Prior evaluation methods rely heavily on general semantic or holistic similarity, which can conflate object identity with shared context. NearID introduces a specialized identity subspace and matched-context evaluation to address this limitation across object-, part-, and human-alignment settings.

  • Visual foundation models: Foundation encoders are widely used for category-level alignment and dense visual representation across discriminative and generative tasks.The related work spans self-supervised, multimodal, and generative pretraining approaches.
  • Metric learning: Binary metric-learning objectives treat semantically related NearID samples and unrelated random negatives alike.This motivates an explicitly structured negative space.
  • Personalized image evaluation: CLIP-I and DINO scores can be inflated when a different instance appears on the correct background.The failure affects automated identity-preservation evaluation in personalization and editing literature.
  • Personalized image evaluation: Holistic metrics such as DreamSim evaluate layout, pose, and composition rather than isolating strict object identity.Consequently, structurally similar distractors sharing the reference context remain problematic.
  • NearID approach: NearID freezes the foundation backbone, tunes a lightweight MAP head, and evaluates identity using matched-context discriminability and human or oracle alignment.The framework targets both representation specialization and evaluation of background conflation.

3 NearID

NearID formulates identity preservation as structured metric learning that separates object identity from background context. It combines matched-context data, a hierarchical contrastive objective, and parameter-efficient adaptation of a frozen vision backbone.

  • NearID combines a hierarchical contrastive objective, a matched-context dataset with synthesized distractors, and margin-based evaluation aligned with human and oracle judgments.The framework is designed to disentangle identity from background context across training and evaluation.
  • 3.1 Overview and Problem Formulation: Each tuple contains an anchor, same-identity views from different backgrounds, and NearID distractors inserted into the anchor’s exact background.This construction removes contextual shortcuts and forces discrimination using intrinsic identity cues.
  • 3.2 NearID Loss: The learning objective enforces the similarity hierarchy same identity > NearID distractor > random batch negative.NearID differs from objectives that treat all negatives uniformly by explicitly preserving graded similarity among candidates.
  • 3.2 NearID Loss: The model freezes the SigLIP2 backbone and trains only a lightweight MAP projection head to reshape normalized embeddings for identity discrimination.The MAP head selectively aggregates identity-salient patch features while updating approximately 3.6% of total parameters.
  • 3.2 NearID Loss: NearID loss combines discrimination and ranking terms so positives outrank distractors while distractors remain closer than generic batch negatives.The ranking regularizer preserves graded local semantic structure instead of treating near-identity distractors as ordinary negatives.
  • 3.3 Dataset Construction: The synthesis pipeline extends SynCD with near-identity distractors and filters its source data to improve identity fidelity and prevent semantic overlap.SynCD supplies multi-view positives across backgrounds, while MTG provides a localized part-edit calibration signal.

4 Experiments

Experiments evaluate NearID with a frozen SigLIP2 backbone and lightweight head across matched-context identity discrimination, part-level transfer, and metric alignment. NearID improves separation and performance, while ablations show that the loss must balance discrimination against representation collapse.

  • Implementation Details: All proposed embedding models use a frozen SigLIP2-so400m-patch14-384 backbone, with optimization focused on the MAP projection head.The head outputs 1152-dimensional ℓ2-normalized embeddings and updates approximately 15M of roughly 428M parameters.
  • Implementation Details: Role-aware foreground masking blacks out backgrounds with different probabilities for anchors, positives, and distractors to support background-invariant transfer.Masking probabilities are 0.5 for anchors, 0.2 for positives, and 0.6 for distractors.
  • Results: NearID reaches 99.17% SSR and 99.71% PA on matched-context evaluation, compared with 30.74% SSR for frozen SigLIP2.The evaluation pools SSR and PA across distractor sources using support-weighted averaging.
  • Results: On MTG, NearID achieves 35.0% SSR, while standard encoders score 0.0% and VSM reaches 7.0% under the same protocol.NearID also improves MTG oracle alignment, with M–O increasing to 0.465 from 0.180.
  • Results: KernelPCA projections show NearID separating matched-context distractors from positive clusters more clearly than the frozen baseline.The visualization uses 12 identities with identical KernelPCA settings in both panels.
  • Training Objective Ablation: InfoNCE raises SSR to 60.97% but leaves near-identity distractors frequently above true positives, whereas the NearID loss reaches 99.17% SSR with M–H = 0.545.Circle + Ranking and Oracle Ranking achieve high discrimination but collapse general-purpose alignment; NearID trades only 0.010 M–H relative to InfoNCE.
  • Training Objective Ablation: Adding positive cohesion produces only marginal gains, reaching 99.31% SSR while MTG SSR changes from 35.0% to 36.0%.The ablation concludes that the base LNearID objective suffices for the primary setting.

5 Conclusion

NearID exposes identity–background entanglement in vision encoders and addresses it with matched-context distractors, a hierarchical contrastive objective, and lightweight adaptation. The resulting representations improve object- and part-level identity discrimination and alignment with human judgments, while remaining bounded in scope.

  • Even the best frozen encoder achieves only 30.74% SSR when NearID distractors share the reference background.
  • NearID combines 19K identities, 316K+ matched-context distractors, a strict similarity hierarchy, and a discriminability-based evaluation protocol.
  • 99.17% SSR is achieved by training only a lightweight MAP head on a frozen backbone, alongside improved MTG discrimination and human-judgment alignment on DreamBench++.
  • NearID is intended as a diagnostic for whether vision representations capture genuine identity rather than contextual shortcuts.
  • The reported scope covers common rigid objects, while distractor realism depends on inpainting fidelity and stylized-data identity remains untrained.

A Contrastive Training Paradigm Comparison

NearID differs from conventional contrastive paradigms in both how positives are generated and what identity variation they represent. Its positives are multi-view, 3D-consistent images rather than augmented copies or caption-conditioned generations.

  • SimCLR uses strong stochastic augmentations of one real image to form positive pairs and encourage pixel-level invariance through pairwise InfoNCE.
  • StableRep replaces hand-crafted augmentations with generative diversity by producing images from one caption under different diffusion noise seeds.
  • NearID uses high-quality multi-view positives generated through SynCD, which leverages Objaverse 3D assets, depth-conditioned diffusion, and cross-view feature warping.
  • NearID provides up to three 3D-consistent views per identity, distinguishing its positives from both augmented copies and caption-conditioned generations.

B Dataset Construction Details

The dataset construction uses multiple inpainting and image-generation systems configured for photorealistic, context-coherent distractor synthesis. The listed pipelines vary substantially in architecture, guidance, and denoising schedules.

  • PowerPaint-v2 uses a Stable Diffusion v1.5 backbone with BrushNet conditioning for mask-image-guided inpainting.Generation uses 45 denoising steps, CFG 7.5, and a BrushNet conditioning scale of 1.0.
  • SDXL-Inpainting uses the SDXL-1.0-inpainting-0.1 checkpoint with 30 denoising steps, CFG 8.0, and inpainting strength 0.99.Negative prompts suppress text, watermarks, and low-resolution artifacts.
  • FLUX.1-Fill and FLUX.1-Canny-Inpaint generate masked or topology-preserving edits with 50 and 28 steps, respectively.The Canny pipeline uses guidance scale 7.0, inpainting strength 0.99, and edge thresholds τ_low = 50 and τ_high = 200.
  • The Qwen-Image pipeline combines InstantX ControlNet with an 8-step Lightning LoRA and Flow-Match Euler Discrete scheduling for high-throughput distractor generation.Its true CFG scale is 1.0.

C Additional Qualitative Results

The qualitative and methodological materials describe per-image comparisons across four metrics, computational costs for NearID and a VLM baseline, and objective-aggregation procedures. They also clarify how ablations differ in their treatment of NearID distractors.

  • Qualitative comparisons on NearID and MTG report anchor-relative scores for SigLIP2, VSM, Qwen3-VL 30B, and NearID.The anchor image itself has no score; MTG and NearID visualizations appear in Figures 8 and 7, respectively.
  • Training the NearID MAP head uses 15M trainable parameters and approximately 6.5 hours on one NVIDIA A100 for 11 epochs.The reported run uses about 3,350 gradient steps and batch size 128, with the backbone frozen.
  • Evaluating the full Qwen3-VL 30B NearID benchmark requires approximately 54 A100-hours, roughly 324× the cost of the NearID training run.
  • The ablation keeps the frozen backbone and MAP head fixed while varying only the loss function and how NearID distractors enter training.The standard InfoNCE baseline receives distractors but ignores them in the loss, relying on in-batch negatives.
  • Per-sample Pearson correlations are aggregated with Fisher’s z-transformation across reported M–O, M–Opair, and M–H metrics.

E.3 Score Distribution Analysis

The evaluation averages identity metrics across multiple inpainting sources, including both training and unseen generators. Frozen baselines show substantial variation across sources, while NearID tracks the oracle curve most closely.

  • NearID follows the oracle curve of scores most closely in Figure 9.
  • SSR and PA are averaged across seven fill-based inpainting sources, including four training sources and three unseen generators.Additional Canny-guided sources are reported separately and excluded from the main evaluation because they provide an extra structural prior.
  • 12–67% SSR: frozen SigLIP2, Qwen3-VL 30B, and VSM exhibit large crosssource variance.

E.5 Training Data Ablation

Training data ablations show that matched-context NearID distractors provide the dominant signal for object-level identity discrimination, while masking and auxiliary data shape broader robustness. Positive cohesion offers only marginal gains over the base objective.

  • Training Data Ablation: 40.8% SSR: removing NearID data while retaining MTG reduces NearID performance from 99.2% to 58.3%.Removing MTG alone costs only 0.8%, establishing NearID distractors as the dominant training signal.
  • Training Data Ablation: 11.5% SSR: training with one inpainting engine instead of four drops performance from 98.4% to 86.9%.The ablation attributes this to reduced source diversity and greater exposure to generator-specific artifacts.
  • Background Masking: Role-aware background masking improves foreground oracle alignment from 0.465 to 0.641 while retaining MTG SSR near 32–35%.Masking rates are 0.5 for anchors, 0.2 for positives, and 0.6 for distractors; the mask-free variant relies more on background context.
  • Positive Cohesion: The cohesion objective averages each positive view’s cosine distance to a shared prototype, with total loss L = LNearID + β Lcoh.Samples with fewer than two valid positives contribute zero cohesion loss.
  • Positive Cohesion: 0.1% SSR: adding positive cohesion raises NearID SSR from 99.2% to 99.3% while MTG SSR remains comparable at 36.0% versus 35.0%.NearID PA, M–O, and M–H also remain comparable, indicating limited benefit over the base objective.
  • Background Dependence: Background removal improves frozen-baseline SSR by +41.0% for CLIP, +43.3% for DINOv2, and +33.9% for SigLIP2.VLM gains are smaller and diminish at scale, from +11.0% for Qwen3-VL 4B to +5.5% for 30B.

E.9 Ranking Weight Ablation (α)

The ranking-weight ablation reveals a trade-off between sharp identity discrimination and graded similarity alignment. A moderate ranking weight is preferred, whereas excessive weighting harms both objectives.

  • Ranking Weight Ablation: 99.6% SSR: the discrimination-only loss with α=0 already achieves near-perfect NearID identity discrimination.
  • Ranking Weight Ablation: Oracle alignment generally increases with α up to 1.0 while SSR degrades mildly, revealing a trade-off between similarity structure and discrimination sharpness.
  • Ranking Weight Ablation: At α=2.0, both SSR and M–O decline, indicating that an overly dominant ranking signal begins to collapse the representation.The adopted default configuration is α=0.5.

E.10 Positive Cohesion Weight Ablation (β)

Positive cohesion was tested as an additional prototype-centered objective. Across the supplied ablation, it does not improve the broader alignment measures and leaves NearID SSR nearly unchanged.

  • Positive Cohesion Weight Ablation: Adding positive cohesion consistently degrades M–O, MTG SSR, and M–H relative to β=0 while NearID SSR remains approximately 98.8%.The result indicates that cohesion does not improve identity discrimination under this ranking-weight setting.
  • Positive Cohesion Weight Ablation: The authors conclude that the ranking term already supplies sufficient identity-consistent pull, making explicit cohesion a redundant gradient signal.

E.11 InfoNCE Component Ablation: Distractors and Ranking

The ablation shows that near-identity distractors improve discrimination, while oracle ranking alone can collapse alignment. The full NearID formulation balances near-perfect identity discrimination with stronger oracle and human alignment.

  • Distractors in the denominator: 99.6% SSR results from adding near-identity distractors to the softmax denominator, but M–H falls to 0.251.The distractors improve discrimination while degrading alignment with human judgments.
  • Oracle ranking alone: 86.3% SSR from oracle ranking alone remains below near-perfect discrimination, while M–H collapses to 0.167.Without distractors in the denominator, pairwise oracle ranking overfits the ranking signal.
  • Combined mechanisms: 99.6% SSR combined with M–H of 0.227 shows that adding oracle ranking to one-directional InfoNCE does not produce well-calibrated embeddings.The combined mechanism recovers discrimination but leaves human alignment degraded.
  • NearID formulation: 99.2% SSR, 0.545 M–H, and 0.465 M–O show that LNearID resolves the discrimination–alignment trade-off.Its multi-positive symmetric softmax and unsupervised softplus ranking regulariser preserve graded similarity while maintaining strong discrimination.
  • NearID formulation: The ranking term controls the discrimination–alignment trade-off smoothly, with α=0.5 providing the best balance in the primary setting.This result comes from the reported α-sweep.
Loading 2604.01973v2…