Preserve-and-Compose Training for Composed Image Retrieval

• CIR must satisfy the requested modification while retaining relevant visual evidence from the reference image.Following an instruction while depicting the wrong person is no more satisfactory than preserving the person while ignoring the requested action.Source: §1 · p. 1
• Target-image triplets provide direct supervision for change and preservation, but finding or generating suitable targets is costly.Target captions offer an alternative, but may omit appearance or clothing details from the source that still matter for retrieval.Source: §1 · p. 1
• PACT aligns composed queries with target captions and uses source-image evidence to preserve information those captions may omit.The source-image representation supervises information shared across multiple modifications of the same source.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Chord combines query–candidate similarity with agreement between the requested change and the candidate’s change from the reference.It can be applied to existing query models without retraining.Source: §1 · p. 2
• PACT achieves state-of-the-art or competitive results across four CIR benchmarks, with strong gains on CIRR and CIRCO.Qualitative retrieval over external image and movie galleries shows better preservation of reference-specific visual identity; ViT-L/14 training takes less than one hour on a single RTX 3090.Source: §1 · p. 2

2.1 CIR WITHOUT TARGET-IMAGE SUPERVISION

• Image–text–text methods retain the source image but replace the target image with a caption; PACT uses that source as direct visual supervision for preservation.Other zero-shot approaches use image–caption pairs for visual inversion or text–text–text triplets for language-based composition.Source: §2.1 · p. 3

2.2 RETRIEVAL SCORING FOR COMPOSED QUERIES

• Unlike methods that use geometric cues in query construction or representation learning, Chord adds reference-relative change only at ranking time.It leaves the learned query unchanged, needs no additional model or retraining, and retains cosine scoring’s asymptotic retrieval complexity.Source: §2.2 · p. 3

3.1 CONSTRUCTING IMAGE–TEXT–TEXT TRAINING DATA

• The ITT corpus pairs source images with modification instructions and target captions, using rule-based caption edits and image-conditioned generation to create 856,089 modification triplets.The corpus contains 313,240 source images, with up to three retained modifications per source; negative captions preserve relevant source information but fail to satisfy the requested modification.Source: §3.1 · p. 3

3.2 CAPTION-GUIDED COMPOSITION: Lcompose

• A lightweight MLP maps each source image to a pseudo-word inserted into the modification prompt, while adapted text-encoder LoRA parameters learn composition and identity queries.The image encoder and teacher text encoder remain frozen; identity queries align with source captions and modified queries with target captions.Source: §3.2 · p. 4

3.3 PRESERVATION OBJECTIVE: Lpreserve

• The preservation objective aligns information shared across multiple modified queries of a source with its image representation, while negative-caption discrimination discourages failing to make the requested change.The identity query is excluded, and sources represented only once do not contribute to the visual preservation loss.Source: §3.3 · p. 5

3.4 INFERENCE WITH CHORD SCORING

• Chord scoring combines composed-query–candidate similarity with agreement between the query change and the candidate’s change from the reference image.Its reference-relative term compares change directions independently of their magnitudes; scoring retains O(Nd) asymptotic complexity with one additional candidate-wise cosine computation.Source: §3.4 · pp. 5–6

4.1 EVALUATION PROTOCOL

• Unless stated otherwise, benchmark comparisons use ViT-L/14; PACT averages five runs, and its Chord weight is validation-selected and fixed for test and transfer evaluation.The main tables compare complete retrieval systems rather than training objectives in isolation.Source: §4.1 · p. 6

4.2 BENCHMARK PERFORMANCE

• PACT achieves the best compared results on CIRCO across reported metrics, including mAP@5 of 23.59, and leads CIRR R@1 and R@5 and FashionIQ category-average R@10.De-MINDS remains higher on CIRR R@10 and FashionIQ R@50.Source: §4.2 · p. 6, Table 1 · p. 6
• On GeneCIS, PACT has the highest R@1 across all four reported marginals and exceeds DiffComp on all four R@1 metrics, though DiffComp leads on Focus and Object R@2.The Chord weight transfers from the other benchmarks without GeneCIS tuning.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 ABLATION OF TRAINING OBJECTIVES AND RETRIEVAL SCORING

• The composition-plus-preservation configuration reaches a three-benchmark mean of 29.70, surpassing composition-only training at 27.49 and improving all three benchmarks.Neither visual nor negative-caption supervision alone recovers the full improvement.Source: §4.3 · p. 7, Table 3 · p. 7
• PACT’s gains over composition-only training persist with both cosine and Chord, separating a learned-query improvement from Chord’s additional inference-time gain.The mean gains are 2.72 points with cosine and 2.21 with Chord; comparisons use matched checkpoints.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 EFFECTIVENESS AND BEHAVIOR OF CHORD SCORING

• Post-hoc Chord improves the three-benchmark mean for Pic2Word, LinCIR, RTD, and PACT, with the largest gain for PACT; its correction rate is higher for smaller cosine deficits.Figure 3 summarizes mean improvements and correction rates for cosine-misordered pairs.Source: §4.4 · p. 8, Figure 3 · p. 8

4.5 MODEL SIZE AND EFFICIENCY

• With ViT-L/14, PACT completes 1,000 joint-training steps in about 52 minutes on one RTX 3090, using 8.19 GiB peak allocated memory.The image encoder remains frozen; timing excludes evaluation, checkpoint saving, and offline data preparation.Source: Figure 4 · p. 8, §4.5 · p. 9

4.6 REFERENCE PRESERVATION BEYOND THE BENCHMARKS

• On MovieNet, the evaluation measures whether top-1 retrieval preserves the annotated reference identity under scene, background, or action edits; PACT improves identity preservation over RTD.Figure 5 provides selected disagreements illustrating identity retention while following scene- and background-level modifications.Source: §4.6 · p. 9, Table 4 · p. 9, Figure 5 · p. 9

5 CONCLUSION

• PACT learns without target-image supervision or gallery updates, aligning composed queries with target captions while using source images and negative-caption discrimination to support preservation and modification.Chord scoring adds reference-relative change agreement at inference and improves multiple query models without retraining.Source: §5 · p. 9
• The authors report using generative AI tools to brainstorm research ideas and assist with experimental design.The reproducibility statement says appendices document data construction, optimization, evaluation protocols, and related procedures.Source: AI USE STATEMENT · p. 10, REPRODUCIBILITY STATEMENT · p. 10
• The experiments use existing datasets and pretrained models, whose biases and errors may affect generated supervision and retrieval results despite automated filtering and evaluation.MovieNet character-preservation results do not establish suitability for real-world person identification.Source: ETHICS STATEMENT · p. 10

A.1 SOURCE IMAGES AND SUPERVISION

• Each image–text–text triplet pairs a source image and modification instruction with a target caption; the source image supplies visual supervision and no target image is required.The corpus contains 313,240 source images and 856,089 modification triplets, plus identity entries for a 1,169,329-entry composition-training pool.Source: §A.1 · p. 13

A.2 MODIFICATION GENERATION

• Caption-based substitutions and language-model generation, alongside image-conditioned generation, produce 856,089 modification records in total.The image-conditioned route contributes 542,849 eligible records.Source: §A.2 · p. 13

A.3 FILTERING AND NEGATIVE CAPTIONS

• Filtering removes malformed, duplicate, unchanged, overlength, or inconsistent proposals, while negative captions must preserve source information but fail the requested modification.Automatic eligibility filters do not constitute human verification of every retained triplet.Source: §A.3 · p. 13

B.1 BACKBONES AND OPTIMIZATION

• PACT initializes from CLIP ViT-B/32 and ViT-L/14 or OpenCLIP ViT-G/14, freezing the image and teacher text encoders while learning the mapper and text-attention LoRA.Training uses 333 identity-warmup updates followed by 1,000 joint updates; Table 5 summarizes the ViT-L/14 configuration.Source: §B.1 · p. 13, Table 5 · p. 14

B.2 COMPOSITION AND PRESERVATION SAMPLING

• Composition batches contain 512 independently sampled identity and modification records; separate preservation batches contain 256 caption-based and 256 image-conditioned records.Sampling is with replacement, so the fixed update schedule does not correspond to an epoch over the eligible pool.Source: §B.2 · p. 14

B.3 VISUAL SUPERVISION WITH REPEATED POSITIVES

• For repeated sources, the visual objective pools their modified queries and treats repeated or near-identical source images as positive mass, not negatives.Sources sampled once remain in the teacher bank but do not contribute pooled queries; the visual loss is zero if no source repeats.Source: §B.3 · p. 14

C.1 BENCHMARKS AND METRICS

• CIRR and CIRCO use test evaluation, while FashionIQ and GeneCIS use validation; FashionIQ reports the unweighted mean of its three category recalls.CIRR reports global top-50 and subset top-3, CIRCO uses a 123,403-image gallery, and GeneCIS reports recall at ranks 1–3 across four task summaries.Source: §C.1 · p. 15

C.2 SOURCES OF GENECIS BASELINE RESULTS

• GeneCIS baseline comparisons combine released-checkpoint evaluations, a reproduced RTD checkpoint, and published results, with unavailable metrics left unreported.The published rows are not presented as local reproductions; MoTa-Adapter has only an R@1 summary of 16.7.Source: §C.2 · p. 15, Table 6 · p. 15

C.3 CHECKPOINT AND SCORING-WEIGHT SELECTION

• Checkpoint and Chord-weight selection uses a validation mean across CIRR, CIRCO, and FashionIQ; each seed’s selected coefficient is then used across datasets, including untuned GeneCIS.The coefficient is selected from λ ∈ {0, 0.02, …, 1}; test scores do not enter selection, and matched checkpoints in Table 3 isolate scoring-rule differences.Source: §C.3 · p. 15

C.4 VARIABILITY ACROSS TRAINING SEEDS

• Across metrics and backbones, five-seed standard deviations range from 0.115 to 1.072 percentage points; CIRCO mAP@5 has the largest standard deviation for every backbone.FashionIQ category-average R@10 standard deviations range from 0.115 to 0.340 points.Source: §C.4 · pp. 15–16, Table 7 · p. 16

C.5 ADDITIONAL BACKBONE RESULTS

• ViT-B/32 and ViT-G/14 results use five-seed means and sample standard deviations, with Chord coefficients selected separately per seed on the three-benchmark validation objective.CIRR and CIRCO scores are from test sets, while FashionIQ scores are from validation; FashionIQ category means and GeneCIS four-task means are averaged within each seed.Source: §C.5 · p. 17, Table 8 · p. 17

C.6 CATEGORY-WISE FASHIONIQ RESULTS

• FashionIQ ViT-L/14 results report validation recall by category and the equally weighted category mean computed within each training run.Table 9 gives means and sample standard deviations across five seeds.Source: §C.6 · p. 17, Table 9 · p. 17

C.7 COMPARISON WITH PUBLISHED RESULTS ACROSS BACKBONES

• Across backbones, published baselines retain their original training and scoring procedures, and unavailable backbone–metric results remain unreported.SEARLE denotes the corresponding SEARLE-XL backbone variant; baseline sources include RTD and LinCIR supplementary tables.Source: §C.7 · p. 18
• PACT has the highest CIRR R@1/5/10 and CIRCO mAP@5/10/25 with both backbones; CIReVL leads FashionIQ with ViT-B/32 and De-MINDS with ViT-G/14.Tables 10 and 11 compare published baselines with five-seed PACT means and standard deviations.Source: Table 10 · p. 18, Table 11 · p. 18, §C.7 · p. 19

C.8 GENECIS COMPARISON ACROSS BACKBONES

• GeneCIS comparisons preserve the published aggregation level: Table 12 uses four-task averages, while Table 13 reports ViT-G/14 task-level results.The ViT-B/32 CIReVL row uses its own evaluation setting, and the LinCIR + MoTa-Adapter summary has no task-level values.Source: §C.8 · p. 19
• PACT has the highest mean in each displayed ViT-G/14 GeneCIS task and recall rank among the compared methods.The four tasks are Focus Attribute, Change Attribute, Focus Object, and Change Object.Source: §C.8 · p. 19, Table 13 · p. 19

D.1 RETRIEVAL IMPLEMENTATION AND PER-DATASET GAINS

• PACT improves over cosine by 6.74 points on CIRR, 5.50 on CIRCO, and 4.14 on FashionIQ.The comparison uses the same query representations and candidate embeddings within each model; coefficients were selected on validation data.Source: §D.1 · p. 20, Table 14 · p. 20
• Chord raises aggregate performance for other query models, though gains need not occur on every dataset: RTD decreases by 0.10 points on FashionIQ while improving on CIRR and CIRCO.Chord uses unit-normalized representations and a reference-relative term; gallery embeddings remain fixed.Source: §D.1 · p. 20

D.2 COSINE AMBIGUITY AND PAIR CORRECTION

• Pair correction compares annotated positives with the highest-cosine unannotated candidate, binning pairs into shared cosine-deficit quintiles where Q5 has the smallest deficits.Rank standardization controls rank composition across models; on CIRCO, candidates outside the annotated positive set are unannotated rather than verified irrelevant.Source: §D.2 · p. 20
• At λ = 0.34, corrected-pair rates are averaged equally across three benchmarks, with 95% reference-cluster bootstrap intervals measuring query-sampling variation.Figure 6 reports benchmark-specific curves; its shading marks the intervals.Source: §D.2 · p. 20, Figure 6 · p. 21

E.1 EVALUATION SETS / E.2 GALLERY AND QUERY CONSTRUCTION

• The annotation-based MovieNet evaluation uses 6,162 frames from 66 films and constructs 200 queries with distinct references from 62 films.Queries pair same-person frames from the same film, with different contents and shot indices at least 40 apart; retrieval excludes the reference and same-shot images.Source: §E.2 · p. 21, §E.1 · p. 21, Table 15 · p. 21

E.3 IDENTITY AND MODIFICATION CRITERIA

• Identity preservation requires the retrieved image’s official annotations to include the reference person’s film-ID/person-ID pair, but does not require preserving clothing, color, or texture.Modification fulfillment is assessed separately by Qwen3.5-27B, and the joint criterion requires both identity retention and modification fulfillment.Source: §E.3 · p. 22

E.4 EFFECT OF VISUAL SUPERVISION

• Adding visual supervision raises Top-1 identity from 58.0% to 78.5% with cosine and from 62.0% to 81.5% with Chord.Full PACT exceeds RTD by 16.5 points with cosine and 18.0 points with Chord.Source: §E.4 · p. 22

F.1 GALLERY CONSTRUCTION

• The qualitative gallery contains 100,000 images from 104 films, including 9,660 images from four query films and 90,340 from 100 additional films.Unlike the annotation-based collection, this gallery is not restricted to frames with unambiguous person-identity annotations.Source: §F.1 · p. 22

F.2 REFERENCE IMAGES AND MODIFICATIONS

• The qualitative set contains 1,151 queries from 90 references across four films, with multiple modifications associated with some references.Instructions request different states of the same person and may isolate action, background, clothing, or pose conditions.Source: §F.2 · p. 22

F.3 BASELINE CONFIGURATIONS

• Qualitative comparisons use the same 100,000-image MovieNet gallery and ViT-L/14 backbone; Image-only and Instruction-only use frozen original CLIP encoders, while Compose-only retains both modalities.Compose-only changes the joint objective by disabling visual and negative-caption preservation terms; full PACT uses the complete objective, whereas RTD uses cosine scoring.Source: §F.3 · p. 23

F.4 COMPARISON AND INTERPRETATION

• The selected comparisons do not estimate overall success rates; Figure 11 likewise shows examples rather than failure frequency.Source: §F.4 · p. 23, §F.5 · p. 28
• In S1–S4, PACT satisfies both requests in each comparison, while composition-only misses character identity, the requested character, or mirror reflections.Source: Figure 7 · p. 24, Figure 8 · p. 25
• In S5–S8, PACT retrieves the pale pillow; both methods retrieve the damaged-car scene and satisfy the ornate-room and work-table requests.Source: Figure 9 · p. 26, Figure 10 · p. 27

F.5 FAILURE CASES: FINE-GRAINED MODIFICATION CONSTRAINTS

• Both methods miss a requested two-fist pose and a looking-down-at-a-wrist action, despite recovering other visual cues in those examples.Source: Figure 11 · p. 28

G.1 TRAINING-TIME MEASUREMENT

• With batch size 512 on one RTX 3090, 333 identity-warmup and 1,000 joint-training steps take 52 min 9 s and peak at 8.19 GiB.Timing excludes evaluation, checkpoint saving, offline data generation, and image-feature preparation; measurements use cached supervision and frozen image features.Source: §G.1 · p. 29

G.2 INFERENCE PARAMETER ACCOUNTING

• PACT-L/14 has approximately 443M inference parameters; Figure 4 uses CIRCO test mAP@5 and distinguishes architecture-derived estimates from lower bounds.Source: §G.2 · p. 29
• MoTa-Adapter estimates are 441.81M (L/14) and 2.57895B (G/14); MCL totals are 2.975B, 6.988B, and 7.068B for its three listed variants.The MCL variants use OPT-2.7B, OPT-6.7B, and Llama2-7B, respectively.Source: §G.2 · p. 29
• PrediCIR component lower bounds are 434.69M for L/14 and 2.54664B for G/14; CoLLM's count omits unspecified adapter and projection overhead.Source: §G.2 · p. 29
• CIReVL's GPT-3.5 component count is undisclosed, so its illustrative 175B plot position is not a measured size or a quantitative parameter-ratio basis.Source: §G.2 · p. 29