ReCoG: Relational Concept Graph for Training-Free Personalization

• Training-free methods retrieve concept entries for a frozen VLM, while fine-tuning can be costly, slow, and difficult to repeat as concepts are added.Source: §1 · p. 1
• Isolated concept descriptions may capture salient shared attributes yet miss subtle cues that distinguish highly similar concepts.Source: §1 · pp. 1–2, Figure 1 · p. 2
• ReCoG stores directed, pair-specific discriminative evidence and uses it to compare competing candidates at inference time.Source: §1 · p. 2
• FINGER evaluates fine-grained personalization, including randomly sampled and highly similar bird species and CLEVR scene configurations.Source: §1 · p. 2
• ReCoG substantially outperforms prior approaches on FINGER and consistently achieves the best performance on recognition, image captioning, and VQA.Source: §1 · pp. 2–3

2.1 Personalization on VLMs and MLLMs

• Personalization methods extend frozen VLM representations through learned embeddings, concept-specific modules, or lightweight alignment, while some approaches require optimization or auxiliary modules.Source: §2.1 · p. 3

2.2 Training-Free Personalization

• Training-free systems such as RAP and R2P store and retrieve concept information, but primarily represent concepts independently with descriptive evidence.Source: §2.2 · p. 3

2.3 Personalization Benchmarks

• Existing personalization benchmarks cover compositional settings and diverse modalities, but carefully controlled fine-grained visual distinctions remain limited, motivating FINGER.Source: §2.3 · p. 3

3.1 Problem Formulation

• Training-free personalization builds a lightweight database from a small support set per concept, then retrieves entries to guide a frozen VLM without updating its parameters.Source: §3.1 · p. 3
• The formulation focuses on one exemplar per concept, with optional associated information such as a textual description or embedding.Source: §3.1 · p. 4

3.2 Conventional Relation-Agnostic Approaches

• Relation-agnostic methods represent concepts independently; CLIP embeddings enable efficient nearest-neighbor retrieval but may be limited for fine-grained personalization.Source: §3.2 · p. 4
• R2P augments concept embeddings with VLM-generated descriptions, retrieves top-K candidates by CLIP similarity, then verifies each candidate independently and selects the highest Yes confidence.Source: §3.2 · p. 4

4.1 Importance of Relation Among Concepts in Personalization

• Isolated descriptions can be salient yet non-discriminative, and independently computed confidence scores may be unreliable for global ranking among similar concepts.Source: §4.1 · p. 5

4.2 Constructing Relation-Aware Database

• ReCoG stores a directed graph whose concept nodes are connected by textual edges encoding how one concept differs from another; opposite directions can hold different evidence.Both directed edges are obtained from one VLM comparison per unordered pair.Source: §4.2 · p. 5, Figure 2 · p. 5

4.3 Discriminative Retrieval via Explicit Comparison

• After CLIP prunes concepts to a candidate set, ReCoG compares each candidate pair using both corresponding relational edges and returns the candidate with the most wins, breaking ties by CLIP similarity.Source: §4.3 · p. 6

4.4 Sparse ReCoG for Computational Efficiency

• Sparse ReCoG selects each concept’s top-k CLIP neighbors for edge construction, yielding at most Nk pair proposals and often fewer VLM calls; missing edges can be generated on demand or treated as ties.The no-call option may slightly reduce performance but stays within the precomputed graph.Source: §4.4 · p. 6

5 FINGER Benchmark

• FINGER targets personalized recognition on concept sets with high visual confusability.It is motivated by fine-grained distinctions such as closely related species or visually similar manufacturing parts.Source: §5 · p. 7
• The CUB tasks use a dataset of 200 bird species with subtle inter-class variation and semantic visual attributes, such as body color.The attributes enable controlled construction of challenging concept sets.Source: §5 · p. 7
• CUB-Easy samples 20 species randomly, while CUB-Hard selects visually similar species using attributes and focuses on predominantly black birds.Both tasks contain 20 concepts, one support image and 19 query images per concept; each has 400 images.Source: §5 · p. 7
• The CLEVR task treats each scene configuration as a concept and uses viewpoint-rendered images, with difficulty adjustable through scene-attribute overlap.It contains 20 concepts with one support image and nine query images per concept, totaling 200 images.Source: §5 · p. 7

6.1 Setup

• Experiments use one reference image per concept; recognition reports positive and negative accuracy and their weighted average, while captioning uses hard recall and VQA uses accuracy.Negative examples exhaustively use images from other concepts; database construction uses CLIP ViT-L and MiniCPM-o-2.6 by default.Source: §6.1 · pp. 7–8

6.2 Results on FINGER Benchmark

• ReCoG’s weighted accuracy reaches 98.1% on CUB-Easy, 92.8% on CUB-Hard, and 93.3% on CLEVR, exceeding R2P by 9.7, 11.8, and 13.7 percentage points, respectively.The gains are even larger on positive accuracy; ReCoG also maintains substantial performance on highly confusable concepts.Source: Table 1 · p. 8, §6.2 · p. 8

6.3 Results on Conventional Personalization Benchmarks

• ReCoG achieves the best performance across recognition, captioning, and VQA on MyVLM, Yo’LLaVA, and PerVA, outperforming prior methods under the same evaluation protocol.The reported results extend to conventional personalization scenarios where concepts are less confusable.Source: Table 3 · p. 9, §6.3 · p. 9

6.4 Analysis on Sparse ReCoG

• With k = 2, Sparse ReCoG reaches 85.0% P.Acc. using 26 VLM calls, close to full ReCoG’s 86.3% using all 190 pairwise edges.Performance largely saturates beyond this point, suggesting much useful relational evidence comes from visually similar neighbors.Source: §6.4 · p. 9, Figure 5 · p. 10

6.5 Isolating Relational Evidence from Pairwise Retrieval

• Method 3 raises P.Acc. from 63.8% to 74.0% with relational evidence, but remains 12.3 percentage points below ReCoG.The ablation separates relational evidence construction from candidate scoring and indicates that pairwise retrieval is more effective when conditioned on pair-specific evidence.Source: §6.5 · pp. 9–10, Table 4 · p. 10

6.6 Scalability to Large and Dynamic Concept Libraries

• On 200 CUB species, the no-call tie strategy reaches 56.2% P.Acc. versus R2P’s 48.6%, while requiring fewer VLM calls in offline construction and online inference.The alternative on-demand strategy caches newly generated missing edges after their first use.Source: §6.6 · p. 10

7 Conclusion

• ReCoG uses directed edges with pair-specific discriminative evidence to compare retrieved candidates, and FINGER evaluates recognition under controlled fine-grained difficulty.The reported gains over prior training-free methods are largest in the most confusable regimes.Source: §7 · p. 10
• Scalability may remain challenging as concept counts grow significantly or databases are updated online.The paper identifies more adaptive and efficient graph sparsification as future work.Source: §7 · p. 10

A Additional Details and Experiments on CLEVR

• The CLEVR pipeline defines concepts using object-level attributes from a controlled vocabulary and renders scenes from a fixed base scene.Attributes include shape, color, material, and size; custom scripts build on the original CLEVR utilities.Source: A Additional Details and Experiments… · p. 14

A.1 CLEVR-Default in the Main Paper

• CLEVR-Default contains 20 chain-organized concepts, each rendered into 10 views, with one support image and nine query images per concept.Each concept changes one or two object attributes from its predecessor; size and positions remain unchanged across neighboring concepts.Source: §A.1 · p. 14
• Neighboring CLEVR-Default scenes differ through local edits to shape, color, or material, keeping consecutive concepts highly similar.Most transitions make one edit, while a smaller number make two simultaneous edits.Source: §A.1 · p. 14

A.2 Difficulty-controlled CLEVR

• The additional CLEVR variants use base scene complexity K and hop size as difficulty controls for scene complexity and inter-concept distance.The figure captions identify performance comparisons across K for both 1-hop and 2-hop settings.Source: §A.2 · pp. 14–15, Figure A6 · p. 15, Figure A7 · p. 15
• An edit unit changes one attribute of an object or adds or removes one object; 1-hop applies one unit and 2-hop applies two sequentially.The variants provide different degrees of separation between concepts.Source: §A.2 · p. 15
• Each setting generates 20 scenes by expanding a root scene in two stages and then removing the root.The 1-hop and 2-hop families differ in the edit size used at each expansion step; duplicate scenes are removed in the 2-hop procedure.Source: §A.2 · p. 15
• Increasing K adds scene complexity, while 1-hop concepts are more locally confusable than 2-hop concepts.These controls enable analysis as tasks become more cluttered or concepts more subtly distinct.Source: §A.2 · p. 15

A.3 Experimental Results on Difficulty-controlled CLEVR

• Positive accuracy drops as CLEVR scenes grow from four to six objects, and 1-hop concepts are harder than 2-hop concepts despite involving fewer edits.The 1-hop concepts differ by a single local change and are described as more confusable.Source: §A.3 · p. 15
• At K = 6, ReCoG reaches 74.4% positive accuracy in 1-hop and 81.4% in 2-hop, with smaller K = 5-to-6 declines than competitors.In 1-hop, ReCoG drops from 76.6% to 74.4%; in 2-hop, it moves from 82.2% to 81.4%.Source: §A.3 · pp. 15–16

A.4 CLEVR with Changing Positional Arrangement

• ReCoG still substantially outperforms the baseline and R2P when candidate scenes differ mainly in object positions, although performance degrades overall.The setting restricts changes to objects’ xy positions, leaving candidate scenes nearly identical in object composition.Source: §A.4 · p. 16, Table A6 · p. 16

B Construction and Variance of CUB-Hard

• On CUB-Hard, ReCoG outperforms R2P with low variation across concept sets built from white, brown, or grey coloration.The original CUB-Hard set uses the 20 species with the highest mean scores across 16 black-coloration attributes.Source: B Construction and Variance of CUB-Hard · p. 16, Table A8 · p. 17

C Impact of Employed VLMs

• On CUB-Hard, ReCoG improves weighted accuracy over the baseline from 69.4 to 92.8 with MiniCPM-o-2.6 and from 66.4 to 94.5 with LLaVA-1.5.The authors report that the gain persists when the recognition backbone is replaced.Source: Table A7 · p. 16, C Impact of Employed VLMs · p. 17

D.1 Prompt Template for Relation-aware Database Construction

• Figure A8’s database-construction prompt asks a frozen VLM to compare support images and identify what makes each concept distinct from the other.This is designed to elicit relational, contrastive evidence rather than generic captions or independent concept descriptions.Source: §D.1 · p. 17
• For each ordered concept pair, the VLM returns two distinct feature lists, one from each concept’s perspective, in a strict JSON-only format.The textual outputs become discriminative edge evidence.Source: §D.1 · pp. 17–18
• Repeating the pairwise process stores explicit inter-concept differences alongside per-concept visual representations.The resulting evidence is intended to support fine-grained personalized recognition.Source: §D.1 · p. 18

D.2 Prompt Template for Comparison-based Retrieval

• Figure A9’s inference prompt asks a frozen VLM to choose which of two candidate support images better matches a query, conditioned on their distinctive features.The choice is not based only on global visual similarity.Source: §D.2 · p. 18
• After CLIP-based pruning leaves a small candidate set, the prompt compares visually similar concepts side by side to resolve remaining ambiguity.The comparison uses the relation-aware textual evidence extracted during database construction.Source: §D.2 · p. 18
• Each comparison returns a short reasoning trace and a final choice, and the pairwise decisions are aggregated to produce the prediction.The aggregation procedure is described in Sec. 4.3.Source: §D.2 · p. 18

E Validation on Realistic Open-Domain Personalized Concepts

• With one support image and 19 query images per identity, ReCoG raises CelebA positive accuracy from 76.1% to 94.2%.The experiment constructs all 190 pairwise relational edges among the 20 concepts.Source: E Validation on Realistic Open-Domain… · p. 18, Table A9 · p. 19

F Sensitivity to the Selected Support Image

• Across five support-image random seeds on CUB-Hard, ReCoG outperforms R2P every time and has lower performance variance, though its absolute performance varies across exemplars.The support image is resampled for every concept while other evaluation settings remain unchanged.Source: F Sensitivity to the Selected Support… · p. 19, Table A10 · p. 19

G Qualitative Results

• ReCoG correctly predicts kie by comparing candidates with relational evidence, while relation-agnostic retrieval focuses on shared attributes and incorrectly predicts nba.The comparison surfaces distinguishing cues such as a replaced green sphere versus nba and an added purple cylinder versus hex.Source: G Qualitative Results · p. 19, Figure A10 · p. 20

NeurIPS Paper Checklist

• The authors report that experimental settings and information needed to reproduce the main results are provided, and that data and code will be public upon acceptance.Source: NeurIPS Paper Checklist · pp. 23–24
• The checklist marks statistical significance reporting N/A because the proposed method is fully deterministic.Source: NeurIPS Paper Checklist · p. 24
• The authors report compliance with the NeurIPS Code of Ethics and state that the work has no notable societal impact or high-risk release concerns.Source: NeurIPS Paper Checklist · pp. 25–26
• The authors report crediting existing assets, documenting the new benchmark, and not conducting crowdsourcing or human-subject research.They state that benchmark documentation will be released upon acceptance.Source: NeurIPS Paper Checklist · pp. 26–27
• The checklist answers [N/A] for LLM involvement, directing readers to the NeurIPS handbook policy.It defines this answer as applying when core method development does not rely on LLMs as important, original, or non-standard components.Source: NeurIPS Paper Checklist · p. 28