PaperlayerView exact revision on arXiv

Source-linked AI summary

Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understanding

Eun Woo Im, Dhruv Madhwal, Vivek Gupta

arXiv:2604.13313v1cs.LG

TL;DR

VLMs struggle with compositional semantics because contrastive pretraining lacks sufficiently informative negatives and existing mining methods do not explicitly control which linguistic elements change. The paper uses lexical concreteness to generate stronger hard negatives and an adaptive-margin loss to address InfoNCE gradient imbalance, with Slipform achieving state-of-the-art compositional-understanding performance. Its scope remains bounded by non-visual components of standard concreteness scores and a reported trade-off with general visual representation.

  • Problem

    VLM contrastive pretraining often lacks effective negatives for distinguishing word order and attribute binding, while existing hard-negative methods do not explicitly control the modified linguistic element.

  • Method

    ConcretePlant targets concrete concepts during hard-negative generation, while Cement loss uses a concreteness-conditioned adaptive margin to regulate gradient magnitudes.

  • Results

    Slipform achieves state-of-the-art performance across evaluated compositional-understanding benchmarks, with a 13.13% relative macro-average gain over the standard CLIP baseline.

  • Takeaways & Limitations

    Concrete concepts produce more structurally discrepant and informative negative pairs, while adaptive margins help models use hard-negative signals more effectively.

  • Takeaways & Limitations

    Standard concreteness scores include tactile, auditory, olfactory, and gustatory perceptibilities, and the method exhibits a trade-off between compositional understanding and general visual representation.

Abstract

from arXiv · show

Vision-Language Models demonstrate remarkable capabilities but often struggle with compositional reasoning, exhibiting vulnerabilities regarding word order and attribute binding. This limitation arises from a scarcity of informative samples needed to differentiate subtle semantic variations during contrastive pretraining. Although hard negative mining offers a promising remedy, existing methods lack explicit mechanisms to dictate which linguistic elements undergo modification. Instead of engineering generative architectures, this study establishes lexical concreteness as a fundamental determinant of negative sample efficacy. Modifying highly concrete terms generates more pronounced structural and visual discrepancies, providing a substantially stronger learning signal. Leveraging this principle, ConcretePlant is proposed to systematically isolate and manipulate perceptually grounded concepts. Analyses of the InfoNCE further reveals a severe gradient imbalance, where easily distinguishable pairs disproportionately overwhelm the optimization process and restrict the bandwidth available for nuanced learning. To resolve this degradation, the Cement loss is formulated utilizing a margin-based approach. By correlating psycholinguistic scores with sample difficulty, this objective dynamically calibrates the penalization applied to individual training pairs. Comprehensive evaluations substantiate these theoretical claims. The integrated framework, designated as Slipform, achieves state-of-the-art accuracy across diverse compositional evaluation benchmarks, general cross-modal retrieval, single and multi label linear probing.

1. Introduction

VLMs often miss word order and attribute binding because random contrastive-pretraining negatives do not distinguish fine-grained compositional semantics. The paper proposes controlling lexical concreteness in hard-negative generation and addressing InfoNCE gradient imbalance with adaptive margins.

  • Randomly sampled negatives often fail to distinguish compositional semantics, encouraging VLMs to match global summaries instead of grammatical structure and relational bindings.
  • Hard negatives improve compositional understanding by pairing misaligned images and captions that retain substantial semantic overlap.Existing generation methods range from POS-based text swaps and spatial heuristics to LLMs, image editing, and generative models.
  • Existing hard-negative generators perturb selected keywords, making sample quality sensitive to which keyword is chosen.This motivates focusing on data quality rather than designing a better generator architecture.
  • Lexical concreteness is linked to hard-negative structural discrepancy, so targeting highly concrete entities encourages localized physical modifications instead of ambiguous global transformations.The study compares datasets generated at controlled concreteness levels to test this hypothesis.
  • Concrete Jungle combines ConcretePlant for concrete-concept hard-negative generation with Cement loss, which uses adaptive margins to regulate gradient magnitudes.The framework is reported to achieve superior performance across benchmarks.

2. Maximizing Hard Negative Efficacy

ConcretePlant improves hard-negative efficacy by selecting highly concrete keywords, which produce clearer visual and structural perturbations. Cement loss then reallocates gradient capacity toward informative hard negatives using an adaptive margin.

  • Concreteness Matters in Hard Negative Mining: Concrete keywords yield larger structural and visual changes than abstract keywords, producing more distinguishable hard-negative images.The method links lexical concreteness to the physical difference between positive and generated negative images.
  • ConcretePlant: Automating Concrete Negative Generation: ConcretePlant uses Qwen3-32B to perturb the selected keyword and SDXL-Turbo to generate the corresponding hard-negative image.The captioner minimizes changes to the remaining caption content, while the image generator produces the visual counterpart.
  • ConcretePlant: Automating Concrete Negative Generation: ConcretePlant samples caption keywords by concreteness while balancing attribute, object, and relation categories.It decomposes captions into semantic keywords, retrieves concreteness ratings, and applies top-K sampling with rule-based category balancing.
  • Learning Slipform from Cement Loss: Easy negatives consume at least 72% of the total gradient signal at N=1024, limiting InfoNCE’s focus on subtle compositional differences.The aggregated probability mass of easy negatives grows with batch size, creating gradient imbalance even in the hard-negative training setup.
  • Learning Slipform from Cement Loss: Cement loss adds a margin to the hard-negative similarity, shifting probability mass from easy negatives toward fine-grained compositional boundaries.The margin is adaptively bounded and modeled with a Fermi-Dirac distribution using concreteness scores because the sampled score distribution is left-skewed.
  • Learning Slipform from Cement Loss: Higher-concreteness samples exhibit larger positive–hard-negative logit gaps and higher retrieval accuracies than low-concreteness samples.The reported dataset comparisons support the claim that concreteness produces more distinguishable visual content in hard-negative images.

3. Experiments

Experiments evaluate concreteness-controlled hard negatives and gradient balancing across dataset analyses, InfoNCE behavior, and benchmark performance. Highly concrete perturbations improve distinguishability, while Cement and Slipform improve compositional results but expose a trade-off with general visual representation.

  • Dataset Analyses: Highly concrete hard negatives produce more distinguishable visual contents and improve T2I and I2T retrieval accuracy compared with low-concreteness negatives.Low-concreteness perturbations yield tighter logit gaps and greater retrieval difficulty.
  • Dataset Analyses: Concrete perturbations can lower BERTScore because they often modify semantically dense multi-word entities, without primarily causing semantic or grammatical invalidity.Visiolinguistic correlations indicate that the lower intra-modal similarity of highly concrete datasets is not mainly due to misalignment.
  • Gradient of InfoNCE: At batch size N=1024, InfoNCE assigns less than 20% of the tracked gradient allocation to hard negatives, confirming worsening imbalance with larger batches.The measured ratio declines as batch size grows.
  • Gradient of InfoNCE: Reducing batch size from 1024 to 256 raises the gradient ratio above 0.32 but causes 2.16% relative loss in general visual representation and 1.18% compositional degradation.The experiments therefore favor margin addition over sacrificing large-batch negative sampling.
  • Quantitative Results: Slipform achieves a 13.13% relative macro-average gain over the standard CLIP baseline on compositional understanding benchmarks.The evaluation also reports consistent performance trends across concreteness sampling strategies.
  • Quantitative Results: Cement improves over InfoNCE, while Cement with highly concrete negatives creates a trade-off between compositional understanding and general visual representation.Slipform nevertheless maintains general visual understanding performance comparable to other methods.

4. Discussion

The discussion identifies unresolved scope and evaluation boundaries for concreteness-based perturbation. Future work targets visual-specific concreteness, the compositional–general representation trade-off, and downstream applications.

  • Limitations and Future Work: Standard concreteness scores include tactile, auditory, olfactory, and gustatory perceptibilities, limiting their precision as measures of latent visual concreteness.The authors propose incorporating visual-specific concreteness measures into ConcretePlant.
  • Limitations and Future Work: The method exhibits an observed performance trade-off between compositional understanding and general visual representation.Further investigation is required to mitigate this trade-off.
  • Limitations and Future Work: Future evaluation will examine generated datasets in downstream tasks, video applications, and use as submodules.These settings are identified as directions for assessing broader utility.

5. Related Work

Related work addresses limitations of contrastive pretraining through objective changes, dataset filtering, and automated hard-negative generation. The cited discussion also notes a trade-off in general visual representation evaluation.

  • Improving Cross-Modal Pretraining: SigLIP replaces softmax with binary cross-entropy to improve training efficiency, while DCL addresses gradient vanishing and InfoNCE push–pull imbalance.These methods target limitations inherent in contrastive cross-modal objectives.
  • Hard Negative Mining: Hard negative mining automates semantic perturbation and incorporates heuristics, generative models, and graph priors to improve structured semantic representations.The approaches target modality bias and scalable hard-negative generation.
  • General Visual Representation: General visual representation results are reported with a macro-average, and the table caption notes a trade-off with compositional understanding.The supplied table passage provides no individual benchmark values.

6. Conclusion

The paper concludes that concreteness-aware hard-negative mining and concreteness-conditioned margin design improve multimodal compositional reasoning. Slipform achieves state-of-the-art compositional accuracy while aligning data generation with objective design.

  • Conclusion: Lexical concreteness is identified as a determinant of hard-negative quality because concrete concepts produce larger visual discrepancies and more informative contrastive supervision.ConcretePlant controls this factor to generate perceptually grounded negative pairs.
  • Conclusion: Slipform uses a concreteness-conditioned adaptive margin to address InfoNCE gradient imbalance and retain hard-negative signals.The conclusion links data generation and objective design within one framework.
  • Conclusion: Empirical evaluations report state-of-the-art accuracy on compositional understanding benchmarks.The conclusion presents this as evidence for the integrated approach.

A.1. Concreteness-Aware Data Generation

The data-generation pipeline selects detailed anchor captions, parses their linguistic structure, and targets candidate entities using concreteness ratings. It balances attribute, relation, and object perturbations while preserving coverage and diversity.

  • The pipeline selects the longest available caption as the anchor because longer captions typically convey richer compositional structure.
  • SpaCy parsing provides tokenization, lemmatization, part-of-speech tags, and dependency structure for enumerating perturbation targets.
  • Candidate entities receive concreteness ratings, and top-k sampling from softmax scores favors perceptible targets while avoiding mode collapse.
  • Perturbation targets are categorized as attribute, relation, or object, with relations prioritized when present because relational descriptions are relatively sparse.
  • Attribute, object, and relation categories are enabled only when caption structure supports grammatically valid substitutions or spatial and interaction relations.

A.2. Prompt

The prompt instructs a captioning model to replace a selected keyword with a physically and grammatically correct phrase that conflicts with the original image. Category-specific examples guide perturbations while preserving the rest of the caption.

  • Different in-context examples are supplied for different compositional categories.
  • The prompt asks for a factually mismatched caption created by replacing a target keyword with a visually conflicting word or phrase.
  • The prompt requires preserving sentence structure, grammar, and all non-target words during hard-negative generation.
  • Filtering retains NOUN, PROPN, ADJ, NUM, ADP, and VERB tags to focus perturbations on semantically meaningful perceptual units.
  • Examples demonstrate attribute, relation, and object substitutions that alter the selected concept while retaining the surrounding caption.

B. Gradient Derivation

The derivation analyzes the InfoNCE loss for a positive pair and its hard negative by expressing the denominator as Z and differentiating with respect to positive-pair similarity. The resulting probability term connects the gradient to the model’s predicted positive-match probability.

  • The loss is differentiated with respect to the positive-pair similarity variable s_i,i.
  • The derivation considers one sample index and its corresponding hard-negative index within the training batch.
  • The InfoNCE denominator is denoted as Z before the loss is reformulated for differentiation.
  • The chain-rule step differentiates the denominator contribution with respect to the positive similarity.
  • The ratio exp(s_i,i)/Z represents the predicted probability that the positive pair is selected among candidates.

C. Qualitative Samples by Datasets

Qualitative comparisons show that the concreteness of the perturbed keyword affects the magnitude of the resulting negative example’s visual or structural change. Highly concrete edits produce larger changes, whereas lower-concreteness edits preserve more of the original scene.

  • Highly concrete keyword perturbations lead to larger visual or structural changes across qualitative negative-pair examples.
  • Lower-concreteness perturbations usually produce subtler modifications that preserve more of the original scene.
  • Blue highlights mark high-concreteness edits, while red highlights mark low-concreteness edits.
Loading 2604.13313v1…