PaperlayerView exact revision on arXiv

Source-linked AI summary

ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension

Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, Anna Rohrbach

arXiv:2204.05991v2cs.CVcs.CL

TL;DR

ReCLIP addresses whether large-scale vision-language models can perform referring expression comprehension zero-shot across visual domains, where collecting domain-specific training data is costly. It isolates proposals for CLIP scoring and adds spatial-relation resolution, outperforming prior zero-shot methods and supervised models evaluated out of domain. The method remains limited by unsupported relation types and inflexible relation handling.

  • Problem

    Domain shift makes supervised ReC models unreliable in new visual domains, while zero-shot performance on complex vision-language reasoning tasks remains limited.

  • Method

    ReCLIP decomposes expressions into subqueries, scores isolated object proposals with CLIP, and combines proposal probabilities using spatial heuristics for relations.

  • Results

    ReCLIP outperforms prior zero-shot ReC approaches and supervised ReC models trained on natural images when evaluated on the out-of-domain RefGTA dataset.

  • Takeaways & Limitations

    ReCLIP provides a strong zero-shot baseline across visual domains, while the findings suggest spatial reasoning should receive greater attention in pre-training.

  • Takeaways & Limitations

    The spatial relation resolver cannot handle several relation types, including counting, and cannot flexibly determine when relations require multiple proposals.

Abstract

from arXiv · show

Training a referring expression comprehension (ReC) model for a new visual domain requires collecting referring expressions, and potentially corresponding bounding boxes, for images in the domain. While large-scale pre-trained models are useful for image classification across domains, it remains unclear if they can be applied in a zero-shot manner to more complex tasks like ReC. We present ReCLIP, a simple but strong zero-shot baseline that repurposes CLIP, a state-of-the-art large-scale model, for ReC. Motivated by the close connection between ReC and CLIP's contrastive pre-training objective, the first component of ReCLIP is a region-scoring method that isolates object proposals via cropping and blurring, and passes them to CLIP. However, through controlled experiments on a synthetic dataset, we find that CLIP is largely incapable of performing spatial reasoning off-the-shelf. Thus, the second component of ReCLIP is a spatial relation resolver that handles several types of spatial relations. We reduce the gap between zero-shot baselines from prior work and supervised models by as much as 29% on RefCOCOg, and on RefGTA (video game imagery), ReCLIP's relative improvement over supervised ReC models trained on real images is 8%.

1 Introduction

ReCLIP addresses the difficulty of zero-shot referring expression comprehension across visual domains by combining CLIP-based proposal scoring with spatial-relation handling. It targets the costly domain-specific data requirements of supervised ReC and improves over prior zero-shot and out-of-domain supervised baselines.

  • Motivation: Domain shift can make supervised ReC models localize the same referring expression correctly in training images but incorrectly in a new visual domain.Collecting referring expressions and corresponding boxes for each domain is expensive.
  • Approach: ReCLIP is a zero-shot ReC method with CLIP-based proposal scoring and a spatial relation resolver.Its proposal-scoring component isolates regions through cropping and blurring before scoring them with CLIP.
  • Spatial reasoning: CLIP and ALBEF are unable to perform their pre-training task on CLEVR examples requiring spatial reasoning.This controlled experiment motivates adding spatial heuristics rather than relying on pretrained models alone for relation resolution.
  • Approach: ReCLIP combines parsed subqueries, CLIP proposal probabilities, and simple spatial rules to select the final proposal.The relation resolver can correct cases where CLIP ranks the same proposal highest for multiple noun chunks.
  • Results: On RefCOCO/g/+, ReCLIP outperforms CPT by about 20% and surpasses UNITER-Large and MDETR on RefGTA by an absolute 4.5%.On RefGTA, the relative improvement over those supervised models is 8%.

2 Background

This section defines referring expression comprehension and the zero-shot setting, then introduces CLIP-based proposal scoring and prior proposal-selection baselines.

  • Task description: ReC selects the bounding-box proposal that best matches a textual referring expression in an image.Accuracy is the percentage of instances where the selected proposal has IoU of at least 0.5 with the ground-truth box.
  • Zero-shot setting: The zero-shot setting applies a pretrained vision-language model to ReC without task-specific training data.The general requirement is an image-text similarity score from the pretrained model.
  • CLIP: CLIP matches images and captions through contrastive pretraining, using image-text similarity scores to identify corresponding pairs.For image i and caption j, the matching probability is based on exp(βx_i^T y_j).
  • Prior methods: CPT-adapted shades each proposal red and asks CLIP to score the prompt '[referring expression] is in red color.'This adapts colorful prompt tuning, originally designed around masked language models, to image-text scoring models.
  • Prior methods: GradCAM uses gradients of the image-text similarity logit to create a visual importance map, then scores proposals by aggregated map values adjusted by proposal area.The highest-scoring proposal is selected.

3 ReCLIP

ReCLIP combines isolated proposal scoring with a rule-based spatial relation resolver because CLIP performs poorly on spatial reasoning. It parses expressions into semantic trees, scores predicates with CLIP, and combines relational constraints recursively.

  • Overview: ReCLIP uses isolated proposal scoring and a rule-based relation resolver as its two main components.The system addresses region matching with CLIP and spatial relations with heuristics.
  • 3.1 Isolated Proposal Scoring: Isolated proposal scoring crops or blurs each object proposal before passing it with the expression through CLIP.Cropping and blurring create isolated visual inputs for proposal scoring; their scores can be ensembled.
  • 3.2 Can we use CLIP to resolve spatial relations?: CLIP’s CLEVR accuracy is near random chance (50%) on spatial image-text matching despite good performance on the non-spatial control task.The result indicates that image-text scores largely omit spatial relations.
  • 3.3 Spatial Relation Resolver: The resolver decomposes expressions into predicate and spatial-relation primitives, using CLIP for predicates and heuristic rules for relations.The supported relation types include left, right, above, below, bigger, smaller, and inside.
  • 3.3 Spatial Relation Resolver: Superlatives such as “leftmost dog” are represented as special relations by copying the predicate into the empty second argument.The semantic-tree procedure converts a superlative into a parent relation node with the original node as its only child.
  • 3.3 Spatial Relation Resolver: Dependency parses become semantic trees whose noun chunks are nodes and whose keyword-bearing dependency paths define relations between them.Node probabilities are computed recursively from predicate probabilities and child-relation constraints.

4 Experiments

Experiments evaluate ReCLIP against zero-shot and supervised baselines across in-domain and video-game imagery, while probing spatial reasoning, proposal isolation, and model architecture. ReCLIP generally performs strongly, though relation handling remains limited and a gap to supervised methods persists.

  • Evaluation setup: RefCOCOg, RefCOCO, and RefCOCO+ experiments compare ReCLIP with zero-shot methods using proposal detections and, in some settings, ground-truth proposals.RefCOCOg includes spatial relations and longer expressions, whereas RefCOCO+ is designed to avoid spatial relations.
  • RefCOCO/g/+ results: ReCLIP outperforms other zero-shot methods on RefCOCOg and RefCOCO, while matching or exceeding GradCAM on average on RefCOCO+ with lower TestA–TestB variance.Using a 5% object-size prior erases GradCAM’s TestA advantage, and ReCLIP’s relation resolver helps on datasets containing spatial relations but not RefCOCO+.
  • RefGTA results: More than 15% higher accuracy than UNITER-Large and roughly 5% more than MDETR are reported for ReCLIP on RefGTA.ReCLIP also outperforms GradCAM by about 20%; with all detected proposals, the relation resolver neither substantially hurts nor significantly improves accuracy.
  • Model transfer: IPS outperforms GradCAM with ALBEF’s ITC output, whereas GradCAM performs better with ALBEF’s ITM output.The result suggests IPS transfers well to contrastively pretrained models with shallow modality interactions, while GradCAM may suit models with deeper interactions.
  • IPS analysis: Cropping plus blurring yields greater proposal-scoring accuracy than either isolation method alone.Qualitative examples indicate that IPS can use local context, but it can also select an incorrect noun chunk.
  • Limitations: ReCLIP remains considerably behind supervised methods because its rule-based relation resolver cannot handle some relation types or determine when multiple proposals are needed.Examples include counting relations such as “the second dog from the right” and ambiguous uses of “with.”

5 Related Work

Related work situates ReCLIP among referring-expression datasets, spatial-heuristic systems, weakly supervised methods, and pretrained vision-language models. ReCLIP differs by applying a pretrained model to ReC without task-specific training and by combining isolated proposal scoring with spatial handling.

  • Referring expression comprehension: ReC datasets span everyday photographs, video games, robotic contexts, and webpages.These datasets represent multiple visual domains in which referring expressions must be grounded.
  • Spatial reasoning: Prior spatial-heuristic work decomposes relations into reasoning steps, while related supervised systems may use learned predicate classifiers or jointly trained language modules.ReCLIP similarly separates parsing from visual execution but uses CLIP-based proposal scoring and simple spatial rules.
  • Training regimes: Weakly supervised ReC uses expression–image pairs without ground-truth boxes, whereas ReCLIP is not trained on the ReC task at all.This differs from another zero-shot setting in which novel objects are tested while the visual domain remains unchanged.
  • Pretrained vision-language models: Contrastive models such as CLIP use separate image and text transformers, unlike earlier cross-modal-transformer approaches using masked language modeling and image-text matching objectives.Prior zero-shot methods include cropped-proposal classification and masked-language-model approaches; ReCLIP instead isolates proposals through cropping and blurring.

6 Conclusion

ReCLIP is a zero-shot ReC method that decomposes expressions, scores isolated proposals with CLIP, and resolves spatial relations with heuristics. It outperforms prior zero-shot approaches and supervised models trained on natural images when evaluated on RefGTA.

  • ReCLIP decomposes referring expressions into subqueries, scores isolated proposals against them with CLIP, and combines the outputs using spatial heuristics.Its design includes both CLIP-based proposal scoring and rule-based spatial relation handling.
  • ReCLIP outperforms zero-shot referring expression comprehension approaches from prior work.
  • ReCLIP outperforms state-of-the-art supervised ReC models trained on natural images when evaluated on RefGTA.RefGTA evaluates the method on video game imagery.
  • CLIP has low zero-shot spatial reasoning performance, suggesting that pre-training methods should account more for spatial reasoning.

7 Ethical and Broader Impacts

The paper highlights ethical risks from biases in pretrained vision-language models and environmental costs associated with model pretraining. It therefore advises analysis and mitigation before deployment rather than immediate real-world use.

  • Pretrained vision-language models can exhibit gender, racial, and other biases because their web-collected training data was not necessarily carefully curated.
  • The authors do not advise deploying ReCLIP directly in the real world before measuring and attempting to mitigate its biases.
  • ReCLIP relies on a pretrained model whose energy-intensive pretraining likely had negative environmental effects.
  • Because ReCLIP does not require training a new model, it could be more environmentally friendly than supervised ReC models depending on inference costs.

A Visualization of Region-Scoring Methods

The region-scoring visualization contrasts proposal representations produced by CPT adaptation with proposals isolated through image blurring. These examples show the visual forms used by the compared scoring methods.

  • Figure 5 visualizes the representation of a proposal using CPT-adapted processing.
  • Figure 6 shows blurred proposal versions generated with σ = 100.

B Synthetic Spatial Reasoning Experiment

The synthetic experiment tests spatial reasoning separately from non-spatial matching using text-pair and image-pair tasks. The setup evaluates six deterministic relations, models inside probabilistically, and compares CLIP-based systems including ALBEF.

  • The synthetic tasks use text pairs or image pairs, with one correct and one incorrect alternative for spatial and non-spatial matching.In the text-pair example, the spatial distinction is whether a yellow cube is in front of or behind a blue cube.
  • The relation resolver computes semantics for left, right, above, below, bigger, and smaller with deterministic heuristics.For directional relations it checks proposal-center positions, while size relations compare proposal areas.
  • Inside is modeled as a random variable using the intersection-area ratio between two boxes, unlike the other deterministic relations.
  • The method extracts noun chunks as predicates and identifies relations through dependency paths containing relation keywords.The listed keywords cover directional, size, and containment relations, including superlative relations.
  • ALBEF combines image-only, text-only, and multimodal transformers and is pretrained with image-text contrastive, image-text matching, and masked-language-modeling losses.The experiments explore both image-text contrastive and image-text matching scores.
  • ALBEF performs well on the non-spatial CLEVR image-text matching task but poorly on the spatial version.The text-pair tasks contain 295 instances each, and the image-pair tasks contain 227 instances each.

D.1 ALBEF Performance on Synthetic Spatial Reasoning Experiment

The synthetic spatial-reasoning experiment reports zero-shot accuracy for ALBEF and describes implementation choices for comparing ALBEF and CLIP-based methods. The experiments also examine input resizing and hyperparameter sensitivity on RefCOCOg validation data.

  • Table 6 reports ALBEF ITM and ITC zero-shot accuracy in the synthetic spatial reasoning experiment.
  • ALBEF receives the expression directly, while CLIP-based methods use the prefix “a photo of.”
  • For all models, images are resized to the visual input resolution using bicubic interpolation; position embeddings are not resized in this setup.
  • RefCOCO+ and RefCOCOg validation/test images are included in COCO’s training set.
  • GradCAM uses α = 0.5 across models, while ViT-B/32 reaches slightly higher RefCOCOg validation accuracy at α = 0.6.
  • IPS performance for CLIP RN50x16 is little changed by blur standard deviation σ above σ = 20.

E.5 Experimentation on validation set

The paper describes validation-set experimentation involving alternative prompting, noun-chunk ensembling, superlative handling, and parser scope. It also notes that relation types, keywords, and some preliminary choices were selected using validation data.

  • The authors experimented on RefCOCOg validation data, and to a lesser extent RefCOCO+ validation data, with several alternative methods.
  • Rectangle-based proposal prompting produced performance somewhat similar to CPT.
  • Ensembling the original expression with a head-word noun-chunk prompt helped IPS and partly contributed to rule-based relation handling.
  • An alternative probabilistic technique for handling superlatives performed worse than the chosen technique on RefCOCOg validation data.
  • The parser and relation-handling pipeline was also tested on all sentences rather than only expressions containing relation or superlative keywords.
  • Relation types, keywords, and most preliminary experiments were selected using validation sets and an area threshold.

G Additional Experiment Results

Additional experiments provide full RefCOCOg, RefCOCO+, and RefCOCO results, alongside qualitative RefGTA examples and an analysis of relation-handling execution errors. ReCLIP is strongest among zero-shot methods on RefCOCOg and competitive on RefCOCO+ with lower split variance than GradCAM.

  • Additional experiment results: The additional-results section reports full results on RefCOCOg, RefCOCO+, and RefCOCO, including single-model and ground-truth-proposal settings.
  • Error analysis: Figure 11 illustrates execution errors involving the meanings of “behind” and “under,” as well as weighting caused by the superlative “smallest.”
  • RefCOCOg and RefCOCO+: ReCLIP outperforms other zero-shot methods on RefCOCOg.
  • RefCOCOg and RefCOCO+: On RefCOCO+, ReCLIP is roughly on par with GradCAM and has lower variance between TestA and TestB.
  • RefCOCO: Table 8 presents RefCOCO accuracy, with separate ground-truth-proposal and detected-proposal results.
  • Qualitative RefGTA examples: Figure 12 compares ground-truth referents with MDETR, UNITER, and ReCLIP predictions on randomly sampled RefGTA validation examples.
Loading 2204.05991v2…