PaperlayerView exact revision on arXiv

Source-linked AI summary

RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs

Maëlic Neau

arXiv:2609.12552v1cs.CV

TL;DR

Relation prediction remains tied to fixed predicate vocabularies and detector labels, limiting open-vocabulary scene-graph evaluation. RelateAnything separates regions, labels, and predicates through inference-time text vocabularies and verified free-text supervision, achieving 2.3–3.5× higher mean recall than the strongest comparable-scale open-vocabulary method across cross-dataset and zero-shot benchmarks.

  • Problem

    Relation prediction remains tied to fixed predicate vocabularies and object-label-conditioned architectures, unlike open-vocabulary detection and promptable segmentation.

    Source: §1 · p. 2

  • Method

    RelateAnything predicts relations from images and arbitrary regions using inference-time string vocabularies, supported by the verified free-text RA-4M corpus and OV-SGG-Bench evaluation.

    Source: §1 · p. 2, §4 · p. 10, §4.1 · p. 10

  • Results

    2.3–3.5× higher mean recall than OvSGTR [Chen et al., 2025], the strongest comparable-scale open-vocabulary method, across three cross-dataset and one zero-shot benchmark.

    Source: §1 · p. 4

  • Takeaways & Limitations

    In-domain measurement overstates transfer gains by about 5×, supporting cross-dataset evaluation for assessing relation-prediction generalization.

    Source: §11 · p. 30

    RA-4M has no human-audited precision estimate for its 4.3M machine-generated annotations.

    Source: §10 · p. 30

Abstract

from arXiv · show

Open-vocabulary detection accepts any class list at inference, and promptable segmentation returns regions without class names: the taxonomy has left the model and become an input. Relation prediction has not. Scene-graph models are still trained and evaluated on the 50 or 56 predicates of one annotation style, their relation head conditioned on object labels and so tied to one detector. Three obstacles explain this, none primarily modelling: no relation corpus is both free-text and verified, a label-conditioned architecture cannot accept a vocabulary it was not trained on, and the standard metric rewards agreement with the training corpus, so a larger vocabulary scores as a regression. We present RelateAnything, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings. Object labels are never an input, so the region source can change without retraining, and the vocabulary is a bank of text embeddings, not a learned classifier. It runs at 20 ms/frame. Training over 19,103 predicates requires positive-unlabeled supervision and a text encoder that separates antonyms, which contrastive encoders embed at cosine 0.95. To supply the supervision we build RA-4M, 474k images and 4.3M relations over 10,102 free-text predicates, generated against numbered box markers and geometrically verified. To measure it we build OV-SGG-Bench, six axes scored across datasets that the priors standard recall rewards cannot satisfy. On three cross-dataset benchmarks and a fourth zero-shot, RelateAnything has 2.3-3.5x the mean recall of the strongest open-vocabulary method of comparable scale, margins that survive a real detector, and leads a 3B-VLM scene-graph model on both metrics at under 2% of its parameters. In-domain measurement overstates transfer gains ~5x. Model, corpus and benchmark are public.

1 Introduction

RelateAnything addresses the data, architecture, and measurement barriers to open-vocabulary relation prediction with a separable 53M-parameter model, the verified RA-4M corpus, and OV-SGG-Bench. Across cross-dataset and zero-shot evaluations, it substantially outperforms comparable open-vocabulary methods while exposing weaknesses in conventional scene-graph recall.

Benchmark

  • OV-SGG-Bench evaluates transfer across datasets, human-adjudicated negatives, and balanced aggregate axes to reduce corpus matching, annotator imitation, and shortcut exploitation.Its design responds to conventional recall’s dependence on shared vocabulary, unconstrained matching, and unreported detector operating points.Source: Figure 1 · p. 2, §1 · pp. 3–4

Model

  • RelateAnything takes an image, regions from any source, and inference-time predicate strings, returning scored pairwise relations without object labels or a learned predicate classifier.Its 53M-parameter interface makes region sources replaceable without retraining and changes vocabularies through text-embedding substitution; inference runs at 20 ms per frame.Source: §1 · pp. 2–3
  • Training over 19,103 predicates uses positive-unlabeled InfoNCE supervision and antonym-repulsive text-encoder distillation to handle unstated true relations and poorly separated inverse predicates.Antonyms such as above and below reach cosine similarity 0.95 in contrastive encoders, while synonym groups remain together after correction.Source: §1 · p. 3

Corpus

  • RA-4M contains 474k images and 4.3M geometrically verified relations over 10,102 free-text predicates, generated from numbered box markers and filtered by deterministic geometric checks.On the same images and boxes, it is 1.7× denser with 107× the vocabulary than the replaced annotations.Source: §1 · pp. 3–4

Motivation and evaluation

  • Conventional scene-graph recall is largely a prior-matching score, while in-domain gains overstate cross-dataset gains by approximately 5×.A category-frequency table can outperform a trained model on leaderboard recall, and in-domain-selected changes can reduce transfer performance.Source: §1 · pp. 3–4

Empirical results

  • 2.3–3.5× higher mean recall than OvSGTR [Chen et al., 2025] and 5–21× higher rare-predicate recall establish RelateAnything as the strongest comparable-scale method across four transfer benchmarks.The margins persist when a real detector supplies regions; all results use cross-dataset transfer except the explicitly noted fine-tuned or partially trained rows.Source: §1 · p. 4

2 Related work

Prior scene-graph work is largely closed-set, label-conditioned, and evaluated against training-vocabulary priors. RelateAnything differs through region-based open-vocabulary prediction, geometry-verified free-text supervision, and vocabulary-scale learning.

Scene graphs and benchmarks

  • Scene-graph benchmarks retain triplet prediction while varying images, regions, sparsity, domains, or annotation construction, with Haystack uniquely publishing explicit negatives.Visual Genome fixed the VG150 vocabulary at 150 object classes and 50 predicates; Haystack enables precision measurement against adjudicated negatives.Source: §2 · p. 4

Closed-set generation and evaluation

  • Closed-set methods classify fixed predicates using object labels and layout, while evaluation evolved from recall@K to mean recall and later debiasing.These corrections estimate training-label statistics; the paper also adopts LVIS practices for federated annotation, support-based frequency splits, and precision over labelled cells.Source: §2 · pp. 4–5

One-stage and real-time generation

  • One-stage models improve speed by jointly querying subjects, objects, and predicates, but remain closed-set, benchmark-specific, and conditioned on predicted object classes.REACT [Neau et al., 2025] and REACT++ [Neau and Falomir, 2026] are identified as the fastest published scene-graph models, with comparisons deferred to § 9.Source: §2 · p. 5

Open-vocabulary relations

  • Open-vocabulary relation methods use text-embedding similarity, but RelateAnything uniquely avoids object labels, trains on free text over a five-figure vocabulary, and evaluates with supplied or withheld answers.OvSGTR [Chen et al., 2024b] defines the reproduced OvR-SGG protocol, while its MegaSG extension [Chen et al., 2025] is the closest supervision-quality control.Source: §2 · p. 5

Machine-generated relation supervision

  • Existing machine-generated supervision either leaves relations ungrounded, learns attachment, or accepts model outputs with filtering; RA-4M instead verifies relations against box geometry and preserves surface forms.GQA and MegaSG project generated relations onto class lists, unlike RA-4M.Source: §2 · p. 5

Learning at vocabulary scale

  • Positive-unlabeled learning handles incomplete relation labels, while contrastive text spaces add anisotropy, hubness, and an unresolved failure in which antonyms embed almost identically.The paper frames federated annotation as the benchmark analogue of positive-unlabeled learning and notes that an annotated riding pair may implicitly represent sitting on.Source: §2 · p. 6

3 The RelateAnything model

RelateAnything predicts relations between region pairs from an inference-supplied predicate vocabulary without object-label inputs or a learned predicate classifier. Its architecture combines context-aware pair representations with predicate-conditioned spatial and semantic branches, while training addresses positive-unlabeled supervision and antonym confusion in text embeddings.

3.1 Problem statement

  • The model outputs scored region-pair triplets over predicate strings supplied at inference, using only image pixels and box coordinates rather than object labels.Object categories appear only in training losses, never as network inputs, allowing deployment with noisy, absent, or out-of-vocabulary detector labels.Source: §3.1 · p. 6
  • Replacing the predicate vocabulary requires substituting one text-embedding matrix, because scores are cosine similarities without a learned predicate classifier or text-side projection.This makes the answer space extensible to strings unseen during training without retraining.Source: §3.1 · p. 6

3.2 Architecture

  • The architecture pools each box from dense patch features with coordinate-aware attention, then refines ordered pairs using geometry, scene context, and features sampled outside their boxes.The external sampling stage supports evidence such as road surfaces for “parked on” and attachment points for “hanging from.”Source: §3.2 · pp. 6–7
  • A predicate-conditioned gate mixes separate spatial and semantic cosine channels, routing geometry-heavy predicates toward spatial evidence and actions toward appearance evidence.The gate depends only on the predicate embedding, so it remains defined for unseen strings; the channels can also be read as layout and content graphs.Source: §3.2 · p. 7

3.3 Training with 104 predicates

  • Training spans 19,103 free-text predicates, using synonym-group InfoNCE positives plus auxiliary sigmoid, directional-swap, background-suppression, and object-text grounding losses.The predicate bank includes 10,102 RA-4M strings and 9,001 strings from three other relation corpora whose relations are not used for training.Source: §3.3 · p. 8
  • With ten thousand predicates, negative labels are often merely unstated positives, so the objective discounts negatives using estimated predicate co-occurrence while preserving annotated predicates and directional inverses.The estimates combine co-annotation counts, text cosine, frequency, and an unlabeled-positive correction.Source: §3.3 · p. 8
  • The teacher encoder places antonyms at cosine 0.95 for above/below, 0.94 for in front of/behind, and 0.99 for left/right, motivating a distilled student text encoder.After distillation, synonym-versus-inverse AUC rises from 0.85 to 0.99, mean inverse cosine falls from 0.92 to 0.09, and hubness falls from 8.1 to 1.3.Source: §3.3 · p. 9
  • Synonym-group alignment makes exact-string closed-vocabulary metrics conservative, because valid alternatives such as “wearing” for VG150’s “wears” can be counted as misses.A corrected evaluation in A3 provides the alternative view.Source: §3.3 · p. 9

4 RA-4M: a verified free-text relation corpus

RA-4M supplies dense, verified free-text relation supervision because existing corpora lack open vocabularies, reliable coverage, and detectable errors. Its geometric checks, direction balancing, leakage controls, and mixed-source training design produce a broad corpus while explicitly preserving residual risks.

4 RA-4M: a verified free-text relation corpus

  • RA-4M contains 474,413 images, 4,282,531 relations, and 10,102 distinct free-text predicates, averaging 9.03 relations per image.The corpus was generated at a cost of 104 GPU-hours to provide dense annotations with a non-benchmark predicate vocabulary.Source: §4 · p. 10
  • Free-text relation annotation is costly and unreliable because each image with N objects contains N(N −1) pairwise claims, motivating dense vision–language-model annotation with geometrically detectable errors.Visual Genome produced 36,549 predicate strings when annotators were unconstrained, but the community retained only the 50 most frequent [Krishna et al., 2017] [Xu et al., 2017].Source: §4 · p. 10

4.1 Generation and geometric verification

  • Numbered box markers ground the annotator’s outputs, and a deterministic geometric gate rejects 11.3% of raw candidates, making RA-4M the only machine-annotated relation corpus described as verifying its annotator.The pipeline uses an open-weight vision–language model [Gemma Team, 2026], while Fig. 14 and Tab. 19 provide the detailed breakdown and comparison.Source: §4.1 · p. 10
  • Geometric verification leaves containment and contact with 0% and 0.1% disjoint boxes, removes duplicate triples and self-loops, and balances inverse spatial directions by swapping roles and predicates.The gate rejects or role-swaps only when geometry constrains the predicate; direction repair preserves the annotator’s surface form before inverse restatement, producing direction balance.Source: §4.1 · p. 10
  • Residual errors remain for geometrically unconstrained relations: roughly 22% of looking at/watching annotations have disjoint boxes, alongside grounding, prior-driven, contact, and source-box errors.One residual error class has a measurable downstream effect traced in § H.3.Source: §4.1 · p. 10

4.2 Statistics and training mixture

  • Compared with source annotations on identical images and boxes, RA-4M is 1.7× denser, has 107× the vocabulary, and raises predicate entropy by 1.4 nats.It reproduces 73 of 94 source predicate classes exactly; its top ten predicates still carry 57% of the mass, while mean object degree rises from 1.88 to 3.20 and connected graphs from 70.9% to 82.6%.Source: §4.2 · p. 11
  • The zero-shot tower mixes RA-4M with leakage-filtered raw Visual Genome free-text annotations, while the released tower additionally includes HICO-DET; both use the same 19,103-string predicate bank.The zero-shot tower sees no evaluation images or benchmark label sets, carries 108 of HICO-DET’s 116 predicate strings without training on its relations, and the towers differ by less than one point on every axis except HICO-DET.Source: §4.2 · p. 11

5 OV-SGG-Bench: analysis of scene-graph evaluation and a cross-dataset protocol

OV-SGG-Bench shows that standard scene-graph recall rewards dataset-specific priors and scoring conventions, then replaces single-number evaluation with a six-axis cross-dataset protocol. The protocol emphasizes prior-blind measurements, reports detector and vocabulary conditions, and uses a chance-corrected composite without replacing the underlying vector.

5.1 Metrics and evaluation modes

  • The benchmark defines R@K, mR@K, F1@K, and wR@K, distinguishes graph-constrained from unconstrained scoring, and separates ground-truth-box evaluation from detector-based SGDet.Graph-constrained scoring changes results by 11.6 R@50 on VG150 and 19.0 on PSG for the model, so all reported numbers use that convention.Source: §5.1 · p. 12

5.2 Three priors rewarded by recall

  • Shared triplet mass exposes in-domain leakage: the baseline reproduces 90.9% of VG150 relation mass as shared triples versus 12.8% for the mixture, while higher VG150 recall can worsen tail metrics.The triple-based statistic captures predicate, object taxonomy, and domain overlap rather than predicate-string overlap alone.Source: §5.2 · p. 12
  • A frequency lookup table beats the trained model on micro accuracy but loses by 29–86% on per-predicate averages, showing that recall rewards object-category priors.The lookup table uses oracle object categories and no pixels, whereas the model receives no labels; the model is correct where the baseline is wrong on 6.9–12.8% of edges.Source: §5.2 · p. 12, Table 1 · p. 13

5.3 Two scoring conventions with effects larger than those measured

  • Detector operating settings move the pair-recall ceiling by 12–49% across four combinations, exceeding the 4.3-point leaderboard spread and potentially reversing detector comparisons.Detection-mode results should report the matcher, detector operating point, and resulting pair-recall ceiling; duplicate-credit correction also lowers published R@50 by 12.0%.Source: §5.3 · pp. 13–14, Figure 4 · p. 14

5.4 The six axes

  • OV-SGG-Bench reports six complementary axes, with prior-blind A2 and A6 underpinning claims about relational competence rather than corpus match.A2 uses human-adjudicated negatives, while A6 is balanced so category frequency cannot help; no single axis is sufficient alone.Source: §5.4 · p. 14
  • OVS combines chance-corrected A1, A2, A4, A5, and A6 by harmonic mean, normalizing A4 and A5 by recoverable quantities while retaining the full axis vector.A3 is reported but excluded because matcher choices can move its result more than system differences; withholding any axis preserves system ordering at 2.0–3.7× ratios.Source: §5.4 · p. 15

6 Results

Across six evaluation axes, RelateAnything transfers across datasets, supports withheld open-vocabulary prediction, preserves performance with detected regions, and improves precision, graph informativeness, and spatial-relation discrimination. Results also expose where recall, matcher choice, detector coverage, and frequency priors can distort conclusions.

6.1 A1: cross-dataset transfer on four benchmarks

  • RelateAnything achieves 2.3–3.5× mean recall and 5–21× rare-bucket recall over OvSGTR across cross-dataset benchmarks, despite receiving no object labels.Head recall is 1.04–1.40× the baseline’s, while the comparison uses held-out images and annotations and ground-truth boxes.Source: §6.1 · p. 16, Table 4 · p. 17

6.2 A2: precision on adjudicated negatives

  • RelateAnything improves mean federated AP by 39%, rare-predicate fAP by 59%, and P-AUC from 83.9 to 93.7 on adjudicated negatives.The evaluation contains 2,870 positives and 23,174 explicit negatives across 26,044 labelled cells; unsampled cells score zero.Source: §6.2 · p. 17

6.3 A3: the full vocabulary with the answer set withheld

  • Using its full 19,103-string vocabulary, RelateAnything reaches 56.0 R@50 on VG150 versus 53.3 when supplied the benchmark’s 50 strings.Its median ground-truth-string rank is 1 on VG150 and IndoorVG and 10 on PSG; synonym-tolerant matching changes system ordering and cross-dataset margins.Source: §6.3 · pp. 18–19

6.4 A4: detection-mode graphs on a shared detector

  • With a shared open-vocabulary detector, every Tab. 4 margin survives: RelateAnything reaches 3.3× mR@50 and 14× rare-bucket recall over OvSGTR.Ground-truth-box retention is 62–63% on PSG and 31–32% on IndoorVG across towers, with detector coverage dominating the loss.Source: §6.4 · p. 19, Table 6 · p. 19

6.5 A5: graph quality judged without ground truth

  • At matched depth, accepted relations yield 18.6 bits per image for RelateAnything versus 13.4 for OvSGTR, corresponding to 0.67 versus 0.48 of annotation information.The metric combines judge-accepted truth with predicate surprisal, avoiding graph-length and frequency biases that affect whole-graph and single-relation judgments.Source: §6.5 · pp. 20–21

6.6 A6: spatial relations against adversarial negatives

  • On adversarial spatial negatives, RelateAnything reaches macro AUC 0.690 and pooled AUC 0.675, outperforming OvSGTR’s 0.591 and 0.617 across all nine predicates.The released tower improves in front of from 0.58 to 0.68, above from 0.54 to 0.60, and to the left of from 0.56 to 0.64, but this remains the weakest axis.Source: §6.6 · p. 22

6.7 The OvR-SGG leaderboard

  • After removing the 15 novel predicates and synonym groups from training, RelateAnything reaches 22.4 Base+Novel and 11.8 Novel R@50, without claiming leaderboard leadership.The held-out model is mid-table on micro recall but exceeds the baseline by nearly fourfold on mean recall, showing the protocol’s frequency sensitivity.Source: §6.7 · pp. 22–23

7 Analysis: decoupling object and relation prediction

RelateAnything decouples object and relation prediction because labels mainly encode transferable category priors, while relation evidence comes from visual pair context rather than shared relational features. This makes region sources interchangeable without retraining, although detector boxes still constrain graph coverage.

7.1 Label dependence of the relation head

  • 88–90% of baseline predictions are reproduced by label-pair predicate lookup, versus 69–70% for RelateAnything, whose shuffled-label behavior confirms image-based prediction.The baseline uses only 20–31 predicates, while RelateAnything uses all 50; its residual entropy is three times larger.Source: §7.1 · pp. 23–24

7.2 Region sources

  • One checkpoint processes both open-vocabulary detector regions and class-agnostic segments, but dense or irrelevant detector boxes still limit the returned interaction graph.The model reads relations from YOLOE regions prompted with 4,585 tags and FastSAM segments without names; its detector vocabulary remains unconstrained.Source: §7.2 · pp. 24–25, Figure 6 · p. 24

7.3 Effect of relation supervision on the dense features

  • Relation fine-tuning increases class selectivity from 0.242 to 0.281 but barely changes relation selectivity, from 0.133 to 0.138, indicating no shared relational representation.Partner-versus-same-class discrimination falls from 0.469 to 0.385, while the interaction region identifies objects rather than verbs: 0.761 versus 0.269 retrieval accuracy.Source: §7.3 · pp. 25–26, Figure 7 · p. 25

7.4 Attribution of the relation logit

  • Pair context explains 87–93% of semantic-logit variance, whereas subject and object identity contribute 0.1% and 0.0%; removing that path costs only 0.1% micro accuracy.Cross-image feature substitution costs 44–68% micro and 85–91% macro accuracy, while removing geometry costs 6–24%, showing that visual pair evidence drives prediction.Source: §7.4 · p. 26
  • Together, these findings show that labels inflate ranking metrics without transferring, so detectors can be replaced without retraining while relation models cannot inherit detector label distributions.The analysis finds no shared representation that justifies a shared backbone and identifies a measurable cost to shared label input.Source: §7 · p. 23, §7.4 · p. 26

8 Analysis: in-domain versus cross-dataset measurement

In-domain measurements substantially overstate transfer gains: corpus scaling improves in-domain recall far more than zero-shot recall, while annotation sparsity and model capacity can reverse or fail to predict cross-dataset performance. Transfer evaluation therefore governs data-scaling and checkpoint decisions, with evidence favoring the released small backbone and detector improvements as the next priority.

8.1 Corpus scale read in-domain and under transfer

  • 5×: Doubling the corpus yields +15%/+56% in-domain micro/macro R@50 gains but only +2.5–3.8%/+9.9–11.9% under transfer, as shown in Fig. 8.The full corpus’s relative gain is therefore much larger on the shared in-domain annotation style than on VG150 and PSG zero-shot tests.Source: Figure 8 · p. 27, §8.1 · p. 27
  • Transfer measurements should guide decisions: final-epoch arm ranking predicts out-of-distribution behavior better than best-epoch selection (ρ = 0.77 versus 0.66), and 500k further images were not generated.The extrapolated transfer gain from those images was only about one point absolute.Source: §8.1 · p. 27
  • Distribution alignment fails as a transfer predictor: increasing PSG-predicate training mass from 27.3% to 39.4% still lowers transfer macro recall by 3.8 points.The change also reduces left/right training mass from 29.5% to 7.4%, showing that predicate overlap alone does not explain transfer.Source: §8.1 · p. 27
  • Sparse annotation is harmful: adding Open Images V6 [Kuznetsova et al., 2020] produces about −6% on the development composite and −40.2% on HICO-DET tail recall.Its 4.33 relations per image versus RA-4M’s 9.03 introduces false negatives under positive-unlabeled training, making sparse annotation worse than none.Source: §8.1 · p. 27
  • Backbone capacity does not improve the composite: a tower with +114% parameters changes it by −0.1%, while smaller towers perform best on the spatial axis.Scaling shifts evidence from box geometry toward pixels, but the largest tower’s individual-axis advantages disappear at deployment; ViT-S/16+ is therefore released.Source: §8.1 · p. 28

9 Inference cost

RelateAnything delivers substantially lower end-to-end cost than OvSGTR under a shared protocol while remaining competitive with closed-set real-time specialists. Its compiled deployment reaches 20 ms per frame and supports large predicate vocabularies with minimal overhead.

Comparison with OvSGTR

  • 7.8× faster end to end than OvSGTR on an A40, despite having more parameters, under the shared protocol.OvSGTR scores about 9,500 pairs among 98 boxes at 800/1333 in fp32, whereas RelateAnything scores at most 128 sampled pairs among 20 boxes at 448 px in bf16.Source: §9 · p. 28

Comparison with closed-set real-time models

  • 21–23 ms versus 35–36 ms for the faster closed-set specialist, while RelateAnything has higher macro recall everywhere but PSG and competitive F1@K.Its F1@K exceeds REACT on all three benchmarks, exceeds REACT++ on VG150, and is 0.2 and 3.7 below REACT++ on IndoorVG and PSG, respectively, without benchmark annotations or object labels.Source: §9 · p. 28, Table 12 · p. 29

Deployment configuration

  • 20 ms per frame after compilation, while scoring 19,103 strings costs under a millisecond at batch 1.Compilation provides a 1.5–1.8× gain across GPUs; an fp16 CPU build reaches 7 frames per second on eight threads with 0.955 top-1 agreement against fp32.Source: §9 · p. 29

10 Discussion and limitations

The discussion identifies limitations in held-out evaluation, corpus precision, relational feature learning, judge-based assessment, detector recall, and reporting practices. It also specifies concrete checks needed before relying on the generated corpus or deployed relation predictions.

Held-out concepts

  • Cross-dataset results use unseen images, object distributions, and annotation styles but not unseen predicate strings, while the general held-out-concept variant remains untested.Several conclusions rely on one seed, and differences within § H’s 1.3% noise floor are identified as such.Source: §10 · p. 29

Corpus precision

  • 11.3% of RA-4M candidates are rejected structurally, but no human-audited precision estimate exists for its 4.3M machine-generated annotations.The gate’s rejections are true negatives up to box error, while geometrically unconstrainable predicates are named and counted rather than checked; human auditing is the first missing measurement.Source: §10 · p. 30

Scope of the decoupling result

  • The decoupling result shows only that the supervision lacks relational dense features, not that relational features are impossible.The recipe has no patch-level relational objective, which probing suggests a shared backbone would require.Source: §10 · p. 30

A5 verdicts

  • Per-relation precision is a genuine shortcoming: the baseline overreaches less at any depth, so false-edge-sensitive deployments should prefer shorter graphs.Whole-graph judgments at deployed length favor the baseline, and one-relation judgments accept slightly more baseline relations; a second judge repeats the latter finding.Source: §10 · p. 30

Detector recall

  • Pair recall is quadratic in object-detector recall, making detector recall the dominant loss between benchmark scores and deployment performance.A larger relation model can improve ranking regions that deployment never emits, but no relation model can exceed the detector’s ceiling.Source: §10 · p. 30

Reporting practices

  • Reporting should include corpus–benchmark shared triplet mass, matcher assignment rules, detector thresholds and caps, pair-recall ceilings, graph constraints, and adjudicated-negative results.These inexpensive disclosures changed a conclusion in this work.Source: §10 · p. 30

11 Conclusion … A.3 Pair head and predicate-conditioned gate

RelateAnything addresses supervision, architecture, and measurement barriers to open-vocabulary relation prediction with RA-4M, inference-time predicate vocabularies, and OV-SGG-Bench. Its evaluation shows strong cross-dataset transfer, while the appendices explain the visual, deformable-read, and predicate-gating mechanisms.

11 Conclusion

  • 2.3–3.5× higher mean recall than the strongest comparable-scale open-vocabulary method across three cross-dataset benchmarks and one zero-shot benchmark, at 20 ms per frame.RA-4M contains 4.3M geometrically verified relations over 10,102 free-text predicates, while OV-SGG-Bench evaluates six axes across datasets.Source: §11 · p. 30
  • In-domain measurement overstates transfer gains by about 5×, while standard scene-graph recall can reward corpus-prior matching and scoring conventions more than method differences.The recommended remedy is to report shared triplet mass between corpus and benchmark alongside recall.Source: §11 · p. 30

A.1 Visual path

  • The visual path individually normalises three feature taps and fully fine-tunes the backbone at a learning rate 8× below the head’s.This avoids last-layer scale domination, and at this scale outperforms both a frozen backbone and LoRA adapters [Hu et al., 2022].Source: §A.1 · p. 36
  • Relation fine-tuning sharpens object boundaries and internal texture but does not introduce pair-spanning structure in patch features.This visual effect is shown in Fig. 9 and quantified in Fig. 26 (§ F).Source: Figure 9 · p. 36

A.2 Deformable scene read

  • The deformable scene read uses an additive zero-initialised gate, unbounded offsets, and distinct ring initialisations so training can learn whether and where to sample around pair anchors.Zero-initialising offsets would collapse each anchor’s points to one location with indistinguishable gradients.Source: §A.2 · pp. 36–37
  • Subject and contact anchors dominate learned sampling, while union reads contribute least because their large box causes 42–53% of samples to clamp at the border.For a person→tie wearing relation, 61% of attention mass lies on contact; reversing roles shifts 52% to null slots.Source: §A.2 · p. 37

A.3 Pair head and predicate-conditioned gate

  • Predicate-conditioned gating routes projective relations toward geometry, actions toward appearance, and contact predicates between them without gate supervision.Median αp is 0.002, while 12% of predicates exceed 0.5; for example, behind has αp=1.000 and carrying αp=0.0004.Source: §A.3 · p. 37, Figure 11 · p. 38
  • One forward pass produces separate spatial layout and semantic content graphs, with the layout graph easier on every source.The two branches are scored before mixing, enabling the benchmark’s spatial-versus-semantic predicate split.Source: §A.3 · p. 37, Table 13 · p. 37

A.4 Training recipe

The released training recipe runs for 5h04m on four A100 GPUs and uses multi-resolution sampling plus photometric, but not geometric, augmentation. These choices regularize training and preserve the correctness of left/right relations while enabling evaluation at other resolutions without retraining.

A.4 Training recipe

  • 5h04m on four A100 GPUs is the runtime of the released training recipe.Source: Table 14 · p. 38
  • Sampling batch resolution from seven values in [0.5, 1.5] × 448 regularizes training and enables evaluation at other resolutions without retraining.Source: §A.4 · p. 38
  • Geometric augmentation is excluded because horizontal flips invert left/right relations and would introduce label noise, while photometric augmentation is retained.Source: §A.4 · p. 38

B Training objective … D.7 OvR-SGG reproduction

This section covers RelateAnything’s training objectives, predicate-text distillation, verified free-text corpus construction, deployment scoring, and evaluation methodology. It addresses supervision, vocabulary ambiguity, calibration, annotation-induced scoring errors, detector and OvR-SGG reproduction choices, and distinctions affecting reported performance.

B.1 Auxiliary objectives and source-aware negatives

  • Source-aware negative masking recovers 12% on the spatial axis and yields the best auxiliary-objective composite when annotation scopes differ across corpora.HICO-DET annotates verbs but not spatial relations, so negatives are restricted to predicates each source could have annotated; the released model uses this mask, unlike the zero-shot model.Source: §B.1 · p. 39

B.2.1 The distillation objective

  • Distillation separates inverse predicates while preserving synonym structure, placing inverses at cosine 0.09 and synonyms at 0.71 in the shipped student.Antonym repulsion and neighbourhood preservation are essential: removing the latter reduces effective dimension from 44 to 24 and synonym-group nearest-neighbour share from 29% to 18%.Source: §B.2.1 · p. 40, Table 16 · p. 40

B.2.2 Alignment to a synonym group

  • Group-aligned training causes exact-string misses despite semantic correctness, while subject–object-swap evaluation gives directional accuracy of 0.97.The model predicts alternative members of synonym or entailment groups, so exact-string closed-vocabulary recall is conservative relative to directional evaluation.Source: §B.2.2 · p. 40, Figure 12 · p. 41

C.1 Source images and generation

  • RA-4M uses three open-vocabulary generation passes with numbered box markers, deterministic geometric verification, and salience-ranked spatial top-up to produce grounded relation supervision.The corpus contains 474,420 training images and 24,968 validation images; geometric top-up contributes 18.4% of spatial relations.Source: §C.1 · p. 42, Figure 13 · p. 42
  • Propose-then-verify preserves relation diversity while balancing spatial forms, yielding 20 surface forms, 32.5% depth, 29% left/right, 14% proximity, and 13% contact.A verify-before-assert prompt reduced semantic diversity from 404 to 135 types, whereas deterministic gates reached the same precision without that diversity loss.Source: §C.1 · pp. 42–43

C.4 Design decisions

  • The deployed predicate bank contains 19,103 strings, separating the answer space from supervision by adding 9,001 predicates beyond RA-4M’s 10,102 strings.This preserves free-text predicates rather than canonicalising synonyms, because arbitrary deployment vocabularies require dense text-space coverage.Source: §C.4 · pp. 43, 46

D.1 Relation score

  • The released score uses additive pair-relatedness fusion, although multiplicative fusion performs better on annotation-derived benchmarks and worse on adjudicated negatives.On adjudicated negatives, additive fusion reaches 0.911 AUC, versus 0.904 for multiplicative fusion and 0.748 for relatedness alone.Source: §D.1 · p. 48

D.4 Calibration

  • Calibration changes threshold meaning without changing rankings: annotation-hit fitting reaches ECE 0.004, while the released adjudicated-negative map reports precision 0.77 at an adjudicated precision of 0.90.The released map estimates judged truth rather than annotation probability, and per-predicate calibration was rejected because it failed to transfer.Source: §D.4 · p. 49, Figure 18 · p. 49

D.5 Output at the operating threshold

  • 18.2, 7.9, and 6.1 edges per image result from calibrated-F1, annotated-precision, and PSG-density criteria, respectively, with τ = 0.45 used for qualitative panels.The operating point therefore matches corpus density rather than a visualization-tuned output budget.Source: §D.5 · p. 50
  • 78.3% of 39,516 emitted edges land on pairs PSG never annotated, showing that the largest apparent error group is unscored rather than wrong.At the calibrated-F1 threshold, 11.1% match annotated triples and 10.6% use a different predicate on annotated pairs; among annotated pairs, 51.0% carry PSG’s predicate.Source: §D.5 · p. 50

D.7 OvR-SGG reproduction

  • The OvR-SGG reproduction recovers twelve published numbers within 0.35 points, while matcher correction reduces baseline Base+Novel by 11–13% and Novel by 13–15% at every K without changing ordering.The evaluation uses OvSGTR’s reproducible 35-base/15-novel split and uncorrected matching for comparability; compliant evaluation removes held-out predicates and synonym groups from training.Source: §D.7 · pp. 50–52, Table 24 · p. 51

D.6 Detector operating point

  • Tab. 23 compares identical detector weights at two operating points, varying only confidence threshold and box cap to expose the resulting pair-recall ceiling.The ceiling is the fraction of ground-truth relations whose endpoints are both detected, bounding relation recall for those boxes.Source: §D.6 · p. 50, Table 23 · p. 51

D.8 Graph-quality oracle (A5) · D.8.1 Per-relation oracle (A5b) · E Additional results

The paper introduces graph-level and per-relation vision–language oracles to evaluate relation quality without relying solely on incomplete annotations. Controls certify judge reliability, while per-relation analysis measures truth, depth, informativeness, and reference-free diversity under explicit limitations.

D.8 Graph-quality oracle (A5)

  • A5 uses Qwen3-VL-8B [Bai et al., 2025a] to judge whole-graph relations from numbered-box images, avoiding annotation-dependent blind spots.The rubric, prompt, and rendering are released with the benchmark.Source: §D.8 · p. 52, Table 25 · p. 52
  • The reported judge passes certification with scramble accuracy 1.00, primacy 0.19, and 56 decisive comparisons out of 103.Admission requires scramble accuracy at least 0.75, primacy at most 0.60, and at least ten decisive control comparisons.Source: §D.8 · p. 53
  • A5 controls graph-length and vocabulary confounds by permitting up to 20 high-scoring relations, testing padding and predicate scrambling, and comparing restricted and full vocabularies.Graphs include every pair scoring within 70% of the image’s best pair, while controls test whether the judge prefers longer or intact graphs.Source: §D.8 · pp. 52–53

D.8.1 Per-relation oracle (A5b)

  • A5b evaluates each asserted relation independently, addressing whether a losing graph fails because its extra relations are false or merely because the judge prefers shorter text.The per-relation judge sees one relation at a time and answers whether it is true of the photograph.Source: §D.8.1 · p. 53
  • Highlighted rendering reduces predicate-swapping acceptance from 0.201 to 0.135, indicating that plain rendering suffers from grounding problems.The swapping rate is an upper bound on false positives because some corrupted predicates can remain true by coincidence.Source: §D.8.1 · p. 53
  • The oracle reports both deployed depth and matched top-ten depth, since neither confidence-based emission nor equal-length comparison supersedes the other.At equal depth, accepted relations equal precision multiplied by depth.Source: §D.8.1 · p. 53
  • Judge-elicited informativeness is rejected because it nearly duplicates truth and cannot observe redundancy, so informativeness is instead measured as surprisal, −log2 p(pred), over accepted relations.For the 200-image run, P(info = 0 | false) = 0.998 under both renderings, while P(info ≥2 | true) is 0.92 and 0.96.Source: §D.8.1 · pp. 53–54
  • The reference-free Krichevsky–Trofimov estimator preserves the ordering: at matched depth, the restricted arm reaches 16.0 bits per image versus 8.0 for baseline.The estimator also reports entropy 4.00 versus 2.14 and 1.9 versus 1.5 distinct accepted predicates per graph.Source: §D.8.1 · p. 54

E.1 Per-predicate results … G Backbone scaling

The analyses identify evaluation, detector, and representation limits while showing that RelateAnything transfers across vocabularies and scales with deployment costs. They also qualify spatial-reasoning claims and compare closed-set, localization, and backbone alternatives.

E.1 Per-predicate results

  • Per-predicate misses largely reflect emission and annotation rules: one emitted verb can replace several annotated verbs, while unannotated or near-synonymous relations receive no credit.On HICO-DET, holding absorbs wielding, carrying, and reading, while riding absorbs straddling and sitting on; on PSG, predictions such as hanging from and in front of can miss unannotated relations.Source: §E.1 · pp. 55–56, Figure 19 · p. 55

E.2 Priors join and detector retention

  • The visual model and frequency prior are complementary: the prior is correct 2.4× as often when the model is wrong, yet model-only correct edges comprise 7–13% of all edges.This indicates different error patterns rather than the model being a weaker version of the prior.Source: Figure 20 · p. 56
  • Under YOLO-World detection, the micro ladder retains its gains while the macro ladder flattens, and IndoorVG loses 70–80% of performance because its categories are poorly covered.Retention uses confidence threshold 0.05, strict class-aware matching, and a 500-pair budget on both sides.Source: Figure 21 · p. 56, §E.2 · p. 56

E.3 Bounds on the spatial-axis claim

  • Spatial-axis evidence is bounded by strong nonvisual and unstable controls: boxes-only scoring reaches 68.8, while a counterfactual grounding probe ranges from 0.01 to 0.81 with backbone learning rate.Appearance versus geometry provides the opposite control, moving the predicate by +34 points relative to its control.Source: §E.3 · pp. 57–58, Figure 23 · p. 57

E.4 Closed-set and localisation legs of the VLM comparison

  • Closed-set VLM margins over three benchmarks are smallest on PSG at 1.1× micro recall and widen to 1.9× on VG150 and 2.2× on IndoorVG.The comparison uses each benchmark’s own protocol, ground-truth regions, and graph-constrained scoring.Source: §E.4 · p. 58, Table 26 · p. 58
  • Genuine detection reaches 23.5 and 25.6 mAP@50 over 133 categories, while reachable annotated pairs scale with the square of box recall.At IoU box recalls of 0.32 and 0.38, only 10% and 14% of annotated pairs have both endpoints present.Source: §E.4 · p. 59

F Representation probes

  • Relation fine-tuning leaves relatedness nearly unchanged but improves predicate decodability, raising macro-F1 from 0.445 to 0.475 on PSG and from 0.264 to 0.305 on VG150.Within-class-pair AUC changes from 0.73 to 0.72, supporting feature readout rather than formation of a new relational representation.Source: F Representation probes · p. 60, Table 28 · p. 60, Figure 25 · p. 60

G Backbone scaling

  • Backbone scaling brackets the optimum: zero-shot train–validation NCE gaps are +0.083/+0.094/+0.159 with best epochs 12/8/7, making ViT-S capacity-limited while ViT-B overfits.A tuned DINOv3-ConvNeXt tiny variant reaches ViT-S accuracy, is fastest at batch size one, and remains useful when dispatch latency dominates.Source: G Backbone scaling · p. 62, Table 29 · p. 62, Figure 27 · p. 62

G.1 Effect of backbone capacity on transfer … H.3 A prior violation traced to one loss term

Backbone capacity does not reliably improve transfer, while recipe attribution identifies mechanisms that help across axes and a loss-term diagnostic exposes a severe prior violation invisible to standard benchmarks. Selection remains reproducible on a measured proxy, with deployment considerations favoring the smaller released tower.

G.1 Effect of backbone capacity on transfer

  • ViT-S/16+ is best on the spatial axis, while ViT-B performs worse despite more than twice the size, indicating capacity cannot replace missing supervision.On the released ladder, S+ gains 7.4% over S, whereas B gains −0.1% over S+; on the zero-shot ladder, S+ is 1.5% below S before B recovers 3.1%.Source: §G.1 · p. 63
  • −35.2% to −9.8% geometry dependence and −53.0% to −58.1% pixel dependence show that larger backbones substitute pixel evidence for explicit box coordinates.The opposing monotonic trends across all three sources rule out a generic robustness explanation.Source: §G.1 · pp. 63–64
  • 0.903 versus 0.893 precision favors S+ after detector deployment, because the larger towers’ benchmark-ranking advantage lies outside the 5–7 edges actually emitted.About 70% of emitted edges fall on boxes unmatched by any ground-truth object, so benchmark recall does not reflect the deployment operating point.Source: §G.1 · p. 64

H.1 Selection protocol and reproducibility

  • 82–86% of above-noise arm comparisons preserve their twelve-epoch ranking from the first epoch, although the winning composite rises from 0.339 on the proxy to 0.371 at full scale.The seed-matched rerun noise floor is 1.3%, which bounds interpretation of reported differences.Source: §H.1 · p. 65
  • Recomputing selection with A1/A2/A4/A6 changes no outcome, and final-epoch selection predicts out-of-distribution behavior better than best-epoch selection.The backbone-learning-rate comparison remains resolved for the default arm, at −0.012 OVS-F1 against a +0.003 threshold.Source: §H.1 · p. 65

H.2 Attributions

  • Deformable scene reading improves every axis and offsets the sigmoid auxiliary’s transfer cost, while multi-scale sampling adds 3.7% with 9 of 11 cells improving.Multi-scale sampling also reduces the proxy train–validation gap by 60%, and layer-normalised taps match a 3×-cost multi-level read.Source: §H.2 · p. 66
  • +10.1% zero-shot tail mean recall comes from same-predicate feature mixing, while source-aware negative masking gives +12% on the spatial axis for mixed annotation scopes.Random-partner mixing gives −2.8%, showing class-conditional matching—not mixing alone—is the mechanism; source-aware masking is the family’s best composite.Source: §H.2 · pp. 66–67

H.3 A prior violation traced to one loss term

  • 11.04% disjoint-box wearing under the gated background-suppression term is 22× the training-data rate, but removing the gate reduces it to 0.00% without other costs.The failure is not explained by vocabulary, graph reasoning, or spatial shortcuts, and A1–A3 cannot detect it because their annotations omit the violated pairs.Source: §H.3 · p. 67

H.4 Negative results

The evaluated directions were not adopted because their single-arm measurements did not support claims beyond the 1.3% noise floor. Evaluation resolution is reported in full for the released tower without retraining.

H.4 Negative results

  • None of the evaluated directions supported a main-text claim, and each measurement was interpreted against the 1.3% noise floor.The directions were evaluated but not adopted based on the listed measurements.Source: §H.4 · p. 67, Table 32 · p. 68
  • Evaluation resolution was reported in full for the released tower trained multi-scale at [0.5, 1.5]×448 without retraining.The evaluation used test splits, ground-truth boxes, and graph-constrained evaluation.Source: §H.4 · p. 67, Table 31 · p. 67

I Inference cost details · J Generation prompts

Inference is dispatch-bound at batch 1, while open-vocabulary scoring adds little live-frame cost but reduces batched throughput. Generation separates semantic, growth, and spatial passes through constrained prompts that enforce precise, nonredundant JSON relations.

I Inference cost details

  • 19.3–20.0 ms batch-1 latency on an A40 persists across towers despite 2.5× parameter and FLOP differences, indicating kernel-dispatch rather than arithmetic bottlenecks.The tower ordering appears only at batch 32, where throughput is 201 / 188 / 130 img/s; the smaller tower is therefore justified by throughput, not batch-1 latency.Source: I Inference cost details · p. 69, Table 34 · p. 69
  • 0.7–0.8 ms per frame is added on an A40 by scoring 19,103 strings instead of 50, while A100 and H100 differences remain within noise.At batch 32, the full vocabulary costs 11–22% throughput across towers, so vocabulary inputs are nearly free live but moderately costly offline.Source: I Inference cost details · p. 69
  • 27.5–28.4 ms A100 batch-1 latency versus 18.9–20.0 ms on A40 and H100 reflects host-CPU dependence, despite A100 batch-32 throughput being 1.9–2.0× higher.The pattern reproduces on an idle second A100 node, so a named GPU alone does not identify an operating point.Source: I Inference cost details · pp. 69–70
  • 20.3 ms on A40 and 18.1 ms on H100 for compiled inference beat eager execution at 30.5 and 32.1 ms, making compilation the fastest tested configuration.Measured detector overlap was inconsistent and costly after compilation, so the sequential path is used.Source: I Inference cost details · p. 70, Table 35 · p. 70

J Generation prompts

  • The generation procedure reproduces three prompts verbatim because calibration selected them through coverage over verify-before-assert, making their exact wording part of interpretation.The prompts are organized as semantic, grow, and spatial passes.Source: J Generation prompts · p. 71

J.1 Semantic pass

  • The semantic pass prioritizes activity, physical interaction, and directed attention, using precise open-vocabulary verb phrases while banning spatial predicates and duplicate subject-object pairs.It targets 6–10 relations, requires every numbered object to appear when informative, and emits one JSON block without extra text.Source: §J.1 · p. 71

J.2 Grow pass

  • The grow pass adds only genuinely visible, missing relationships, emphasizing underrepresented objects and distinct pair relations while forbidding repeats, inverses, synonyms, and vague predicates.If nothing is missing, it outputs an empty list and always returns only new relationships in JSON.Source: §J.2 · p. 72

J.3 Spatial pass

  • The spatial pass handles layout separately, preferring depth, contact, and proximity, covering major objects with one viewer-perspective relation per pair and excluding action verbs and body-part objects.It targets 6–12 relations, avoids inverse or redundant pairs, and outputs only new spatial relationships in JSON.Source: §J.3 · pp. 72–73

Related papers

Loading 2609.12552v1…