PaperlayerView exact revision on arXiv

Source-linked AI summary

RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs

Maëlic Neau

arXiv:2609.12552v1cs.CV

TL;DR

Relation prediction remains tied to fixed predicate vocabularies, label-conditioned architectures, and evaluation protocols that reward corpus agreement rather than broader relational description. RelateAnything combines a separable text-conditioned relation model with verified free-text supervision and cross-dataset evaluation. It achieves 2.3–3.5× the mean recall of a comparable open-vocabulary baseline across three cross-dataset benchmarks and one zero-shot benchmark, while in-domain measurement overstates transfer gains by about 5×.

  • Problem

    Scene-graph relation prediction remains tied to fixed predicate vocabularies, object labels, and annotation styles, while standard recall rewards agreement with the training corpus.

  • Method

    RelateAnything takes images and regions from any source, scores relations over inference-time predicate strings using text embeddings, and is trained with RA-4M’s verified free-text relations.

  • Results

    2.3–3.5× higher mean recall than OvSGTR across three cross-dataset benchmarks and one zero-shot benchmark, with margins persisting under real detector regions.

  • Takeaways & Limitations

    Open-vocabulary relation prediction benefits from separating region inputs, predicate vocabularies, supervision, and evaluation from fixed detector and corpus assumptions.

  • Takeaways & Limitations

    RA-4M has no human-audited precision estimate, and the general held-out-concept variant across all four benchmarks has not been run.

Abstract

from arXiv · show

Open-vocabulary detection accepts any class list at inference, and promptable segmentation returns regions without class names: the taxonomy has left the model and become an input. Relation prediction has not. Scene-graph models are still trained and evaluated on the 50 or 56 predicates of one annotation style, their relation head conditioned on object labels and so tied to one detector. Three obstacles explain this, none primarily modelling: no relation corpus is both free-text and verified, a label-conditioned architecture cannot accept a vocabulary it was not trained on, and the standard metric rewards agreement with the training corpus, so a larger vocabulary scores as a regression. We present RelateAnything, a 53M-parameter model taking an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings. Object labels are never an input, so the region source can change without retraining, and the vocabulary is a bank of text embeddings, not a learned classifier. It runs at 20 ms/frame. Training over 19,103 predicates requires positive-unlabeled supervision and a text encoder that separates antonyms, which contrastive encoders embed at cosine 0.95. To supply the supervision we build RA-4M, 474k images and 4.3M relations over 10,102 free-text predicates, generated against numbered box markers and geometrically verified. To measure it we build OV-SGG-Bench, six axes scored across datasets that the priors standard recall rewards cannot satisfy. On three cross-dataset benchmarks and a fourth zero-shot, RelateAnything has 2.3-3.5x the mean recall of the strongest open-vocabulary method of comparable scale, margins that survive a real detector, and leads a 3B-VLM scene-graph model on both metrics at under 2% of its parameters. In-domain measurement overstates transfer gains ~5x. Model, corpus and benchmark are public.

1 Introduction

RelateAnything addresses three non-modelling barriers to open-vocabulary relation prediction: limited verified supervision, label-conditioned architectures, and recall metrics that reward corpus agreement. It combines a separable relation model, the RA-4M corpus, and OV-SGG-Bench to evaluate transfer beyond fixed predicate sets.

  • OV-SGG-Bench aggregates six chance-corrected axes across datasets because standard recall cannot reliably measure open-vocabulary relation prediction.The benchmark is designed to expose prior matching and scoring-convention effects rather than only annotation agreement.
  • The paper identifies three obstacles: absent free-text verified supervision, architectures tied to object labels, and metrics that penalize answers outside the benchmark vocabulary.Standard scene-graph recall can reward agreement with corpus priors rather than relations visible in the image.
  • RelateAnything takes images, regions from any source, and inference-time predicate strings, while discarding object class labels and avoiding a learned predicate classifier.Its separable interface supports replacing the region source or vocabulary without retraining, and runs at 20 ms per frame.
  • RA-4M provides 474k images and 4.3M geometrically verified relations over 10,102 free-text predicates, generated using numbered box markers.The corpus is 1.7× denser and has 107× the vocabulary of the annotations it replaces on the same images and boxes.
  • 2.3–3.5× higher mean recall than OvSGTR across three cross-dataset benchmarks and one zero-shot benchmark, with margins persisting when a real detector supplies regions.Rare-predicate recall is higher by 5–21×, and the model leads on every reported recall and precision metric.

2 Related work

Prior work largely treats scene-graph generation as fixed-vocabulary prediction conditioned on object labels and corpus statistics. Open-vocabulary methods and machine-generated supervision relax parts of this setup, but do not jointly provide verified free-text relations and evaluation beyond corpus agreement.

  • Scene-graph benchmarks generally retain fixed predicate vocabularies, while dominant models condition relation prediction on object labels and shared visual representations.VG150 uses 50 predicates and PSG uses 56, each derived from a particular annotation style.
  • Debiasing and label-correction methods estimate corrections from training-label statistics, preserving the same prior structure that standard evaluation rewards.These approaches do not address shared training–benchmark vocabularies or the limitations of standard recall identified by the paper.
  • One-stage and real-time scene-graph models improve efficiency but remain primarily closed-set systems.The cited designs predict subjects, objects, and predicates from a shared query set or analogous fixed-vocabulary structures.
  • Open-vocabulary relation approaches replace predicate classification with text similarity, prompting, language-model expansion, or compositional text embeddings.These methods admit new predicate strings at inference, but the related-work passage does not establish verified supervision or evaluation beyond corpus agreement for them.
  • Machine-generated relation corpora differ in grounding and checking, ranging from caption parsing to multimodal-model annotation with optional filtering.The paper positions its own corpus against this literature by requiring a subsequent verification step.
  • Vocabulary-scale training requires positive-unlabeled learning because unannotated predicates may still be true, while contrastive text spaces can collapse antonyms together.This creates both supervision and representation challenges for open-vocabulary relation prediction.

3 The RelateAnything model

RelateAnything separates visual inputs, region sources, and predicate vocabularies, scoring region pairs against frozen text embeddings rather than object-label-conditioned classifiers. Its training addresses unlabeled positives and antonym collapse through synonym-group objectives and text-embedding distillation.

  • RelateAnything scores ordered region pairs against predicate strings supplied at inference, using pixels and box coordinates without object labels.Replacing the predicate vocabulary is a matrix substitution, and the region producer is external to the model.
  • A visual path pools backbone features for each box and pair, then refines sampled pairs using pairwise and scene-level attention.The representation includes union and contact regions plus scale-invariant geometry.
  • Two cosine branches separately model spatial and semantic evidence, with a predicate-conditioned gate mixing them from the predicate embedding alone.Projective and proximity predicates favor geometry, whereas actions favor appearance; the gate is defined for unseen strings.
  • At vocabulary scale, InfoNCE uses synonym groups as positives, while positive-unlabeled weighting prevents co-true but unannotated predicates from being treated as negatives.Annotated predicates retain full weight, and directional inverses are never discounted.
  • Teacher text embeddings place spatial antonyms near synonyms, including above/below at cosine 0.95, limiting the separability of visual relation targets.The limitation applies to methods that score relations against off-the-shelf text embeddings.
  • Distillation raises synonym-versus-inverse AUC from 0.85 to 0.99 and lowers mean inverse cosine from 0.92 to 0.09 while preserving synonym structure.The distilled student is used offline; inference uses the resulting embedding bank.
  • Exact-string evaluation can undercount correct synonym-group predictions, such as wearing when the benchmark annotation is wears.The paper reports a corrected view for this effect in A3.

4 RA-4M: a verified free-text relation corpus

RA-4M is a large free-text relation corpus generated with vision–language annotation and deterministic geometric verification. It expands relation coverage while preserving explicit safeguards against grounding and directional inconsistencies.

  • 474,413 images and 4,282,531 relations cover 10,102 free-text predicates, averaging 9.03 relations per image.The corpus was generated at a cost of 104 GPU-hours.
  • A vision–language model annotates relations densely and uniformly, while geometric checks detect errors that the model cannot reliably avoid.The paper uses the vision–language model as annotator rather than predictor because relation annotation is expensive and difficult to verify manually.
  • Numbered box markers provide grounding to the annotator, and a deterministic geometric gate rejects 11.3% of raw relation candidates.The gate rejects or swaps roles only when box geometry logically constrains the predicate; unconstrained predicates remain residual risk.
  • Residual errors remain concentrated in geometrically unconstrained relations, including gaze, wrong-instance grounding, prior-driven labels, contact predicates, and mislabelled boxes.The paper identifies a measurable downstream effect for one residual error class but does not quantify it in these passages.
  • Compared with source annotations on identical images and boxes, RA-4M is 1.7× denser, has 107× the vocabulary, and raises predicate entropy by 1.4 nats.Its mean object degree increases from 1.88 to 3.20, while single-component graphs rise from 70.9% to 82.6%.
  • The training mixture uses free-text annotations from RA-4M and leakage-filtered Visual Genome, with the released tower additionally incorporating HICO-DET.The zero-shot tower uses 40,615 Visual Genome images and 17,742 predicate strings, while evaluation splits are excluded by image identity and near-duplicate filtering.

5 OV-SGG-Bench: analysis of scene-graph evaluation and a cross-dataset protocol

OV-SGG-Bench shows that standard scene-graph recall is strongly shaped by dataset and annotation priors, detector settings, and scoring conventions. It therefore uses six complementary axes, including prior-blind measurements, to evaluate relational competence across conditions.

  • 5.2 Three priors rewarded by recall: A frequency baseline beats the trained model on micro scoring but loses on macro scoring, showing that standard recall rewards object-category priors over balanced relation understanding.The frequency lookup uses ground-truth labels and no pixels, while the trained model receives no labels.
  • 5.2 Three priors rewarded by recall: 78% of emitted edges fall on pairs never annotated in PSG test, so predicting annotation propensity is not equivalent to predicting relations.Only 11% match an annotated triple in either direction at the checkpoint’s maximum-F1 threshold.
  • 5.3 Two scoring conventions with effects larger than those measured: Restoring one-to-one detection matching reduces published R@50 by 12.0% and novel R@50 by 13.5%, exceeding the apparent gain from a larger backbone.The correction changes Base+Novel R@50 by 2.45 points, versus 2.39 points from upgrading Swin-T to Swin-B.
  • 5.3 Two scoring conventions with effects larger than those measured: Detector confidence thresholds and box caps move the pair-recall ceiling by 12–49% across four detector-benchmark combinations, dwarfing the 4.3-point published-method spread.On VG150, the operating-point change is 19 points, so detection-mode comparisons must report these settings.
  • 5.4 The six axes: OV-SGG-Bench reports six axes because no single score can address all priors; A2 uses adjudicated negatives and A6 is balanced against category-prior exploitation.The paper treats these two axes as the basis for claims about relational competence rather than corpus matching.
  • 5.4 The six axes: The composite OVS combines chance-corrected A1, A2, A4, A5, and A6 with a harmonic mean, while A4 and A5 use measured normalization quantities.A4 is normalized by the shared detector’s pair-recall ceiling, and A5 by information in the images’ annotations.

6 Results

RelateAnything consistently improves open-vocabulary relation prediction across transfer, free-text, detector-mode, and adversarial evaluations, while exposing important metric and judge limitations. Its gains include stronger rare-predicate and mean recall, robustness to vocabulary and region-source changes, and better information per accepted relation.

  • 6.1 A1: cross-dataset transfer on four benchmarks: 2.3–3.5× mean recall over OvSGTR extends across benchmarks, while rare-bucket recall reaches 5–21× the baseline.RelateAnything outperforms OvSGTR on every reported metric and source, despite receiving no object labels while the baseline receives ground-truth labels.
  • 6.2 A2: precision on adjudicated negatives: 39% higher mean federated AP, 59% higher rare-predicate fAP, and P-AUC rising from 83.9 to 93.7 demonstrate improved precision on adjudicated negatives.The comparison spans 55 predicates with support of at least five, including 31 rare predicates.
  • 6.3 A3: the full vocabulary with the answer set withheld: 56.0 R@50 with the full 19,103-string vocabulary exceeds 53.3 with benchmark strings, while synonym matching changes rankings substantially.Synonym mapping adds 11.7 R@50 to ROBIN and 23.2 to RelateAnything; the reported band is needed because scorer choice changes the result.
  • 6.4 A4: detection-mode graphs on a shared detector: 3.3× mR@50 and 14× rare-bucket recall survive when a shared detector supplies regions, showing that the relation comparison retains its structure across box sources.Detector losses dominate: pair-recall retention is 62–63% on PSG and 31–32% on IndoorVG when detected classes are ignored.
  • 6.5 A5: information-weighted human preference: 18.6 versus 13.4 bits per image at matched depth and 28.7 versus 14.1 at deployed depth favor RelateAnything’s accepted relations over OvSGTR’s.OvSGTR receives slightly more accepted relations at matched depth, but each accepted relation carries less information; the axis avoids graph-length dependence.
  • 6.6 A6: spatial relations against adversarial negatives: A6 remains the weakest axis despite RelateAnything’s macro AUC of 0.690 versus OvSGTR’s 0.591, with gains on all nine spatial predicates.The auxiliary and deformable components raise AUCs for in front of, above, and to the left of, while the adversarial setup prevents frequency priors from helping.

7 Analysis: decoupling object and relation prediction

RelateAnything separates relation prediction from object labels and detector identity, testing whether relational evidence comes from image regions rather than category priors. The analysis finds that pair context and geometry carry the relation signal, while shared dense features do not.

  • 7.1 Label dependence of the relation head: 88–90% of the baseline’s predictions are reproduced by label-pair frequency tables, whereas RelateAnything uses all 50 predicates and retains three times more residual entropy.Shuffling labels leaves the baseline’s output statistics unchanged, while RelateAnything never observes them.
  • 7.2 Interchangeable region sources: One checkpoint reads relations from both a 4,585-tag detector and a class-agnostic segmenter, showing that the region source can change without retraining.The returned edges remain the same kind of relation output despite the different region interfaces.
  • 7.3 Effect of relation supervision on the dense features: Fine-tuning increases class selectivity from 0.242 to 0.281 but leaves relation selectivity nearly unchanged, from 0.133 to 0.138.The learned dense features become less able to distinguish a relation partner from an unrelated object of the same class.
  • 7.3 Effect of relation supervision on the dense features: The interaction region becomes more contact-focused, yet its feature retrieves object class at 0.761 versus 0.068 chance, indicating object identity rather than relational structure.The contact advantage rises from 0.356 to 0.438 as the feature shifts toward hand and object.
  • 7.4 Attribution of the relation logit: Pair context explains 87–93% of semantic-logit variance, while removing the subject–object identity path changes micro accuracy by only 0.1%.Cross-image features reduce micro accuracy by 44–68%, and removing box geometry costs 6–24%.
  • Summary: The analysis concludes that relation supervision does not create relational dense features, so a shared backbone is not justified and detector replacement is practical.Object labels provide a transferable prior for frequent relations, whereas visual evidence contributes in the tail.

8 Analysis: in-domain versus cross-dataset measurement

The study compares in-domain improvements with transfer behavior while varying corpus scale, mixture composition, annotation quantity, and backbone capacity. In-domain scores consistently provide an unreliable guide to cross-dataset performance, and detector quality becomes the practical bottleneck.

  • 8.1 Corpus scale read in-domain and under transfer: +15% in-domain micro recall and +56% macro recall from doubling the corpus become only +2.5 to +3.8% and +9.9 to +11.9% under transfer.The in-domain slope therefore overstates the transfer slope by approximately 5×.
  • 8.1 Corpus scale read in-domain and under transfer: Removing automatically derived left/right edges improves in-domain micro and macro recall by +2.0% and +27.0% but reduces transfer macro recall by −5.9 to −0.3%.The extrapolated gain from 500k additional annotated images was about one absolute point, so the authors did not generate them.
  • 8.1 Corpus scale read in-domain and under transfer: Final-epoch checkpoint ranking predicts out-of-distribution behavior better than best-epoch selection, with ρ = 0.77 versus 0.66.This supports making data-scaling and checkpoint decisions using transfer measurements.
  • 8.2 Annotation quantity under sparse annotation: Adding sparse Open Images relation annotations harms every axis, including approximately −6% on the development composite and −40.2% on HICO-DET tail recall.The extension was tested at an amplified 25% relation share to exceed the noise floor.
  • 8.3 Backbone capacity: Increasing backbone capacity does not improve the composite: a tower with 114% more parameters changes it by −0.1%, while smaller towers perform best on the spatial axis.Scaling shifts evidence from box geometry toward pixels, but the advantage disappears at the deployment operating point.

9 Inference cost

RelateAnything is evaluated as a real-time open-vocabulary system against an open-vocabulary baseline and closed-set specialists. Its speed advantage depends on a smaller deployment configuration, while specialists retain higher in-domain micro recall.

  • Comparison with OvSGTR: RelateAnything is 7.8× faster end to end than OvSGTR under the shared protocol.It scores at most 128 sampled pairs among 20 boxes at 448 px in bf16, whereas OvSGTR scores about 9,500 pairs among 98 boxes at 800/1333 in fp32.
  • Comparison with closed-set real-time models: REACT++ runs at 21–23 ms versus RelateAnything’s 35–36 ms in the cited specialist comparison, while the specialists achieve higher micro recall on every trained benchmark.RelateAnything has higher macro recall everywhere except PSG.
  • Deployment configuration: Compilation brings the released tower with detector and decoding to 20 ms per frame, while scoring 19,103 strings costs under a millisecond at batch 1.At batch 1, kernel dispatch rather than FLOPs dominates the relation head’s latency.

10 Discussion and limitations

The discussion bounds the evidence behind RelateAnything’s claims, especially concept novelty, corpus precision, shared representations, judge-based informativeness, and detector recall. It also proposes reporting practices intended to make scene-graph comparisons more diagnostically useful.

  • Discussion and limitations: Cross-dataset results reuse predicate strings seen during training, while the general variant holding concepts out across all four benchmarks has not been run.Several conclusions also rest on one seed, with differences within a 1.3% noise floor treated as indistinguishable.
  • Discussion and limitations: RA-4M has no human-audited precision estimate for its 4.3M machine-generated relations, leaving structural filtering as the basis for its precision claims.The authors identify human auditing as the first measurement to add before relying on the corpus.
  • Discussion and limitations: The decoupling result shows that this recipe does not produce relational dense features, not that relational features could never be learned.The missing ingredient is a patch-level relational objective, which the probes suggest a shared backbone would require.
  • Discussion and limitations: Per-relation precision is a genuine shortcoming: the baseline overreaches less at every depth, so false-edge-sensitive deployments should prefer shorter graphs.A one-relation-at-a-time judge cannot perceive repetition, motivating surprisal-based informativeness instead.
  • Discussion and limitations: Detector recall is the dominant deployment loss because pair recall is quadratic in object recall and relation models cannot exceed it.A larger relation model can improve rankings that deployment never emits when detector boxes are missing.
  • Discussion and limitations: The paper recommends reporting corpus–benchmark triplet overlap, matcher rules, detector thresholds and pair-recall ceilings, graph constraints, and adjudicated-negative performance.These inexpensive disclosures changed conclusions in this study.

11 Conclusion

The paper removes supervision, architecture, and measurement barriers to open-vocabulary relation prediction, while exposing remaining calibration and evaluation constraints.

  • 11 Conclusion: 2.3–3.5× mean recall over the strongest comparable open-vocabulary method is achieved across three cross-dataset benchmarks and a fourth zero-shot benchmark.The gains remain at 20 ms per frame, while OV-SGG-Bench evaluates six axes across datasets.
  • 11 Conclusion: OV-SGG-Bench addresses standard recall’s prior-matching behavior by evaluating relation prediction across datasets and multiple axes.A frequency table can outperform trained models on leaderboard recall while underperforming per predicate.
  • 11 Conclusion: The deformable read concentrates on subject and contact anchors, while union sampling contributes least because large union boxes extend beyond the image.PSG shares are 0.43–0.48 for subject, 0.17–0.21 for contact, and 0.04–0.05 for union.
  • 11 Conclusion: The predicate-conditioned gate routes spatial relations toward geometry and actions toward appearance, with contact relations receiving intermediate mixing.These routing weights are learned from text embeddings without direct supervision.

A.4 Training recipe

The released training recipe uses variable resolution and photometric, but not geometric, augmentation to preserve directional relation labels and support evaluation at other resolutions.

  • A.4 Training recipe: The training run takes 5h04m on four A100 GPUs.
  • A.4 Training recipe: Resolution is sampled from seven values in [0.5, 1.5] × 448 as regularization, allowing evaluation at other resolutions without retraining.
  • A.4 Training recipe: Geometric augmentation is excluded because horizontal flips invert left/right relations without rewriting balanced labels, whereas photometric augmentation is retained.

B Training objective

The training objective aligns pair representations with predicate text regions while handling positive-unlabeled supervision, directional order, source-specific annotation scope, and synonym variation.

  • B Training objective: A swap hinge makes subject–object order contribute to directional-predicate training, while background suppression downweights likely-unannotated pairs without treating them as certainly negative.
  • B Training objective: Source-aware negative masks recover +12% on the spatial axis by excluding predicates that a source could not have annotated.HICO-DET annotates verbs but not spatial relations, making unannotated spatial predicates unreliable negatives.
  • B Training objective: The objective treats synonym groups as text-space targets rather than single predicate points, improving semantic grouping but causing exact-string metrics to count valid alternatives as misses.Non-winning surface forms are pushed to ranks 90–338.
  • B Training objective: 19,103 predicate strings define the answer bank, including 10,102 RA-4M predicates and 9,001 strings from additional corpora without requiring those relations as training supervision.
  • B Training objective: The pair-relatedness score improves recall on annotation-derived benchmarks but removing it improves every projective predicate by +0.03 to +0.05 AUC on adjudicated negatives.
  • B Training objective: The released model ranks relations well but is uncalibrated because mass-balanced training assumes an implicit 50/50 prior while real frames are 0.2–4% positive.

D.8 Graph-quality oracle (A5)

The graph-quality oracle compares complete relation graphs with certified judge controls, while a matched-depth arm separates graph quality from graph length.

  • D.8 Graph-quality oracle (A5): The oracle’s top section reports graph shape, while its bottom section reports win rate among decisive comparisons stratified by which graph was longer.The comparison uses PSG test images, shared YOLO-World boxes, and Qwen3-VL-8B.
  • D.8 Graph-quality oracle (A5): The judge passed certification with scramble accuracy 1.00, primacy 0.19, and 56 of 103 decisive control comparisons.
  • D.8 Graph-quality oracle (A5): A5b scores each asserted relation separately, avoiding the pairwise oracle’s inability to distinguish false extra relations from a preference for shorter graphs.
  • D.8 Graph-quality oracle (A5): Informativeness is measured as surprisal because the judge’s direct informativeness question nearly duplicated truth judgments.The original question had P(info = 0 | false) = 0.998.
  • D.8 Graph-quality oracle (A5): The normalized informativeness denominator includes all annotated relations, including those whose endpoints the shared detector failed to find.

E.1 Per-predicate results

RelateAnything’s per-predicate behavior reflects both genuine relational performance and evaluation artifacts, while transfer remains robust across datasets and detector settings. Spatial reasoning improves over the baseline but remains bounded by annotation and grounding limitations.

  • Synonym substitutions create many zero-recall predicates, while multi-label annotations and unannotated true relations also count correct predictions as misses.HICO-DET’s single emitted verb cannot receive credit for alternative annotated verbs, and PSG provides no credit for unannotated relations or near-synonyms.
  • The model’s edge-level complementarity matters: the frequency baseline is correct 2.4× as often on model-wrong edges, yet model-only edges comprise 7–13% of all edges.The two systems therefore succeed on partly different edges rather than forming a simple strong-versus-weak ordering.
  • Detection preserves the micro ladder but flattens the macro ladder, with IndoorVG losing 70–80% of performance because its categories are poorly covered.The retention measurement uses YOLO-World detections with strict class-aware matching and a 500-pair budget.
  • Grounding probes indicate that spatial decisions depend mainly on box geometry and semantic decisions on pixels, while retaining pixels inside the union box preserves 75–92% of top-1 decisions.The context added by deformable reading rarely changes these decisions in the reported probe.
  • 2.2× mean-recall margin on IndoorVG versus the strongest comparable open-vocabulary system, with smaller gains of 1.9× on VG150 and 1.1× on PSG.These comparisons use ground-truth regions and graph-constrained evaluation across three benchmarks.

G.1 Effect of backbone capacity on transfer

Backbone scaling does not reliably improve transfer: the released middle tower outperforms the largest model at much lower cost, while larger models shift from geometry toward pixel evidence. Recipe components and supervision quality matter more than capacity alone.

  • A 114% parameter increase from S+ to B yields −0.1% composite change, while S+ itself costs 15% more parameters and yields +7.4%.The largest tower also requires 1.33× memory and 1.22× wall-clock time, and does not outperform the released tower.
  • ViT-S/16+ is best on A6 on both ladders, while ViT-B is worse than a model less than half its size, supporting a supervision-ceiling interpretation.Capacity cannot substitute for absent projective-relation supervision in this axis.
  • Larger backbones become less dependent on box geometry and more dependent on pixels, with geometry effects changing from −35.2% to −9.8% and pixel effects from −53.0% to −58.1%.The opposite trends indicate substitution rather than general robustness to either lesion.
  • The deformable scene read improves every axis and recovers the sigmoid auxiliary’s transfer cost, making the pair the full-scale recipe.The sigmoid auxiliary specifically moves the projective spatial band and makes the output-head scale trainable.
  • Class-conditional feature mixing raises zero-shot tail mean recall by 10.1%, while random-partner mixing decreases it by 2.8%, identifying matching rather than mixing as the mechanism.Source-aware negative masking adds 12% on the spatial axis when annotation scopes are mixed.
  • The model violates the garment-body prior 22× more often than its training data under a gated background-suppression variant, with 11.04% disjoint-box wearing assertions.The error is mainly co-reference fan-in rather than vocabulary selection or relational inference.

H.4 Negative results

Several evaluated directions were not adopted, and the reported measurements are explicitly bounded by a 1.3% noise floor and single-arm comparisons.

  • Evaluation-resolution results use a released tower trained multi-scale at [0.5, 1.5]×448 without retraining, on graph-constrained test splits with ground-truth boxes.The supplied passage identifies the protocol but does not report the resulting values.
  • The evaluated but rejected directions are documented with the measurement that determined each decision rather than presented as main-text claims.The negative-result table should therefore be read as recipe-screening evidence, not as a broad performance conclusion.

I Inference cost details

Inference cost is dominated by deployment conditions rather than parameter count alone. The open vocabulary is inexpensive at batch 1, while host CPU, compilation, and batching materially affect reported latency and throughput.

  • 19.3–20.0 ms batch-1 latency on an A40 is nearly unchanged across towers spanning a 2.5× parameter and FLOP range.Tower ordering appears at batch 32, where throughput is 201, 188, and 130 images per second.
  • Using 19,103 predicates instead of 50 adds 0.7–0.8 ms per frame on an A40 and no measurable cost on A100 or H100 at batch 1.At batch 32, the same vocabulary head costs 11–22% throughput across towers.
  • Batch-1 latency depends on the host CPU: A100 measurements are 27.5–28.4 ms versus 18.9–20.0 ms on A40 and H100 despite higher A100 throughput at batch 32.The pattern reproduces on a second idle A100 node, consistent with dispatch-bound host dependence.
  • Compilation reduces end-to-end latency from 30.5 to 20.3 ms on A40 and from 32.1 to 18.1 ms on H100, but measured detector overlap varies by hardware.Overlap recovers 2.6 ms on an A40 under eager execution but costs 27 ms on an H100.
Loading 2609.12552v1…