PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition

Geo Ahn, Inwoong Lee, Taeoh Kim, Minho Shim, Dongyoon Wee, Jinwoo Choi

arXiv:2601.16211v3cs.CVcs.AI

TL;DR

ZS-CAR models can rely on object-driven shortcuts instead of temporal evidence, weakening recognition of unseen verb–object compositions. RCORE addresses this with co-occurrence and temporal-order regularization, consistently improving unseen-composition performance across two benchmarks.

  • Problem

    Sparse compositional supervision and asymmetric verb–object learning promote object-driven shortcuts that replace temporal evidence with co-occurrence priors in ZS-CAR.

  • Method

    RCORE combines CPR, which expands supervision and treats frequent pairs as hard negatives, with TORC, which enforces temporal-order sensitivity.

  • Results

    Across Sth-com and EK100-com, RCORE reduces shortcut diagnostics and consistently improves unseen-composition generalization across datasets and backbones.

  • Takeaways & Limitations

    RCORE supports more reliable verb–object compositional reasoning under an open-world evaluation protocol without test-tuned bias calibration.

  • Takeaways & Limitations

    Conventional closed-world evaluation can hide over-prediction of seen compositions, while test-label bias calibration can inflate unseen accuracy and undermine fair comparison.

Abstract

from arXiv · show

Zero-Shot Compositional Action Recognition (ZS-CAR) requires recognizing novel verb-object combinations composed of previously observed primitives. In this work, we tackle a key failure mode: models predict verbs via object-driven shortcuts (i.e., relying on the labeled object class) rather than temporal evidence. We argue that sparse compositional supervision and verb-object learning asymmetry can promote object-driven shortcut learning. Our analysis with proposed diagnostic metrics shows that existing methods overfit to training co-occurrence patterns and underuse temporal verb cues, resulting in weak generalization to unseen compositions. To address object-driven shortcuts, we propose Robust COmpositional REpresentations (RCORE) with two components. Co-occurrence Prior Regularization (CPR) adds explicit supervision for unseen compositions and regularizes the model against frequent co-occurrence priors by treating them as hard negatives. Temporal Order Regularization for Composition (TORC) enforces temporal-order sensitivity to learn temporally grounded verb representations. Across Sth-com and EK100-com, RCORE reduces shortcut diagnostics and consequently improves compositional generalization.

1 Introduction

The paper identifies object-driven verb shortcuts as a central obstacle to ZS-CAR generalization and proposes RCORE to counter them through co-occurrence and temporal-order regularization. Across Sth-com and EK100-com, RCORE reduces shortcut diagnostics and improves unseen-composition generalization.

  • FSP and FCP diagnose unseen inputs collapsing to seen, especially frequent, training compositions in existing ZS-CAR models.These ratios quantify shortcut-driven failures and expose overreliance on training co-occurrence patterns.
  • RCORE combines CPR, which expands supervision over absent compositions and treats frequent pairs as hard negatives, with TORC, which enforces temporal-order sensitivity.Together, the components target sparse compositional supervision, co-occurrence priors, and weak temporal grounding.
  • The paper identifies skewed verb–object co-occurrence and asymmetric learning difficulty as root causes of object-driven shortcuts.Objects are visually explicit, whereas verbs require multi-frame temporal reasoning, causing object features to dominate training signals.
  • RCORE improves unseen-composition performance over baselines across Sth-com and EK100-com using multiple VLM backbones.The method also mitigates co-occurrence biases and learns temporally grounded verb features.

2 Related Work

Prior work studies compositional action recognition, zero-shot composition, and shortcut learning, but ZS-CAR additionally requires temporally grounded verbs under a fixed label space. This setting enables diagnosing object-driven shortcuts and designing backbone-transferable training regularizers.

  • Compositional Action Recognition: CAR factors actions into verbs and objects, while prior methods seek verb representations robust to static cues and sensitive to temporal dynamics.Approaches include modeling object-level interactions.
  • Compositional Action Recognition: ZS-CAR holds out verb–object pairs while sharing verb and object vocabularies across splits; conditional learning and disentanglement-oriented designs improve joint modeling.The study highlights that sparse and skewed compositional supervision can still produce object-driven shortcuts.
  • Compositional Zero-Shot Learning: Image-based CZSL recognizes novel compositions of seen primitives under sparse supervision, whereas ZS-CAR additionally requires temporal grounding of verbs.Image CZSL uses feature disentanglement, modular factorization, and CLIP-based prompt learning.
  • Shortcut learning: Shortcut learning exploits spurious correlations, with skewed distributions and co-occurrence statistics encouraging reliance on dataset priors.Prior compositional work reweights rare combinations, while video studies report reliance on static scene cues.
  • Vision-Language Model: VLMs use encoder-based or decoder-based paradigms, while ZS-CAR evaluates fixed verb–object logits over YV × YO for controlled diagnosis and transferable training regularizers.Encoder examples include CLIP and video extensions; decoder examples include LLaVA and Qwen-VL [4].

3 Diagnosis: Why ZS-CAR Models Fail?

The diagnosis shows that ZS-CAR models exploit easier object cues and training co-occurrence priors instead of learning temporally grounded verbs, producing shortcut-driven failures on unseen compositions. Training-bias metrics and the Compositional Gap quantify this behavior and link it to weak compositional generalization.

  • Training-bias metrics: Training-bias diagnostics quantify reliance on co-occurrence priors using False Seen Prediction and False Composition Prediction ratios on unseen compositions.Their component-wise breakdown distinguishes verb-collapse, object-collapse, and dual-collapse errors to identify whether bias primarily harms verb or object learning.
  • Object-driven shortcuts indeed exist in ZS-CAR: Shortcut reliance correlates with poor unseen generalization: the seen–unseen accuracy gap grows with FSP/FCP, and verb-collapse dominates CLIP’s final errors.This pattern remains notable with the video-pretrained InternVideo2 backbone, which still exhibits a seen–unseen gap and high FSP.
  • Compositional Gap: The Compositional Gap measures whether joint verb–object prediction exceeds the independent reference AccV × AccO, and is reported alongside FSP/FCP and unseen accuracy.Negative ∆CG on unseen compositions indicates that AccC falls below the independent reference, including with the video-pretrained backbone; ∆CG is diagnostic rather than standalone.
  • Objects are easier to learn than verbs: Controlled experiments show that objects are learned faster and more accurately than verbs, enabling object-driven shortcuts under skewed verb–object co-occurrences.On a perfectly biased split, CLIP achieves high object accuracy while verb accuracy on bias-conflict unseen compositions falls below chance.

4 RCORE

RCORE mitigates object-driven shortcuts in ZS-CAR under sparse, skewed compositional supervision by strengthening verb–object representations. It combines CPR, which expands and regularizes composition supervision, with TORC, which enforces temporally grounded verb representations.

  • 4 RCORE: RCORE mitigates object-driven shortcuts in ZS-CAR by strengthening verb–object representations under sparse compositional supervision.The framework uses AIM for CLIP and LoRA for InternVideo2 backbones, following prior work.
  • Co-occurrence Prior Regularization (CPR): CPR synthesizes unseen verb–object pairs, uses frequent seen pairs as hard negatives, and regularizes co-occurrence priors with a margin constraint.It combines synthesized composition supervision, margin-based regularization, and batch-adaptive expansion of the composition label space.
  • Co-occurrence Prior Regularization (CPR): CPR constructs synthesized videos by injecting another sample’s static object cue into high-motion regions, then assigns soft labels for the resulting compositions.The high-motion mask is extracted with a learning-free estimator.
  • Temporal Order Regularization for Composition (TORC): TORC reduces reliance on static object cues by separating forward and reversed features and encouraging high-entropy verb predictions after temporal disruption.Its objective combines cosine-similarity and entropy terms: LTORC = Lcos + Lent.
  • Training Objective: RCORE jointly optimizes verb, object, composition, TORC, and CPR losses through a weighted total objective.Composition logits aggregate factorized verb and object logits, and the composition loss uses a dynamically constructed denominator Yexp.

5 Experimental Results

RCORE improves open-world zero-shot compositional recognition by reducing co-occurrence-driven and object-driven shortcuts while strengthening temporal verb representations. Its components complement each other, with TORC improving verb generalization and CPR targeting frequent confounders.

  • 5.2 Diagnostic Analyses: RCORE reduces co-occurrence reliance: on Sth-com, FSP and FCP fall from 47% to 44% and remain at 23%, unlike C2C’s increases from 53% to 63% and 26% to 29%.These curves indicate markedly less dependence on co-occurrence statistics during training.
  • 5.3 Quantitative Comparisons: RCORE improves Sth-com unseen recognition over C2C, raising verb@unseen-comp and unseen composition accuracy with both CLIP and InternVideo2.Gains are 4.6 and 3.8 points with CLIP, and 3.4 and 4.5 points with InternVideo2, respectively.
  • 5.2 Diagnostic Analyses: RCORE grounds verb representations in temporal dynamics, showing a larger original-versus-shuffled feature gap and better discrimination of opposite verbs than C2C.On unseen Sth-com compositions, RCORE more effectively distinguishes pairs such as ‘unfolding’ versus ‘folding’.
  • 5.4 Ablation Studies: TORC alone improves verb@unseen-comp by 3.5 points and unseen composition accuracy by 3.8 points, whereas CPR alone raises unseen composition accuracy by 3.1 points but lowers seen accuracy by 4.2 points.Combining CPR and TORC achieves the best overall results; the ablation uses C2C with CLIP [34] in the unbiased open-world setting.
  • 5.4 Ablation Studies: Penalizing frequent hard negatives in CPR achieves the best trade-off, including the highest H.M. of 37.7%, while penalizing all hard negatives harms object and composition performance.The result supports restricting penalties to frequent confounders.

6 Conclusions · Appendix

The paper identifies object-driven shortcuts as a key failure mode in zero-shot compositional action recognition and proposes RCORE to mitigate them. The appendix provides evaluation, dataset, implementation, diagnostic, and additional-result details supporting the main paper.

  • 6 Conclusions: The study attributes object-driven shortcuts to compositional sparsity and asymmetric verb–object learning difficulty, which promote overfitting to co-occurrence priors and harm unseen-composition generalization.Diagnostic metrics reveal that existing models underuse verb learning by relying on labeled object classes.
  • 6 Conclusions: RCORE addresses these shortcuts through CPR, which expands supervision to synthesized compositions and regularizes against co-occurrence priors.The supplied conclusion introduces CPR as one of RCORE’s two components, but does not provide further implementation or quantitative details.
  • Appendix: The appendix supplements the main paper with comprehensive evaluation-protocol, dataset, implementation, and additional-results details.It is organized to provide supporting material rather than introduce a separate research contribution.
  • Appendix: Section A documents the evaluation setting, while Section B details the EK100-com dataset.These sections define the evaluation and dataset context used by the paper’s experiments.
  • Appendix: Section C provides complete implementation details for reproducing the reported methods and experiments.The passage identifies implementation documentation as a dedicated appendix component.
  • Appendix: Sections D and E add evidence of object-driven shortcuts and additional results beyond those presented in the main paper.The supplied passage specifies their scope but does not report individual findings or numerical outcomes.

A Overview of our evaluation setting

The evaluation adopts an open-world protocol over the full verb-object composition space and measures composition, verb, and object recognition across seen and unseen compositions.

  • Evaluation protocol: The ZS-CAR label space is the Cartesian product of verb and object primitives, with test samples divided into seen and unseen compositions based on training exposure.
  • Evaluation protocol: Open-world evaluation computes logits for every possible verb-object combination, avoiding ground-truth test-label assumptions and exposing possible-but-unseen compositions.Closed-world evaluation restricts assessment to compositions present in the data, whereas the adopted open-world protocol better reflects real-world applications.
  • Evaluation metrics: Evaluation reports composition accuracy, verb@seen/unseen-comp, and object@seen/unseen-comp to assess compositional recognition and primitive degradation on unseen compositions.These metrics are derived from the model’s composition predictions by checking whether the composition, verb, or object is correct across seen and unseen splits.

B Details on EK100-com dataset · C Complete Implementation Details · C.1 Implementation details on our diagnosis

The appendix defines EK100-com, a curated ZS-CAR benchmark, and details diagnosis protocols using C2C, controlled toy data, and shortcut-focused misclassification metrics. It also shows that RCORE suppresses co-occurrence shortcuts and narrows seen–unseen performance gaps on EK100-com.

  • B Details on EK100-com dataset: EK100-com repurposes EK100 using filtered training and validation splits, retaining compositions with more than five samples and ensuring validation compositions occur in training.After filtering, training contains 62,790 samples and 1,331 compositions, while validation contains 8,657 samples.
  • B Details on EK100-com dataset: The benchmark swaps randomly selected composition subsets between training and validation, then splits validation and test data at a 3:4 ratio.The resulting EK100-com statistics and seen/unseen examples are provided in Table 6 and Figures 9–10.
  • B Details on EK100-com dataset: EK100-com provides explicit object supervision because EK100 videos depict single-verb, single-object actions, while EK100’s long-tailed distribution motivates the benchmark’s construction.These properties distinguish it from datasets where videos may involve multiple objects associated with a verb.
  • C.1 Implementation details on our diagnosis: The diagnosis uses C2C with separate linear verb and object classifiers operating on the ViT CLS token.This setup supports the appendix’s implementation details for the diagnosis described in the main paper.
  • C.1 Implementation details on our diagnosis: A controlled 4 × 4 Sth-Com toy set contains four verbs, four objects, and four diagonal training pairs with 40 samples per pair.The toy set controls sample counts for each verb–object pair when analyzing object-driven shortcuts.
  • C.1 Implementation details on our diagnosis: FSP measures unseen samples misclassified as seen compositions, while FCP measures those whose predictions are frequent training co-occurrences.Both ratios are further decomposed into verb-collapse, object-collapse, and dual-collapse cases.
  • C.1 Implementation details on our diagnosis: RCORE suppresses the increase of FCP during training and narrows the seen–unseen composition validation-accuracy gap versus C2C [19] on EK100-com.These trends are reported in the baseline-versus-RCORE learning curves.

C.2 Implementation details on RCORE

RCORE is implemented with CLIP-B/16 and InternVideo2 backbones using uniformly sampled video clips, specialized frame-level InternVideo2 pooling, and CPR/TORC training configurations. Inference uses a single-view, single-crop protocol with harmonic-mean selection over seen and unseen compositions.

  • Experimental setup: RCORE uses CLIP-B/16 as its visual/text backbone and processes clips of 3 × 16 × 224 × 224 for CLIP or 3 × 8 × 224 × 224 for InternVideo2.Experiments use PyTorch on 8 NVIDIA Tesla V100 GPUs.
  • InternVideo2 adaptation: InternVideo2 is modified to produce frame-level features by applying attention pooling separately to each frame’s CLS token and spatial tokens.The original mechanism instead concatenates the CLS token with video tokens to produce one video-level representation.
  • CPR augmentation: CPR mixes samples using a Beta(2.0, 2.0)-sampled coefficient scaled by 0.5 and applies the augmentation within batches with probability paug.FAME identifies high-motion foreground regions for generating the mixed samples.
  • Loss configuration: The total loss sets α = 0.2 for verb and object components and β = 1.0 for composition loss, while warming γ from 0.0 to 1.0 over 5 to 10 epochs.The LCPR scale δ varies by dataset and backbone according to Table 7.
  • Inference: Inference uses single-view, single-crop evaluation and selects models using the harmonic mean of seen- and unseen-composition accuracies.The same criterion is used for bias calibration.

D Additional evidence of object-driven shortcuts … E.2 Shortcut diagnosis on another ZS-CAR method

The additional analyses show that object-driven shortcuts affect multiple ZS-CAR methods, while RCORE improves temporal sensitivity and compositional reasoning on unseen compositions. Macro Compositional Gap results further show RCORE’s robustness across datasets and backbones.

  • D Additional evidence of object-driven shortcuts: Top-10 failure-case analysis evaluates C2C on unseen EK100-com compositions using unbiased open-world inference scores and reports prediction rates with frequent co-occurring counterparts.The analysis identifies the most frequent training-set counterpart and its co-occurrence ratio for each mispredicted component.
  • D Additional evidence of object-driven shortcuts: RCORE drives original–reversed verb-feature similarity to −0.79, distinguishing opposite temporal semantics such as ‘opening’ and ‘closing’.This result indicates that RCORE learns temporally grounded verb representations rather than relying on object-driven shortcuts.
  • E.1 Macro Compositional Gap: Macro Compositional Gap compares joint composition accuracy with the product of verb and object accuracies for each test composition, then averages across composition classes.The macro formulation is designed to ensure robustness to class imbalance.
  • E.1 Macro Compositional Gap: RCORE consistently achieves the highest Macro Compositional Gap on unseen compositions across Sth-com and EK100-com and different backbones.The metric averages compositional gaps across classes to reduce sensitivity to class imbalance.
  • D Additional evidence of object-driven shortcuts: On Sth-com, C2C maintains +0.92 cosine similarity between original and reversed verb features throughout training, indicating limited temporal sensitivity.Figure 13 compares C2C and RCORE using the CLIP [34] backbone.
  • E.2 Shortcut diagnosis on another ZS-CAR method: Jung et al. with the CLIP backbone also shows severe co-occurrence overfitting: the seen–unseen accuracy gap grows together with FSP/FCP despite enhanced disentangling training.This reproduces the trend observed for C2C, indicating shortcut reliance is not specific to that method.

E.3 Additional ablation results

Additional ablations on Sth-com show that attention pooling alone does not close the unseen-composition gap, while CPR sample design and pCPR tuning materially affect compositional performance. The results support explicit temporal-order modeling and composition-aware inputs rather than relying on generic aggregation or image augmentations.

  • Effects of TORC with different temporal aggregation methods: Attention pooling slightly improves verb accuracy but fails to reduce the compositional gap on unseen compositions, unlike the motivation for TORC.The study compares temporal average pooling with frame-wise attention pooling; the passage does not provide the completed TORC result.
  • Effects of input samples in CPR: Mixup produces lower verb@unseen-comp than the baseline, 52.6% vs. 53.0%, because concurrent verb-object blending fails to preserve temporal differences.The ablation argues that conventional image-based augmentations are insufficient for newly created unseen-composition labels.
  • Additional ablation results: Table 11 comprehensively ablates RCORE design choices on Sth-com using C2C with CLIP-B/16 as the baseline and reports seen, unseen, and harmonic-mean performance.The ablations cover temporal aggregation, CPR input samples, and RCORE hyperparameters.
  • Effects of hyperparameters of RCORE: Setting pCPR slightly below 1.0, such as 0.95 or 0.9, improves unseen accuracy with minimal seen-performance loss, whereas pCPR = 0.8 degrades the gains.The authors tune pCPR per dataset and backbone using the best validation H.M. of seen and unseen accuracies.

E.4 Ablation studies on EK100-com

On EK100-com, combining CPR and TORC produces the strongest ablation performance, while Lcos is the best individual TORC loss and LCPR further improves harmonic mean by 0.3 points.

  • Effects of CPR and TORC: Combining CPR and TORC outperforms the baseline by 3.2 H.M. points on EK100-com.The baseline uses C2C with CLIP-B/16 [34], and Table 12 reports seen, unseen, and harmonic-mean performance.
  • Effects of each TORC loss term: Using only Lcos in TORC yields the highest H.M. on EK100-com.
  • Effects of LCPR in CPR: Suppressing co-occurrence priors with LCPR improves H.M. by 0.3 points, consistent with results on Sth-com.

E.5 Results with validation-set-tuned bias calibration · E.6 Results with large video-pretrained VLM backbone · E.7 Temporal/Static split of Sth-com

RCORE remains competitive under validation-set-tuned open-world bias calibration, scales to a substantially larger video-pretrained backbone, and improves verb recognition while relying more on temporal dynamics than static cues across Temporal/Static Sth-com splits.

  • E.5 Results with validation-set-tuned bias calibration: The calibration adds a scalar bias exclusively to unseen-composition logits, compensating for their naturally lower training-unoptimized logits before test-time prediction.Unlike prior calibration, the bias is tuned on validation-set open-world logits and then applied to the test set, retaining applicability to real-world open-world scenarios.
  • E.6 Results with large video-pretrained VLM backbone: RCORE consistently improves C2C on Sth-com and EK100-com with InternVideo2-1B, raising unseen composition accuracy, unseen ∆CG, and H.M.The gains are +3.4p. and +2.1p. for unseen composition accuracy, +0.8p. and +1.1p. for unseen ∆CG, and +1.5p. and +0.4p. for H.M., respectively.
  • E.5 Results with validation-set-tuned bias calibration: With validation-set-tuned open-world calibration, RCORE’s calibrated performance matches its original unbiased performance on Sth-com.This suggests CPR’s training-label-space expansion provides meaningful supervision for unseen compositions instead of relying only on seen-composition co-occurrences.
  • E.6 Results with large video-pretrained VLM backbone: The InternVideo2-1B experiment uses an approximately 11× larger 1B-parameter vision encoder than the 87M-parameter encoders used in the main paper.RCORE’s improvements over C2C remain consistent on both Sth-com and EK100-com.
  • E.7 Temporal/Static split of Sth-com: The extended temporal-modeling evaluation partitions Sth-com into Temporal and Static splits and tests both original and temporally shuffled inputs.The evaluation follows prior work and includes both the reconstructed splits and those proposed by Sevilla et al..
  • E.7 Temporal/Static split of Sth-com: Across both reconstructed and Sevilla et al. Temporal/Static splits, RCORE performs worse on temporally shuffled inputs but better than C2C on original inputs.The larger original-versus-shuffled difference indicates stronger reliance on temporal dynamics rather than static cues, yielding robust verb improvements across splits.
Loading 2601.16211v3…