PaperlayerView exact revision on arXiv

Source-linked AI summary

Concentrate After Imagination: Text-Conditioned Evidence Grounding for Partially Relevant Video Retrieval

Shuaiqi Cheng, Siyu You, Yanbi Wu, Yuxi Chen, Jiahao Zhang, Xuming Hu

arXiv:2609.08999v1cs.CV

TL;DR

PRVR ranks untrimmed videos when a query may match only a brief moment, but max-style scoring can trust an unsupported local peak. TRACE uses query-conditioned global-register evidence to verify frame-level support before localized selection. Across three benchmarks, it achieves the highest SumR and improves DreamPRVR by +1.2, +1.1, and +1.5.

  • Problem

    PRVR needs to rank videos from brief matching moments, yet local concentration can promote a negative video based on a coincidentally similar fragment without query-relevant global verification.

    Source: Introduction · p. 1

  • Method

    TRACE activates query-relevant global registers, routes their support to frame evidence, and marginalizes alternative paths before localized temporal selection.

    Source: Introduction · p. 1, Overview · p. 3

  • Results

    TRACE achieves the highest SumR on ActivityNet Captions, Charades-STA, and TVR, improving DreamPRVR by +1.2, +1.1, and +1.5, respectively.

    Source: Introduction · p. 2, Conclusion · p. 7

  • Takeaways & Limitations

    Ablation, routing-corruption, transfer, and hard-negative analyses support query-conditioned evidence verification rather than a generic score offset as the explanation for the gains.

    Source: Introduction · p. 2

    Main-text results use diffusion-generated registers, although lightweight anchors also yield positive ΔSumR across four backbones.

    Source: §A.4 · p. 12

Abstract

from arXiv · show

Partially Relevant Video Retrieval (PRVR) retrieves untrimmed videos when queries describe only short moments. Although recent methods improve local representations, uncertainty modeling, and global context, final ranking often still trusts the strongest local response; a coincidentally similar fragment can therefore produce an unsupported peak. We identify this failure as the query-agnostic concentration bottleneck and propose TRACE, a score-level evidence verification operator for PRVR. Given a query and global video registers, TRACE activates query-relevant registers, routes their support to frame-level evidence, and smoothly marginalizes alternative query-to-register-to-frame paths before localized temporal selection. Unlike representation-level feature fusion, TRACE uses this evidence only as a query-conditioned residual calibration of the original local score. On ActivityNet Captions, Charades-STA, and TVR, TRACE achieves the best SumR on all three benchmarks and improves the DreamPRVR backbone by 1.2, 1.1, and 1.5 points, respectively. Ablation, routing-corruption, hard-negative, and cross-backbone transfer analyses support the interpretation that the gains arise from query-conditioned evidence verification rather than a generic score offset.

Introduction

PRVR must rank untrimmed videos even when queries match only brief moments, but local concentration can trust a coincidentally similar fragment without global support. TRACE verifies local responses using query-conditioned global evidence while retaining localized selection, and improves SumR across three benchmarks.

Introduction

  • PRVR scoring must identify a query-matching local response and verify that query-relevant global evidence corroborates it.Queries may match only a brief segment, so local evidence matters even though a coincidentally similar fragment can promote a negative video.Source: Introduction · p. 1
  • TRACE activates query-relevant registers, grounds their support in temporal evidence, and marginalizes alternative query-to-register-to-frame paths before localized maximization.Its smooth semantic-path aggregation retains a hard temporal choice.Source: Introduction · p. 1
  • Unlike representation-level fusion, TRACE adds query-conditioned verification as a residual while keeping the original local matching score primary.The figure's score curves are schematic and illustrate a changed peak choice, not additional quantitative results.Source: Introduction · p. 1
  • TRACE improves DreamPRVR SumR by +1.2, +1.1, and +1.5 on ActivityNet Captions, Charades-STA, and TVR, respectively.Routing-corruption, transfer, ablation, and hard-negative diagnostics support the verification-based explanation rather than a generic score offset.Source: Introduction · p. 2

Related Work

Prior PRVR work improves local temporal evidence, semantic and uncertainty modeling, or global representations, while evidence aggregation specifies how local responses become video-level scores. The related-work discussion distinguishes these directions from TRACE's explicit verification of whether a local peak has compatible global support.

Local Temporal Evidence Modeling in PRVR

Semantic Structure and Uncertainty in PRVR

Global Context and Register-Based PRVR

  • Global-context methods and DreamPRVR enrich representations before ranking, whereas evidence-aggregation methods pool, compress, or assign local responses without explicitly checking whether the selected peak has compatible global support.TRACE instead marginalizes q→R→F paths into per-frame evidence before temporal max selection.Source: Related Work · pp. 2–3

Preliminaries

The formulation treats PRVR as ranking videos that contain a local moment relevant to a query, using frame, clip, and optionally global-register representations. TRACE retains the register-based backbone and adds the registers as query-selected evidence anchors at the scoring boundary.

Problem Formulation

  • PRVR ranks an untrimmed video according to whether it contains a local temporal moment relevant to the query, using frame and clip responses in the original max-style score.The frame–clip aggregation weights remain unchanged by TRACE.Source: Problem Formulation · p. 3

Register-Based PRVR

  • DreamPRVR generates global registers and fuses them with frame and clip embeddings, while TRACE additionally uses them as query-selected anchors to verify the local peak at scoring time.This complements representation-level context with score-level verification.Source: Register-Based PRVR · p. 3

Method

TRACE adds query-conditioned global-to-local evidence to the original retrieval branch, routing register support to frame evidence before localized temporal selection.

Text-Conditioned Latent Evidence Paths

  • The latent-path view combines query-conditioned anchor activation, anchor-to-frame grounding, and direct query–frame evidence before concentration.The scoring view is energy-based, not a calibrated likelihood.Source: Text-Conditioned Latent Evidence Paths · p. 3

Cross-Granular Evidence Routing

Overview

  • TRACE adds query-conditioned evidence through residual score fusion while retaining the original local-matching branch over clip and frame embeddings.The overview depicts the video and text encoders producing clip embeddings, frame embeddings, registers, and query embeddings.Source: Overview · p. 3, Figure 2 · p. 4

Register-to-Frame Routing

  • Register–frame compatibility routes activated register support to frames, with log-sum-exp combining joint query-to-register-to-frame path energies.This differentiable soft maximum supports frames compatible with query-relevant registers while smoothing register paths.Source: Register-to-Frame Routing · p. 4

Hierarchical Evidence Concentration

Soft Register Evidence Route.

  • The auxiliary soft route aggregates frame values into one query-scored representation, complementing rather than replacing the sparse routed selector.The soft branch uses query-conditioned register-to-frame attention.Source: Soft Register Evidence Route · p. 4

Backbone Integration.

  • The TRACE evidence score combines routed concentration with weighted soft evidence, then enters residual fusion with the backbone score.η weights the auxiliary branch relative to routed evidence.Source: Backbone Integration · pp. 4–5

Model Learning

  • TRACE retains the backbone objective and trains its evidence score with InfoNCE and bidirectional triplet losses before progressively introducing fusion.The evidence branch is warmed up without fusion, its auxiliary loss is enabled, and fusion is ramped after stabilization.Source: Model Learning · p. 5

Experiments

TRACE improves retrieval across three PRVR benchmarks, with analyses examining its efficiency, evidence routing, query-conditioned register activation, and transfer to other backbones.

Experimental Setup / Baselines.

  • Evaluation covers ActivityNet Captions, Charades-STA, and TVR with Recall@K and SumR, comparing TRACE with conventional T2VR, VCMR, and PRVR baselines.The DreamPRVR configuration uses 384 hidden dimensions, four attention heads, and 4, 6, and 8 registers for the three datasets; pre-extracted features are used.Source: Experimental Setup · p. 5, Baselines · p. 5, Implementation Details · p. 5

Retrieval Performance.

  • TRACE achieves the highest SumR on all three datasets, improving DreamPRVR by +1.2, +1.1, and +1.5 on ActivityNet Captions, Charades-STA, and TVR.It also improves R@1 and R@100 on all three datasets; TVR R@1 reaches 17.8%.Source: Retrieval Performance · p. 5, Table 1 · p. 6

Qualitative Visualization.

  • TRACE raises the Charades-STA ground-truth–hard-negative margin from −0.0009 to +0.0021, restoring the ground truth from rank 2 to rank 1.The hard negative’s direct peak is 0.88 but its register support is 0.74.Source: Qualitative Visualization · pp. 5–6, Figure 3 · p. 6

Model Efficiency.

  • TRACE increases DreamPRVR’s parameter count by approximately 1.6%, feature extraction time by 2.9%, retrieval time by 0.6%, and per-epoch training time by 5.3%.The comparison reports feature extraction over 1,334 evaluation videos and retrieval time including query encoding, similarity computation, and ranking.Source: Table 2 · p. 6, Model Efficiency · p. 6

Ablation Study

  • TRACE’s routed max improves SumR over SqF by +0.9, +0.7, and +1.0, while the final combination reaches 157.3/81.1/194.6.Ablations also find global matching alone insufficient, query-independent register weights weaker than query-conditioned activation, and temporal averaging harmful to short moments.Source: Ablation Study · p. 6, Table 3

Register Activation Visualization

  • Figure 4 shows different queries emphasizing different register indices, while aggregate usage across 10,895 TVR validation pairs spans all indices.The visualization includes pair weights, dominant registers by query group, and aggregate dominant-register usage.Source: Figure 4 · p. 7, Register Activation Visualization · p. 7

Cross-Backbone Transferability

  • All four evaluated non-register backbones gain ∆SumR on all three datasets, with average gains ranging from +0.17 on MS-SL to +0.96 on HLFormer.Anchor construction and fusion weight were selected on validation data and frozen for evaluation.Source: Table 4 · p. 7, Cross-Backbone Transferability · p. 7

Conclusion

The conclusion presents TRACE as score-level evidence verification for PRVR, combining query-conditioned register activation and frame grounding before localized selection. The supplement covers additional diagnostics and method details.

Conclusion

  • TRACE improves DreamPRVR SumR by +1.2, +1.1, and +1.5 across three PRVR benchmarks, with analyses supporting query-conditioned evidence grounding.The conclusion frames global context as participating in final ranking verification, in addition to enriching video representations.Source: Conclusion · p. 7

Appendix

  • The supplement reports additional experiments and diagnostics alongside implementation, training-schedule, and theoretical details of the concentration operator.Its experiments include sensitivity, transfer, hard-negative, bootstrap, rank-correction, and routing-corruption analyses.Source: Appendix · p. 9

A.1 Hyper-Parameter Sensitivity

TRACE uses one score-level configuration across datasets and remains stable across reasonable hyperparameter ranges. Nonzero evidence-fusion weights outperform disabling fusion.

A.1 Hyper-Parameter Sensitivity

  • λ = 0.03, γ = 0.3, and η = 0.2 are used across datasets; register counts are inherited from DreamPRVR.The inherited counts are 4 for ActivityNet Captions, 6 for Charades-STA, and 8 for TVR.Source: §A.1 · p. 9
  • Figure 5 reports stable validation SumR around the unified setting and consistent gains from nonzero fusion weights over λ = 0.Moderate routing and soft-evidence weights also perform competitively, supporting a single cross-dataset configuration rather than dataset-specific tuning.Source: §A.1 · p. 9

A.2 Transfer to Earlier Non-Register-Based PRVR Backbones

The transfer diagnostic applies query-anchor-frame verification to frozen, earlier PRVR backbones using lightweight anchors derived from their encoded tokens. It yields positive gains across evaluated backbones, though smaller than TRACE with explicit diffusion registers.

A.2 Transfer to Earlier Non-Register-Based PRVR Backbones

  • The diagnostic freezes official checkpoints and selects anchor constructions and fusion weights using validation SumR before evaluation.It tests anchor-agnostic scoring rather than a full backbone-level reimplementation.Source: §A.2 · p. 9
  • Anchors follow each backbone’s representation structure: frame tokens for MS-SL and GMMFormerV2, clip tokens for GMMFormer, and frame tokens for HLFormer.HLFormer’s encoded frame tokens provide the anchors for its hierarchical partial-relevance model.Source: §A.2 · p. 9
  • Positive retrieval gains across earlier PRVR backbones support score-time verification with query-selected anchors, while the gains are smaller than with explicit diffusion registers.The authors attribute the smaller gains to simple pooling rather than learned global register generation; some released methods lacked comparable frozen-backbone evaluations with exposed local tokens.Source: §A.2 · p. 9

A.3 Additional Diagnostics

Additional diagnostics examine hyperparameter stability, hard-negative ranking corrections, and whether gains depend on semantically grounded routing. TRACE’s gains grow on difficult unsupported hard-negative subsets, while routing corruption weakens them.

A.3 Additional Diagnostics

  • Figure 6 summarizes strict top-1 correction cases in which TRACE gives the ground-truth video a larger calibration boost than the hard negative.The decomposition compares changes to ground-truth and hard-negative scores and their relative advantage.Source: §A.3 · p. 9
  • TRACE’s SumR gains increase on baseline-error queries and further on the top 25% unsupported hard-negative subset across all three datasets.The subsets rank baseline-error queries by the unsupported gap between a hard negative’s strongest local evidence and register support.Source: §A.3 · p. 9
  • Full-query SumR improvements are statistically reliable on all three datasets, and the top-25% unsupported hard-negative subsets show larger positive effects.The analysis uses paired bootstrap confidence intervals and paired sign-flip tests.Source: §A.3 · pp. 9–10
  • Figure 5 reports robustness within a reasonable range of λ, γ, and η, with a unified score-level configuration working consistently across datasets.Source: Figure 5 · p. 10
  • On TVR, removing query-specific activation or corrupting register-to-frame grounding weakens gains; grounding corruption reduces SumR to 192.6 and degrades more ranks than it improves.The corruptions retain an added score branch while disrupting semantic correspondence in the route.Source: §A.3 · p. 10

A.4 Scope and Limitations

TRACE calibrates ranking scores rather than replacing the backbone, so its direct scope is PRVR systems with local temporal tokens and global evidence anchors. Its behavior depends on anchor relevance, and its gains coexist with recall-cutoff trade-offs and bounded added computation.

A.4 Scope and Limitations

  • TRACE is most directly applicable to PRVR systems that expose both local temporal tokens and global evidence anchors, because it verifies scores rather than replacing the backbone.The score-level design diagnoses whether a local peak has global support without retraining a heavy generator or altering backbone architecture.Source: §A.4 · pp. 10–11
  • When a query is distant from every active register, activation approaches uniformity and frame support becomes query-agnostic.The discussion identifies a richer register vocabulary, such as mixed global, local, and event-level anchors, as a possible remedy.Source: §A.4 · p. 11
  • Without explicit learned registers, simple pooled anchors yield roughly +0.13 to +1.43 SumR, with HLFormer TVR’s +1.43 approaching but not matching diffusion-register TRACE.The lightweight-anchor diagnostic reports positive ΔSumR across four backbones.Source: §A.4 · pp. 11–12
  • Charades-STA R@10 changes from 14.5 to 14.3 despite TRACE’s best SumR across all three datasets.TRACE improves R@1 and R@100, but does not dominate every recall cutoff.Source: §A.4 · p. 11
  • TRACE’s added computation is an Nr × F register-frame aggregation and one temporal max-reduction, with memory proportional to Nr × F per video.Because Nr is small, verification cost remains bounded by the base forward pass as the video encoder scales.Source: §A.4 · p. 12
  • The discussion reports a TVR gain of up to +1.43 ΔSumR on HLFormer and suggests richer query-relevant anchor generation contributes to TRACE’s headline gains.It also states that the verification rule itself is anchor-agnostic.Source: §A.4 · p. 12

B.1 Implementation and Computation Details

Implementation combines benchmark-specific backbone inputs and register configurations with staged TRACE training and routing-based diagnostics.

B.1 Implementation and Computation Details

  • ActivityNet Captions and Charades-STA use I3D video features and 1,024-dimensional RoBERTa queries, while TVR uses concatenated ResNet152/I3D features and 768-dimensional RoBERTa queries.The register counts are 4, 6, and 8 for ActivityNet Captions, Charades-STA, and TVR, respectively.Source: §B.1 · p. 12
  • Diagnostics stratify baseline-error queries by unsupported-gap quartiles, summarize rank changes, and test query-register and register-to-frame routing corruption on TVR.Larger unsupported gaps yield larger rank gains after TRACE, and Q4 aligns with the Top-25 subset in Table 6.Source: Figure 7 · p. 13, Table 8 · p. 13, Table 9 · p. 14
  • TRACE computes local query-frame scores, routes query-selected register support to frames, combines local and routed evidence, then adds soft evidence to the base score.The algorithm selects the strongest routed evidence before combining it with register-guided soft evidence.Source: §B.1 · p. 13
  • The score-level branch uses evidence fusion weight λ=0.03, routing strength γ=0.3, soft evidence weight η=0.2, and register activation temperature τr=0.07.The auxiliary objective weights are λnce=0.02 and λtri=0.05, with triplet margins 0.2 for ActivityNet Captions and Charades-STA and 0.1 for TVR.Source: §B.1 · p. 13
  • Training warms up the backbone for 100 epochs, freezes it for the first five TRACE epochs, jointly optimizes both branches from epoch five, and ramps score integration from epoch 30 over 20 epochs.The ranking score remains Sbase during the initial five TRACE-stage epochs.Source: §B.1 · p. 13
  • Each experiment uses three independent seeds, and final checkpoints average the last three TRACE-stage epochs to reduce single-seed variance without changing the mean.Training uses AdamW, gradient clipping, and bfloat16 mixed precision on eight NVIDIA A800 GPUs.Source: §B.1 · pp. 13–14

B.2 Theoretical Properties of Hierarchical Evidence Concentration

The analysis frames ordinary max-style PRVR scoring as vulnerable to unsupported local peaks and characterizes TRACE as a smooth, query-conditioned evidence selector. It also gives a condition under which routed support can reverse a hard-negative ranking.

B.2 Theoretical Properties of Hierarchical Evidence Concentration

  • The analysis treats cosine similarities as unnormalized energy scores, assumes bounded similarities and fixed positive γ and τr, and requires no score-distribution assumption.The selected frame indices are well-defined because the frame set is finite.Source: §B.2 · p. 14
  • A negative video can dominate standard max-style PRVR scoring when one coincidentally similar local fragment wins without verification by query-relevant global evidence.The local maximum remains suitable for partial relevance because only one moment may match the query.Source: §B.2 · p. 14
  • Register-path log-sum-exp smoothly approximates the strongest query-to-register-to-frame path, with per-frame gap τr log Nr and routed-score gap γτr log Nr.The outer temporal maximum selects a frame after considering register support.Source: §B.2 · p. 14
  • Setting γ=0 recovers standard max scoring, query-independent activation gives global support, removing frame compatibility gives a global register bias, and temporal averaging dilutes evidence.These reductions isolate query-conditioned path selection, register-to-frame grounding, and localized temporal concentration.Source: §B.2 · pp. 14–15
  • A routed selector can reverse a hard negative's raw-score lead when the positive video's routed support exceeds the negative's; it should not force correction when that support is comparable or weaker.The correction is targeted evidence verification, not a uniform score shift.Source: §B.2 · p. 15
Loading 2609.08999v1…