Source-linked AI summary
CurvSpec: Adaptive Multi-Curvature Learning for Partial Relevant Video Retrieval
Zhen Liu, Letian Li, Jinpeng Wang, Shuzhao Xie, Yuzhi Huang, Jingyan Jiang, Zhi Wang
TL;DR
PRVR seeks videos containing a query-matching moment, which may be brief within a much longer untrimmed video; existing methods face signal dilution and fixed-geometry representations for structurally varied videos. CurvSpec combines content-adaptive Euclidean and hyperbolic representations with semantic-centroid matching. It reports state-of-the-art retrieval performance across three benchmarks, with efficiency comparable to single-geometry baselines.
Problem
PRVR must retrieve untrimmed videos containing a query-matching moment that may last only seconds within a several-minute video, while fixed geometry does not accommodate varied structural depths.
Source: §1 · p. 1Method
CurvSpec combines parallel Euclidean and independently curved hyperbolic attention layers with content-aware fusion and semantic-centroid matching.
Source: §1 · p. 2Results
Takeaways & Limitations
The paper concludes that partial-relevance retrieval depends on retrieval-space geometry as well as finding the right temporal evidence.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Partially Relevant Video Retrieval (PRVR) seeks to retrieve untrim-med videos containing a moment that matches a text query, without temporal annotations. The relevant moment may last only seconds within a video spanning several minutes, creating an extremely low signal-to-noise ratio that makes PRVR more challenging than standard full-video retrieval. This task presents two intertwined challenges: (1) signal dilution, where coarse global representations blur the brief relevant signal into the dominant irrelevant surroundings;(2) curvature rigidity, where embedding all videos in the same fixed-geometry space distorts representations for videos that range from flat atomic events to deep compositional hierarchies. Existing PRVR methods have improved moment selection and cross-modal matching, but they still typically encode all videos in a single fixed-curvature retrieval space, limiting their ability to model diverse video structures. To address both challenges, we propose CurvSpec, a framework that learns content-adaptive curvature for video retrieval representations rather than imposing a fixed geometric prior. CurvSpec processes features through parallel Euclidean and hyperbolic attention layers, with independently learned curvatures assigned to the hyperbolic layers, and a content-aware fusion mechanism routes each input to its most suitable geometric regime. To further suppress signal dilution, CurvSpec represents each video with semantic centroids whose number is determined by the video's content complexity, projects them onto the learned manifold, and matches each query against its nearest centroid by geodesic distance. Experiments on ActivityNet Captions, TVR, and Charades-STA demonstrate state-of-the-art retrieval performance.
1 Introduction
PRVR must find brief relevant moments in untrimmed videos, while existing representations face both signal dilution and geometric rigidity. CurvSpec addresses these challenges with content-adaptive geometry and semantic-centroid matching, and reports state-of-the-art performance across three benchmarks.
1 Introduction
- PRVR retrieves untrimmed videos containing a query-matching moment that may last only seconds within a video spanning several minutes.This creates an extremely low signal-to-noise ratio compared with standard full-video retrieval.Source: §1 · p. 1
- Global representations dilute brief relevant signals, while forcing all videos into one fixed geometry fails to accommodate their differing structural depths.Flat relationships fit Euclidean space, whereas deep hierarchies are more faithfully captured by hyperbolic manifolds.Source: §1 · pp. 1–2
- CurvSpec uses parallel Euclidean and hyperbolic attention layers with independently learned hyperbolic curvatures, then routes each video through content-aware fusion.The approach adapts geometry to video structure rather than requiring manual geometric tuning.Source: §1 · p. 2
- Semantic centroids represent coherent thematic regions, which are refined through cross-attention and matched to queries on the learned manifold.Nearest-centroid matching aims to isolate relevant moments from irrelevant content.Source: §1 · p. 2
- CurvSpec achieves state-of-the-art performance on ActivityNet Captions, TVR, and Charades-STA, with gains concentrated on structurally complex videos.Its computational cost is comparable to single-geometry baselines.Source: §1 · p. 2
2 Related Work
PRVR research has advanced matching to relevant temporal regions, while many methods retain global or fixed-geometry representations. Hyperbolic video-retrieval work models hierarchies, but CurvSpec instead adapts across geometric regimes.
2.1 Partially Relevant Video Retrieval
- PRVR methods improve partial matching through multi-scale segments, Gaussian-weighted temporal context [37] [38], and approaches including distillation, event alignment, uncertainty, and event modeling [6] [44] [3] [33] [46].Many methods still rely on global or fixed-geometry representations, which may dilute short moments and fail to accommodate heterogeneous video structures.Source: §2.1 · p. 3
2.2 Hyperbolic Representation Learning
- Hyperbolic geometry supports hierarchical video structures, but HLFormer uses one fixed curvature; CurvSpec learns a curvature spectrum and fuses geometric regimes by content.Hyperbolic spaces' exponential volume growth preserves branching relations [12] [16] [26] [32].Source: §2.2 · p. 3
3 Method
CurvSpec combines mixed-geometry attention with content-adaptive semantic centroids for partial-relevance matching, and trains these representations with cross-modal matching, centroid alignment, and geometric regularization.
3.1 Preliminaries
- PRVR retrieves videos containing a moment that matches a query without temporal boundary annotations, and CurvSpec uses the Lorentz model alongside Euclidean geometry with learnable curvature.Curvature governs hyperbolic volume growth; the framework learns distinct curvature values to reflect varied geometric demands.Source: §3.1 · p. 3
3.2 Overall Architecture
- Text and video are encoded separately, then retrieval combines max-pooled cosine similarities over frame- and clip-level features to isolate the strongest match at each granularity.The query is encoded with frozen RoBERTa; video features at both temporal granularities are processed by the CurvSpec block.Source: §3.2 · p. 3
3.3 CurvSpec Block
- Parallel Euclidean and Lorentz attention layers capture multiple geometric scales, with independently learned Lorentz curvatures and content-aware weights routing each token toward a suitable branch.Euclidean layers use global attention first and then Gaussian masks with widths σ_n = 2^n; Lorentz layers begin with identical curvatures and diversify during training.Source: §3.3 · p. 4
3.4 Semantic Centroid Decomposition
- The number of semantic centroids is adapted using the effective rank of clip features, selecting the smallest K in {2, 4, 6} whose singular values explain at least a threshold θ of variance.The criterion is computed offline from pre-extracted features without inference overhead.Source: §3.4 · p. 5
- K-means prototypes are refined by cross-attending to frame and clip features, projected onto the hyperboloid, and matched to a query by minimum geodesic distance without temporal boundary annotations.A learned sigmoid gate modulates each granularity’s contribution during centroid refinement.Source: §3.4 · p. 5
3.5 Training Objective
- The training objective combines frame- and clip-level matching losses, centroid retrieval and alignment losses, and regularizers for query diversity and bounded curvature gaps.The curvature regularizer constrains pairwise curvature gaps to a target band to prevent geometric collapse and uncontrolled divergence.Source: §3.5 · p. 5
4 Experiments
CurvSpec ranks first in SumR across three benchmarks, and the experiments examine its efficiency, component contributions, and adaptation to video complexity.
4.1 Experimental Setup
- Evaluation uses standard splits across ActivityNet Captions, TVR, and Charades-STA, with R@K and SumR; a shared threshold adapts centroid counts across datasets.The centroid-count threshold is selected on the Charades-STA validation set and applied across corpora.Source: §4.1 · pp. 5–6
4.2 Comparison with Prior Works
- CurvSpec ranks first in SumR on all three benchmarks; on TVR it achieves the best retrieval accuracy with comparable FLOPs and online query latency.The comparison reports best performance on the majority of metrics across the benchmarks.Source: §4.2 · pp. 6–7, Table 1 · p. 6, Table 2 · p. 6
4.3 Ablation Study
- The learnable curvature spectrum consistently beats all-Euclidean and single-fixed-curvature variants, supporting adaptive geometry over either fixed alternative.The ablation study spans ActivityNet Captions, TVR, and Charades-STA.Source: §4.3 · p. 7, Table 3 · p. 7
- Einstein-midpoint aggregation improves results across all datasets over tangent-space weighted summation, whose geometric distortions accumulate in high-curvature regions.The alternative maps values to the tangent space, sums them, then projects them back.Source: §4.3 · p. 7
- Removing semantic centroids and using globally pooled video representations causes consistent, notable performance drops across all three datasets.The study attributes the gains to more precise query-to-segment matching.Source: §4.3 · p. 7, Table 3 · p. 7
- Clip features are more critical overall than frame features; Lreg prevents curvature collapse, while Ldiv encourages discriminative query embeddings.Removing clip features causes larger degradation than removing frame features; both auxiliary losses contribute to performance.Source: §4.3 · pp. 7–8
4.4 Analysis
- Identically initialized Lorentz layers learn distinct stable curvatures; fusion assigns more weight to hyperbolic layers and higher effective curvature as complexity rises.The reported curvature diversification emerges from the retrieval objective without explicit geometric supervision.Source: Figure 5 · p. 7, Figure 4 · p. 7, §4.4 · p. 8
- On TVR, SumR gains are over 5× larger for complex than simple videos; t-SNE shows centroid queries compact around matched regions versus scattered global-pooling embeddings.The complexity analysis stratifies TVR videos into Simple, Medium, and Complex groups.Source: §4.4 · p. 8, Table 4 · p. 8, Figure 6 · p. 8
5 Conclusion
CurvSpec combines a learnable curvature spectrum with semantic centroid decomposition for partially relevant video retrieval. Experiments on three benchmarks show state-of-the-art performance with efficiency comparable to single-geometry baselines.
5 Conclusion
- CurvSpec combines a learnable curvature spectrum with semantic centroid decomposition for partially relevant video retrieval.The curvature spectrum uses multiple Euclidean and hyperbolic attention layers with end-to-end optimized curvatures, while centroids represent coherent video regions for fine-grained matching.Source: §5 · p. 8
- Experiments on three benchmarks show state-of-the-art performance with efficiency comparable to single-geometry baselines.The benchmarks are ActivityNet Captions, TVR, and Charades-STA.Source: §5 · p. 8
- The conclusion frames partial relevance as depending both on finding temporal evidence and on matching retrieval-space geometry to video semantic structure.It presents learning curvature from content as a way to align representation geometry with heterogeneous real-world video structures.Source: §5 · p. 8