PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Class-Time Learning for Video Classification with Multi-Instance Partial-Label Learning

Lingyu Shen, Wei Tang, Fakhri Karray, Min-Ling Zhang

arXiv:2610.06234v1cs.CVcs.AI

TL;DR

Video-level labels leave both the true class and its supporting temporal evidence unresolved when supervision supplies a candidate set. PIVOTMIPL jointly assigns class and time through candidate-restricted inference and a structured teacher; it achieves the highest mean ACC in 10 of 12 dataset–feature configurations.

  • Problem

    In VCMIPL, candidate labels leave the true class ambiguous, while bag-level labels do not specify which temporal evidence supports each candidate class.

    Source: §1 · p. 1

  • Method

    PIVOTMIPL uses occupancy-regularized spherical matching and candidate-restricted assignment to construct a structured teacher, then aligns the full-space predictor with a plan-level KL objective.

    Source: §1 · p. 2

  • Results

    PIVOTMIPL achieves the highest mean ACC in 10 of 12 dataset–feature configurations.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The work contributes three VCMIPL benchmarks built with model-generated candidate labels and evaluated across four feature representations.

    Source: §1 · p. 2

    The study is offline within a fixed label space and does not establish robustness to distribution shifts or evolving class vocabularies.

    Source: D Limitations · p. 20

Abstract

from arXiv · show

Multi-instance partial-label learning (MIPL) addresses inexact supervision in both the instance and label spaces, which can be applied to video classification. However, bag-level labels do not explicitly supervise the correspondence between candidate classes and temporal evidence. We propose {\ours}, which couples label disambiguation with temporal evidence allocation through a joint class--time assignment. Occupancy-regularized spherical matching associates contextualized video features while learning nonuniform temporal mass and discouraging excessive concentration. During training, candidate-restricted inference recomputes the assignment within the candidate label set. A dual-marginal KL projection then constructs a structured teacher that incorporates momentum-refined class beliefs while preserving the proposal's temporal occupancy. A single plan-level KL objective aligns the full-space predictor with this teacher. Our analysis characterizes when candidate re-solving differs from masking and shows that, under the stated construction, the joint objective decomposes into class-marginal and class-conditional temporal supervision. We construct VCMIPL benchmarks from Breakfast, DoTA, and FineAction using model-generated candidate labels and evaluate the method across four feature representations. Extensive experimental results demonstrate that PIVOTMIPL outperforms existing MIPL algorithms in both effectiveness and efficiency.

1 Introduction

Video classification with candidate labels requires resolving both the true class and the temporal evidence supporting it. PIVOTMIPL couples these decisions through class–time assignments and outperforms existing MIPL methods across the constructed benchmarks.

1 Introduction

  • Bag-level class probabilities alone do not specify how temporal evidence is allocated to each candidate class; distinct class–time assignments can yield identical class probabilities.The motivating gap is the missing correspondence between candidate classes and temporal evidence.Source: §1 · pp. 1–2
  • PIVOTMIPL uses occupancy-regularized spherical matching and candidate-restricted inference to build a structured teacher with refined class beliefs and preserved temporal occupancy.A plan-level KL objective aligns the full-space predictor with that teacher.Source: §1 · p. 2
  • PIVOTMIPL achieves the highest mean ACC in 10 of 12 dataset–feature configurations, averaging a 3.16% gain over the strongest baseline in each configuration.The benchmarks cover Breakfast, DoTA, and FineAction with four feature representations.Source: §1 · p. 2
  • The paper contributes a joint-assignment framework, analyses of its inference and supervision mechanisms, and three VCMIPL benchmarks with model-generated candidate labels.The benchmarks use four feature representations.Source: §1 · p. 2

2 Related work

Prior work addresses inexact supervision through MIPL methods and weakly supervised video learning, while this setting must link an ambiguous candidate class to temporal evidence.

2.1 Multi-Instance Partial-Label Learning

  • MIPL research spans probabilistic disambiguation, attention and momentum, candidate-label information, graph propagation, feature geometry, and calibration; this paper focuses on coupling candidate-restricted temporal occupancy with historical class beliefs.The cited approaches are described as complementary developments in MIPL.Source: §2.1 · p. 2

2.2 Weakly Supervised Video Learning

  • Weakly supervised video methods learn temporal evidence from video-level labels, but generally assume unambiguous labels; VCMIPL also identifies the true class within a candidate set and links it to supporting evidence.Source: §2.2 · p. 2

3 Methodology

PIVOTMIPL jointly assigns video instances to classes and time, linking candidate-label disambiguation with temporal evidence allocation. Its training constructs a structured teacher from candidate-restricted inference and aligns the full-space predictor to that teacher.

3.1 Problem Formulation and Overview

  • Each video has one target class within a nonempty candidate set, and multiple temporal positions may jointly support the class without every frame belonging to it.This is the paper’s collective bag-classification setting.Source: §3.1 · p. 3

3.2 Temporal Encoder

  • The temporal encoder contextualizes video instances with bidirectional sequence processing; frame-level features also use local differences, while clip-level features bypass that module.The two paths then use bidirectional Mamba-2 and a gated residual adapter.Source: §3.2 · pp. 3–4

3.3 Occupancy-Regularized Spherical Transport

  • Spherical matching assigns contextualized features to multiple class prototypes while learning temporal occupancy, whose regularization discourages concentration without enforcing uniform timing.The occupancy penalty is defined relative to the uniform temporal distribution.Source: §3.3 · pp. 4–5
  • Candidate-restricted inference re-solves the assignment and can change temporal allocation; unlike re-solving, globally masking need not optimize the restricted occupancy-regularized objective.The paper gives equivalence conditions, including the λ = 0 boundary.Source: §3.3 · p. 5

3.4 Candidate-Restricted Teacher Construction

  • A momentum-refined class belief and the candidate proposal’s temporal occupancy constrain a KL-projected teacher plan.The stored moving average is over class marginals, not full plans or network parameters.Source: §3.3 · p. 5, §3.4 · p. 6
  • The detached teacher supervises the full-space actor through plan-level KL, covering class marginals and class-conditional temporal allocation while penalizing non-candidate class mass.Under the same-cost construction, actor and teacher prototype-conditional distributions coincide on candidate support, so that KL component is zero.Source: §3.4 · p. 6

3.5 Joint Class–Time Learning and Prediction

  • Training processes one complete video bag per optimization step, applies one plan-level loss and gradient-clipped update, and introduces no temporal padding.At test time, the procedure computes the matching costs and full-space actor before predicting.Source: §3.5 · p. 6

Algorithm 1: PIVOTMIPL training and prediction

  • Algorithm 1 computes a full-space actor, re-solves on candidate labels, updates class belief, constructs the teacher, then backpropagates the actor KL and stores the refined belief.The test procedure returns a prediction from the full-space actor.Source: Algorithm 1: PIVOTMIPL training and… · p. 7

4 Experiments

Across three VCMIPL benchmarks and four feature representations, PIVOTMIPL generally outperforms the strongest MIPL baselines. Ablations indicate that temporal encoding, joint supervision, and the dual-marginal teacher contribute to the observed performance pattern.

4.1 Experimental Setup

  • The benchmarks use Breakfast, DoTA, and FineAction, with model-generated candidate labels and four feature representations; FineAction excludes multi-label videos and classes with fewer than 50 videos.The three retained subsets contain 1,989, 4,577, and 10,675 videos, respectively.Source: §4.1 · p. 7

4.2 Main Experimental Results

  • PIVOTMIPL has the highest mean ACC in 10 of 12 dataset–feature configurations, gaining 3.16% on average over the strongest baseline per configuration.It is significantly superior to the strongest competing method in eight configurations and is inferior to AGOPMIPL in one.Source: §4.2 · pp. 7–8, Table 1 · p. 8

4.3 Ablation Study

  • Every ablation reduces mean ACC across 12 configurations; temporal encoding has the largest average impact, while the joint objective and dual-marginal teacher deliver the most consistent gains.Removing the joint KL or occupancy-preserving teacher reduces performance across all configurations; removing the temporal encoder or using one prototype reduces ACC in 11.Source: §4.3 · p. 8, Table 2 · p. 9
  • PIVOTMIPL achieves the highest mean BACC in 10 of 12 configurations, and it outperforms AGOPMIPL in 10 of 12 when that baseline uses the same temporal encoder.The paper also reports favorable accuracy-efficiency trade-offs in most evaluated settings and limited variation across tested transport-hyperparameter ranges.Source: §4.3 · p. 8

5 Conclusion

PIVOTMIPL couples candidate-label disambiguation with temporal allocation; the appendix characterizes occupancy optimization and when candidate-restricted re-solving differs from masking.

5 Conclusion

  • PIVOTMIPL achieves the highest mean ACC and BACC in 10 of 12 configurations across Breakfast-MIPL, DoTA-MIPL, and FineAction-MIPL.The conclusion reports evaluations using four feature representations.Source: §5 · p. 9

A Structural Properties and Proofs

  • The appendix analyzes optimization problems rather than network-training convergence, and its properties do not establish true-label recovery, temporal localization, or generalization.Source: A Structural Properties and Proofs · p. 12

A.1 Factorization and Uniqueness

  • For finite costs, the occupancy objective has a unique interior minimizer, with only its normalization multiplier requiring a scalar root solve.Source: §A.1 · p. 12

A.2 Candidate-Restricted Re-solving and Global Masking

  • With λ = 0, candidate re-solving equals global masking; with λ > 0, they are equal if and only if the retained occupancy is constant across valid positions.This characterization assumes the stated uniform prior and exact objectives.Source: §A.2 · p. 13
  • Matching class and temporal marginals does not determine their class–time association: the example has zero class-marginal KL but positive joint KL.Source: §A.2 · p. 13

A.3 Teacher projection and the effective supervision space

The appendix analyzes the teacher’s supervision and gradients, the effect of occupancy regularization, and computational costs under stated conditions.

A.3 Teacher projection and the effective supervision space

  • With the teacher frozen, the three-dimensional and two-dimensional losses have equal first derivatives with respect to actor parameters at θ0.Source: §A.3 · p. 14
  • Under the stated same-point construction and correction conditions, joint supervision comprises class-marginal alignment and class-conditional temporal supervision, without an additional prototype-conditional KL term.The temporal distribution is parameterized by multi-prototype geometry.Source: §A.3 · p. 14

A.4 Teacher Projection and Class–Time Supervision

  • When µ = 0 and λ = 0, the teacher equals the candidate proposal; this boundary case is not an identity for the full method with positive momentum or occupancy regularization.Source: §A.4 · p. 14

A.5 Effective-Support Control by Occupancy Regularization

  • With geometry fixed, increasing occupancy regularization reduces effective-support concentration; this does not establish monotonic support during learning or monotonic accuracy in λ.Effective support measures mass concentration, not semantic diversity or true-action coverage.Source: §A.5 · p. 15

A.6 Differentiability and Computational Complexity

  • A Jacobian–vector product for occupancy costs O(n); dense spherical matching costs O(CKnd) operations and O(CKn) storage, excluding the temporal encoder.The complexity counts also depend on requested numerical precision.Source: §A.6 · p. 15

B.1 More Information on Benchmark Construction

The section describes the constructed VCMIPL benchmarks and the procedure used to generate their candidate label sets.

B.1 More Information on Benchmark Construction

  • The benchmarks use Breakfast, DoTA, and FineAction; FineAction retains single-label videos and excludes classes with fewer than 50 videos.The resulting subsets contain 1,989 Breakfast videos, 4,577 DoTA videos, and 10,675 FineAction videos.Source: §B.1 · pp. 15–16
  • Candidate labels are generated from out-of-fold predictions of a VideoMAEv2 recognizer with a TimeSformerHead, using five-fold cross-fitting.The recognizer is trained on four folds and predicts the held-out fold.Source: §B.1 · p. 16
  • Candidate sets combine the smallest sample-wise class set reaching 95% cumulative probability with class-wise top-50% selections, then add ground truth if absent.The inclusion guarantee applies to corrected sets; exact labels are used during benchmark construction, while downstream MIPL learning uses the candidate sets.Source: §B.1 · p. 16

B.2 Feature Representations and Dataset Statistics

The benchmarks span four feature representations with different temporal granularities, and evaluation uses a fixed five-fold protocol with out-of-fold candidate labels.

B.2 Feature Representations and Dataset Statistics

  • ResNet and DINOv3 provide frame-level features, while SlowFast and VideoMAEv2 provide clip-level features across twelve dataset–feature configurations.The configurations span different temporal granularities and feature dimensions.Source: §B.2 · p. 16
  • Table 4 summarizes extracted-instance counts and per-video maximum, minimum, and mean instance counts, while feature representations share each dataset’s video, class, and candidate-set statistics.Feature-vector dimension is also reported.Source: §B.2 · p. 16, Table 4 · p. 17

B.3 More Implementation Settings

  • Each fold trains on four predefined folds and tests on the fifth; candidate labels are generated out of fold, and Table 5 summarizes implementation settings.The same training protocol applies across feature representations unless otherwise specified; ablations use Fold 1 and are evaluated at epoch 40.Source: §B.3 · p. 16, Table 5 · p. 17

C.1 Balanced Accuracy Results

PIVOTMIPL leads on balanced accuracy in most configurations, generally offers a favorable accuracy–efficiency trade-off, and benefits from several complementary design components.

C.1 Balanced Accuracy Results

  • 10 of 12 configurations: PIVOTMIPL achieves the highest mean BACC, with an average gain of 3.26% over the strongest baseline.It ranks first in all four DoTA-MIPL configurations; the BACC improvement pattern closely follows ACC rather than being driven primarily by dominant classes.Source: §C.1 · p. 17, Table 6 · p. 18

C.2 Efficiency Analysis

  • PIVOTMIPL has a favorable accuracy–efficiency trade-off in most settings, while AGOPMIPL can reach a Pareto-superior operating point in VideoMAEv2 settings.Figure 2 compares classification accuracy with training time.Source: §C.2 · p. 18

C.3.1 Ablation Summaries

  • Replacing the plan-level objective with class-level KL or removing the teacher’s temporal-marginal constraint degrades performance in all 12 configurations.Removing the temporal encoder causes the largest average drop; occupancy regularization, adaptive temporal allocation, and multiple prototypes provide complementary gains.Source: §C.3.1 · p. 18

C.3.2 Analysis Of The Effectiveness Of The Temporal Encoder

With the same temporal encoder architecture, PIVOTMIPL retains an ACC advantage over AGOPMIPL in most configurations.

C.3.2 Analysis Of The Effectiveness Of The Temporal Encoder

  • 10 configurations: PIVOTMIPL retains higher ACC than encoder-augmented AGOPMIPL, with an average advantage of 2.83%.The comparison suggests temporal encoding contributes to performance, but the shared encoder alone does not account for PIVOTMIPL’s advantage across most configurations.Source: §C.3.2 · p. 18
  • Figure 2 compares mean classification accuracy with total training time across twelve dataset–feature settings, with higher accuracy and shorter training time preferred.Each panel represents one setting and reports results over five folds.Source: Figure 2 · p. 19

C.4 Parameter Sensitivity Analysis

PIVOTMIPL’s performance varies moderately across the evaluated entropic and occupancy coefficients, which regulate assignment sharpness and temporal-mass concentration.

C.4 Parameter Sensitivity Analysis

  • ε controls class–prototype–time assignment sharpness, while λ regulates temporal-marginal concentration and discourages mass from concentrating on a few positions.A very small ε favors sharper matching; increasing ε smooths assignments.Source: §C.4 · p. 19
  • The structured teacher and plan-level supervision remain effective across a range of assignment smoothness and temporal occupancy regularization strengths.The paper identifies this robustness as desirable when sequence length, temporal redundancy, and feature granularity vary across datasets.Source: §C.4 · p. 19
  • Across three representative configurations, performance varies only moderately over the evaluated ranges of ε and λ.The configurations span different dataset scales and feature representations, and the broad plateaus suggest no narrowly tuned balance is required.Source: §C.4 · p. 19, Figure 3 · p. 20

D Limitations

The study is limited to offline VCMIPL with a fixed label space; robustness to distribution shifts or evolving vocabularies and strictly online prediction remain unestablished.

D Limitations

  • The study evaluates offline VCMIPL within a fixed label space, without establishing robustness to distribution shifts or evolving class vocabularies.Transfer of class–time associations across such changes remains future work.Source: D Limitations · p. 20
  • Strictly online prediction would require causal contextualization and incremental assignment updates that account for incomplete temporal evidence.The current formulation uses bidirectional temporal context and normalizes assignments over the observed video.Source: D Limitations · p. 20
Loading 2610.06234v1…