PaperlayerView exact revision on arXiv

Source-linked AI summary

When Semantically Consistent Encoding Meets View-Label Heterogeneity Modeling: A Unified Framework for Incomplete Multi-View Multi-Label Learning

Chengliang Liu, Bo Li, Bob Zhang, Yanghao Zhou, Jie Wen, Wenwu Wang

arXiv:2609.07525v1cs.CV

TL;DR

Incomplete multi-view multi-label learning must balance robust shared semantics from missing views with label- and instance-specific use of view evidence. V2L combines perturbation-aware semantic encoding with active view-label relevance modeling in a unified hybrid framework, achieving leading performance on five benchmarks under incomplete and complete settings.

  • Problem

    Existing approaches emphasize either shared representations that may compress label-discriminative view-specific cues or decision fusion with fixed or globally learned weights that overlook instance- and label-specific view needs.

  • Method

    V2L constructs semantically consistent variational posteriors through source-perturbation invariant encoding and adaptively fuses view-specific predictions using instance-wise, label-aware relevance modeling.

  • Results

    V2L achieves leading performance on five benchmarks under both incomplete and complete settings.

  • Takeaways & Limitations

    V2L unifies shared semantic consistency with adaptive use of complementary view-specific evidence rather than treating representation fusion and decision fusion as isolated alternatives.

Abstract

from arXiv · show

Incomplete multi-view multi-label learning requires not only robust semantic aggregation from partially observed views, but also label-aware exploitation of view-specific evidence. Existing approaches usually emphasize either shared representation learning or decision-level fusion. The former improves robustness against missing views, yet tends to compress label-discriminative view-specific cues into a single latent representation. The latter preserves individual view predictions, but often relies on fixed or globally learned fusion weights, ignoring that different labels of different instances may require different views. To address these limitations, this paper presents V2L, a unified representation-decision framework for incomplete multi-view multi-label classification. On the representation side, V2L constructs semantically consistent variational posteriors from incomplete views through a perturbation-aware encoding mechanism, which provides a stable shared semantic basis. On the decision side, V2L introduces an active view-label relevance modeling strategy that estimates instance-wise and label-wise view contributions, allowing each label prediction to adaptively select useful view-specific evidence. From the perspective of model architecture, these two important strategies are integrated into a unified framework through a hybrid fusion architecture, simultaneously meeting the requirements of cross-view semantic consistency and representational complementarity. Extensive experiments under both incomplete and complete settings show that V2L achieves leading performance on five benchmarks. Code is available at: https://github.com/justsmart/V2L.

I. INTRODUCTION

Incomplete multi-view multi-label classification must handle missing views and labels while exploiting both shared cross-view semantics and label-specific view evidence. V2L unifies robust semantic encoding with adaptive view-label fusion to address these complementary needs.

  • Motivation: Incomplete multi-view multi-label classification must infer complete labels from partially observed views and incomplete supervision simultaneously.Missing views can result from acquisition failures, cost, privacy, or occlusion, while missing labels commonly arise from sparse annotation and human omission.
  • Motivation: Shared representation methods improve robustness to missing views but can compress label-discriminative, view-specific cues into one latent representation.Existing approaches commonly construct a shared latent representation from incomplete views before applying a multi-label classifier.
  • Motivation: View-label heterogeneity means useful evidence can vary across labels and instances, so globally fixed fusion weights are insufficient.Different labels may favor different modalities, and the most reliable view for one label can change with noise, occlusion, or missing observations.
  • Contributions: V2L uses source-perturbation invariant encoding to preserve task-relevant shared information and strengthen semantic consistency under missing views and labels.The encoding mechanism is designed to provide robust shared semantics across incomplete observations.
  • Contributions: V2L models sample-specific and label-aware view contributions, enabling adaptive predictive fusion beyond static or globally learnable weighting.The framework integrates mid-level semantic fusion with late-stage decision fusion in a unified representation-decision architecture.

B. Task-Relevant Common Semantics in Multi-View Data

Multi-view observations share task-related semantic information, motivating a common latent representation that preserves predictive content while limiting irrelevant view-specific variation. Because semantic consistency alone can neglect useful view-specific cues, the framework supplements it with view-label heterogeneity modeling.

  • Task-Related Common Semantics: Different views describing the same instance are assumed to share latent semantic information related to the labels.The conceptual variable s captures common semantic factors supported by each view and related to the label variable y.
  • Task-Related Common Semantics: A multi-view representation can serve as a common semantic representation when it preserves the shared semantic information across observed views.The paper uses this proposition to motivate learning a unified representation under incomplete observations.
  • Representation Challenges: Naive fusion can introduce irrelevant view-specific variation, obscuring shared task-relevant signals and reducing latent predictability for labels.Raw multi-view observations may contain substantial nuisance variation unrelated to the target.
  • Representation Challenges: The information-bottleneck formulation combines compression of encoded information with stronger correlation between the latent representation and the task.The desired representation is constrained by both sufficiency and compactness.
  • Representation Challenges: Semantic consistency assumptions are limited because enforcing shared semantics can neglect view-specific information, motivating complementary view-label heterogeneity modeling.The framework therefore combines multi-view information-bottleneck learning with explicit modeling of view-specific cues.

IV. METHODOLOGY

V2L formulates incomplete multi-view learning as shared semantic extraction under missing observations, then realizes it with a multi-view variational encoder and cross-view posterior modeling. The framework preserves view-valid information while enforcing semantic consistency across views.

  • IV. METHODOLOGY: V2L combines an information-theoretic objective with source-perturbation invariant encoding, posterior regularization, active view-label relevance modeling, and multi-label classification.These components form the unified framework shown in Fig. 2.
  • IV. METHODOLOGY: The representation z is designed to preserve shared task-related information while suppressing view-private redundancy unsupported by other views.The objective is motivated by minimal sufficiency and conditional mutual information principles.
  • IV. METHODOLOGY: Cross-view semantic consistency is imposed by maintaining corresponding elements of distribution clusters under source-view exchanges.Missing index matrices prevent unavailable views and labels from participating in training.
  • IV. METHODOLOGY: Instead of a single probabilistic encoder, V2L uses clusters of probabilistic encoders that map each source view into target-view-aware proposals on a shared latent space.Each proposal cluster provides multiple latent projections from one observed view.
  • IV. METHODOLOGY: Precision-weighted Gaussian aggregation consolidates each proposal cluster into a view-specific posterior, assigning larger weights to proposals with smaller posterior variance.The consolidated posterior serves as the expert associated with its source view.

1) Variational Realization of Cross-View Semantic Consistency:

V2L makes cross-view consistency tractable by combining reconstruction-based information preservation with a posterior-consistency surrogate. This balances retaining view-valid information against aligning target-wise semantics induced from different source views.

  • 1) Variational Realization of Cross-View Semantic Consistency:: The reconstruction objective acts as a variational surrogate for preserving mutual information between each observed view and the latent representation.It prevents the latent variable from being overly compressed.
  • 1) Variational Realization of Cross-View Semantic Consistency:: The consistency term reduces conditional mutual information by bringing posteriors conditioned on multiple views closer to posteriors conditioned on a single view.This provides a tractable alternative to directly evaluating intractable conditional posteriors.
  • 1) Variational Realization of Cross-View Semantic Consistency:: Comparing proposals from different source views under the same target-aware head encourages target-wise semantic posteriors to remain invariant to the source view.The compared proposals correspond to the same target-aware head.
  • 1) Variational Realization of Cross-View Semantic Consistency:: The consistency regularizer is a consistency surrogate rather than an exact estimator or variational bound of conditional mutual information.This scope qualification limits how the regularizer should be interpreted.
  • 1) Variational Realization of Cross-View Semantic Consistency:: The information bottleneck loss combines view-valid information preservation with latent-posterior consistency across source views.Together, these terms implement minimum sufficiency under incomplete multi-view observations.

2) Source-Perturbation Invariance Modeling:

V2L replaces exhaustive ordered view-pair alignment with random exchange over source-anchored posterior clusters. This perturbation-aware mechanism preserves target-relevant semantics while allowing moderate view-specific information through reconstruction.

  • 2) Source-Perturbation Invariance Modeling:: Exhaustively enumerating ordered view pairs may be computationally burdensome and overly rigid, motivating random exchange over source-anchored posterior clusters.The exchange strategy is introduced as a more scalable and robust consistency mechanism.
  • 2) Source-Perturbation Invariance Modeling:: Each available source view induces a proposal cluster that provides a structured semantic description projected onto all target latent subspaces.Corresponding cluster elements should remain distributionally consistent when source associations are perturbed.
  • 2) Source-Perturbation Invariance Modeling:: A random mapping exchanges source-view identities for target-aware proposals, producing a perturbed cluster for consistency regularization.The mapping is defined over available views and excludes the original source view.
  • 2) Source-Perturbation Invariance Modeling:: When views are semantically equivalent at the target-relevant level, exchanging their source roles does not alter the induced proposal cluster.This is the stated source-perturbation invariance principle.
  • 2) Source-Perturbation Invariance Modeling:: Exchanged sources assigned to different targets are required to be nonrepeated, promoting diversity across target-aware proposals.The rewritten consistency loss regularizes the discrepancy between original and permuted clusters.
  • 2) Source-Perturbation Invariance Modeling:: The perturbation-aware loss aligns latent semantic proposals without assuming raw-input identity, while reconstruction preserves moderate view-specific information.Its consistency requirement applies at the target-relevant latent level.

C. Dual-Level Posterior Regularization

V2L combines semantically consistent representation learning with instance- and label-specific decision fusion. Its hybrid architecture preserves shared semantic evidence while adaptively retaining complementary view-specific cues.

  • Posterior regularization: Dual-level posterior regularization complements perturbation invariance by improving within-view proposal coherence and cross-view instance discriminability.The posterior loss is defined as Lpost = σLintra + γLcross.
  • Posterior regularization: Cross-view discrimination treats different views of one instance as positives and different instances as negatives, encouraging semantic consistency without losing instance separation.The objective regularizes view-level posterior means using a temperature-scaled contrastive formulation.
  • View-label relevance modeling: V2L models view-label heterogeneity because view utility can vary across labels and instances, making joint representation fusion alone insufficient.The framework therefore combines mid-level representation fusion with late decision fusion.
  • View-label relevance modeling: The active perception network estimates relevance for each instance, label, and available view from stacked view-wise logits, excluding unavailable views during normalization.This differs from static late fusion and view-level dynamic fusion with equal instance weights within a view.
  • Hybrid fusion: Hybrid prediction combines mid-level semantic logits with late-stage decision logits, preserving stable shared evidence alongside label-specific contributions from individual views.The resulting architecture supports both cross-view semantic consistency and representational complementarity.
  • Relevance supervision: Prediction-error-derived relevance targets supervise the active perception network without entering the main variational framework, preserving training stability and inference independence.Views with smaller prediction errors receive larger target relevance weights.

E. Multi-Label Classification and Overall Losses

V2L defines classification on observed labels and hybrid predictions, while its unified objective combines predictive, relevance, and representation-consistency terms. Training jointly optimizes label prediction and semantically consistent representations.

  • Classification loss: The final multi-label classification loss uses only labels observed for each sample and the hybrid prediction.Single-view classifiers are also trained to retain sufficient predictive ability for the adaptive decision branch.
  • Unified objective: The overall objective combines information bottleneck, posterior regularization, view-label relevance, and classification losses.The representation terms and predictive terms are optimized together in one unified framework.
  • Training procedure: Each training iteration first learns consistent latent representations from incomplete views, then performs label-aware decision fusion using the learned relevance graph.All parameters are updated by minimizing the unified objective.

F. Complexity analysis.

The computational cost of V2L depends on the number of available views, target-aware encoder heads, latent dimension, and batch size. Cross-view discrimination has the highest stated worst-case batch complexity.

  • Complexity terms: Proposal generation requires O(b|V|mCenc) operations when each available source view uses m target-aware encoder heads.Here b is mini-batch size, |V| is the number of available views, and Cenc is the cost of one probabilistic encoder.
  • Complexity terms: Reconstruction and posterior alignment require O(b|V|Cdec) and O(b|V|md) operations, respectively.Cdec denotes the cost of one probabilistic decoder and d is the latent dimension.
  • Complexity terms: Batch-wise cross-view discrimination has worst-case complexity O(b^2|V|^2d).The quadratic dependence on batch size and number of views comes from comparing instances across view pairs.

G. Remarks: On Shared Semantics and View-Label Relevance

V2L treats shared semantics as a stable representation-level basis rather than an assumption that views contribute equally. Its view-specific branches and relevance graph preserve and adaptively use complementary evidence across five benchmark datasets.

  • Shared semantics: Semantic consensus is imposed at the representation level to preserve label-relevant information commonly supported across incomplete views.This does not require different views to contribute equally to all labels.
  • View-label relevance: View-specific posterior branches retain label-discriminative evidence outside the shared posterior, while the relevance graph weights each available view separately for every instance and label.View complementarity therefore means adaptive use of useful predictive differences, not arbitrary disagreement.
  • View-label relevance: Global view weights are overly restrictive because view importance can vary with both the target label and the input instance.Instance-wise label-wise relevance also avoids suppressing fine-grained view-specific cues useful for particular labels.
  • Benchmarks: The evaluation uses Corel5k, Pascal07, ESPGame, IAPRTC12, and MIRFLICKR, with six visual views per instance and benchmark settings matched to prior iM3C studies.The six views are GIST, HSV, DenseHue, DenseSIFT, RGB, and LAB.
  • Benchmarks: The five datasets differ substantially in scale and label structure, ranging from 20 categories in Pascal07 to 291 categories in IAPRTC12.Corel5k, ESPGame, IAPRTC12, and MIRFLICKR contain 4,999, 20,770, 19,627, and 25,000 images, respectively.

2) Construction of incomplete data:

Experiments evaluate V2L under incomplete and complete multi-view multi-label settings, varying missingness and examining view-label relevance patterns. The results support adaptive, instance-wise and label-aware fusion alongside shared semantic encoding.

  • Construction of incomplete data: The standard incomplete-data protocol randomly removes view instances while ensuring each sample retains one view and masks positive and negative labels as unknown.Unless otherwise specified, both missing-view and missing-label ratios are 50%.
  • Incomplete settings: V2L improves across multiple evaluation criteria because semantic encoding stabilizes latent representations while adaptive view-label relevance modeling refines label-wise decisions.The reported gains span holistic ranking criteria such as AP and AUC and decision-oriented criteria such as 1-OE and 1-Cov.
  • Complete settings: On fully observed datasets, V2L remains among the top-performing methods across datasets and metrics, indicating that its strategy also applies beyond missing-data conditions.The complete-data comparison uses five datasets without missing views or labels, visualized with six-metric radar maps.
  • Varying missing settings: Performance declines as either view or label missingness increases, with increasing view missingness generally causing the more pronounced degradation.The varying-rate study fixes one missing rate at 50% and changes the other from 0 to 0.7 on MIRFLICKR and Pascal07.
  • View-label heterogeneity: View contributions vary across instances and labels, while some views are consistently weaker or redundant, supporting instance-wise label-aware post-fusion rather than identical fusion weights.Heat maps use two random instances from Pascal07 and MIRFLICKR and display correlations between six views and labels.

D. Study of Information Bottleneck Design

The information bottleneck study examines how β balances view-specific reconstruction information against cross-view semantic consistency. Intermediate weighting performs best because the two objectives play complementary roles.

  • Information bottleneck formulation: The information bottleneck loss is Lib = Lre + βLconsis, where reconstruction preserves valid view-specific information and consistency suppresses redundancy while promoting shared semantics.The study analyzes the first two objective terms and their roles in self-supervised representation learning.
  • Trade-off analysis: When β is too large, semantic compression raises Lre by discarding reconstruction-relevant information and reduces predictive performance.Overemphasizing consistency minimizes Lconsis but can remove view-valid information.
  • Trade-off analysis: The best results occur at β = 1e0, where reconstruction and consistency reach a favorable balance.This middle range avoids both excessive view-specific redundancy and over-compression of the latent representation.

E. Ablation Study

Ablation and sensitivity analyses show that V2L’s losses, adaptive fusion, missing-index guidance, and moderate coefficient settings each contribute to performance, while the full design retains manageable computational cost.

  • Loss-function ablation: The complete model achieves the best performance across all metrics on Corel5k and Pascal07 under 50% missing views and labels.Removing auxiliary losses individually or jointly reduces performance, with the largest decline when all three are removed.
  • Fusion-strategy ablation: Static or passive fusion degrades overall performance on both datasets compared with V2L’s instance-wise label-aware fusion.The comparison evaluates equal-weight late fusion and globally learned view-level parameters against the proposed active strategy.
  • Missing-index guidance: Removing missing-view or missing-label index guidance also degrades performance, with missing-view masking causing the more obvious overall drop.The results support explicitly identifying available views for semantic aggregation and adaptive decision fusion.
  • Hyper-parameter sensitivity: V2L remains relatively stable for moderate γ and σ values, whereas excessively large values of either coefficient cause clear performance degradation.Very small γ provides insufficient cross-view posterior regularization, leaving more view-specific disturbance in the latent representations.
  • Hyper-parameter sensitivity: The best results generally occur around γ = 10^-2–10^-1 and σ = 10^-1–10^0 across the evaluated datasets.This range provides a favorable balance and consistently strong performance.
  • Computational cost: V2L requires 0.03 seconds for inference, with 73.377 million trainable parameters and 1803.68 MB peak GPU memory on Corel5k.Its parameter count and memory consumption are lower than several compared methods, while training overhead remains moderate.
  • Overall design: The unified architecture combines semantically consistent representation learning with instance-dependent, label-aware view complementarity through mid-level and late-stage fusion.The paper also identifies extension to other incomplete multi-view or multimodal tasks as future work requiring task-specific relevance supervision and prediction objectives.
Loading 2609.07525v1…