PaperlayerView exact revision on arXiv

Source-linked AI summary

Kronecker-Structured Video Completion with Cross-Fitted Self-Supervised Residual Learning

Qi Qi, Yanfei Wang, Yanjun Cui, Jun-e Feng

arXiv:2610.35065v1cs.CV

TL;DR

Tube-wise video completion must reconstruct temporal trajectories at spatial locations unobserved in every frame, while residual training needs targets that reflect errors at unseen tubes rather than in-sample fitting errors. The paper combines regularized masked KPD with cross-fitted residual learning and reports strong mean reconstruction results on grayscale and selected DAVIS RGB benchmarks.

  • Problem

    At missing tubes, the desired correction concerns the backbone’s out-of-sample prediction error, whereas residuals on tubes used to fit the backbone are in-sample errors.

    Source: A. Residual modeling and target mismatch · p. 6

  • Method

    The method fits a regularized masked KPD backbone to observed entries and trains a residual predictor on cross-fitted targets made by withholding each target fold from its backbone fit.

    Source: §I · p. 2

  • Results

    The complete method achieves the highest mean PSNR and SSIM at the higher grayscale sampling rates and on the eight selected DAVIS RGB sequences at every evaluated rate.

    Source: §VI · p. 13

  • Takeaways & Limitations

    The framework combines compact structural estimation with held-out residual supervision and exact data consistency for tube-wise video completion.

    Source: §VI · p. 13

Abstract

from arXiv · show

Tube-wise video completion reconstructs temporal trajectories at spatial locations unobserved in every frame. We propose a two-stage framework combining regularized masked Kronecker product decomposition (KPD) with cross-fitted self-supervised residual learning. First, we estimate a compact structural backbone directly from observed entries using cyclic component-wise least-squares updates, spatial and temporal factor regularization, and regularizer-optimal scale balancing. The exact updates yield a nonincreasing objective. Second, we address the mismatch between in-sample fitting errors and the prediction errors encountered at missing tubes. We partition observed tubes into folds and construct residual targets using backbone models fitted without the corresponding target tubes. A lightweight 3-D convolutional neural network then predicts corrections from the backbone, visible residual context, observation mask, and local detail features. Exact data consistency preserves observed measurements. Experiments include ablations, sensitivity analysis, grayscale and RGB video completion, and structured missing patterns. Our method achieves the highest mean PSNR and SSIM at the higher tested sampling rates on the grayscale benchmark. On eight selected DAVIS RGB sequences, it achieves the highest mean missing-region PSNR and full-frame SSIM and the lowest mean missing-tube temporal-gradient MAE at every evaluated sampling rate.

I. INTRODUCTION

The paper addresses tube-wise video completion by fitting a compact structural model to observed data and learning residual corrections that reflect prediction errors at unseen tubes. Its introduction presents the masked KPD estimator, cross-fitted supervision, and an observation model in which the same spatial mask applies across frames.

A. Video tensor and observation model

  • In tube-wise missingness, each spatial location is either observed throughout the video or missing across all frames, leaving its full temporal trajectory unavailable.The reconstruction therefore relies on observed spatial tubes and relationships among frames.Source: §I · p. 1

I. INTRODUCTION

  • Cross-fitting withholds each target tube fold from its corresponding KPD fit, producing held-out structural prediction errors for training a per-video 3-D CNN.At inference, the backbone is fitted to all available tubes and the network predicts corrections at missing locations.Source: §I · p. 2
  • The framework combines the KPD backbone with residual context, a sampling mask, and local high-frequency features, then applies exact data consistency to preserve observed measurements.The introduction identifies structural estimation and recovery of details beyond the chosen rank as complementary tasks.Source: §I · pp. 1–2
  • The estimator fits a joint observed-entry KPD objective with spatial and temporal factor regularization, cyclic component-wise least-squares updates, and regularizer-optimal scale balancing.The exact update steps have a nonincreasing objective property.Source: §I · p. 2

B. Kronecker-structured representation

The section presents a separable KPD representation and compares its storage and approximation fidelity with matrix, tensor-spectral, and multilinear alternatives. The completion experiments use a compact KPD backbone and learn a residual correction.

B. Kronecker-structured representation

  • The KPD factors define a separable model with the same parameterization as third-order CP decomposition, while Stage I fits individual components under a regularized masked objective.The fitted tensor provides the structural backbone for residual prediction.Source: B. Kronecker-structured representation · p. 3
  • Figure 1 compares separable KPD, matrix SVD, truncated t-SVD, and Tucker-HOSVD using average approximation PSNR and SSIM against normalized storage ratio.The plotted sequence-level curves are averaged over the storage interval shared by Akiyo, Foreman, and Mobile.Source: B. Kronecker-structured representation · p. 3, Figure 1 · p. 3
  • At the K = 4 operating point, KPD stores about 0.062% of the voxel count and has approximation fidelity comparable to Tucker-HOSVD.Spectral and multilinear models reach higher fidelity at larger storage budgets.Source: B. Kronecker-structured representation · p. 3, Figure 1 · p. 3

III. MASKED KRONECKER PRODUCT DECOMPOSITION

This section develops a regularized masked KPD estimator that fits separable factors to observed entries through cyclic component-wise updates. Spatial and temporal factor penalties constrain the model, and exact update steps are designed to avoid increasing the objective.

III. MASKED KRONECKER PRODUCT DECOMPOSITION

  • The workflow fits the masked KPD backbone from observed entries before the later section develops the residual predictor.The estimator's cyclic updates and scale-balancing step are part of backbone estimation.Source: §III · pp. 3–4

A. Observed-entry objective

  • The estimator fits all K components to a joint observed-entry objective with spatial and temporal smoothness penalties and Tikhonov regularization.The penalties act on factors rather than individual voxels; positive Tikhonov regularization bounds sublevel sets and prevents unbounded equivalent factor rescalings.Source: A. Observed-entry objective · p. 4

B. Component-wise masked least-squares updates

3) Update of the temporal factor:

The section completes the cyclic masked KPD updates with a temporal-factor step and regularizer-optimal scale balancing. The algorithm initializes factors from a smoothed, mean-filled tensor and returns a raw reconstruction whose observed-entry residuals remain available for the next stage.

3) Update of the temporal factor:

  • After the spatial factors are fixed, the cyclic masked least-squares step updates the temporal factor using only entries selected by the observation mask.The component-wise step comprises the updates in Equations (26)–(32).Source: 3) Update of the temporal factor · p. 5

C. Regularizer-optimal scale balancing

  • Equivalent rescaling preserves each rank-one component, and the chosen scaling minimizes its regularization penalty without increasing the objective.The scaling equalizes the three regularization contributions; the step is omitted if any q is numerically zero.Source: C. Regularizer-optimal scale balancing · p. 5

D. Cyclic masked KPD algorithm

  • The algorithm initializes factors from a smoothed, mean-filled tensor and then cycles through components and modes, applying scale balancing after initialization and each component update.The raw reconstruction retains observed-entry fitting residuals for the next stage.Source: D. Cyclic masked KPD algorithm · p. 5

E. Monotonicity of the exact cyclic updates

Exact cyclic factor updates and optional scale balancing make the KPD objective nonincreasing. Cross-fitting addresses the mismatch between in-sample residuals and errors at missing tubes by withholding each target fold from its backbone fit.

E. Monotonicity of the exact cyclic updates

  • Each exact factor-block update minimizes its subproblem, and optional scale balancing preserves the represented component while not increasing the regularizer.The objective sequence is nonincreasing and converges to a finite limit.Source: E. Monotonicity of the exact cyclic… · p. 5

IV. CROSS-FITTED SELF-SUPERVISED RESIDUAL LEARNING

  • Stage II trains the residual predictor on observed tubes withheld from fold-specific KPD fits, using their prediction errors as targets for missing-tube correction.The predictor is trained separately for each video using observed tubes and cross-fitted targets.Source: §IV · p. 5

A. Residual modeling and target mismatch

  • In-sample fitting residuals differ from the out-of-sample backbone errors needed at missing tubes, so each measured target tube is withheld from its KPD fit.Cross-fitting constructs training targets for the latter prediction errors.Source: A. Residual modeling and target mismatch · p. 6

B. Tube-wise cross-fitting

  • Observed training tubes are partitioned into folds; each fold is predicted using a KPD model fitted only on its context tubes.Validation tubes are excluded from fold-specific KPD fits and residual-network training; after configuration selection, all observed tubes are used for cross-fitting.Source: B. Tube-wise cross-fitting · p. 6

C. No-leakage property of the cross-fitted target

The cross-fitted target is a held-out prediction error when fold-specific KPD operations use only context measurements. Under tube sampling independent of video values and spatial content, pseudo-missing tubes provide an analogous held-out task; the network input combines four distinct signals.

C. No-leakage property of the cross-fitted target

  • The cross-fitted target is an out-of-sample prediction error of the fold-specific backbone when its data-dependent operations use only context measurements.Fold assignment independent of video values and spatial content makes pseudo-missing tubes analogous to genuinely unobserved locations, though fold and final errors may differ because their backbones use different context sizes.Source: C. No-leakage property of the… · p. 6

D. Residual network input

  • The four-channel input combines the bounded backbone, observed context residuals, a context mask, and a spatial high-pass feature of the backbone.The high-pass operator extracts local spatial variation without temporal pooling; the feature highlights variation attenuated in the low-rank backbone.Source: D. Residual network input · pp. 6–7

E. Lightweight 3-D residual predictor

A compact 3-D CNN predicts residual corrections from per-video cross-fitted training. Its loss is evaluated on held-out tubes, and final inference combines the refined estimate with observed measurements to preserve them exactly.

E. Lightweight 3-D residual predictor

  • The compact 3-D residual predictor is initialized to leave the bounded KPD backbone unchanged before training and is trained separately for each video.Its final convolution's weights and bias are initialized to zero.Source: E. Lightweight 3-D residual predictor · p. 7

F. Cross-fitted self-supervised loss

  • Training samples a fold and spatiotemporal patch, computes fitting loss only on held-out tubes, and keeps held-out measurements out of network inputs.Inputs combine the fold-specific backbone, context residuals, context mask, and spatial high-pass feature; the magnitude penalty covers the entire predicted patch.Source: F. Cross-fitted self-supervised loss · p. 7

G. Final inference and data consistency

  • The final output retains every observed entry exactly by combining observed measurements with the bounded refined estimate at unobserved locations.The output construction is stated to satisfy exact data consistency.Source: G. Final inference and data consistency · p. 7

H. Training and inference

Inference adds a CNN-predicted residual to the KPD backbone, then applies data consistency. Cross-fitting supplies held-out residual targets, and the final backbone is fitted using all observed tubes.

H. Training and inference

  • The estimator combines the KPD backbone with a CNN-predicted correction before applying data consistency.The network uses the backbone, observed residual context, mask, and local detail features as inputs.Source: H. Training and inference · p. 7
  • For each fold, auxiliary KPD fits provide context residuals and pseudo-missing targets for training the residual network.After training, the algorithm fits the final KPD reconstruction on all observed tubes, predicts residuals, and applies data consistency.Source: §I · p. 8

I. Computational complexity

  • Dense implementation costs O(SKN) when N dominates factor dimensions; cross-fitting adds F auxiliary KPD fits, approximately F times a single-fit cost.A residual-network forward pass costs O(Nh^2) for fixed 3 × 3 × 3 kernels and hidden width h.Source: §I · p. 8

V. EXPERIMENTS

The experiments evaluate tube-wise completion on grayscale and RGB videos, compare residual-supervision and network-input variants, and examine visual reconstructions and parameter sensitivity.

A. Protocol and implementation

  • Evaluation covers 23 grayscale videos and eight selected DAVIS RGB sequences under mode-3 tube missingness, with masks shared across frames and RGB channels.Grayscale settings are selected on three development sequences; RGB settings are selected on Blackswan and applied unchanged to the eight reported sequences.Source: A. Protocol and implementation · p. 8
  • The experiments use full-video PSNR, frame-averaged full-frame SSIM, missing-entry RGB PSNR, and missing-tube temporal-gradient MAE; higher PSNR/SSIM and lower T-MAE indicate better quality.The grayscale benchmark uses five sampling rates and the RGB benchmark uses three.Source: A. Protocol and implementation · pp. 8–9

D. Stage-wise visual ablation

  • At SR = 0.3, Fig. 3 compares ground truth, observed input, KPD backbone, simple residual propagation, and CrossFit reconstructions using enlarged common ROIs.The accompanying text reports that the backbone recovers main scene geometry and residual refinement recovers additional edge and texture detail.Source: Figure 3 · p. 9

B. Ablation of residual supervision

C. Ablation of residual-network inputs

  • Visible residuals raise mean PSNR from 19.360 dB (B) to 22.354 dB (BR); adding the mask (BRM) and high-pass channel (FULL) raises it to 23.944 and 24.380 dB.FULL also has the highest SSIM and lowest validation MAE among the four input configurations.Source: Table III · p. 9, C. Ablation of residual-network inputs · p. 10

E. Sensitivity analysis

  • Figure 5 examines KPD rank, cross-fitting folds, optimization steps, and spatial and temporal regularization on three grayscale development sequences at SR = 0.2; stars mark selected settings.The supplied caption identifies the settings examined but does not report their comparative outcomes.Source: Figure 5 · p. 10

F. Comparison on the 23-video benchmark

The grayscale benchmark favors S2NTNN at lower sampling rates and Ours at higher rates, while Ours leads the reported mean metrics on eight selected DAVIS RGB sequences. Structured-mask results differ by sampling geometry.

F. Comparison on the 23-video benchmark

  • Ours has the highest mean PSNR and SSIM on the grayscale benchmark at SR = 0.3, 0.4, and 0.5, while S2NTNN leads at SR = 0.1 and 0.2.Ours exceeds the respective second-best methods by 0.314, 0.686, and 0.529 dB in PSNR and by 0.0104, 0.0023, and 0.0007 in SSIM at the three higher rates.Source: F. Comparison on the 23-video benchmark · pp. 10–11

G. DAVIS 2017 RGB completion at 360p

  • On eight selected DAVIS RGB sequences, Ours has the highest mean missing-region PSNR and full-frame SSIM and the lowest mean missing-tube T-MAE at all three evaluated sampling rates.Mean missing-region PSNR is 27.07, 28.12, and 28.99 dB at SR = 0.2, 0.3, and 0.4; mean T-MAE is 0.03079, 0.02732, and 0.02495.Source: G. DAVIS 2017 RGB completion at 360p · p. 11, Table V · p. 12

H. Qualitative RGB comparisons

  • At SR = 0.3, Ours exceeds the strongest baseline in missing-region PSNR by 1.71 dB on Bear, 1.43 dB on Drift-chicane, and 1.18 dB on Cows.The comparison uses a common frame and two fixed ROIs per sequence; PSNR is computed over missing tubes across all 50 frames.Source: H. Qualitative RGB comparisons · p. 11, Figure 6 · p. 11

I. Structured-mask experiments on color videos

  • Under grid missingness, Ours has the highest PSNR and SSIM on Coastguard and Container; for contiguous blocks, KBR leads on Coastguard and SALTS leads on Container.Grid results are 30.15 dB/0.9880 on Coastguard and 26.40 dB/0.9858 on Container; the grid leaves observed content around distributed missing locations, unlike contiguous blocks.Source: H. Qualitative RGB comparisons · p. 12, §I · p. 13, Table VI · p. 13

VI. CONCLUSION

  • The conclusion reports that cross-fitted supervision improves mean reconstruction quality over direct residual learning, while the complete method leads on higher-rate grayscale metrics and all evaluated DAVIS RGB rates.The structured-mask experiments also show that method rankings differ between grid and contiguous-block missingness.Source: §VI · p. 13

B. KPD Rank and Spatial Convolutional Kernel

The RGB sensitivity study selects rank 12 and a 3 × 3 × 3 kernel, then evaluates changes to CNN width, training budget, and patch size. The selected configuration is applied to eight DAVIS sequences, with T-MAE reported for missing tubes.

B. KPD Rank and Spatial Convolutional Kernel

  • With the smaller spatial kernel, PSNR increases with rank; the 3 × 3 × 3 kernel gives the highest PSNR at K = 12 and is retained.The study varies K ∈ {4, 8, 12} and spatial kernel width ks ∈ {3, 5}, fixing kt = 3, h = 12, and 1200 training steps.Source: B. KPD Rank and Spatial Convolutional… · p. 15, Table I · p. 15

C. Structural Rank Versus CNN Width

D. Optimization Steps and Training-Patch Size

  • In the tested ranges, larger patches and longer training generally increase PSNR, with nonmonotone behavior at the largest budget; the highest measured PSNR uses 3600 steps and 96 × 96 spatial patches.Changing patch size alters training context but leaves the CNN receptive field unchanged.Source: D. Optimization Steps and… · p. 15

E. Final RGB Configuration

  • The selected RGB configuration improves Blackswan missing-region PSNR from 27.1250 to 28.9023 dB and full-frame SSIM from 0.8510 to 0.9047 relative to the initial configuration.These measurements are from the development sequence used for selection; the selected settings are then applied to the eight reported sequences at all three rates.Source: E. Final RGB Configuration · p. 16

A. Selected-Sequence Quantitative Comparison

  • The selected configuration is applied to eight selected DAVIS sequences at three sampling rates, using each sequence’s own self-supervised fit with shared hyperparameter settings.All methods use the same tube-wise mask for each video and rate.Source: A. Selected-Sequence Quantitative… · p. 16

B. Temporal-gradient MAE on Missing Tubes

  • Ours has the lowest missing-tube T-MAE in 19 of 24 video–rate cases; the metric measures reconstruction error in frame-to-frame intensity changes at missing tubes.The missing pixel–channel index set remains fixed across frames under tube-wise sampling.Source: B. Temporal-gradient MAE on Missing… · p. 16

C. Quality Trends and Computational Cost

The supplemental comparison reports RGB quality trends, per-sequence PSNR margins, and runtime across the selected DAVIS sequences. Its visual comparisons use common frames and fixed regions of interest across methods.

C. Quality Trends and Computational Cost

  • Ours has a median wall-clock time of 1859.2 s per RGB video, compared with 466.9 s for S2NTNN and 45.4 s for LRTFR under the reported implementations.Figure 7 shows mean PSNR, SSIM, and T-MAE trends, while Figure 3 shows per-sequence missing-region PSNR margins over the strongest alternative.Source: C. Quality Trends and Computational Cost · p. 16

Sequence DRO-TFF HLRTF KBR LRTFR OTLRM TNN SALTS S2NTNN Ours

  • The per-sequence missing-region PSNR tables cover the eight selected DAVIS RGB videos at SR = 0.2, 0.3, and 0.4.The corresponding full-frame SSIM and missing-tube temporal-gradient MAE tables report the same selected-video scope.Source: Table V · p. 17, Table VI · p. 17, Table VII · p. 17

D. Additional RGB Visual Comparisons

Related papers

Loading 2610.35065v1…