PaperlayerView exact revision on arXiv

Source-linked AI summary

Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation

Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, Michael J. Black

arXiv:1805.09806v3cs.CV

TL;DR

The paper asks how depth, camera motion, optical flow, and motion segmentation can be learned jointly without ground-truth supervision. It introduces Competitive Collaboration, whose competing and collaborating networks exploit scene geometry and a moderator to assign static and moving regions. The resulting system achieves top performance among unsupervised methods across the subproblems while learning motion segmentation without direct supervision.

  • Problem

    The paper addresses unsupervised learning of depth, camera motion, optical flow, and static-versus-moving scene segmentation despite limited or unavailable ground truth.

  • Method

    Competitive Collaboration trains a static-scene reconstructor, moving-region reconstructor, and motion-segmentation moderator as competing and collaborating neural networks.

  • Results

    Top performance among unsupervised methods is reported across all subproblems, with the moderator learning static and moving regions without direct supervision.

  • Takeaways & Limitations

    Joint learning uses geometric coupling among depth, camera motion, and optical flow, while segmentation partitions training data into relevant static and moving regions.

  • Takeaways & Limitations

    The paper’s automotive depth formulation is not yet extended to world coordinates for integrating depth over long image sequences.

Abstract

from arXiv · show

We address the unsupervised learning of several interconnected problems in low-level vision: single view depth prediction, camera motion estimation, optical flow, and segmentation of a video into the static scene and moving regions. Our key insight is that these four fundamental vision problems are coupled through geometric constraints. Consequently, learning to solve them together simplifies the problem because the solutions can reinforce each other. We go beyond previous work by exploiting geometry more explicitly and segmenting the scene into static and moving regions. To that end, we introduce Competitive Collaboration, a framework that facilitates the coordinated training of multiple specialized neural networks to solve complex problems. Competitive Collaboration works much like expectation-maximization, but with neural networks that act as both competitors to explain pixels that correspond to static or moving regions, and as collaborators through a moderator that assigns pixels to be either static or independently moving. Our novel method integrates all these problems in a common framework and simultaneously reasons about the segmentation of the scene into moving objects and the static background, the camera motion, depth of the static scene structure, and the optical flow of moving objects. Our model is trained without any supervision and achieves state-of-the-art performance among joint unsupervised methods on all sub-problems.

1. Introduction

The paper targets four difficult unsupervised vision problems whose ground-truth data are expensive, sparse, inaccurate, or unavailable. It proposes Competitive Collaboration, where specialized networks compete and collaborate through motion segmentation to exploit their geometric coupling.

  • Problem: Ground-truth data are impractical or expensive for depth, camera motion, optical flow, and motion segmentation.Depth scans can be sparse, optical flow lacks direct sensors, and motion segmentation requires pixel-level manual labeling.
  • Problem: Unsupervised estimation is challenging because depth, flow, and camera motion are ambiguous without additional geometric constraints.Prior explainability and consistency masks do not reliably distinguish independently moving objects from model failures.
  • Idea: Jointly solving the four problems lets coupled networks partition unlabeled data and use relevant pixels for more accurate learning.Static regions are explained through depth and camera motion, while optical flow covers all scene regions and segmentation separates static from moving areas.
  • Approach: Competitive Collaboration is a three-player framework with static-scene reconstructor R, moving-region reconstructor F, and moderator M.R and F compete for static and moving-region pixels, while M distributes training data and is trained collaboratively by the competitors.
  • Approach: The framework resembles expectation-maximization but uses neural networks that compete and collaborate during unsupervised training.The networks alternate between explaining regions and training the moderator to classify them.
  • Contributions: The contributions include unsupervised motion segmentation from low-level geometric information and state-of-the-art results among unsupervised methods across several tasks.The authors also report a first baseline for fully unsupervised motion segmentation and performance against larger or cascaded competing systems.

2. Related Work

Prior unsupervised methods often couple only some tasks or rely on underconstrained photometric losses, while explainability masks can confuse motion with model failures. The paper extends this line with a competitive-collaborative framework that jointly uses scene geometry, flow, and segmentation.

  • Prior unsupervised learning: Unsupervised optical-flow methods that minimize photometric loss are highly underconstrained and perform poorly.Supervised alternatives also face difficult labeling requirements and often depend on synthetic data that may not generalize to real imagery.
  • Prior joint methods: Earlier joint methods coupled depth and camera motion, sometimes adding optical flow, but explainability masks did not provide reliable motion segmentation.Such masks also captured occlusions and other model failures, while recent methods showed strong performance on either geometry or flow rather than both.
  • Paper framework: Figure 2’s architecture combines a static-scene reconstructor, a whole-image flow network, and a motion segmentation network under a composite-flow loss.The static reconstructor uses depth and camera motion; the moderator masks static pixels from the flow network before training on neighboring frames.
  • Generalization: Competitive Collaboration generalizes to intersecting-goal problems in which competitors learn different modes while collaborating through data assignment.The paper connects this design to expectation-maximization and to layered scene models originating in optical-flow research.

3. Competitive Collaboration

Competitive Collaboration jointly trains specialized networks for static-scene reconstruction, moving-region flow, and motion segmentation through alternating competition and collaboration. A moderator partitions pixels between competitors, while consensus and geometric constraints coordinate their learning.

  • Framework: Competitive Collaboration formulates joint learning as a three-player game in which static-scene and moving-region networks compete for data under moderator regulation.The framework is presented as a general method for coordinating task-specific networks.
  • Training cycle: Training alternates between fixing the moderator while training competitors and fixing competitors while training the moderator from their consensus.This cycle is described as competition followed by collaboration, analogous to neural expectation maximization.
  • Network roles: The static-scene reconstructor R combines depth and camera-motion networks, whereas F estimates optical flow for independently moving regions.The segmentation network M selects R for static pixels and F for moving pixels.
  • Objective: The joint objective combines reconstruction losses for static and moving regions, moderation, consensus collaboration, and smoothness regularization.The weighting terms control the relative contributions of these energy components.
  • Geometric consensus: Consensus compares static-scene flow induced by depth and camera motion with optical-flow estimates, favoring static labels when photometric errors or flows agree.The smoothness term regularizes depth, segmentation, and flow while image edges guide the regularization.
  • Inference: At inference, depth and camera motion come directly from network outputs, while motion masks combine moderator predictions with forward/backward mask intersection and flow consensus.The full optical flow is composed from static-scene and independently moving-region flows.

4. Experiments

Experiments evaluate the network architecture, Competitive Collaboration training procedure, and joint performance on depth, camera motion, optical flow, and motion segmentation. Results show gains from joint training, complementary static and moving-region reconstructions, and strong performance among unsupervised or joint methods.

  • Architecture and training: The system uses specialized depth, camera-motion, flow, and motion-segmentation networks trained with the paper’s staged procedure.Depth and camera motion use Dθ and Cφ, flow uses Fψ, and Mχ predicts motion masks; Algorithm 1 alternates specialized and joint updates.
  • Qualitative results: Qualitative outputs are presented as sample images, estimated depth, soft consensus masks, motion-segmented flow, and combined flow.
  • Benchmark results: The method achieves state-of-the-art results among unsupervised methods for depth and camera motion, and among joint methods for optical flow.Depth is evaluated on the Eigen KITTI split, camera motion on KITTI Odometry, and optical flow on the KITTI 2015 training set.
  • Depth and camera motion: Competitive Collaboration with joint loss produces larger depth gains than adding SSIM alone, with further improvements from DispResNet and a stronger flow network.The depth ablation reports cross-task improvement when the flow network is improved, supporting the framework’s joint-training effect.
  • Depth and camera motion: Competitive Collaboration further improves camera-motion estimation beyond the basic model and the addition of SSIM loss.The camera-motion trends are reported in Table 3 alongside the depth experiments.
  • Optical flow and motion segmentation: Jointly combining static-scene and moving-region reconstructions improves optical flow, while Joint masks substantially improve motion-segmentation IoU over MaskNet and Consensus.Independent training performs worse because each reconstructor is specialized for different scene regions; the final masks intersect the two estimates.

5. Conclusions and Discussion

The paper combines geometric relationships among depth, camera motion, optical flow, and motion segmentation through Competitive Collaboration. It reports top performance among unsupervised methods across the subproblems while learning static-versus-moving segmentation without direct supervision.

  • Conclusions and Discussion: Depth, camera motion, and optical flow are related through camera motion in static scenes, while moving regions require a generic flow network.For static scenes observed by a moving camera, depth and camera motion determine 2D optical flow; independently moving regions are handled separately.
  • Conclusions and Discussion: Competitive Collaboration trains networks that both compete and cooperate to address the interconnected vision problems.The framework uses coordinated learning across the subproblems rather than treating them independently.
  • Conclusions and Discussion: The method achieves top performance among unsupervised methods for all subproblems and learns static-versus-moving segmentation without direct supervision.The moderator provides the segmentation component within the joint framework.
  • Conclusions and Discussion: Future work includes adding limited supervision, richer segmentation, world-coordinate depth, and extending the method beyond automotive scenes.The authors specifically mention sparse depth and flow, Cityscapes segmentation, non-rigid segmentation, long-sequence integration, and generic scenes and camera motions.

A.1. Competitive Collaboration as a General Learning Framework

Competitive Collaboration is presented as a general framework for training multiple task-specific networks. The paper demonstrates this generality through mixed-domain classification and analyzes convergence properties.

  • A.1. Competitive Collaboration as a General Learning Framework: Competitive Collaboration generalizes to training multiple task-specific networks beyond the paper’s low-level vision setting.The appendix demonstrates the framework on mixed-domain classification and studies its convergence properties.

A.1.1 Mixed Domain Classification

The mixed-domain experiment trains competing classifiers on shuffled MNIST and SVHN samples, with a moderator assigning examples between them. The appendix also analyzes the objectives and assumptions underlying this alternating procedure.

  • A.1.1 Mixed Domain Classification: The experiment combines MNIST and SVHN samples by shuffling them so the source domain is not indicated by each image-target pair.The task is digit classification into 10 classes, using samples from both datasets.
  • A.1.1 Mixed Domain Classification: Competitive Collaboration uses two competing networks, R_x and F_x, regulated by moderator M_y, which assigns training data during alternating optimization.The competitors are trained against the mixed-domain dataset, while the moderator controls their assignments.
  • A.1.1 Mixed Domain Classification: The moderator assigns each sample toward the competitor with the lower classification loss during the collaboration phase.The output m_i is the probability of assigning a sample to R_x; the loss compares L_Ri and L_Fi.
  • A.1.1 Mixed Domain Classification: A variance constraint encourages the moderator to distribute images across both competitors instead of selecting only one.The constraint increases the variance of moderator outputs within a batch.
  • A.1.1 Mixed Domain Classification: The intended outcome is specialization, with each competitor learning one domain and potentially outperforming a single network trained on the mixture.This is stated as the ideal case for the MNIST+SVHN experiment.
  • A.1.1 Mixed Domain Classification: The convergence analysis requires competing and collaboration objectives to have suitably aligned gradients and similar scales.The appendix describes these as conditions for the alternating optimization procedure to converge.
  • A.1.1 Mixed Domain Classification: Under Lipschitz-smoothness assumptions, the proposition shows that the relevant objectives satisfy the stated convergence conditions for individual data points.The smoothness assumptions are described as standard for gradient-descent convergence.
  • A.1.1 Mixed Domain Classification: The theory does not directly apply to multiple data points because the summed gradients make their inner product difficult to characterize definitively.The appendix considers practical convergence plausible but does not establish it through Theorem 1 for the summed objectives.

A.1.3 Proofs

The proofs establish convergence of the alternating optimization under smoothness, boundedness, and gradient-alignment assumptions. The resulting gradient measure tends to zero, implying convergence to a first-order stationary point.

  • A.1.3 Proofs: The x-update is analyzed as a gradient-descent step on f using a Lipschitz bound.This begins the proof by controlling the change in the objective during the competitor update.
  • A.1.3 Proofs: The proof combines the two update inequalities using C = max(A, B) and defines G_t from the relevant gradient norms.This produces a per-iteration bound that can be summed over t.
  • A.1.3 Proofs: Since f is lower-bounded, G_t tends to zero, implying convergence to a first-order stationary point of f.This is the proof’s final convergence conclusion.
  • A.1.3 Proofs: The proposition establishes Lipschitz continuity of the x-gradient when M is bounded and L_R and L_F are Lipschitz smooth.This verifies a smoothness requirement used in the convergence theorem.
  • A.1.3 Proofs: The y-gradient analysis separates the cases determined by which competing loss is smaller.The proof explicitly treats L_R(x) < L_F(x), with the reverse case analogous.
  • A.1.3 Proofs: Boundedness of M, L_R, and L_F permits choices of β and B that satisfy the required gradient conditions.The proof uses these bounds to establish the constants needed by the convergence argument.

A.2. The camera warping function wc and static flow transformer ν

The camera-motion and depth predictions support geometric transformations from image coordinates to world coordinates and warped image locations. The static flow transformer then uses this geometry to represent static-scene motion.

  • Geometric representation: Camera motion is represented by rotations and translations, while depth provides the scene structure needed for geometric transformation.The motion vector is e = (sinα, sinβ, sinγ, tx, ty, tz).
  • Coordinate transformation: Image coordinates are transformed into world coordinates using predicted camera motion and depth, with camera intrinsics included in the mapping.The intrinsics are represented by (cx, cy, f).
  • Coordinate transformation: The world coordinates are then transformed using a 3D rotation and translation in SE3 before being expressed again in image coordinates.The transformation uses (RxRyRz, t) ∈ SE3.
  • Static-scene motion: The resulting image-coordinate mapping is used to perform camera warping and define the static flow transformer.The passages introduce the warping operation and static flow transformer after the coordinate transformations.

A.3. The flow warping function, wf

The flow-warping function operates in the image’s spatial coordinate system and uses optical flow to warp image locations. This provides the flow-based transformation used in the framework.

  • Flow warping: The flow warping function wf is defined as the operation used to warp image coordinates according to optical flow.The passage introduces wf directly as the flow-warping function.
  • Flow variables: Optical flow is represented by (ux, uy), while (x, y) denotes the spatial coordinate system on which the warping operates.These symbols identify the motion vector and image location used by wf.

A.4. Network Architectures

The framework combines specialized depth, camera-motion, flow, and mask networks with qualitative and cross-dataset evaluations. The reported results show sharper scene and motion predictions, while shared encoder features can degrade camera-motion estimates.

  • Network architectures: The framework uses DispNet-style depth prediction, a convolutional camera-motion network, FlowNetC-style flow prediction, and a U-Net mask network.Depth outputs are produced at six scales, flow also has six output scales, and the mask decoder uses upconvolutions with sigmoid prediction layers.
  • Motion segmentation: The method segments moving cars and bicyclists while leaving parked cars unsegmented, demonstrating a distinction between independently moving and static regions.The qualitative result specifically contrasts a moving car with parked cars on the roads.
  • Depth prediction: The joint model qualitatively improves depth around moving objects and surface edges, including cars, bikes, trees, pavements, and buildings.The comparison includes independently trained basic models and the joint-loss model.
  • Optical flow: The flow estimates are reported as better than UnFlow, Geonet, and DF-Net, especially at car and pavement boundaries where competing fields appear blurry.The comparison is qualitative and is shown in Figure 10.
  • Additional experiments: On Make3D, the model is evaluated without training on that dataset and is reported to outperform prior work; pose is separately evaluated on Sintel alley sequences.The Make3D comparison uses a model trained only on Cityscapes and KITTI, while Sintel evaluation reports relative pose errors.
  • Architecture ablation: Sharing an encoder between camera motion and motion segmentation degrades camera-motion estimates because feature encodings may not be optimal for both networks.The authors relate this observation to prior depth and camera-motion sharing results.
  • Additional experiments: 0.320 is the reported CC value alongside Zhou 0.383, DF-Net 0.331, and Godard 0.361 in the listed comparison.The passage presents these values in the order Zhou, DF-Net, Godard, and CC.
  • Network outputs: The qualitative prediction outputs include depth, consensus masks, static-scene flow, moving-region flow, and full optical flow.These outputs are presented together in the network-prediction visualization.
Loading 1805.09806v3…