Unsupervised Learning of Predictors from Unpaired Input-Output Samples

1 Introduction

• Large-scale prediction systems face prohibitive annotation costs, motivating direct use of abundant unlabeled data.Speech recognition may require thousands of hours of manually labeled acoustic data, while millions of hours of natural speech are available.Source: §1 · p. 1
• Prior unsupervised approaches commonly learn input representations for a separate classifier or pre-train models, but usually remain less effective than end-to-end supervised learning.Pre-training is reported as effective mainly when only a small amount of labeled data is available.Source: §1 · pp. 1–2
• Unsupervised prediction aims to learn input-to-output mappings from unpaired samples rather than use unsupervised learning only as an auxiliary step for supervised learning.The proposed objective fits predicted outputs to output structure while preserving input-output correlation.Source: §1 · p. 2

2 Related Work

• Existing unsupervised methods model input distributions with maximum likelihood or related generative approaches, but face difficulties such as exact likelihood-gradient computation.Examples include restricted Boltzmann machines, deep belief networks, and topic models, with variational and Monte Carlo approximations used for optimization.Source: §2 · p. 2
• Autoencoders, variational autoencoders, and GANs avoid some likelihood difficulties but still target input-distribution modeling rather than unpaired input-to-output prediction.These methods include autoencoder, denoising autoencoder, variational autoencoder, and generative adversarial network approaches.Source: §2 · p. 2
• Output distribution matching [27] is closer, but it does not exploit output structure; this paper instead uses a sequence prior over outputs.The sequence prior can be learned from large text collections for applications such as speech recognition, machine translation, and captioning.Source: §2 · p. 2
• The proposed objective also regularizes input-output correlation through a generative model and can be optimized end-to-end with stochastic gradient descent.The generative-model term provides an additional source of prior information beyond the structured output distribution.Source: §2 · pp. 2–3

3 Problem Formulation

• The predictor p(yt|xt, Wd) maps an M-dimensional input vector xt to a C-class one-hot output yt through a parametric model.Wd denotes the model parameters, and the conditional model may be implemented by a neural network.Source: §3 · p. 3
• The supervised objective requires each input xt to be paired with and labeled by an output yt.T denotes the number of training examples in the supervised formulation.Source: §3 · p. 3
• Unlike supervised learning, the proposed setting provides separate input and output sequences that need only match their respective marginal distributions, not the true joint distribution.The two sequences may therefore come from completely independent sources.Source: §3 · p. 3
• The unsupervised objective learns the predictor from the input sequence while exploiting a temporally structured output distribution learned from another unpaired output sequence.The output distribution is assumed known a priori and may be estimated from a different data source with the same distribution.Source: §3 · p. 3

Output-distribution term

• The first term maximizes the expected log-likelihood of outputs generated from inputs under the output prior p(y1, . . . , yT).Its global optimum is the ground-truth predictor when the parametric predictor family contains that solution.Source: §4 · pp. 3–4

Optimization challenge

• The output-fitting objective alone has many bad local optima, including trivial predictors that ignore inputs while matching the output distribution.The correlation term is introduced specifically to penalize this decoupling.Source: §4 · p. 4

Generative regularization

• The generative term scores how well p(xt|yt, Wg) reconstructs each input from a sampled output and jointly learns the best reconstruction model.Maximizing this term with respect to the predictor increases the modeled correlation between inputs and predicted outputs.Source: §4 · p. 4

Cost-function design

• The objective combines expected output-distribution fitness with a generative reconstruction term that regularizes the predictor toward input-output correlation.The two terms are optimized jointly over the discriminative predictor and generative model; λ controls their relative weight.Source: §4 · pp. 3–4

Optimization

• If the discriminative and generative models are deep neural networks, stochastic-gradient optimization can compute the learning gradients by back propagation.Source: §4 · p. 4

5.1 Experimental setup

• The synthetic setup generates four-class one-hot outputs from a Markov chain, transforms them through a fixed unknown permutation, and trains on unpaired input and output sequences.The predictors use softmax models, with γ controlling softmax sharpness.Source: §5.1 · p. 5

5.2 The landscape of the proposed unsupervised cost function

• Even with simple linear classifiers, the unsupervised cost is highly non-convex, unlike the convex supervised cost for a linear classifier.The landscape is evaluated by comparing the unsupervised objective with supervised cross-entropy along a line between the ground-truth and converged solutions.Source: §5.1 · p. 5, §5.2 · p. 5, Figure 2 · p. 5
• The global optimum of the unsupervised objective's first term coincides with the supervised optimum, but optimization can reach a near-equivalent local solution that ignores the inputs.Repeated experiments found rank-1 local solutions, for which p(yt|xt, Wd) becomes independent of xt.Source: §5.2 · p. 6

5.3 The importance of regularization

• The unsupervised solution with λ = 30 is very close to the supervised solution, whereas regularization smooths local optima and creates a slope that helps escape the trivial solution.The comparison uses landscapes along lines through the ground-truth solution and shows the regularized solution approaching the supervised optimum.Source: §5.3 · pp. 6–7, Figure 3 · p. 6

5.4 The impact of imperfect p(y1, . . . , yT )

• As estimation error in p(y1, . . . , yT ) increases, performance degrades, and λ must be set reasonably because overly weak or strong regularization is ineffective.Small λ produces an insufficient escape slope, while large λ overwhelms information about p(yt|xt).Source: §5.4 · p. 7, Figure 4 · p. 7

• The proposed objective makes predicted outputs fit output structure while preserving input-output correlation, enabling prediction without paired input-label data.Source: §6 · p. 7
• On a synthetic structural prediction problem, the objective is highly non-convex even for simple linear classifiers.Source: §6 · p. 7
• Regularization via a generative model makes learning with the proposed unsupervised objective converge to an optimal solution.Source: §6 · p. 7
• Other local optima represent trivial solutions despite having cost values close to the global optimum.Source: §6 · p. 7
• Designing better objective functions to eliminate trivial local optima remains an open direction, alongside testing more realistic models and real-world data.Source: §6 · p. 7