PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual Supervised Learning

Yingce Xia, Tao Qin, Wei Chen, Jiang Bian, Nenghai Yu, Tie-Yan Liu

arXiv:1707.00415v1cs.LGstat.ML

TL;DR

Many supervised tasks have dual counterparts, yet their probabilistic relationship is often unused because the models are trained separately. Dual supervised learning jointly trains both models while regularizing them through this relationship, and the paper reports improved performance across translation, image processing, and sentiment analysis.

  • Problem

    Dual tasks are commonly trained separately even though their models have an intrinsic probabilistic connection, leaving this relationship underused.

  • Method

    Dual supervised learning jointly trains primal and dual conditional models and uses their probabilistic duality as a regularization term.

  • Results

    DSL improves both tasks across machine translation, image processing, and sentiment analysis, including reported BLEU gains, lower image-classification error, and lower sentiment-classification error.

  • Takeaways & Limitations

    DSL provides a supervised-learning framework that exploits structural duality to improve practical performance for both members of a task pair.

  • Takeaways & Limitations

    DSL requires task duality, trainable primal and dual models, and available empirical marginal distributions, and may work poorly when these conditions are absent.

Abstract

from arXiv · show

Many supervised learning tasks are emerged in dual forms, e.g., English-to-French translation vs. French-to-English translation, speech recognition vs. text to speech, and image classification vs. image generation. Two dual tasks have intrinsic connections with each other due to the probabilistic correlation between their models. This connection is, however, not effectively utilized today, since people usually train the models of two dual tasks separately and independently. In this work, we propose training the models of two dual tasks simultaneously, and explicitly exploiting the probabilistic correlation between them to regularize the training process. For ease of reference, we call the proposed approach \emph{dual supervised learning}. We demonstrate that dual supervised learning can improve the practical performances of both tasks, for various applications including machine translation, image processing, and sentiment analysis.

1. Introduction

Many supervised learning tasks have dual forms whose probabilistic connection is usually ignored because the models are trained separately. Dual supervised learning jointly trains the two tasks using this connection, improving performance across translation, image processing, and sentiment analysis.

  • 1. Introduction: Dual supervised learning jointly trains primal and dual models while exploiting their structural and probabilistic relationship.The primal maps X to Y, the dual maps Y to X, and DSL uses their relationship to improve learning effectiveness.
  • 1. Introduction: DSL uses the equality of two factorizations of P(x, y) as a regularization constraint linking the conditional models.The models are jointly optimized subject to the probabilistic relationship rather than trained independently.
  • 1. Introduction: +2.07/0.86 BLEU points for English↔French, +1.37/0.12 for English↔Germen, and +0.74/1.69 for English↔Chinese are reported with DSL.These improvements are reported for the two translation directions in each language pair.
  • 1. Introduction: On CIFAR-10, DSL reduces ResNet-110 error from 6.43 to 5.40 and produces clearer images with smaller bits per dimension.The image tasks are classification and category-conditioned generation, despite information loss from images to labels.
  • 1. Introduction: On IMDB, DSL improves sentiment-classification error by 0.9 point and generates sentences with clearer, richer sentiment styles.The dual tasks are sentiment classification and sentence generation conditioned on sentiment polarity.
  • 1. Introduction: Experiments across real artificial-intelligence applications report improved practical performance for both dual tasks simultaneously.

2. Framework

The framework formalizes paired primal and dual prediction tasks and reinforces their probabilistic duality during joint supervised training. DSL uses a data-dependent regularizer, subject to requirements including available empirical marginals and trainable dual models.

  • 2. Framework: DSL formulates a primal task from X to Y and a dual task from Y to X, with losses measuring prediction errors on paired training data.The framework assumes i.i.d. training pairs and parameterized conditional distributions for the two directions.
  • 2. Framework: Standard supervised learning trains the two models independently, providing no guarantee that they satisfy probabilistic duality.DSL addresses this gap by explicitly reinforcing empirical probabilistic duality.
  • 2. Framework: DSL jointly minimizes the original task losses plus a regularization term derived from probabilistic duality, using empirical marginal distributions when groundtruth marginals are unavailable.The algorithm applies the duality term through a weighted objective and updates both models with task-specific optimizers.
  • 2. Framework: The duality regularizer is data-dependent: each training sample contributes to it, and each model regularizes the other.This distinguishes DSL from data-independent regularizers such as Lasso or SVM.
  • 2. Framework: DSL differs from co-training, multi-task learning, and transfer learning because it targets structurally dual tasks without requiring shared inputs or an auxiliary main-task relationship.
  • 2. Framework: DSL requires task duality, trainable primal and dual models, and available empirical marginal distributions; otherwise it might not work well.The paper notes that many image, speech, and text tasks satisfy these conditions.

3. Application to Machine Translation

DSL is evaluated on three symmetric machine-translation pairs using dual-task datasets, standard BLEU evaluation, and jointly trained models. Across these experiments, DSL improves both translation directions and strengthens probabilistic consistency relative to RNNSearch.

  • Settings: DSL uses symmetric translation tasks, language-model marginal distributions, GRU sequence-to-sequence models, and BLEU evaluation with beam width 12.The experiments cover En↔Fr, En↔De, and En↔Zh using the stated datasets and training procedure.
  • Results: DSL improves both directions across all three translation pairs compared with RNNSearch.Table 1 reports BLEU scores for En↔Fr, En↔De, and En↔Zh; the authors state that both dual tasks improve simultaneously.
  • Results: On En→Fr, DSL reduces ℓduality from 1545.68 to 1468.28, indicating greater coherence between the two models.
  • Results: With UNK post-processing, DSL outperforms all compared baseline methods on En→Fr.
  • Results: Using stronger initializations raises BLEU from 34.83 to 35.95 for En→Fr and from 32.94 to 33.40 for Fr→En.

4. Application to Images Processing

DSL jointly applies dual supervised learning to image classification and class-conditional image generation on CIFAR-10. It improves classification accuracy and image-generation performance, with clearer generated images than the baseline.

  • Settings: The image-processing setup pairs image classification with class-conditional image generation on the 10-class CIFAR-10 dataset.ResNet models serve classification, while PixelCNN++ models image generation conditioned on category labels.
  • Settings: Classification error rates and image-generation bits per dimension are the evaluation metrics, with average bpd reported on the CIFAR-10 test set.CIFAR-10 images contain 3072 pixels, and bpd evaluates image-generation performance.
  • Results: Test bpd decreases from 2.94 to 2.93 with DSL for ResNet-110, reported as a new CIFAR-10 state-of-the-art result.No significant bpd improvement is observed with ResNet-32.
  • Results: DSL produces clearer and more category-distinguishable images than the baseline, especially for birds, cats, and dogs in the illustrated rows.Figure 2 places five baseline images on the left and five DSL images on the right for each category.

5. Application to Sentiment Analysis

The sentiment-analysis experiments jointly train sentiment classification and sentiment-conditioned sentence generation with DSL. On IMDB, DSL lowers classification error and modestly improves generation quality, while also working without warm-start models.

  • 5. Application to Sentiment Analysis: The task pairs sentiment classification with sentence generation conditioned on sentiment polarity.Classification predicts a sentence’s sentiment label, while generation produces sentences from a sentiment polarity.
  • 5.1. Experimental Setup: The IMDB experiments use 25k training sentences, 25k test sentences, and 3,750 validation sentences.Each sentence has either a positive or negative sentiment label.
  • 5.1. Experimental Setup: The contextual language model generates sentences sequentially from word and sentiment embeddings using an LSTM.The paper calls this sentiment-conditioned generation model the contextual language model, or CLM.
  • 5.2. Results: 0.90 points: DSL reduces IMDB sentiment-classification error without modifying the LSTM model structure.The baseline error is 10.10, and the paper reports a 0.90-point reduction.
  • 5.2. Results: DSL slightly improves sentence-generation perplexity, but the improvement is not very significant.The authors attribute the limited gain to sentiment labels providing at most 1 bit of information.
  • 5.3. Discussions: DSL succeeds from warm-start and cold-start configurations, with cold-start models reaching 9.50% classification error and 58.82 perplexity.Warm starts can speed training, but the reported results do not require them.

6. Conclusions and Future Work

The paper concludes that dual supervised learning jointly improves paired primal and dual tasks through a probabilistic duality regularizer. Future work targets broader task coverage, theory, unlabeled data, and dual inference.

  • Conclusions: Dual supervised learning improves both primal and dual tasks simultaneously by using probabilistic duality as a data-dependent regularizer.The authors report empirical validation of the framework’s effectiveness.
  • Future Work: Future work includes testing DSL on more dual tasks, strengthening its theory, and combining it with unsupervised dual learning and dual inference.The proposed combinations aim to use unlabeled data and structural duality during training and inference.

A. Theoretical Analysis

The theoretical analysis compares DSL with standard supervised learning through approximation error and Rademacher-complexity bounds. It shows a smaller generation-error bound for DSL, while equal approximation error holds under a realizability condition.

  • Hypothesis Class: The analysis defines DSL’s hypothesis class as the product of primal and dual model classes intersected with the probabilistic-duality constraint.The expected loss is taken over the underlying distribution, with separate parameter spaces for the two models.
  • Complexity Analysis: The theoretical framework introduces Rademacher complexity for the constrained dual hypothesis class and compares it with standard supervised learning complexity.The complexity is defined over paired samples from the joint input-output distribution.
  • Generalization Bound: DSL has a smaller generation-error bound than supervised learning under the stated Rademacher-complexity analysis.The bound is obtained because the dual hypothesis class is restricted by the probabilistic-duality constraint.
  • Approximation Error: If the target conditional distributions belong to the corresponding model classes, supervised learning and DSL have the same approximation error.The theorem states that both approximation errors are then zero.

B. Details about the Language Models for Marginal Distributions

The marginal sentence distributions are modeled with LSTM language models. The implementation uses a 1024-dimensional embedding and hidden state for the main language models, while sentiment classification uses a 500-dimensional embedding and achieves perplexity 58.74.

  • Marginal Sentence Models: LSTM language models characterize sentence marginal distributions by multiplying word-level conditional probabilities across each sentence.The model uses 1024-dimensional embeddings and hidden nodes with 0.5 dropout in specified layers.
  • Evaluation: Validation perplexities of the language models are reported in Table 7 using the same validation sets.The table summarizes validation performance for the marginal-distribution models.
  • Sentiment Analysis: The sentiment-analysis marginal language model uses vocabulary size 10000, word-embedding dimension 500, and perplexity 58.74.These settings differ from the main language-model configuration described for other applications.
Loading 1707.00415v1…