PaperlayerView exact revision on arXiv

Source-linked AI summary

Supervised, semi-supervised and unsupervised inference of gene regulatory networks

Stefan R. Maetschke, Piyush B. Madhamshettiwar, Melissa J. Davis, Mark A. Ragan

arXiv:1301.1083v1q-bio.MNq-bio.QMstat.ML

TL;DR

Inferring gene regulatory networks from expression data lacks a comprehensive comparison across unsupervised, semi-supervised, and supervised methods. The paper evaluates these approaches on simulated expression data across network and experimental settings, finding that supervised methods generally achieve the highest accuracies while z-score is a notable exception on knock-out data.

  • Problem

    A comprehensive evaluation comparing unsupervised, semi-supervised, and supervised network-inference methods across experimental data types was lacking.

  • Method

    The study evaluates inference methods on simulated expression data from known regulatory sub-networks across knock-out, knock-down, and multi-factorial experiments.

  • Results

    Supervised methods generally achieve the highest accuracies, while unsupervised methods show very low accuracy except for z-score on knock-out data; semi-supervised methods outperform unsupervised methods with as few as 10% known interactions.

  • Takeaways & Limitations

    Small networks can reach AUC ≈0.9 with supervised techniques or z-score, while knock-out data support higher-quality inference than multi-factorial data.

  • Takeaways & Limitations

    The evaluation relies on incomplete simulation models and imperfectly known network structures, so real-data assessment would be preferable.

Abstract

from arXiv · show

Inference of gene regulatory network from expression data is a challenging task. Many methods have been developed to this purpose but a comprehensive evaluation that covers unsupervised, semi-supervised and supervised methods, and provides guidelines for their practical application, is lacking. We performed an extensive evaluation of inference methods on simulated expression data. The results reveal very low prediction accuracies for unsupervised techniques with the notable exception of the z-score method on knock-out data. In all other cases the supervised approach achieved the highest accuracies and even in a semi-supervised setting with small numbers of only positive samples, outperformed the unsupervised techniques.

1 Introduction

Inferring gene regulatory networks is important but difficult, and existing evaluations had not comprehensively compared unsupervised, semi-supervised, and supervised methods across experimental data types. This study extends prior comparisons to assess method suitability and sample requirements.

  • Gene regulatory network topology influences cellular behaviors including development, differentiation, homeostasis, and stimulus responses.
  • Experimental approaches can identify regulatory relationships but are time-consuming, expensive, and require transcription-factor-specific antibodies.
  • A comprehensive evaluation spanning unsupervised, semi-supervised, and supervised inference methods was previously lacking.
  • The study extends prior work from eight unsupervised and one supervised method to 17 unsupervised methods plus supervised and semi-supervised comparisons.
  • The evaluation covers knock-out, knock-down, and multi-factorial data across networks and asks which methods suit each data type and how many samples they require.
  • Earlier assessments found network inference challenging, although z-score methods performed well under certain knock-out conditions.

2 Methods

The study evaluates inference methods with AUC on simulated expression data generated from known network subgraphs, varying network size, experimental type, and sample count. This design enables comparison against a known true topology while retaining simulation-based scope.

  • Prediction performance is assessed using area under the receiver operator characteristic curve (AUC).AUC compares ranked predicted edge weights with the true network topology without requiring a threshold.
  • Simulated steady-state expression data are generated from E. coli and S. cerevisiae sub-networks so predictions can be compared with a perfectly known true network.
  • GeneNetWeaver generates knock-out, knock-down, and multi-factorial expression data using models of transcription, translation, and chemical kinetics.
  • SynTReN simulates transcription and multi-factorial experiments and allows sample number to vary independently of network size.
  • The comparison varies network sizes from 10 to 110 nodes, sample sizes from 10 to 110, and three experimental data types.
  • Unsupervised methods receive no parameter optimization, whereas supervised and semi-supervised methods use five-fold cross-validation with training-set optimization.

2.1 Unsupervised

The unsupervised methods infer regulatory relationships from expression profiles using correlation, mutual information, feature selection, regression, similarity, or knock-out effects. Their designs differ in how they transform expression relationships into interaction weights.

  • Correlation: Correlation-based methods use expression association as evidence of regulation, with positive and negative coefficients indicating activating and inhibitory interactions.
  • Correlation: Absolute correlation values are used as interaction weights because prediction accuracy does not distinguish inhibiting from activating interactions.
  • Correlation: WGCNA applies a monotonic softpower transformation to correlation coefficients, so its AUC is identical to that of the underlying correlation method.
  • Mutual information: Mutual-information methods quantify dependence between expression profiles, while CLR adjusts scores using each gene’s background distribution.
  • Mutual information: ARACNE examines gene triplets and prunes interactions judged indirect by the Data Processing Inequality beyond eps = 0.2.
  • Feature selection: MRNET selects regulators with Minimum-Redundancy-Maximum-Relevance, whereas MRNET-B replaces forward selection with backward selection and sequential replacement.
  • Feature selection: GENIE uses Random Forests and Extra-Trees for regression and feature selection instead of mutual information and MRMR.
  • Z-SCORE: Z-SCORE normalizes the effect of a gene knock-out on another gene relative to that gene’s average expression.

2.2 Supervised

The supervised approach uses linear SVMs to classify gene-pair feature vectors from labeled interaction data, while cross-validation estimates prediction accuracy. Expression profiles are transformed into symmetric features so predicted interactions are undirected.

  • The evaluation uses Support Vector Machines because they have been applied to gene regulatory network inference and can be trained semi-supervised.
  • The SVM decision distance is interpreted as a confidence value and used as the interaction weight.
  • Each gene pair is represented by a feature vector constructed from the outer product of its two expression profiles.
  • The outer-product representation makes predicted interactions symmetric and undirected.
  • Training labels assign yi = +1 to interacting gene pairs and yi = −1 to non-interacting pairs.
  • In practice, only a subset of gene pairs is labeled for training, while remaining pairs serve as test samples.
  • A supervised-learning sample is a gene-pair feature vector, whereas an expression-data sample is the gene-expression profile from one experiment.
  • Five-fold cross-validation repeatedly uses four parts of the labeled sample set for training and one part for testing.

2.3 Semi-supervised

Semi-supervised inference uses partially labeled training data, including positive-only labels, to incorporate information from otherwise unlabeled samples. The evaluation varies labeling schemes and proportions under matched cross-validation conditions.

  • Practical setting: Known negative interactions are especially scarce because experiments primarily detect interactions rather than exclude them.Consequently, fully labeled supervised training sets are rarely available in practical network inference.
  • Labeling strategies: Semi-supervised methods use unlabeled training samples and can operate with either positive and negative labels or positive labels only.Supervised methods require every training sample to be labeled, whereas unsupervised methods require none.
  • Method: The evaluated semi-supervised method adapts a supervised SVM by relabeling unlabeled training samples as negatives.This preserves the same prediction algorithm for direct comparison with fully labeled training.
  • Evaluation: Training used 10%–100% of true positive and negative or positive-only labels, with five-fold cross-validation and equal training/test sizes across approaches.The design compares supervised, semi-supervised, positives-only, and unsupervised methods under matched set sizes.

3 Results

The results show generally weak and unstable unsupervised inference, with performance depending on experimental type, network size, and sample number. Supervised methods perform well broadly, while semi-supervised methods can outperform unsupervised methods with limited labels.

  • 3.1 Unsupervised methods: Large standard deviations make unsupervised accuracy highly variable, and method rankings depend on experimental type, source network, and sub-network size.Although many differences are statistically significant, the differences are often small and rankings are unstable.
  • 3.1 Unsupervised methods: Rank-based correlations perform poorly on knock-out and knock-down data but well on multi-factorial experiments, while Pearson correlation is consistently the second-best performer.This demonstrates strong dependence of method behavior on experimental data type.
  • 3.1 Unsupervised methods: Most unsupervised methods have low accuracy, whereas Z-SCORE achieves AUC = 0.9 on knock-out data.Z-SCORE was designed specifically for knock-out data and clearly outperforms the other unsupervised methods there.
  • 3.2 Network size: SPEARMAN-C has the highest mean AUC on multi-factorial data, whereas EUCLID performs best on 10-node networks and correlation methods improve on larger networks.Performance therefore changes with network size rather than following one stable method ranking.
  • 3.3 Sample number: For 50-node networks, increasing samples from 50 to 110 raises AUC only from roughly 0.65 to around 0.67.The evaluation attributes the small improvement to unsupervised methods reliably inferring only simple network topologies.
  • 3.4 Supervised methods: Supervised methods achieve good accuracy across experimental types, improve with more labeled data, and show little difference between positive-only and positive-negative labeling.The supervised evaluation used 30-node E. coli networks and repeated each experiment 10 times.
  • 3.4 Supervised methods: With 10% of known interactions, semi-supervised methods outperform unsupervised methods on multi-factorial data; on knock-down data, at least 70% labeling performs better.Z-SCORE remains strongest on knock-out data, while supervised methods are not far behind.
  • 3.4 Supervised methods: Overall, supervised and semi-supervised approaches considerably outperform unsupervised methods except for Z-SCORE on knock-out data.This conclusion applies to networks of the evaluated size.

4 Discussion

The evaluation highlights important limitations of simulated-data benchmarks and linear SVM choices, while showing that simple methods can match complex ones and that mixed regulation challenges unsupervised inference.

  • Benchmark scope: Simulated benchmarks omit posttranscriptional regulation and chromatin states, while real reference networks are fragmentary and partially incorrect.The study therefore limits evaluation to in silico benchmarks.
  • Classifier choice: Linear SVMs were chosen because they train faster, require fewer parameters, and suit complex systems with many variables and few samples.Transductive SVMs were not fully evaluated because they were time-consuming and less accurate than semi-supervised SVMs.
  • Feature construction: Outer-product feature vectors are larger than concatenated profiles but are order-independent and produced higher linear-SVM prediction accuracies.The outer product has N^2 features versus 2N for concatenation, while halving the training-set size relative to the concatenation approach.
  • Method comparison: Simple methods were no less accurate than complex methods; Z-SCORE and Pearson correlation were the strongest unsupervised performers.MINE did not outperform linear Pearson correlation.
  • Network inference: Mixed activating and inhibiting interactions can make a non-regulator’s expression profile resemble a regulated gene, misleading similarity-based inference.In the example, gene D is regulated by A and B but is most similar to non-regulator C.

5 Conclusion

The study finds that reliable inference depends strongly on method and experimental design: supervised and semi-supervised approaches generally outperform unsupervised methods, while knock-out data are especially informative.

  • Evaluation reliability: Large variations in prediction accuracy require many replicates across networks of varying size for reliable method estimates.Single-data-set evaluations, especially on real data, are unsuitable for establishing accuracy differences.
  • Overall performance: Unsupervised methods generally achieve very low accuracy, except Z-SCORE, and are considerably outperformed by supervised and semi-supervised methods.Pearson correlation is as accurate as more complex methods while being faster and parameterless.
  • Knock-out data: Z-SCORE achieved the best accuracy on knock-out data but fails when a gene is regulated by an OR-junction of two other genes.The method could be generalized to multi-knock-out experiments.
  • Multi-factorial data: With only 10% of known interactions, semi-supervised methods still outperformed all unsupervised approaches on multi-factorial data.Using only positive labels produced little accuracy difference from using positive and negative samples.
  • Practical implications: Large-scale or genome-scale inference from expression data alone is currently not feasible because even the best methods are accurate mainly for small, relatively simple networks.Feature selection is usually required for datasets at modern microarray scale.
  • Experimental design: Knock-out data support higher-quality inference than multi-factorial data, whereas increasing sample numbers does not provide the expected improvement.These limitations should inform experimental design.
  • Summary: Small networks can reach AUC ≈0.9 with supervised techniques or Z-SCORE, but accuracy varies widely and predictions may be limited to undirected networks without self-interactions.The reported high accuracy applies to the small networks evaluated here.

Funding

The study acknowledges funding from the Australian Research Council.

  • Funding was provided by Australian Research Council grants DP110103384 and CE0348221.

Supplementary material

The supplied supplementary-material passages identify the preprint and its authors but provide no supplementary findings.

  • The paper is identified as arXiv:1301.1083v1, dated 7 January 2013.
  • The paper title begins “Supervised, semi-supervised and unsupervised inference of.”
  • The listed authors are Stefan R. Maetschke, Piyush B. Madhamshettiwar, Melissa J. Davis and Mark A. Ragan.

1 Unsupervised

Unsupervised inference accuracy varies strongly with experimental data type, network size, topology, and interaction signs. Z-SCORE is especially effective on knock-out and knock-down data, while simple topologies are easier to recover than complex mixed-sign networks.

  • Z-SCORE and PEARSON remain the best-performing unsupervised methods across metrics, with Z-SCORE dominating on knock-out data.
  • Multi-factorial data: On multi-factorial data, EUCLID performs best only on very small networks with 10 to 30 nodes.
  • Knock-out data: On knock-out data, Z-SCORE outperforms all other methods across network sizes despite declining accuracy as networks grow.
  • Knock-down data: On knock-down data, Z-SCORE remains the best-performing method, while most methods perform best on the 10-node network.
  • Topology effects: Spearman’s correlation ranged from AUC = 0.508 on a failed topology recovery to AUC = 0.971 on an accurate recovery.
  • Topology effects: Simple networks with exclusively activating or inhibiting interactions can be inferred accurately, whereas complex mixed-sign topologies are typically unreliable.

2 Supervised

The study compares supervised, semi-supervised, and unsupervised methods across multiple metrics and simulated experimental data types. Using repeated cross-validation on 30-node networks, it finds that Z-SCORE is strongest on knock-out data but weakest on multi-factorial data.

  • The comparison evaluates supervised, semi-supervised, and unsupervised methods using AUC, MCC, and F1-score.
  • Methods were evaluated on multi-factorial, knock-out, knock-down, and averaged GenNetWeaver data using 5-fold cross-validation repeated 10 times.
  • Z-SCORE achieves the highest accuracies on knock-out data but performs worst on multi-factorial data.
Loading 1301.1083v1…