PaperlayerView exact revision on arXiv

Source-linked AI summary

Biases in the Experimental Annotations of Protein Function and their Effect on Our Understanding of Protein Function Space

Alexandra M. Schnoes, David C. Ream, Alexander W. Thorman, Patricia C. Babbitt, Iddo Friedberg

arXiv:1301.1740v4q-bio.GNcs.DLcs.ITmath.IT

TL;DR

Protein-function annotation depends on experimental findings, but high-throughput studies can bias the resulting picture. The paper examines GO Consortium annotations and reports that high-throughput experiments bias understanding toward selected aspects of protein function.

  • Problem

    Experimental assays have limits on the kinds of information they can discover, and high-throughput annotations may bias understanding of protein function.

    Source: Author Summary · p. 2

  • Method

    The paper examines GO Consortium annotations to identify biases associated with high-throughput experiments.

    Source: Discussion · p. 12

  • Results

    High-throughput experiments bias understanding of protein function toward certain aspects.

    Source: Author Summary · pp. 2–3

  • Takeaways & Limitations

    The authors advise caution with sweeping measures intended to address these biases.

    Source: Conclusions · p. 15

    The authors identify a circularity problem in the use of the metric.

    Source: Quantifying GO-term Information · p. 18

Abstract

from arXiv · show

The ongoing functional annotation of proteins relies upon the work of curators to capture experimental findings from scientific literature and apply them to protein sequence and structure data. However, with the increasing use of high-throughput experimental assays, a small number of experimental studies dominate the functional protein annotations collected in databases. Here we investigate just how prevalent is the "few articles -- many proteins" phenomenon. We examine the experimentally validated annotation of proteins provided by several groups in the GO Consortium, and show that the distribution of proteins per published study is exponential, with 0.14% of articles providing the source of annotations for 25% of the proteins in the UniProt-GOA compilation. Since each of the dominant articles describes the use of an assay that can find only one function or a small group of functions, this leads to substantial biases in what we know about the function of many proteins. Mass-spectrometry, microscopy and RNAi experiments dominate high throughput experiments. Consequently, the functional information derived from these experiments is mostly of the subcellular location of proteins, and of the participation of proteins in embryonic developmental pathways. For some organisms, the information provided by different studies overlap by a large amount. We also show that the information provided by high throughput experiments is less specific than those provided by low throughput experiments. Given the experimental techniques available, certain biases in protein function annotation due to high-throughput experiments are unavoidable. Knowing that these biases exist and understanding their characteristics and extent is important for database curators, developers of function annotation programs, and anyone who uses protein function annotation data to plan experiments.

OH, USA

The listed affiliations include Alexander W. Thorman in Miami University's Department of Microbiology.

OH, USA

  • Alexander W. Thorman is listed with Miami University's Department of Microbiology.Source: OH, USA · p. 1

Author Summary

Experimental assays are limited in the kinds of information they can discover. The Author Summary says reliance on high-throughput experiments biases our understanding of protein function, while noting that recognizing these biases may help mitigate their effects.

Author Summary

Introduction

The paper examines whether a small number of high-throughput studies dominate experimental protein annotations. It reports that these annotations are less informative, focus on limited functions, and often overlap.

Introduction

  • Functional annotation remains an open problem and a primary challenge in molecular biology.Source: Introduction · p. 3
  • A concern is that a small number of high-throughput experiments could dominate the protein-function landscape in the GO Consortium database.Source: Introduction · p. 4
  • The authors analyze the relative contributions of peer-reviewed articles to experimentally derived UniProt-GOA annotations.Source: Introduction · p. 4
  • High-throughput annotations are less informative than low-throughput annotations, biased toward limited functions, and often overlapping.Source: Introduction · p. 4
  • The findings offer a picture of how literature generates the protein-function annotation landscape and caution producers and consumers of high-throughput data.Source: Introduction · p. 4

Results

The Results examine how annotation coverage across studies and model organisms relates to represented functions and ontologies, including repeated protein selection and annotation consistency. They also assess annotation information across throughput cohorts and describe methods used to identify proteins in high-throughput experiments.

Articles and Proteins

  • The number of proteins annotated per article follows a power-law distribution, indicating that a few articles annotate many proteins.A log-axis linear regression fit had p < 1.18 × 10−8 and R2 = −0.72.Source: Articles and Proteins · p. 5
  • Only 0.14% of articles account for 25% of GO annotations, while 96% of single- and low-throughput articles annotate 53% of proteins.Moderate- and high-throughput studies together account for almost 47% of annotations but only 3.66% of published studies.Source: Articles and Proteins · p. 5
  • Studies are specific to a single species (typically a model organism) and one assaying pipeline; one of three GO ontologies is used in any single experiment, and one study can dominate a species’ functional aspect.Source: Articles and Proteins · p. 6
  • High-throughput articles (over 99 terms per article) have 5% Molecular Function and 38% Biological Process terms; similar-ratio articles have 22–26% Molecular Function, 51–57% Biological Process, and 17–25% Cellular Component terms.Source: Articles and Proteins · p. 7

Repetition and Consistency in Top-50 Annotations

  • Protein selection by high-throughput studies is repetitive in most model organisms, but the rate varies widely, reaching 65% in C. elegans.When normalized by the number of annotating articles, repetition was highest in mouse at 15.33% across three articles, followed by M. tuberculosis at 14% across two articles.Source: Repetition and Consistency in Top-50… · p. 8
  • Annotation consistency varied: the mean maximum-consistency was 0.251 for 1,941 Arabidopsis proteins, while the highest mean consistency was 0.832 for mouse proteins.The mouse result covered 807 proteins in Cellular Component, annotated by only three articles using two terms.Source: Repetition and Consistency in Top-50… · p. 8

Quantifying Annotation Information

  • The subsection sets up an investigation of whether annotation information differs across throughput levels.It frames the question against the assumption that high-throughput experiments provide shallower functional predictions than low-throughput experiments.Source: Quantifying Annotation Information · pp. 8–9
  • Biological Process annotation information decreased significantly with proteins per article, driven by high-throughput papers, while Cellular Component showed no GO-depth trend but lower information content in high-throughput cohorts.Molecular Function edge counts and log-frequency scores also decreased as annotated proteins per article increased; the edge-count and information-content analyses generally agreed.Source: Quantifying Annotation Information · p. 9

Exclusive High Throughput Annotations

  • Although about 80% of experimentally determined proteins were annotated through medium-, low- or single-throughput experiments, few species had many proteins annotated exclusively by high-throughput studies.Those few species included important study and model species; possible redundancy from isoforms, mutants and duplications was noted.Source: Exclusive High Throughput Annotations · pp. 9–10

Frequently Used High-Throughput Experiments

  • Imaging assay evidence ranked second but covered microscopy, RNAi, some mass-spectrometry studies and yeast-2-hybrid; RNAi studies mostly examined embryonic or post-embryonic development.Loss-of-function mutant and RNAi evidence terms ranked fourth and fifth and were closely associated; reuse of RNAi libraries contributed to the large redundancy observed for C. elegans.Source: Frequently Used High-Throughput… · p. 11
  • Computational evidence and sequence- or motif-similarity methods were prominent in high-throughput papers for identifying proteins, rather than directly assigning their functions.Computational combinatorial evidence was the third most frequently used assertion, reflecting combinations of experimental evidence rather than computational prediction alone.Source: Frequently Used High-Throughput… · p. 12

Discussion

High-throughput experiments contribute substantially to experimental protein annotations, but provide less information per protein and concentrate knowledge in a limited set of functions. The discussion links these biases to assay capabilities, GO’s scope, and the uneven distribution of studies across organisms, then considers implications for annotation and prediction.

Discussion

  • Up to 20% of experimentally annotated proteins are annotated by high-throughput assays, which produce less information per protein than moderate-, low-, and single-throughput experiments.The comparison is evidenced by the GO terms produced in the Molecular Function and Biological Process ontologies.Source: Discussion · p. 13
  • High-throughput studies enrich low-information-content GO terms and use fewer terms across many proteins, limiting the specificity and diversity of their annotations.The authors attribute these patterns to the inherent capabilities and limitations of high-throughput experiments.Source: Discussion · p. 13
  • The annotation landscape favors Cellular Component and, to a lesser extent, Biological Process, while few studies address Molecular Function.Drosophila melanogaster and C. elegans are exceptions, with Biological Process terms dominant.Source: Discussion · p. 13
  • Studies of C. elegans and A. thaliana comprise 36 of the top 50 annotating articles, or 72%.The authors identify organism studied as a related source of bias.Source: Discussion · p. 13

Information Capture and Scope of GO

  • GO captures only molecular function, biological process, and cellular component, so some experimental information—such as protein complexes and their network relationships—falls outside its scope.In a yeast study, the only GO term captured in curation was “protein binding” [9].Source: Information Capture and Scope of GO · pp. 13–14

Conclusions

  • The concentrated annotations provide only a partial picture of protein function, dominated by subcellular localization and developmental phenotypes.The conclusion identifies cell fractionation and mass-spectrometry-based localization among the sources of localization information.Source: Conclusions · p. 15
  • The authors warn that training sets containing disproportionate high-throughput annotations can bias machine-learning predictions toward a few frequent, shallow GO terms.They note that this can adversely affect programs that employ prior probabilities.Source: Conclusions · p. 15
  • They recommend recording annotation provenance, including how many proteins each source article annotates, while cautioning that sweeping remedies can cause unintended information loss.The authors suggest this provenance could support less biased benchmarks or be incorporated into annotation procedures.Source: Conclusions · p. 15

Methods

The study analyzed UniProt-GOA annotations using literature-based evidence codes, two measures of GO-term information, and a leaf-term measure of annotation consistency.

Methods

  • The analysis used UniProt-GOA data from December 2011; ECO terms for top experiments were assigned manually after reading the articles.The authors report that the data and scripts were made available online.Source: Methods · p. 16

Use of GO evidence codes

  • GO annotations carry evidence codes describing how the protein–term association was established, including experimental codes such as direct assay (IDA) and physical interaction (IPI).The curation process includes literature-based curation, attribution of article PubMed IDs, and assignment of an evidence code to each protein–GO-term association.Source: Use of GO evidence codes · p. 16

Quantifying GO-term Information

  • The study measured GO-term information by counting edges from the ontology root, using the minimal path when multiple paths lead to a term.Greater edge distance was treated as greater specificity, though ontology structure can make term depths imperfect measures.Source: Quantifying GO-term Information · pp. 17–18
  • The authors also measured information content as the log inverse of term frequency and used both measures because neither was wholly satisfactory.Term frequency may be strongly influenced by top annotating articles, creating a circularity problem for information content.Source: Quantifying GO-term Information · p. 18

Annotation Consistency

  • Annotation consistency was the largest count of an identical terminal GO term divided by the total annotations for a protein within an ontology.The authors compared leaf terms rather than GO subgraphs because comparing subgraphs without manual curation is unreliable and may skew perceived similarity.Source: Annotation Consistency · pp. 18–19

Funding

The research received partial support from NSF and NIH awards.

Funding

  • Funding included NSF DBI 1146960 to IF, and NIH R01 GM60595 and NSF DBI 0640476 to PCB.Source: Funding · p. 20

Figures

The figures depict annotation distribution, contributions by highly annotating articles, redundancy among proteins, and information measures across article-size cohorts.

Figure 1

  • The number of proteins annotated per article follows a logarithmic distribution in the UniProt-GOA experimental annotations.The caption reports a significant linear fit to the log-log plot (R2 = 0.72; p < 1.10 × 10−18).Source: Figure 1 · p. 29

Figure 2

  • Figure 2 bars show the percentage of proteins annotated by the top 50 articles for each organism and GO term, with terms found in multiple species highlighted.Source: Figure 2 · p. 30

Figure 3

  • Figure 3 links proteins with 100% sequence identity; red lines mark annotations in the same ontology and black lines mark different ontologies.The caption illustrates overlap between two S. pombe articles, including one article covering many proteins.Source: Figure 3 · p. 31

Figure 4

  • Figure 4 compares information content and GO depth across Molecular Function, Biological Process, and Cellular Component article-size cohorts.Cohorts range from one annotated protein per article to 100 or more.Source: Figure 4 · p. 33

Tables

The tables organize annotations by article throughput and examine sequence redundancy, annotation consistency, and exclusive high-throughput coverage.

Table 1. Annotation Cohorts

  • 0.14% of experimental-annotation articles account for 25.13% of UniProt-GOA proteins in the cohort breakdown.The table groups articles by the number of proteins they annotate, from singletons to high-throughput studies.Source: Tables · p. 34, Table 1 · p. 34

Table 2. Sequence Redundancy in Top-50 Annotating Articles

  • The top-50 article analysis reports species-level annotation and sequence-redundancy measures, including the percentage of proteins annotated more than once.It also lists the number of annotating articles and proteins, identical-protein clusters, and mean genes per cluster.Source: Tables · p. 35, Table 2 · p. 35

Table 3. Annotation Consistency in Top 50 articles

  • The annotation-consistency analysis summarizes mean consistency and its standard deviation and standard error by species and ontology.Protein–ontology combinations with fewer than two annotating articles or fewer than two terms are omitted.Source: Tables · p. 36, Table 3 · p. 36

Table 4. Fraction of Proteins Exclusively Annotated by High Throughput Studies

  • The exclusive high-throughput analysis reports, by species, the number and percentage of proteins annotated only by studies annotating at least 100 proteins per study.The percentage is measured against the species’ total experimentally annotated proteins.Source: Tables · p. 37, Table 4 · p. 37

Supplementary Material Legends

The supplementary legends explain the tables of the 50 most-annotating papers, defining their fields and mapping ECO evidence terms to annotation types. They also describe how ECO terms assigned to these papers are counted and ranked by frequency.

Supplementary Material Legends

  • Table S1 lists the top 50 annotating articles with protein, GO-annotation, species, reference, and ontology-specific protein counts.MFO, BPO, and CCO denote Molecular Function, Biological Process, and Cellular Component, respectively.Source: Table S1 · p. 39, Supplementary Material Legends · p. 41
  • Table S2 ranks ECO terms assigned to the top-50 papers by assignment frequency and boldfaces computational methods, which were often combined with experimental methods.The legends give examples of term counts and identify the table as listing ECO terms.Source: Supplementary Material Legends · p. 38, Table S2 · p. 42
  • The legends pair ECO codes with evidence types including imaging assays, RNAi, affinity chromatography, sequence similarity, and computational evidence.They also list protein BLAST, cell fractionation, and localization-related evidence types.Source: Supplementary Material Legends · pp. 47–48
  • Table S3 is described as counting ECO terms in the top-50 papers.The accompanying legend identifies the listed terms and IDs as those assigned to the articles.Source: Supplementary Material Legends · p. 52, Table S3 · p. 53

ECO ID

  • Listed evidence categories include epitope-tagged and YFP-fusion protein localization, sequence orthology, biological-system reconstruction, and reverse-transcription polymerase-chain-reaction transcription evidence.The legend also names microarray RNA-expression, fusion-protein localization, nucleic-acid hybridization, enzyme-inhibition, immunolocalization, and hybrid-interaction evidence.Source: ECO ID · p. 54
  • Other listed ECO evidence categories are microarray RNA expression, fusion-protein localization, nucleic-acid hybridization, enzyme inhibition, immunolocalization, and hybrid interaction.These are categories named in the supplementary legend.Source: ECO ID · p. 54
  • The authors assigned ECO terms to the top 50 annotating papers and ranked table entries by assignment frequency; 27 papers had ECO:0000160 and 21 had ECO:0000004.Bold entries indicate computational methods, which were used alongside experimental methods in many papers; Table S2 lists the ECO terms.Source: ECO ID · p. 55
Loading 1301.1740v4…