Repeatability and reproducibility in proteomic identifications by liquid chromatography-tandem mass spectrometry

David L. Tabb, Lorenzo Vega-Montoto, Paul A. Rudnick, Asokan Mulayath Variyath, Amy Joan L Ham, David M. Bunk, Lisa E. Kilpatrick, David D Billheimer, Ronald K. Blackman, Helene L. Cardasis, Steven A. Carr, Karl R. Clauser, Jacob D. Jaffe, Kevin A. Kowalski, Thomas A. Neubert, Fred E. Regnier, Birgit Schilling, Tony J. Tegeler, Mu Wang, Pei Wang & 12 others Jeffrey R. Whiteaker, Lisa J. Zimmerman, Susan J. Fisher, Bradford W. Gibson, Christopher R. Kinsinger, Mehdi Mesri, Henry Rodriguez, Stephen E. Stein, Paul Tempst, Amanda G. Paulovich, Daniel C. Liebler, Cliff Spiegelman

Research output: Contribution to journalArticle

281 Citations (Scopus)

Abstract

The complexity of proteomic instrumentation for LC-MS/MS introduces many possible sources of variability. Data-dependent sampling of peptides constitutes a stochastic element at the heart of discovery proteomics. Although this variation impacts the identification of peptides, proteomic identifications are far from completely random. In this study, we analyzed interlaboratory data sets from the NCI Clinical Proteomic Technology Assessment for Cancer to examine repeatability and reproducibility in peptide and protein identifications. Included data spanned 144 LC-MS/MS experiments on four Thermo LTQ and four Orbitrap instruments. Samples included yeast lysate, the NCI-20 defined dynamic range protein mix, and the Sigma UPS 1 defined equimolar protein mix. Some of our findings reinforced conventional wisdom, such as repeatability and reproducibility being higher for proteins than for peptides. Most lessons from the data, however, were more subtle. Orbitraps proved capable of higher repeatability and reproducibility, but aberrant performance occasionally erased these gains. Even the simplest protein digestions yielded more peptide ions than LC-MS/MS could identify during a single experiment. We observed that peptide lists from pairs of technical replicates overlapped by 35-60%, giving a range for peptide-level repeatability in these experiments. Sample complexity did not appear to affect peptide identification repeatability, even as numbers of identified spectra changed by an order of magnitude. Statistical analysis of protein spectral counts revealed greater stability across technical replicates for Orbitraps, making them superior to LTQ instruments for biomarker candidate discovery. The most repeatable peptides were those corresponding to conventional tryptic cleavage sites, those that produced intense MS signals, and those that resulted from proteins generating many distinct peptides. Reproducibility among different instruments of the same type lagged behind repeatability of technical replicates on a single instrument by several percent. These findings reinforce the importance of evaluating repeatability as a fundamental characteristic of analytical technologies.

Original languageEnglish (US)
Pages (from-to)761-776
Number of pages16
JournalJournal of Proteome Research
Volume9
Issue number2
DOIs
StatePublished - Feb 5 2010

Fingerprint

Liquid chromatography
Tandem Mass Spectrometry
Liquid Chromatography
Proteomics
Mass spectrometry
Peptides
Proteins
Biomedical Technology Assessment
Experiments
Biomarkers
Yeast
Proteolysis
Statistical methods
Yeasts
Ions
Sampling
Technology

Keywords

  • Biomarkers
  • Complexity
  • Repeatability
  • Reproducibility
  • Sampling

ASJC Scopus subject areas

  • Biochemistry
  • Chemistry(all)

Cite this

Tabb, D. L., Vega-Montoto, L., Rudnick, P. A., Variyath, A. M., Ham, A. J. L., Bunk, D. M., ... Spiegelman, C. (2010). Repeatability and reproducibility in proteomic identifications by liquid chromatography-tandem mass spectrometry. Journal of Proteome Research, 9(2), 761-776. https://doi.org/10.1021/pr9006365

Repeatability and reproducibility in proteomic identifications by liquid chromatography-tandem mass spectrometry. / Tabb, David L.; Vega-Montoto, Lorenzo; Rudnick, Paul A.; Variyath, Asokan Mulayath; Ham, Amy Joan L; Bunk, David M.; Kilpatrick, Lisa E.; Billheimer, David D; Blackman, Ronald K.; Cardasis, Helene L.; Carr, Steven A.; Clauser, Karl R.; Jaffe, Jacob D.; Kowalski, Kevin A.; Neubert, Thomas A.; Regnier, Fred E.; Schilling, Birgit; Tegeler, Tony J.; Wang, Mu; Wang, Pei; Whiteaker, Jeffrey R.; Zimmerman, Lisa J.; Fisher, Susan J.; Gibson, Bradford W.; Kinsinger, Christopher R.; Mesri, Mehdi; Rodriguez, Henry; Stein, Stephen E.; Tempst, Paul; Paulovich, Amanda G.; Liebler, Daniel C.; Spiegelman, Cliff.

In: Journal of Proteome Research, Vol. 9, No. 2, 05.02.2010, p. 761-776.

Research output: Contribution to journalArticle

Tabb, DL, Vega-Montoto, L, Rudnick, PA, Variyath, AM, Ham, AJL, Bunk, DM, Kilpatrick, LE, Billheimer, DD, Blackman, RK, Cardasis, HL, Carr, SA, Clauser, KR, Jaffe, JD, Kowalski, KA, Neubert, TA, Regnier, FE, Schilling, B, Tegeler, TJ, Wang, M, Wang, P, Whiteaker, JR, Zimmerman, LJ, Fisher, SJ, Gibson, BW, Kinsinger, CR, Mesri, M, Rodriguez, H, Stein, SE, Tempst, P, Paulovich, AG, Liebler, DC & Spiegelman, C 2010, 'Repeatability and reproducibility in proteomic identifications by liquid chromatography-tandem mass spectrometry', Journal of Proteome Research, vol. 9, no. 2, pp. 761-776. https://doi.org/10.1021/pr9006365
Tabb, David L. ; Vega-Montoto, Lorenzo ; Rudnick, Paul A. ; Variyath, Asokan Mulayath ; Ham, Amy Joan L ; Bunk, David M. ; Kilpatrick, Lisa E. ; Billheimer, David D ; Blackman, Ronald K. ; Cardasis, Helene L. ; Carr, Steven A. ; Clauser, Karl R. ; Jaffe, Jacob D. ; Kowalski, Kevin A. ; Neubert, Thomas A. ; Regnier, Fred E. ; Schilling, Birgit ; Tegeler, Tony J. ; Wang, Mu ; Wang, Pei ; Whiteaker, Jeffrey R. ; Zimmerman, Lisa J. ; Fisher, Susan J. ; Gibson, Bradford W. ; Kinsinger, Christopher R. ; Mesri, Mehdi ; Rodriguez, Henry ; Stein, Stephen E. ; Tempst, Paul ; Paulovich, Amanda G. ; Liebler, Daniel C. ; Spiegelman, Cliff. / Repeatability and reproducibility in proteomic identifications by liquid chromatography-tandem mass spectrometry. In: Journal of Proteome Research. 2010 ; Vol. 9, No. 2. pp. 761-776.
@article{5b5b4851aa8d42b69ece1a347385b71e,
title = "Repeatability and reproducibility in proteomic identifications by liquid chromatography-tandem mass spectrometry",
abstract = "The complexity of proteomic instrumentation for LC-MS/MS introduces many possible sources of variability. Data-dependent sampling of peptides constitutes a stochastic element at the heart of discovery proteomics. Although this variation impacts the identification of peptides, proteomic identifications are far from completely random. In this study, we analyzed interlaboratory data sets from the NCI Clinical Proteomic Technology Assessment for Cancer to examine repeatability and reproducibility in peptide and protein identifications. Included data spanned 144 LC-MS/MS experiments on four Thermo LTQ and four Orbitrap instruments. Samples included yeast lysate, the NCI-20 defined dynamic range protein mix, and the Sigma UPS 1 defined equimolar protein mix. Some of our findings reinforced conventional wisdom, such as repeatability and reproducibility being higher for proteins than for peptides. Most lessons from the data, however, were more subtle. Orbitraps proved capable of higher repeatability and reproducibility, but aberrant performance occasionally erased these gains. Even the simplest protein digestions yielded more peptide ions than LC-MS/MS could identify during a single experiment. We observed that peptide lists from pairs of technical replicates overlapped by 35-60{\%}, giving a range for peptide-level repeatability in these experiments. Sample complexity did not appear to affect peptide identification repeatability, even as numbers of identified spectra changed by an order of magnitude. Statistical analysis of protein spectral counts revealed greater stability across technical replicates for Orbitraps, making them superior to LTQ instruments for biomarker candidate discovery. The most repeatable peptides were those corresponding to conventional tryptic cleavage sites, those that produced intense MS signals, and those that resulted from proteins generating many distinct peptides. Reproducibility among different instruments of the same type lagged behind repeatability of technical replicates on a single instrument by several percent. These findings reinforce the importance of evaluating repeatability as a fundamental characteristic of analytical technologies.",
keywords = "Biomarkers, Complexity, Repeatability, Reproducibility, Sampling",
author = "Tabb, {David L.} and Lorenzo Vega-Montoto and Rudnick, {Paul A.} and Variyath, {Asokan Mulayath} and Ham, {Amy Joan L} and Bunk, {David M.} and Kilpatrick, {Lisa E.} and Billheimer, {David D} and Blackman, {Ronald K.} and Cardasis, {Helene L.} and Carr, {Steven A.} and Clauser, {Karl R.} and Jaffe, {Jacob D.} and Kowalski, {Kevin A.} and Neubert, {Thomas A.} and Regnier, {Fred E.} and Birgit Schilling and Tegeler, {Tony J.} and Mu Wang and Pei Wang and Whiteaker, {Jeffrey R.} and Zimmerman, {Lisa J.} and Fisher, {Susan J.} and Gibson, {Bradford W.} and Kinsinger, {Christopher R.} and Mehdi Mesri and Henry Rodriguez and Stein, {Stephen E.} and Paul Tempst and Paulovich, {Amanda G.} and Liebler, {Daniel C.} and Cliff Spiegelman",
year = "2010",
month = "2",
day = "5",
doi = "10.1021/pr9006365",
language = "English (US)",
volume = "9",
pages = "761--776",
journal = "Journal of Proteome Research",
issn = "1535-3893",
publisher = "American Chemical Society",
number = "2",

}

TY - JOUR

T1 - Repeatability and reproducibility in proteomic identifications by liquid chromatography-tandem mass spectrometry

AU - Tabb, David L.

AU - Vega-Montoto, Lorenzo

AU - Rudnick, Paul A.

AU - Variyath, Asokan Mulayath

AU - Ham, Amy Joan L

AU - Bunk, David M.

AU - Kilpatrick, Lisa E.

AU - Billheimer, David D

AU - Blackman, Ronald K.

AU - Cardasis, Helene L.

AU - Carr, Steven A.

AU - Clauser, Karl R.

AU - Jaffe, Jacob D.

AU - Kowalski, Kevin A.

AU - Neubert, Thomas A.

AU - Regnier, Fred E.

AU - Schilling, Birgit

AU - Tegeler, Tony J.

AU - Wang, Mu

AU - Wang, Pei

AU - Whiteaker, Jeffrey R.

AU - Zimmerman, Lisa J.

AU - Fisher, Susan J.

AU - Gibson, Bradford W.

AU - Kinsinger, Christopher R.

AU - Mesri, Mehdi

AU - Rodriguez, Henry

AU - Stein, Stephen E.

AU - Tempst, Paul

AU - Paulovich, Amanda G.

AU - Liebler, Daniel C.

AU - Spiegelman, Cliff

PY - 2010/2/5

Y1 - 2010/2/5

N2 - The complexity of proteomic instrumentation for LC-MS/MS introduces many possible sources of variability. Data-dependent sampling of peptides constitutes a stochastic element at the heart of discovery proteomics. Although this variation impacts the identification of peptides, proteomic identifications are far from completely random. In this study, we analyzed interlaboratory data sets from the NCI Clinical Proteomic Technology Assessment for Cancer to examine repeatability and reproducibility in peptide and protein identifications. Included data spanned 144 LC-MS/MS experiments on four Thermo LTQ and four Orbitrap instruments. Samples included yeast lysate, the NCI-20 defined dynamic range protein mix, and the Sigma UPS 1 defined equimolar protein mix. Some of our findings reinforced conventional wisdom, such as repeatability and reproducibility being higher for proteins than for peptides. Most lessons from the data, however, were more subtle. Orbitraps proved capable of higher repeatability and reproducibility, but aberrant performance occasionally erased these gains. Even the simplest protein digestions yielded more peptide ions than LC-MS/MS could identify during a single experiment. We observed that peptide lists from pairs of technical replicates overlapped by 35-60%, giving a range for peptide-level repeatability in these experiments. Sample complexity did not appear to affect peptide identification repeatability, even as numbers of identified spectra changed by an order of magnitude. Statistical analysis of protein spectral counts revealed greater stability across technical replicates for Orbitraps, making them superior to LTQ instruments for biomarker candidate discovery. The most repeatable peptides were those corresponding to conventional tryptic cleavage sites, those that produced intense MS signals, and those that resulted from proteins generating many distinct peptides. Reproducibility among different instruments of the same type lagged behind repeatability of technical replicates on a single instrument by several percent. These findings reinforce the importance of evaluating repeatability as a fundamental characteristic of analytical technologies.

AB - The complexity of proteomic instrumentation for LC-MS/MS introduces many possible sources of variability. Data-dependent sampling of peptides constitutes a stochastic element at the heart of discovery proteomics. Although this variation impacts the identification of peptides, proteomic identifications are far from completely random. In this study, we analyzed interlaboratory data sets from the NCI Clinical Proteomic Technology Assessment for Cancer to examine repeatability and reproducibility in peptide and protein identifications. Included data spanned 144 LC-MS/MS experiments on four Thermo LTQ and four Orbitrap instruments. Samples included yeast lysate, the NCI-20 defined dynamic range protein mix, and the Sigma UPS 1 defined equimolar protein mix. Some of our findings reinforced conventional wisdom, such as repeatability and reproducibility being higher for proteins than for peptides. Most lessons from the data, however, were more subtle. Orbitraps proved capable of higher repeatability and reproducibility, but aberrant performance occasionally erased these gains. Even the simplest protein digestions yielded more peptide ions than LC-MS/MS could identify during a single experiment. We observed that peptide lists from pairs of technical replicates overlapped by 35-60%, giving a range for peptide-level repeatability in these experiments. Sample complexity did not appear to affect peptide identification repeatability, even as numbers of identified spectra changed by an order of magnitude. Statistical analysis of protein spectral counts revealed greater stability across technical replicates for Orbitraps, making them superior to LTQ instruments for biomarker candidate discovery. The most repeatable peptides were those corresponding to conventional tryptic cleavage sites, those that produced intense MS signals, and those that resulted from proteins generating many distinct peptides. Reproducibility among different instruments of the same type lagged behind repeatability of technical replicates on a single instrument by several percent. These findings reinforce the importance of evaluating repeatability as a fundamental characteristic of analytical technologies.

KW - Biomarkers

KW - Complexity

KW - Repeatability

KW - Reproducibility

KW - Sampling

UR - http://www.scopus.com/inward/record.url?scp=76149142168&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=76149142168&partnerID=8YFLogxK

U2 - 10.1021/pr9006365

DO - 10.1021/pr9006365

M3 - Article

VL - 9

SP - 761

EP - 776

JO - Journal of Proteome Research

JF - Journal of Proteome Research

SN - 1535-3893

IS - 2

ER -