Libra

Improved partitioning strategies for massive comparative metagenomics analysis

Illyoung Choi, Mahew Bomhoff, Alise J. Ponsero, Bonnie L Hurwitz, Ken Youens-Clark, John H Hartman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Big-data analytics platforms, such as Hadoop, are appealing for scientific computation because they are ubiquitous, well-supported, and well-understood. Unfortunately, load-balancing is a common challenge of implementing large-scale scientific computing applications on these platforms. In this paper we present the design and implementation of Libra, a Hadoop-based tool for comparative metagenomics (comparing samples of genetic material collected from the environment). We describe the computation that Libra performs and how that computation is implemented using Hadoop tasks, including the techniques used by Libra to ensure that the task workloads are balanced despite nonuniform sample sizes and skewed distributions of genetic material in the samples. On a 10-machine Hadoop cluster Libra can analyze the entire Tara Ocean Viromes of ~4.2 billion reads in fewer than 20 hours.

Original languageEnglish (US)
Title of host publicationProceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450358637
DOIs
StatePublished - Jun 11 2018
Event9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Tempe, United States
Duration: Jun 11 2018 → …

Other

Other9th Workshop on Scientific Cloud Computing, ScienceCloud 2018
CountryUnited States
CityTempe
Period6/11/18 → …

Fingerprint

Natural sciences computing
Resource allocation
Big data

Keywords

  • Comparative genomics
  • Genome distance
  • K-mer
  • Metagenomics
  • Parallel
  • Task-partitioning

ASJC Scopus subject areas

  • Computer Science Applications
  • Software
  • Computational Theory and Mathematics

Cite this

Choi, I., Bomhoff, M., Ponsero, A. J., Hurwitz, B. L., Youens-Clark, K., & Hartman, J. H. (2018). Libra: Improved partitioning strategies for massive comparative metagenomics analysis. In Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018 [a2] Association for Computing Machinery, Inc. https://doi.org/10.1145/3217880.3217882

Libra : Improved partitioning strategies for massive comparative metagenomics analysis. / Choi, Illyoung; Bomhoff, Mahew; Ponsero, Alise J.; Hurwitz, Bonnie L; Youens-Clark, Ken; Hartman, John H.

Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018. Association for Computing Machinery, Inc, 2018. a2.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Choi, I, Bomhoff, M, Ponsero, AJ, Hurwitz, BL, Youens-Clark, K & Hartman, JH 2018, Libra: Improved partitioning strategies for massive comparative metagenomics analysis. in Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018., a2, Association for Computing Machinery, Inc, 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018, Tempe, United States, 6/11/18. https://doi.org/10.1145/3217880.3217882
Choi I, Bomhoff M, Ponsero AJ, Hurwitz BL, Youens-Clark K, Hartman JH. Libra: Improved partitioning strategies for massive comparative metagenomics analysis. In Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018. Association for Computing Machinery, Inc. 2018. a2 https://doi.org/10.1145/3217880.3217882
Choi, Illyoung ; Bomhoff, Mahew ; Ponsero, Alise J. ; Hurwitz, Bonnie L ; Youens-Clark, Ken ; Hartman, John H. / Libra : Improved partitioning strategies for massive comparative metagenomics analysis. Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018. Association for Computing Machinery, Inc, 2018.
@inproceedings{fbce02285def45e8b2038de7708109b8,
title = "Libra: Improved partitioning strategies for massive comparative metagenomics analysis",
abstract = "Big-data analytics platforms, such as Hadoop, are appealing for scientific computation because they are ubiquitous, well-supported, and well-understood. Unfortunately, load-balancing is a common challenge of implementing large-scale scientific computing applications on these platforms. In this paper we present the design and implementation of Libra, a Hadoop-based tool for comparative metagenomics (comparing samples of genetic material collected from the environment). We describe the computation that Libra performs and how that computation is implemented using Hadoop tasks, including the techniques used by Libra to ensure that the task workloads are balanced despite nonuniform sample sizes and skewed distributions of genetic material in the samples. On a 10-machine Hadoop cluster Libra can analyze the entire Tara Ocean Viromes of ~4.2 billion reads in fewer than 20 hours.",
keywords = "Comparative genomics, Genome distance, K-mer, Metagenomics, Parallel, Task-partitioning",
author = "Illyoung Choi and Mahew Bomhoff and Ponsero, {Alise J.} and Hurwitz, {Bonnie L} and Ken Youens-Clark and Hartman, {John H}",
year = "2018",
month = "6",
day = "11",
doi = "10.1145/3217880.3217882",
language = "English (US)",
booktitle = "Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018",
publisher = "Association for Computing Machinery, Inc",

}

TY - GEN

T1 - Libra

T2 - Improved partitioning strategies for massive comparative metagenomics analysis

AU - Choi, Illyoung

AU - Bomhoff, Mahew

AU - Ponsero, Alise J.

AU - Hurwitz, Bonnie L

AU - Youens-Clark, Ken

AU - Hartman, John H

PY - 2018/6/11

Y1 - 2018/6/11

N2 - Big-data analytics platforms, such as Hadoop, are appealing for scientific computation because they are ubiquitous, well-supported, and well-understood. Unfortunately, load-balancing is a common challenge of implementing large-scale scientific computing applications on these platforms. In this paper we present the design and implementation of Libra, a Hadoop-based tool for comparative metagenomics (comparing samples of genetic material collected from the environment). We describe the computation that Libra performs and how that computation is implemented using Hadoop tasks, including the techniques used by Libra to ensure that the task workloads are balanced despite nonuniform sample sizes and skewed distributions of genetic material in the samples. On a 10-machine Hadoop cluster Libra can analyze the entire Tara Ocean Viromes of ~4.2 billion reads in fewer than 20 hours.

AB - Big-data analytics platforms, such as Hadoop, are appealing for scientific computation because they are ubiquitous, well-supported, and well-understood. Unfortunately, load-balancing is a common challenge of implementing large-scale scientific computing applications on these platforms. In this paper we present the design and implementation of Libra, a Hadoop-based tool for comparative metagenomics (comparing samples of genetic material collected from the environment). We describe the computation that Libra performs and how that computation is implemented using Hadoop tasks, including the techniques used by Libra to ensure that the task workloads are balanced despite nonuniform sample sizes and skewed distributions of genetic material in the samples. On a 10-machine Hadoop cluster Libra can analyze the entire Tara Ocean Viromes of ~4.2 billion reads in fewer than 20 hours.

KW - Comparative genomics

KW - Genome distance

KW - K-mer

KW - Metagenomics

KW - Parallel

KW - Task-partitioning

UR - http://www.scopus.com/inward/record.url?scp=85050131781&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050131781&partnerID=8YFLogxK

U2 - 10.1145/3217880.3217882

DO - 10.1145/3217880.3217882

M3 - Conference contribution

BT - Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018

PB - Association for Computing Machinery, Inc

ER -