Libra: Improved partitioning strategies for massive comparative metagenomics analysis

Illyoung Choi, Mahew Bomhoff, Alise J. Ponsero, Bonnie L. Hurwitz, Ken Youens-Clark, John H. Hartman

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Big-data analytics platforms, such as Hadoop, are appealing for scientific computation because they are ubiquitous, well-supported, and well-understood. Unfortunately, load-balancing is a common challenge of implementing large-scale scientific computing applications on these platforms. In this paper we present the design and implementation of Libra, a Hadoop-based tool for comparative metagenomics (comparing samples of genetic material collected from the environment). We describe the computation that Libra performs and how that computation is implemented using Hadoop tasks, including the techniques used by Libra to ensure that the task workloads are balanced despite nonuniform sample sizes and skewed distributions of genetic material in the samples. On a 10-machine Hadoop cluster Libra can analyze the entire Tara Ocean Viromes of ~4.2 billion reads in fewer than 20 hours.

Original languageEnglish (US)
Title of host publicationProceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450358637
DOIs
StatePublished - Jun 11 2018
Event9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Tempe, United States
Duration: Jun 11 2018 → …

Publication series

NameProceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018

Other

Other9th Workshop on Scientific Cloud Computing, ScienceCloud 2018
CountryUnited States
CityTempe
Period6/11/18 → …

Keywords

  • Comparative genomics
  • Genome distance
  • K-mer
  • Metagenomics
  • Parallel
  • Task-partitioning

ASJC Scopus subject areas

  • Computer Science Applications
  • Software
  • Computational Theory and Mathematics

Fingerprint Dive into the research topics of 'Libra: Improved partitioning strategies for massive comparative metagenomics analysis'. Together they form a unique fingerprint.

Cite this