Exploiting redundancy for cost-effective, time-constrained execution of HPC applications on Amazon EC2

Aniruddha Marathe, Rachel Harris, David K. Lowenthal, Bronis R. De Supinski, Barry Rountree, Martin Schulz

Research output: Chapter in Book/Report/Conference proceedingConference contribution

34 Scopus citations

Abstract

The use of clouds to execute high-performance computing (HPC) applications has greatly increased recently. Clouds provide several potential advantages over traditional supercomputers and in-house clusters. The most popular cloud is currently Amazon EC2, which provides a fixed-cost option (called on-demand) and a variable-cost, auction-based option (called the spot market). The spot market trades lower cost for potential interruptions that necessitate checkpointing; if the market price exceeds the bid price, a node is taken away from the user without warning. We explore techniques to maximize performance per dollar given a time constraint within which an application must complete. Specifically, we design and implement multiple techniques to reduce expected cost by exploiting redundancy in the EC2 spot market. We then design an adaptive algorithm that selects a scheduling algorithm and determines the bid price. We show that our adaptive algorithm executes programs up to 7x cheaper than using the on-demand market and up to 44% cheaper than the best non-redundant, spot-market algorithm.

Original languageEnglish (US)
Title of host publicationHPDC 2014 - Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing
PublisherAssociation for Computing Machinery
Pages279-290
Number of pages12
ISBN (Print)9781450327480
DOIs
StatePublished - Jan 1 2014
Event23rd ACM Symposium on High-Performance Parallel and Distributed Computing, HPDC 2014 - Vancouver, BC, Canada
Duration: Jun 23 2014Jun 27 2014

Publication series

NameHPDC 2014 - Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing

Other

Other23rd ACM Symposium on High-Performance Parallel and Distributed Computing, HPDC 2014
CountryCanada
CityVancouver, BC
Period6/23/146/27/14

    Fingerprint

Keywords

  • Cloud
  • Cost
  • Fault-tolerance
  • Resource provisioning

ASJC Scopus subject areas

  • Software

Cite this

Marathe, A., Harris, R., Lowenthal, D. K., De Supinski, B. R., Rountree, B., & Schulz, M. (2014). Exploiting redundancy for cost-effective, time-constrained execution of HPC applications on Amazon EC2. In HPDC 2014 - Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing (pp. 279-290). (HPDC 2014 - Proceedings of the 23rd International Symposium on High-Performance Parallel and Distributed Computing). Association for Computing Machinery. https://doi.org/10.1145/2600212.2600226