Distributed Asynchronous Array Computing with the JetLag Environment

Steven R. Brandt, Bita Hasheminezhad, Nanmiao Wu, Sayef Azad Sakin, Alex R. Bigelow, Katherine E. Isaacs, Kevin Huck, Hartmut Kaiser

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We describe JetLag, a Python-based environment that provides access to a distributed, interactive, asynchronous many-task (AMT) computing framework called Phylanx. This environment encompasses the entire computing process, from a Jupyter front-end for managing code and results to the collection and visualization of performance data.We use a Python decorator to access the abstract syntax tree of Python functions and transpile them into a set of C++ data structures which are then executed by the HPX runtime. The environment includes services for sending functions and their arguments to run as jobs on remote resources.A set of Docker and Singularity containers are used to simplify the setup of the JetLag environment. The JetLag system is suitable for a variety of array computational tasks, including machine learning and exploratory data analysis.

Original languageEnglish (US)
Title of host publicationProceedings of PYHPC 2020
Subtitle of host publication9th Workshop on Python for High-Performance and Scientific Computing, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages49-57
Number of pages9
ISBN (Electronic)9780738110868
DOIs
StatePublished - Nov 2020
Event9th IEEE/ACM Workshop on Python for High-Performance and Scientific Computing, PYHPC 2020 - Virtual, Atlanta, United States
Duration: Nov 13 2020 → …

Publication series

NameProceedings of PYHPC 2020: 9th Workshop on Python for High-Performance and Scientific Computing, Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference9th IEEE/ACM Workshop on Python for High-Performance and Scientific Computing, PYHPC 2020
CountryUnited States
CityVirtual, Atlanta
Period11/13/20 → …

Keywords

  • array computing
  • asynchronous
  • distributed
  • python

ASJC Scopus subject areas

  • Computer Science Applications
  • Software

Fingerprint Dive into the research topics of 'Distributed Asynchronous Array Computing with the JetLag Environment'. Together they form a unique fingerprint.

Cite this