EMP: execution time measurement protocol for compute-bound programs

Young Kyoon Suh, Richard Thomas Snodgrass, John D Kececioglu, Peter J. Downey, Robert S Maier, Cheng Yi

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

Measuring execution time is one of the most used performance evaluation techniques in computer science research. Inaccurate measurements cannot be used for a fair performance comparison between programs. Despite the prevalence of its use, the intrinsic variability in the time measurement makes it hard to obtain repeatable and accurate timing results of a program running on an operating system. We propose a novel execution time measurement protocol (termed EMP) for measuring the execution time of a compute-bound program on Linux, while minimizing that measurement's variability. During the development of execution time measurement protocol, we identified several factors that disturb execution time measurement. We introduce successive refinements to the protocol by addressing each of these factors, in concert, reducing variability by more than an order of magnitude. We also introduce a new visualization technique, what we term ‘dual-execution scatter plot’ that highlights infrequent, long-running daemons, differentiating them from frequent and/or short-running daemons. Our empirical results show that the proposed protocol successfully achieves three major aspects—precision, accuracy, and scalability—in execution time measurement that can work for open-source and proprietary software.

Original languageEnglish (US)
Pages (from-to)559-597
Number of pages39
JournalSoftware - Practice and Experience
Volume47
Issue number4
DOIs
StatePublished - Apr 1 2017

Fingerprint

Time measurement
Network protocols
Computer science
Visualization

Keywords

  • compute-bound programs
  • execution time
  • measurement
  • protocol

ASJC Scopus subject areas

  • Software

Cite this

EMP : execution time measurement protocol for compute-bound programs. / Suh, Young Kyoon; Snodgrass, Richard Thomas; Kececioglu, John D; Downey, Peter J.; Maier, Robert S; Yi, Cheng.

In: Software - Practice and Experience, Vol. 47, No. 4, 01.04.2017, p. 559-597.

Research output: Contribution to journalArticle

@article{06655cd1daad49e6a7838c5a33d0f299,
title = "EMP: execution time measurement protocol for compute-bound programs",
abstract = "Measuring execution time is one of the most used performance evaluation techniques in computer science research. Inaccurate measurements cannot be used for a fair performance comparison between programs. Despite the prevalence of its use, the intrinsic variability in the time measurement makes it hard to obtain repeatable and accurate timing results of a program running on an operating system. We propose a novel execution time measurement protocol (termed EMP) for measuring the execution time of a compute-bound program on Linux, while minimizing that measurement's variability. During the development of execution time measurement protocol, we identified several factors that disturb execution time measurement. We introduce successive refinements to the protocol by addressing each of these factors, in concert, reducing variability by more than an order of magnitude. We also introduce a new visualization technique, what we term ‘dual-execution scatter plot’ that highlights infrequent, long-running daemons, differentiating them from frequent and/or short-running daemons. Our empirical results show that the proposed protocol successfully achieves three major aspects—precision, accuracy, and scalability—in execution time measurement that can work for open-source and proprietary software.",
keywords = "compute-bound programs, execution time, measurement, protocol",
author = "Suh, {Young Kyoon} and Snodgrass, {Richard Thomas} and Kececioglu, {John D} and Downey, {Peter J.} and Maier, {Robert S} and Cheng Yi",
year = "2017",
month = "4",
day = "1",
doi = "10.1002/spe.2476",
language = "English (US)",
volume = "47",
pages = "559--597",
journal = "Software - Practice and Experience",
issn = "0038-0644",
publisher = "John Wiley and Sons Ltd",
number = "4",

}

TY - JOUR

T1 - EMP

T2 - execution time measurement protocol for compute-bound programs

AU - Suh, Young Kyoon

AU - Snodgrass, Richard Thomas

AU - Kececioglu, John D

AU - Downey, Peter J.

AU - Maier, Robert S

AU - Yi, Cheng

PY - 2017/4/1

Y1 - 2017/4/1

N2 - Measuring execution time is one of the most used performance evaluation techniques in computer science research. Inaccurate measurements cannot be used for a fair performance comparison between programs. Despite the prevalence of its use, the intrinsic variability in the time measurement makes it hard to obtain repeatable and accurate timing results of a program running on an operating system. We propose a novel execution time measurement protocol (termed EMP) for measuring the execution time of a compute-bound program on Linux, while minimizing that measurement's variability. During the development of execution time measurement protocol, we identified several factors that disturb execution time measurement. We introduce successive refinements to the protocol by addressing each of these factors, in concert, reducing variability by more than an order of magnitude. We also introduce a new visualization technique, what we term ‘dual-execution scatter plot’ that highlights infrequent, long-running daemons, differentiating them from frequent and/or short-running daemons. Our empirical results show that the proposed protocol successfully achieves three major aspects—precision, accuracy, and scalability—in execution time measurement that can work for open-source and proprietary software.

AB - Measuring execution time is one of the most used performance evaluation techniques in computer science research. Inaccurate measurements cannot be used for a fair performance comparison between programs. Despite the prevalence of its use, the intrinsic variability in the time measurement makes it hard to obtain repeatable and accurate timing results of a program running on an operating system. We propose a novel execution time measurement protocol (termed EMP) for measuring the execution time of a compute-bound program on Linux, while minimizing that measurement's variability. During the development of execution time measurement protocol, we identified several factors that disturb execution time measurement. We introduce successive refinements to the protocol by addressing each of these factors, in concert, reducing variability by more than an order of magnitude. We also introduce a new visualization technique, what we term ‘dual-execution scatter plot’ that highlights infrequent, long-running daemons, differentiating them from frequent and/or short-running daemons. Our empirical results show that the proposed protocol successfully achieves three major aspects—precision, accuracy, and scalability—in execution time measurement that can work for open-source and proprietary software.

KW - compute-bound programs

KW - execution time

KW - measurement

KW - protocol

UR - http://www.scopus.com/inward/record.url?scp=85013674795&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85013674795&partnerID=8YFLogxK

U2 - 10.1002/spe.2476

DO - 10.1002/spe.2476

M3 - Article

AN - SCOPUS:85013674795

VL - 47

SP - 559

EP - 597

JO - Software - Practice and Experience

JF - Software - Practice and Experience

SN - 0038-0644

IS - 4

ER -