Ground truth construction and parameter tuning for the detection of sleep spindle timing in rodents

Blaine Harper, Jean-Marc Fellous

Research output: Contribution to journalArticle

Abstract

Background: The precise detection of cortical sleep spindles is critical to basic research on memory consolidation in rodents. Previous research using automatic spindle detection algorithms often lacks systematic parameter variations and validations. New Method: We present a method to systematically tune and validate algorithm parameters in automatic spindle detection algorithms using a moderate number of human raters. Results: Comparing a Hilbert transform-based algorithm to a ground truth constructed by six human raters, this method produced a parameter set yielding an F1 score of 0.82 at 10 ms resolution. The algorithm performance fell within the range of human agreement with the ground truth. Both human and algorithm failures arose largely from disagreement in spindle boundaries rather than spindle occurrence. With no additional tuning, the algorithm performed similarly in recordings from different days or rats. Comparison with existing methods: Most spindle detection algorithms do not perform systematic parameter variations and validation using a ground truth. To our knowledge, our study is the first in which rodent spindle data is scored by humans, and in which an automatic spindle detection algorithm is evaluated with respect to this ground truth. The rodent data from this study make it possible to compare our algorithm with others previously tested on human data. Conclusions: We present a general ground truth based approach for the tuning and validation of spindle extraction algorithms and suggest that algorithms aimed at extracting precise spindle timing in rats should use a systematic approach for parameter tuning.

Original languageEnglish (US)
Pages (from-to)13-23
Number of pages11
JournalJournal of Neuroscience Methods
Volume313
DOIs
StatePublished - Feb 1 2019

Fingerprint

Rodentia
Sleep
Research

Keywords

  • Consolidation
  • Event detection
  • Rat
  • Sleep spindle
  • Slow wave sleep

ASJC Scopus subject areas

  • Neuroscience(all)

Cite this

Ground truth construction and parameter tuning for the detection of sleep spindle timing in rodents. / Harper, Blaine; Fellous, Jean-Marc.

In: Journal of Neuroscience Methods, Vol. 313, 01.02.2019, p. 13-23.

Research output: Contribution to journalArticle

@article{9beaea853c864148b85a92c9ba883977,
title = "Ground truth construction and parameter tuning for the detection of sleep spindle timing in rodents",
abstract = "Background: The precise detection of cortical sleep spindles is critical to basic research on memory consolidation in rodents. Previous research using automatic spindle detection algorithms often lacks systematic parameter variations and validations. New Method: We present a method to systematically tune and validate algorithm parameters in automatic spindle detection algorithms using a moderate number of human raters. Results: Comparing a Hilbert transform-based algorithm to a ground truth constructed by six human raters, this method produced a parameter set yielding an F1 score of 0.82 at 10 ms resolution. The algorithm performance fell within the range of human agreement with the ground truth. Both human and algorithm failures arose largely from disagreement in spindle boundaries rather than spindle occurrence. With no additional tuning, the algorithm performed similarly in recordings from different days or rats. Comparison with existing methods: Most spindle detection algorithms do not perform systematic parameter variations and validation using a ground truth. To our knowledge, our study is the first in which rodent spindle data is scored by humans, and in which an automatic spindle detection algorithm is evaluated with respect to this ground truth. The rodent data from this study make it possible to compare our algorithm with others previously tested on human data. Conclusions: We present a general ground truth based approach for the tuning and validation of spindle extraction algorithms and suggest that algorithms aimed at extracting precise spindle timing in rats should use a systematic approach for parameter tuning.",
keywords = "Consolidation, Event detection, Rat, Sleep spindle, Slow wave sleep",
author = "Blaine Harper and Jean-Marc Fellous",
year = "2019",
month = "2",
day = "1",
doi = "10.1016/j.jneumeth.2018.11.023",
language = "English (US)",
volume = "313",
pages = "13--23",
journal = "Journal of Neuroscience Methods",
issn = "0165-0270",
publisher = "Elsevier",

}

TY - JOUR

T1 - Ground truth construction and parameter tuning for the detection of sleep spindle timing in rodents

AU - Harper, Blaine

AU - Fellous, Jean-Marc

PY - 2019/2/1

Y1 - 2019/2/1

N2 - Background: The precise detection of cortical sleep spindles is critical to basic research on memory consolidation in rodents. Previous research using automatic spindle detection algorithms often lacks systematic parameter variations and validations. New Method: We present a method to systematically tune and validate algorithm parameters in automatic spindle detection algorithms using a moderate number of human raters. Results: Comparing a Hilbert transform-based algorithm to a ground truth constructed by six human raters, this method produced a parameter set yielding an F1 score of 0.82 at 10 ms resolution. The algorithm performance fell within the range of human agreement with the ground truth. Both human and algorithm failures arose largely from disagreement in spindle boundaries rather than spindle occurrence. With no additional tuning, the algorithm performed similarly in recordings from different days or rats. Comparison with existing methods: Most spindle detection algorithms do not perform systematic parameter variations and validation using a ground truth. To our knowledge, our study is the first in which rodent spindle data is scored by humans, and in which an automatic spindle detection algorithm is evaluated with respect to this ground truth. The rodent data from this study make it possible to compare our algorithm with others previously tested on human data. Conclusions: We present a general ground truth based approach for the tuning and validation of spindle extraction algorithms and suggest that algorithms aimed at extracting precise spindle timing in rats should use a systematic approach for parameter tuning.

AB - Background: The precise detection of cortical sleep spindles is critical to basic research on memory consolidation in rodents. Previous research using automatic spindle detection algorithms often lacks systematic parameter variations and validations. New Method: We present a method to systematically tune and validate algorithm parameters in automatic spindle detection algorithms using a moderate number of human raters. Results: Comparing a Hilbert transform-based algorithm to a ground truth constructed by six human raters, this method produced a parameter set yielding an F1 score of 0.82 at 10 ms resolution. The algorithm performance fell within the range of human agreement with the ground truth. Both human and algorithm failures arose largely from disagreement in spindle boundaries rather than spindle occurrence. With no additional tuning, the algorithm performed similarly in recordings from different days or rats. Comparison with existing methods: Most spindle detection algorithms do not perform systematic parameter variations and validation using a ground truth. To our knowledge, our study is the first in which rodent spindle data is scored by humans, and in which an automatic spindle detection algorithm is evaluated with respect to this ground truth. The rodent data from this study make it possible to compare our algorithm with others previously tested on human data. Conclusions: We present a general ground truth based approach for the tuning and validation of spindle extraction algorithms and suggest that algorithms aimed at extracting precise spindle timing in rats should use a systematic approach for parameter tuning.

KW - Consolidation

KW - Event detection

KW - Rat

KW - Sleep spindle

KW - Slow wave sleep

UR - http://www.scopus.com/inward/record.url?scp=85058531859&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85058531859&partnerID=8YFLogxK

U2 - 10.1016/j.jneumeth.2018.11.023

DO - 10.1016/j.jneumeth.2018.11.023

M3 - Article

VL - 313

SP - 13

EP - 23

JO - Journal of Neuroscience Methods

JF - Journal of Neuroscience Methods

SN - 0165-0270

ER -