More on the language-as-fixed-effect fallacy

Monte Carlo estimates of error rates for F1,F2,F′, and min F′

Kenneth I Forster, R. G. Dickinson

Research output: Contribution to journalArticle

74 Citations (Scopus)

Abstract

The properties of four different tests of the treatment effect in experiments using linguistic materials are examined using Monte Carlo procedures for estimating Type I error rates. It is shown that: (a) in extreme cases, the Type I error rates for F1 and F2 can exceed the desired rate by a factor of at least 10; (b) minF′ tends to be a very close estimate of F′; (c) both minF′ and F′ are very conservative tests when between item variance or subject-by-treatment variance is low; (d) requiring both F1 and F2 to be significant before H0 is rejected does not prevent the nominal Type I error rate from being exceeded; (e) most of these problems can be minimized by using multistage decision rules which select the most appropriate test on the basis of preliminary tests of item variance and subject-by-treatment variance.

Original languageEnglish (US)
Pages (from-to)135-142
Number of pages8
JournalJournal of Verbal Learning and Verbal Behavior
Volume15
Issue number2
DOIs
StatePublished - 1976
Externally publishedYes

Fingerprint

Linguistics
Language
language
linguistics
Fixed Effects
Fallacies
experiment
Experiments

Cite this

@article{da067507b0c6407095303b37e5e512bc,
title = "More on the language-as-fixed-effect fallacy: Monte Carlo estimates of error rates for F1,F2,F′, and min F′",
abstract = "The properties of four different tests of the treatment effect in experiments using linguistic materials are examined using Monte Carlo procedures for estimating Type I error rates. It is shown that: (a) in extreme cases, the Type I error rates for F1 and F2 can exceed the desired rate by a factor of at least 10; (b) minF′ tends to be a very close estimate of F′; (c) both minF′ and F′ are very conservative tests when between item variance or subject-by-treatment variance is low; (d) requiring both F1 and F2 to be significant before H0 is rejected does not prevent the nominal Type I error rate from being exceeded; (e) most of these problems can be minimized by using multistage decision rules which select the most appropriate test on the basis of preliminary tests of item variance and subject-by-treatment variance.",
author = "Forster, {Kenneth I} and Dickinson, {R. G.}",
year = "1976",
doi = "10.1016/0022-5371(76)90014-1",
language = "English (US)",
volume = "15",
pages = "135--142",
journal = "Journal of Memory and Language",
issn = "0749-596X",
publisher = "Academic Press Inc.",
number = "2",

}

TY - JOUR

T1 - More on the language-as-fixed-effect fallacy

T2 - Monte Carlo estimates of error rates for F1,F2,F′, and min F′

AU - Forster, Kenneth I

AU - Dickinson, R. G.

PY - 1976

Y1 - 1976

N2 - The properties of four different tests of the treatment effect in experiments using linguistic materials are examined using Monte Carlo procedures for estimating Type I error rates. It is shown that: (a) in extreme cases, the Type I error rates for F1 and F2 can exceed the desired rate by a factor of at least 10; (b) minF′ tends to be a very close estimate of F′; (c) both minF′ and F′ are very conservative tests when between item variance or subject-by-treatment variance is low; (d) requiring both F1 and F2 to be significant before H0 is rejected does not prevent the nominal Type I error rate from being exceeded; (e) most of these problems can be minimized by using multistage decision rules which select the most appropriate test on the basis of preliminary tests of item variance and subject-by-treatment variance.

AB - The properties of four different tests of the treatment effect in experiments using linguistic materials are examined using Monte Carlo procedures for estimating Type I error rates. It is shown that: (a) in extreme cases, the Type I error rates for F1 and F2 can exceed the desired rate by a factor of at least 10; (b) minF′ tends to be a very close estimate of F′; (c) both minF′ and F′ are very conservative tests when between item variance or subject-by-treatment variance is low; (d) requiring both F1 and F2 to be significant before H0 is rejected does not prevent the nominal Type I error rate from being exceeded; (e) most of these problems can be minimized by using multistage decision rules which select the most appropriate test on the basis of preliminary tests of item variance and subject-by-treatment variance.

UR - http://www.scopus.com/inward/record.url?scp=0001431847&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0001431847&partnerID=8YFLogxK

U2 - 10.1016/0022-5371(76)90014-1

DO - 10.1016/0022-5371(76)90014-1

M3 - Article

VL - 15

SP - 135

EP - 142

JO - Journal of Memory and Language

JF - Journal of Memory and Language

SN - 0749-596X

IS - 2

ER -