Why meta-analysis doesn't tell us what the data really mean

distinguishing between moderator effects and moderator processes

Craig J. Russell, Stephen W Gilliland

Research output: Contribution to journalArticle

16 Citations (Scopus)

Abstract

Traditional approaches to detecting the presence of moderators in meta-analyses involve inferences drawn from the residual variance in criterion-related validities (ri) after correcting for sampling error and statistical artifacts. James, Demaree, Mulaik, and Ladd (1992) argued that these residualized interpretations of meta-analytic results may be spurious when statistical artifacts covary with true moderators. We extend their model to suggest that situational moderators might also covary with sample size and content (i.e., nonrandom sample selection error), causing meta-analysis to be uninterpretable and a significant correlation between criterion-related validities and ni. We investigate this possibility on studies examining criterion-related validities of peer nominations originally reported by Kane and Lawler (1978). Application of residualized meta-analysis suggests the presence of moderator effects, but a significant correlation between ri and ni precludes interpretation of the moderator process behind these effects. More generally, we argue that the nature of true contingencies cannot be inferred from meta-analytic summaries of traditional criterion-related validity studies. Primary research with appropriate controls is the only means of identifying true moderator effects and processes on criterion-related validity.

Original languageEnglish (US)
Pages (from-to)813-831
Number of pages19
JournalJournal of Management
Volume21
Issue number4
DOIs
StatePublished - 1995
Externally publishedYes

Fingerprint

Moderator effects
Meta-analysis
Moderator
Contingency
Sampling
Peers
Inference
Sample selection
Sample size

ASJC Scopus subject areas

  • Strategy and Management
  • Finance

Cite this

@article{fb7172ab77de4f4983db85b238e507e5,
title = "Why meta-analysis doesn't tell us what the data really mean: distinguishing between moderator effects and moderator processes",
abstract = "Traditional approaches to detecting the presence of moderators in meta-analyses involve inferences drawn from the residual variance in criterion-related validities (ri) after correcting for sampling error and statistical artifacts. James, Demaree, Mulaik, and Ladd (1992) argued that these residualized interpretations of meta-analytic results may be spurious when statistical artifacts covary with true moderators. We extend their model to suggest that situational moderators might also covary with sample size and content (i.e., nonrandom sample selection error), causing meta-analysis to be uninterpretable and a significant correlation between criterion-related validities and ni. We investigate this possibility on studies examining criterion-related validities of peer nominations originally reported by Kane and Lawler (1978). Application of residualized meta-analysis suggests the presence of moderator effects, but a significant correlation between ri and ni precludes interpretation of the moderator process behind these effects. More generally, we argue that the nature of true contingencies cannot be inferred from meta-analytic summaries of traditional criterion-related validity studies. Primary research with appropriate controls is the only means of identifying true moderator effects and processes on criterion-related validity.",
author = "Russell, {Craig J.} and Gilliland, {Stephen W}",
year = "1995",
doi = "10.1016/0149-2063(95)90012-8",
language = "English (US)",
volume = "21",
pages = "813--831",
journal = "Journal of Management",
issn = "0149-2063",
publisher = "SAGE Publications Inc.",
number = "4",

}

TY - JOUR

T1 - Why meta-analysis doesn't tell us what the data really mean

T2 - distinguishing between moderator effects and moderator processes

AU - Russell, Craig J.

AU - Gilliland, Stephen W

PY - 1995

Y1 - 1995

N2 - Traditional approaches to detecting the presence of moderators in meta-analyses involve inferences drawn from the residual variance in criterion-related validities (ri) after correcting for sampling error and statistical artifacts. James, Demaree, Mulaik, and Ladd (1992) argued that these residualized interpretations of meta-analytic results may be spurious when statistical artifacts covary with true moderators. We extend their model to suggest that situational moderators might also covary with sample size and content (i.e., nonrandom sample selection error), causing meta-analysis to be uninterpretable and a significant correlation between criterion-related validities and ni. We investigate this possibility on studies examining criterion-related validities of peer nominations originally reported by Kane and Lawler (1978). Application of residualized meta-analysis suggests the presence of moderator effects, but a significant correlation between ri and ni precludes interpretation of the moderator process behind these effects. More generally, we argue that the nature of true contingencies cannot be inferred from meta-analytic summaries of traditional criterion-related validity studies. Primary research with appropriate controls is the only means of identifying true moderator effects and processes on criterion-related validity.

AB - Traditional approaches to detecting the presence of moderators in meta-analyses involve inferences drawn from the residual variance in criterion-related validities (ri) after correcting for sampling error and statistical artifacts. James, Demaree, Mulaik, and Ladd (1992) argued that these residualized interpretations of meta-analytic results may be spurious when statistical artifacts covary with true moderators. We extend their model to suggest that situational moderators might also covary with sample size and content (i.e., nonrandom sample selection error), causing meta-analysis to be uninterpretable and a significant correlation between criterion-related validities and ni. We investigate this possibility on studies examining criterion-related validities of peer nominations originally reported by Kane and Lawler (1978). Application of residualized meta-analysis suggests the presence of moderator effects, but a significant correlation between ri and ni precludes interpretation of the moderator process behind these effects. More generally, we argue that the nature of true contingencies cannot be inferred from meta-analytic summaries of traditional criterion-related validity studies. Primary research with appropriate controls is the only means of identifying true moderator effects and processes on criterion-related validity.

UR - http://www.scopus.com/inward/record.url?scp=0001430686&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0001430686&partnerID=8YFLogxK

U2 - 10.1016/0149-2063(95)90012-8

DO - 10.1016/0149-2063(95)90012-8

M3 - Article

VL - 21

SP - 813

EP - 831

JO - Journal of Management

JF - Journal of Management

SN - 0149-2063

IS - 4

ER -