Why Meta-Analysis Doesn’t Tell Us What the Data Really Mean: Distinguishing between Moderator Effects and Moderator Processes

Craig J. Russell, Stephen W. Gilliland

Research output: Contribution to journalArticle

16 Scopus citations

Abstract

Traditional approaches to detecting the presence of moderators in meta-analyses involve inferences drawn from the residual variance in criterion-related validities (a) after correcting for sampling error and statistical artifacts. James, Demaree, Mulaik, and Ladd (1992) argued that these residualized interpretations of meta-analytic results may be spurious when statistical artifacts covary with true moderators. We extend their model to suggest that situational moderators might also covary with sample size and content (i.e., nonrandom sample selection error), causing meta-analysis to be uninterpretable and a significant correlation between criterion-related validities and ni. We investigate this possibility on studies examining criterion-related validities ofpeer nominations originally reported by Kane and Lawler (1978). Application of residualized meta-analysis suggests the presence of moderator effects, but a significant correlation between ri and ni precludes interpretation of the moderator process behind these effects. More generally, we argue that the nature of true contingencies cannot be inferred from meta-analytic summaries of traditional criterion-related validity studies. Primary research with appropriate controls is the only means of identifying true moderator effects and processes on criterion-related validity.

Original languageEnglish (US)
Pages (from-to)813-831
Number of pages19
JournalJournal of Management
Volume21
Issue number4
DOIs
StatePublished - Aug 1995
Externally publishedYes

ASJC Scopus subject areas

  • Finance
  • Strategy and Management

Fingerprint Dive into the research topics of 'Why Meta-Analysis Doesn’t Tell Us What the Data Really Mean: Distinguishing between Moderator Effects and Moderator Processes'. Together they form a unique fingerprint.

  • Cite this