On model selection criteria in multimodel analysis

Ming Ye, Philip D. Meyer, Shlomo P Neuman

Research output: Contribution to journalArticle

155 Citations (Scopus)

Abstract

Hydrologic systems are open and complex, rendering them prone to multiple conceptualizations and mathematical descriptions. There has been a growing tendency to postulate several alternative hydrologic models for a site and use model selection criteria to (1) rank these models, (2) eliminate some of them, and/or (3) weigh and average predictions and statistics generated by multiple models. This has led to some debate among hydrogeologists about the merits and demerits of common model selection (also known as model discrimination or information) criteria such as AIC, AICc, BIC, and KIC and some lack of clarity about the proper interpretation and mathematical representation of each criterion. We examine the model selection literature to find that (1) all published rigorous derivations of AIC and AICc require that the (true) model having generated the observational data be in the set of candidate models; (2) though BIC and KIC were originally derived by assuming that such a model is in the set, BIC has been rederived by Cavanaugh and Neath (1999) without the need for such an assumption; and (3) KIC reduces to BIC as the number of observations becomes large relative to the number of adjustable model parameters, implying that it likewise does not require the existence of a true model in the set of alternatives. We explain why KIC is the only criterion accounting validly for the likelihood of prior parameter estimates, elucidate the unique role that the Fisher information matrix plays in KIC, and demonstrate through an example that it imbues KIC with desirable model selection properties not shared by AIC, AICc, or BIC. Our example appears to provide the first comprehensive test of how AIC, AICc, BIC, and KIC weigh and rank alternative models in light of the models' predictive performance under cross validation with real hydrologic data.

Original languageEnglish (US)
Article numberW03428
JournalWater Resources Research
Volume44
Issue number3
DOIs
StatePublished - Mar 2008

Fingerprint

selection criteria
imidazole mustard
analysis
Fisher information matrix
hydrologic data
rendering
Open systems
hydrologic models
statistics

ASJC Scopus subject areas

  • Environmental Science(all)
  • Environmental Chemistry
  • Aquatic Science
  • Water Science and Technology

Cite this

On model selection criteria in multimodel analysis. / Ye, Ming; Meyer, Philip D.; Neuman, Shlomo P.

In: Water Resources Research, Vol. 44, No. 3, W03428, 03.2008.

Research output: Contribution to journalArticle

@article{c7c8e4d10eda4254b779c8170d667b2a,
title = "On model selection criteria in multimodel analysis",
abstract = "Hydrologic systems are open and complex, rendering them prone to multiple conceptualizations and mathematical descriptions. There has been a growing tendency to postulate several alternative hydrologic models for a site and use model selection criteria to (1) rank these models, (2) eliminate some of them, and/or (3) weigh and average predictions and statistics generated by multiple models. This has led to some debate among hydrogeologists about the merits and demerits of common model selection (also known as model discrimination or information) criteria such as AIC, AICc, BIC, and KIC and some lack of clarity about the proper interpretation and mathematical representation of each criterion. We examine the model selection literature to find that (1) all published rigorous derivations of AIC and AICc require that the (true) model having generated the observational data be in the set of candidate models; (2) though BIC and KIC were originally derived by assuming that such a model is in the set, BIC has been rederived by Cavanaugh and Neath (1999) without the need for such an assumption; and (3) KIC reduces to BIC as the number of observations becomes large relative to the number of adjustable model parameters, implying that it likewise does not require the existence of a true model in the set of alternatives. We explain why KIC is the only criterion accounting validly for the likelihood of prior parameter estimates, elucidate the unique role that the Fisher information matrix plays in KIC, and demonstrate through an example that it imbues KIC with desirable model selection properties not shared by AIC, AICc, or BIC. Our example appears to provide the first comprehensive test of how AIC, AICc, BIC, and KIC weigh and rank alternative models in light of the models' predictive performance under cross validation with real hydrologic data.",
author = "Ming Ye and Meyer, {Philip D.} and Neuman, {Shlomo P}",
year = "2008",
month = "3",
doi = "10.1029/2008WR006803",
language = "English (US)",
volume = "44",
journal = "Water Resources Research",
issn = "0043-1397",
publisher = "American Geophysical Union",
number = "3",

}

TY - JOUR

T1 - On model selection criteria in multimodel analysis

AU - Ye, Ming

AU - Meyer, Philip D.

AU - Neuman, Shlomo P

PY - 2008/3

Y1 - 2008/3

N2 - Hydrologic systems are open and complex, rendering them prone to multiple conceptualizations and mathematical descriptions. There has been a growing tendency to postulate several alternative hydrologic models for a site and use model selection criteria to (1) rank these models, (2) eliminate some of them, and/or (3) weigh and average predictions and statistics generated by multiple models. This has led to some debate among hydrogeologists about the merits and demerits of common model selection (also known as model discrimination or information) criteria such as AIC, AICc, BIC, and KIC and some lack of clarity about the proper interpretation and mathematical representation of each criterion. We examine the model selection literature to find that (1) all published rigorous derivations of AIC and AICc require that the (true) model having generated the observational data be in the set of candidate models; (2) though BIC and KIC were originally derived by assuming that such a model is in the set, BIC has been rederived by Cavanaugh and Neath (1999) without the need for such an assumption; and (3) KIC reduces to BIC as the number of observations becomes large relative to the number of adjustable model parameters, implying that it likewise does not require the existence of a true model in the set of alternatives. We explain why KIC is the only criterion accounting validly for the likelihood of prior parameter estimates, elucidate the unique role that the Fisher information matrix plays in KIC, and demonstrate through an example that it imbues KIC with desirable model selection properties not shared by AIC, AICc, or BIC. Our example appears to provide the first comprehensive test of how AIC, AICc, BIC, and KIC weigh and rank alternative models in light of the models' predictive performance under cross validation with real hydrologic data.

AB - Hydrologic systems are open and complex, rendering them prone to multiple conceptualizations and mathematical descriptions. There has been a growing tendency to postulate several alternative hydrologic models for a site and use model selection criteria to (1) rank these models, (2) eliminate some of them, and/or (3) weigh and average predictions and statistics generated by multiple models. This has led to some debate among hydrogeologists about the merits and demerits of common model selection (also known as model discrimination or information) criteria such as AIC, AICc, BIC, and KIC and some lack of clarity about the proper interpretation and mathematical representation of each criterion. We examine the model selection literature to find that (1) all published rigorous derivations of AIC and AICc require that the (true) model having generated the observational data be in the set of candidate models; (2) though BIC and KIC were originally derived by assuming that such a model is in the set, BIC has been rederived by Cavanaugh and Neath (1999) without the need for such an assumption; and (3) KIC reduces to BIC as the number of observations becomes large relative to the number of adjustable model parameters, implying that it likewise does not require the existence of a true model in the set of alternatives. We explain why KIC is the only criterion accounting validly for the likelihood of prior parameter estimates, elucidate the unique role that the Fisher information matrix plays in KIC, and demonstrate through an example that it imbues KIC with desirable model selection properties not shared by AIC, AICc, or BIC. Our example appears to provide the first comprehensive test of how AIC, AICc, BIC, and KIC weigh and rank alternative models in light of the models' predictive performance under cross validation with real hydrologic data.

UR - http://www.scopus.com/inward/record.url?scp=43249109413&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=43249109413&partnerID=8YFLogxK

U2 - 10.1029/2008WR006803

DO - 10.1029/2008WR006803

M3 - Article

AN - SCOPUS:43249109413

VL - 44

JO - Water Resources Research

JF - Water Resources Research

SN - 0043-1397

IS - 3

M1 - W03428

ER -