Variance estimation using refitted cross-validation in ultrahigh dimensional regression

Jianqing Fan, Shaojun Guo, Ning - Hao

Research output: Contribution to journalArticle

82 Citations (Scopus)

Abstract

Variance estimation is a fundamental problem in statistical modelling. In ultrahigh dimensional linear regression where the dimensionality is much larger than the sample size, traditional variance estimation techniques are not applicable. Recent advances in variable selection in ultrahigh dimensional linear regression make this problem accessible. One of the major problems in ultrahigh dimensional regression is the high spurious correlation between the unobserved realized noise and some of the predictors. As a result, the realized noises are actually predicted when extra irrelevant variables are selected, leading to a serious underestimate of the level of noise. We propose a two-stage refitted procedure via a data splitting technique, called refitted cross-validation, to attenuate the influence of irrelevant variables with high spurious correlations. Our asymptotic results show that the resulting procedure performs as well as the oracle estimator, which knows in advance the mean regression function. The simulation studies lend further support to our theoretical claims. The naive two-stage estimator and the plug-in one-stage estimators using the lasso and smoothly clipped absolute deviation are also studied and compared. Their performances can be improved by the refitted cross-validation method proposed.

Original languageEnglish (US)
Pages (from-to)37-65
Number of pages29
JournalJournal of the Royal Statistical Society. Series B: Statistical Methodology
Volume74
Issue number1
DOIs
StatePublished - Jan 2012

Fingerprint

Variance Estimation
Cross-validation
Regression
Estimator
Linear regression
Two-stage Procedure
Lasso
Statistical Modeling
Plug-in
Regression Function
Variable Selection
Dimensionality
Predictors
Sample Size
Deviation
Simulation Study
Variance estimation
Spurious correlation

Keywords

  • Data splitting
  • Dimension reduction
  • High dimensionality
  • Refitted cross-validation
  • Sure screening
  • Variable selection
  • Variance estimation

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Cite this

Variance estimation using refitted cross-validation in ultrahigh dimensional regression. / Fan, Jianqing; Guo, Shaojun; Hao, Ning -.

In: Journal of the Royal Statistical Society. Series B: Statistical Methodology, Vol. 74, No. 1, 01.2012, p. 37-65.

Research output: Contribution to journalArticle

@article{da235cdcb203445c9fc2a267dd8157b4,
title = "Variance estimation using refitted cross-validation in ultrahigh dimensional regression",
abstract = "Variance estimation is a fundamental problem in statistical modelling. In ultrahigh dimensional linear regression where the dimensionality is much larger than the sample size, traditional variance estimation techniques are not applicable. Recent advances in variable selection in ultrahigh dimensional linear regression make this problem accessible. One of the major problems in ultrahigh dimensional regression is the high spurious correlation between the unobserved realized noise and some of the predictors. As a result, the realized noises are actually predicted when extra irrelevant variables are selected, leading to a serious underestimate of the level of noise. We propose a two-stage refitted procedure via a data splitting technique, called refitted cross-validation, to attenuate the influence of irrelevant variables with high spurious correlations. Our asymptotic results show that the resulting procedure performs as well as the oracle estimator, which knows in advance the mean regression function. The simulation studies lend further support to our theoretical claims. The naive two-stage estimator and the plug-in one-stage estimators using the lasso and smoothly clipped absolute deviation are also studied and compared. Their performances can be improved by the refitted cross-validation method proposed.",
keywords = "Data splitting, Dimension reduction, High dimensionality, Refitted cross-validation, Sure screening, Variable selection, Variance estimation",
author = "Jianqing Fan and Shaojun Guo and Hao, {Ning -}",
year = "2012",
month = "1",
doi = "10.1111/j.1467-9868.2011.01005.x",
language = "English (US)",
volume = "74",
pages = "37--65",
journal = "Journal of the Royal Statistical Society. Series B: Statistical Methodology",
issn = "1369-7412",
publisher = "Wiley-Blackwell",
number = "1",

}

TY - JOUR

T1 - Variance estimation using refitted cross-validation in ultrahigh dimensional regression

AU - Fan, Jianqing

AU - Guo, Shaojun

AU - Hao, Ning -

PY - 2012/1

Y1 - 2012/1

N2 - Variance estimation is a fundamental problem in statistical modelling. In ultrahigh dimensional linear regression where the dimensionality is much larger than the sample size, traditional variance estimation techniques are not applicable. Recent advances in variable selection in ultrahigh dimensional linear regression make this problem accessible. One of the major problems in ultrahigh dimensional regression is the high spurious correlation between the unobserved realized noise and some of the predictors. As a result, the realized noises are actually predicted when extra irrelevant variables are selected, leading to a serious underestimate of the level of noise. We propose a two-stage refitted procedure via a data splitting technique, called refitted cross-validation, to attenuate the influence of irrelevant variables with high spurious correlations. Our asymptotic results show that the resulting procedure performs as well as the oracle estimator, which knows in advance the mean regression function. The simulation studies lend further support to our theoretical claims. The naive two-stage estimator and the plug-in one-stage estimators using the lasso and smoothly clipped absolute deviation are also studied and compared. Their performances can be improved by the refitted cross-validation method proposed.

AB - Variance estimation is a fundamental problem in statistical modelling. In ultrahigh dimensional linear regression where the dimensionality is much larger than the sample size, traditional variance estimation techniques are not applicable. Recent advances in variable selection in ultrahigh dimensional linear regression make this problem accessible. One of the major problems in ultrahigh dimensional regression is the high spurious correlation between the unobserved realized noise and some of the predictors. As a result, the realized noises are actually predicted when extra irrelevant variables are selected, leading to a serious underestimate of the level of noise. We propose a two-stage refitted procedure via a data splitting technique, called refitted cross-validation, to attenuate the influence of irrelevant variables with high spurious correlations. Our asymptotic results show that the resulting procedure performs as well as the oracle estimator, which knows in advance the mean regression function. The simulation studies lend further support to our theoretical claims. The naive two-stage estimator and the plug-in one-stage estimators using the lasso and smoothly clipped absolute deviation are also studied and compared. Their performances can be improved by the refitted cross-validation method proposed.

KW - Data splitting

KW - Dimension reduction

KW - High dimensionality

KW - Refitted cross-validation

KW - Sure screening

KW - Variable selection

KW - Variance estimation

UR - http://www.scopus.com/inward/record.url?scp=84855983910&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84855983910&partnerID=8YFLogxK

U2 - 10.1111/j.1467-9868.2011.01005.x

DO - 10.1111/j.1467-9868.2011.01005.x

M3 - Article

AN - SCOPUS:84855983910

VL - 74

SP - 37

EP - 65

JO - Journal of the Royal Statistical Society. Series B: Statistical Methodology

JF - Journal of the Royal Statistical Society. Series B: Statistical Methodology

SN - 1369-7412

IS - 1

ER -