Component selection and smoothing in multivariate nonparametric regression

Yi Lin, Hao Zhang

Research output: Contribution to journalArticle

243 Citations (Scopus)

Abstract

We propose a new method for model selection and model fitting in multivariate nonparametric regression models, in the framework of smoothing spline ANOVA. The "COSSO" is a method of regularization with the penalty functional being the sum of component norms, instead of the squared norm employed in the traditional smoothing spline method. The COSSO provides a unified framework for several recent proposals for model selection in linear models and smoothing spline ANOVA models. Theoretical properties, such as the existence and the rate of convergence of the COSSO estimator, are studied. In the special case of a tensor product design with periodic functions, a detailed analysis reveals that the COSSO does model selection by applying a novel soft thresholding type operation to the function components. We give an equivalent formulation of the COSSO estimator which leads naturally to an iterative algorithm. We compare the COSSO with MARS, a popular method that builds functional ANOVA models, in simulations and real examples. The COSSO method can be extended to classification problems and we compare its performance with those of a number of machine learning algorithms on real datasets. The COSSO gives very competitive performance in these studies.

Original languageEnglish (US)
Pages (from-to)2272-2297
Number of pages26
JournalAnnals of Statistics
Volume34
Issue number5
DOIs
StatePublished - Oct 2006
Externally publishedYes

Fingerprint

Multivariate Regression
Nonparametric Regression
Smoothing
Smoothing Splines
Model Selection
Estimator
Norm
Functional Model
Model Fitting
Nonparametric Model
Product Design
Thresholding
Periodic Functions
Classification Problems
Tensor Product
Iterative Algorithm
Penalty
Learning Algorithm
Linear Model
Regression Model

Keywords

  • Machine learning
  • Method of regularization
  • Model selection
  • Nonparametric classification
  • Nonparametric regression
  • Smoothing spline ANOVA

ASJC Scopus subject areas

  • Mathematics(all)
  • Statistics and Probability

Cite this

Component selection and smoothing in multivariate nonparametric regression. / Lin, Yi; Zhang, Hao.

In: Annals of Statistics, Vol. 34, No. 5, 10.2006, p. 2272-2297.

Research output: Contribution to journalArticle

@article{9b0cdc4e03fc4253addea5b97bd94253,
title = "Component selection and smoothing in multivariate nonparametric regression",
abstract = "We propose a new method for model selection and model fitting in multivariate nonparametric regression models, in the framework of smoothing spline ANOVA. The {"}COSSO{"} is a method of regularization with the penalty functional being the sum of component norms, instead of the squared norm employed in the traditional smoothing spline method. The COSSO provides a unified framework for several recent proposals for model selection in linear models and smoothing spline ANOVA models. Theoretical properties, such as the existence and the rate of convergence of the COSSO estimator, are studied. In the special case of a tensor product design with periodic functions, a detailed analysis reveals that the COSSO does model selection by applying a novel soft thresholding type operation to the function components. We give an equivalent formulation of the COSSO estimator which leads naturally to an iterative algorithm. We compare the COSSO with MARS, a popular method that builds functional ANOVA models, in simulations and real examples. The COSSO method can be extended to classification problems and we compare its performance with those of a number of machine learning algorithms on real datasets. The COSSO gives very competitive performance in these studies.",
keywords = "Machine learning, Method of regularization, Model selection, Nonparametric classification, Nonparametric regression, Smoothing spline ANOVA",
author = "Yi Lin and Hao Zhang",
year = "2006",
month = "10",
doi = "10.1214/009053606000000722",
language = "English (US)",
volume = "34",
pages = "2272--2297",
journal = "Annals of Statistics",
issn = "0090-5364",
publisher = "Institute of Mathematical Statistics",
number = "5",

}

TY - JOUR

T1 - Component selection and smoothing in multivariate nonparametric regression

AU - Lin, Yi

AU - Zhang, Hao

PY - 2006/10

Y1 - 2006/10

N2 - We propose a new method for model selection and model fitting in multivariate nonparametric regression models, in the framework of smoothing spline ANOVA. The "COSSO" is a method of regularization with the penalty functional being the sum of component norms, instead of the squared norm employed in the traditional smoothing spline method. The COSSO provides a unified framework for several recent proposals for model selection in linear models and smoothing spline ANOVA models. Theoretical properties, such as the existence and the rate of convergence of the COSSO estimator, are studied. In the special case of a tensor product design with periodic functions, a detailed analysis reveals that the COSSO does model selection by applying a novel soft thresholding type operation to the function components. We give an equivalent formulation of the COSSO estimator which leads naturally to an iterative algorithm. We compare the COSSO with MARS, a popular method that builds functional ANOVA models, in simulations and real examples. The COSSO method can be extended to classification problems and we compare its performance with those of a number of machine learning algorithms on real datasets. The COSSO gives very competitive performance in these studies.

AB - We propose a new method for model selection and model fitting in multivariate nonparametric regression models, in the framework of smoothing spline ANOVA. The "COSSO" is a method of regularization with the penalty functional being the sum of component norms, instead of the squared norm employed in the traditional smoothing spline method. The COSSO provides a unified framework for several recent proposals for model selection in linear models and smoothing spline ANOVA models. Theoretical properties, such as the existence and the rate of convergence of the COSSO estimator, are studied. In the special case of a tensor product design with periodic functions, a detailed analysis reveals that the COSSO does model selection by applying a novel soft thresholding type operation to the function components. We give an equivalent formulation of the COSSO estimator which leads naturally to an iterative algorithm. We compare the COSSO with MARS, a popular method that builds functional ANOVA models, in simulations and real examples. The COSSO method can be extended to classification problems and we compare its performance with those of a number of machine learning algorithms on real datasets. The COSSO gives very competitive performance in these studies.

KW - Machine learning

KW - Method of regularization

KW - Model selection

KW - Nonparametric classification

KW - Nonparametric regression

KW - Smoothing spline ANOVA

UR - http://www.scopus.com/inward/record.url?scp=33847350805&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33847350805&partnerID=8YFLogxK

U2 - 10.1214/009053606000000722

DO - 10.1214/009053606000000722

M3 - Article

AN - SCOPUS:33847350805

VL - 34

SP - 2272

EP - 2297

JO - Annals of Statistics

JF - Annals of Statistics

SN - 0090-5364

IS - 5

ER -