Variable selection for the multicategory SVM via adaptive sup-norm regularization

Hao Zhang, Yufeng Liu, Yichao Wu, Ji Zhu

Research output: Contribution to journalArticle

53 Citations (Scopus)

Abstract

The Support Vector Machine (SVM) is a popular classification paradigm in machine learning and has achieved great success in real applications. However, the standard SVM can not select variables automatically and therefore its solution typically utilizes all the input variables without discrimination. This makes it difficult to identify important predictor variables, which is often one of the primary goals in data analysis. In this paper, we propose two novel types of regularization in the context of the multicategory SVM (MSVM) for simultaneous classification and variable selection. The MSVM generally requires estimation of multiple discriminating functions and applies the argmax rule for prediction. For each individual variable, we propose to characterize its importance by the supnorm of its coefficient vector associated with different functions, and then minimize the MSVM hinge loss function subject to a penalty on the sum of supnorms. To further improve the supnorm penalty, we propose the adaptive regularization, which allows different weights imposed on different variables according to their relative importance. Both types of regularization automate variable selection in the process of building classifiers, and lead to sparse multiclassifiers with enhanced interpretability and improved accuracy, especially for high dimensional low sample size data. One big advantage of the sup-norm penalty is its easy implementation via standard linear programming. Several simulated examples and one real gene data analysis demonstrate the outstanding performance of the adaptive supnorm penalty in various data settings.

Original languageEnglish (US)
Pages (from-to)149-167
Number of pages19
JournalElectronic Journal of Statistics
Volume2
DOIs
StatePublished - Jan 1 2008
Externally publishedYes

Fingerprint

Variable Selection
Support Vector Machine
Regularization
Penalty
Norm
Data analysis
Interpretability
Loss Function
Discrimination
Linear programming
Predictors
Machine Learning
Sample Size
High-dimensional
Classifier
Paradigm
Support vector machine
Variable selection
Gene
Minimise

Keywords

  • Classification
  • L-norm penalty
  • Multicategory
  • Sup-norm
  • SVM

ASJC Scopus subject areas

  • Statistics and Probability

Cite this

Variable selection for the multicategory SVM via adaptive sup-norm regularization. / Zhang, Hao; Liu, Yufeng; Wu, Yichao; Zhu, Ji.

In: Electronic Journal of Statistics, Vol. 2, 01.01.2008, p. 149-167.

Research output: Contribution to journalArticle

@article{8b3897a5ea374d2bab85cd8375767e8f,
title = "Variable selection for the multicategory SVM via adaptive sup-norm regularization",
abstract = "The Support Vector Machine (SVM) is a popular classification paradigm in machine learning and has achieved great success in real applications. However, the standard SVM can not select variables automatically and therefore its solution typically utilizes all the input variables without discrimination. This makes it difficult to identify important predictor variables, which is often one of the primary goals in data analysis. In this paper, we propose two novel types of regularization in the context of the multicategory SVM (MSVM) for simultaneous classification and variable selection. The MSVM generally requires estimation of multiple discriminating functions and applies the argmax rule for prediction. For each individual variable, we propose to characterize its importance by the supnorm of its coefficient vector associated with different functions, and then minimize the MSVM hinge loss function subject to a penalty on the sum of supnorms. To further improve the supnorm penalty, we propose the adaptive regularization, which allows different weights imposed on different variables according to their relative importance. Both types of regularization automate variable selection in the process of building classifiers, and lead to sparse multiclassifiers with enhanced interpretability and improved accuracy, especially for high dimensional low sample size data. One big advantage of the sup-norm penalty is its easy implementation via standard linear programming. Several simulated examples and one real gene data analysis demonstrate the outstanding performance of the adaptive supnorm penalty in various data settings.",
keywords = "Classification, L-norm penalty, Multicategory, Sup-norm, SVM",
author = "Hao Zhang and Yufeng Liu and Yichao Wu and Ji Zhu",
year = "2008",
month = "1",
day = "1",
doi = "10.1214/08-EJS122",
language = "English (US)",
volume = "2",
pages = "149--167",
journal = "Electronic Journal of Statistics",
issn = "1935-7524",
publisher = "Institute of Mathematical Statistics",

}

TY - JOUR

T1 - Variable selection for the multicategory SVM via adaptive sup-norm regularization

AU - Zhang, Hao

AU - Liu, Yufeng

AU - Wu, Yichao

AU - Zhu, Ji

PY - 2008/1/1

Y1 - 2008/1/1

N2 - The Support Vector Machine (SVM) is a popular classification paradigm in machine learning and has achieved great success in real applications. However, the standard SVM can not select variables automatically and therefore its solution typically utilizes all the input variables without discrimination. This makes it difficult to identify important predictor variables, which is often one of the primary goals in data analysis. In this paper, we propose two novel types of regularization in the context of the multicategory SVM (MSVM) for simultaneous classification and variable selection. The MSVM generally requires estimation of multiple discriminating functions and applies the argmax rule for prediction. For each individual variable, we propose to characterize its importance by the supnorm of its coefficient vector associated with different functions, and then minimize the MSVM hinge loss function subject to a penalty on the sum of supnorms. To further improve the supnorm penalty, we propose the adaptive regularization, which allows different weights imposed on different variables according to their relative importance. Both types of regularization automate variable selection in the process of building classifiers, and lead to sparse multiclassifiers with enhanced interpretability and improved accuracy, especially for high dimensional low sample size data. One big advantage of the sup-norm penalty is its easy implementation via standard linear programming. Several simulated examples and one real gene data analysis demonstrate the outstanding performance of the adaptive supnorm penalty in various data settings.

AB - The Support Vector Machine (SVM) is a popular classification paradigm in machine learning and has achieved great success in real applications. However, the standard SVM can not select variables automatically and therefore its solution typically utilizes all the input variables without discrimination. This makes it difficult to identify important predictor variables, which is often one of the primary goals in data analysis. In this paper, we propose two novel types of regularization in the context of the multicategory SVM (MSVM) for simultaneous classification and variable selection. The MSVM generally requires estimation of multiple discriminating functions and applies the argmax rule for prediction. For each individual variable, we propose to characterize its importance by the supnorm of its coefficient vector associated with different functions, and then minimize the MSVM hinge loss function subject to a penalty on the sum of supnorms. To further improve the supnorm penalty, we propose the adaptive regularization, which allows different weights imposed on different variables according to their relative importance. Both types of regularization automate variable selection in the process of building classifiers, and lead to sparse multiclassifiers with enhanced interpretability and improved accuracy, especially for high dimensional low sample size data. One big advantage of the sup-norm penalty is its easy implementation via standard linear programming. Several simulated examples and one real gene data analysis demonstrate the outstanding performance of the adaptive supnorm penalty in various data settings.

KW - Classification

KW - L-norm penalty

KW - Multicategory

KW - Sup-norm

KW - SVM

UR - http://www.scopus.com/inward/record.url?scp=47749133494&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=47749133494&partnerID=8YFLogxK

U2 - 10.1214/08-EJS122

DO - 10.1214/08-EJS122

M3 - Article

AN - SCOPUS:47749133494

VL - 2

SP - 149

EP - 167

JO - Electronic Journal of Statistics

JF - Electronic Journal of Statistics

SN - 1935-7524

ER -