Model building with likelihood basis pursuit

Michael C. Ferris, Meta M. Voelker, Hao Zhang

Research output: Contribution to journalArticle

3 Scopus citations

Abstract

We consider a non-parametric penalized likelihood approach for model building called likelihood basis pursuit (LBP) that determines the probabilities of binary outcomes given explanatory vectors while automatically selecting important features. The LBP model involves parameters that balance the competing goals of maximizing the log-likelihood and minimizing the penalized basis pursuit terms. These parameters are selected to minimize a proxy of misclassification error, namely, the randomized, generalized approximate cross validation (ranGACV) function. The ranGACV function is not easily represented in compact form; its functional values can only be obtained by solving two instances of the LBP model, which may be computationally expensive. A grid search is typically used to find appropriate parameters, requiring the solutions to hundreds or thousands of instances of the LBP model. Since only parameters (data) are changed between solves, the resulting problem is a nonlinear slice model in the parameter space. We show how slice-modeling techniques significantly improve the efficiency of individual solves and thus speed-up the grid search. In addition, we consider using derivative-free optimization algorithms for parameter selection, replacing the grid search. We show how, by seeding the derivative-free algorithms with a coarse grid search, these algorithms can find better solutions with fewer function evaluations. Our interest in this area comes directly from the seminal work that Olvi and his collaborators have carried out designing and applying optimization techniques to problems in machine learning and data mining.

Original languageEnglish (US)
Pages (from-to)577-594
Number of pages18
JournalOptimization Methods and Software
Volume19
Issue number5 SPEC. ISS.
DOIs
Publication statusPublished - Oct 2004
Externally publishedYes

    Fingerprint

Keywords

  • Basis pursuit
  • Model building
  • Parameter selection
  • Slice models

ASJC Scopus subject areas

  • Computer Graphics and Computer-Aided Design
  • Software
  • Applied Mathematics
  • Control and Optimization
  • Management Science and Operations Research

Cite this