Subspace information criterion for nonquadratic regularizers-model selection for sparse regressors

Koji Tsuda, Masashi Sugiyama, Klaus Muller

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

Nonquadratic regularizers, in particular the l1 norm regularizer can yield sparse solutions that generalize well. In this work we propose the generalized subspace information criterion (GSIC) that allows to predict the generalization error for this useful family of regularizers. We show that under some technical assumptions GSIC is an asymptotically unbiased estimator of the generalization error. GSIC is demonstrated to have a good performance in experiments with the l1 norm regularizer as we compare with the network information criterion (NIC) and cross- validation in relatively large sample cases. However in the small sample case, GSIC tends to fail to capture the optimal model due to its large variance. Therefore, also a biased version of GSIC is introduced, which achieves reliable model selection in the relevant and challenging scenario of high-dimensional data and few samples.

Original languageEnglish
Pages (from-to)70-80
Number of pages11
JournalIEEE Transactions on Neural Networks
Volume13
Issue number1
DOIs
Publication statusPublished - 2002 Jan 1
Externally publishedYes

Fingerprint

Information Criterion
Information Services
Model Selection
Subspace
Generalization Error
L1-norm
Experiments
Unbiased estimator
High-dimensional Data
Cross-validation
Small Sample
Biased
Tend
Predict
Scenarios
Generalise
Experiment

Keywords

  • Kernel methods
  • Model selection
  • Regularization
  • Sparse regressors
  • Subspace information criterion (SIC)

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Theoretical Computer Science
  • Electrical and Electronic Engineering
  • Artificial Intelligence
  • Computational Theory and Mathematics
  • Hardware and Architecture

Cite this

Subspace information criterion for nonquadratic regularizers-model selection for sparse regressors. / Tsuda, Koji; Sugiyama, Masashi; Muller, Klaus.

In: IEEE Transactions on Neural Networks, Vol. 13, No. 1, 01.01.2002, p. 70-80.

Research output: Contribution to journalArticle

Tsuda, Koji ; Sugiyama, Masashi ; Muller, Klaus. / Subspace information criterion for nonquadratic regularizers-model selection for sparse regressors. In: IEEE Transactions on Neural Networks. 2002 ; Vol. 13, No. 1. pp. 70-80.
@article{20993473d485492cb694ef052013cc9a,
title = "Subspace information criterion for nonquadratic regularizers-model selection for sparse regressors",
abstract = "Nonquadratic regularizers, in particular the l1 norm regularizer can yield sparse solutions that generalize well. In this work we propose the generalized subspace information criterion (GSIC) that allows to predict the generalization error for this useful family of regularizers. We show that under some technical assumptions GSIC is an asymptotically unbiased estimator of the generalization error. GSIC is demonstrated to have a good performance in experiments with the l1 norm regularizer as we compare with the network information criterion (NIC) and cross- validation in relatively large sample cases. However in the small sample case, GSIC tends to fail to capture the optimal model due to its large variance. Therefore, also a biased version of GSIC is introduced, which achieves reliable model selection in the relevant and challenging scenario of high-dimensional data and few samples.",
keywords = "Kernel methods, Model selection, Regularization, Sparse regressors, Subspace information criterion (SIC)",
author = "Koji Tsuda and Masashi Sugiyama and Klaus Muller",
year = "2002",
month = "1",
day = "1",
doi = "10.1109/72.977272",
language = "English",
volume = "13",
pages = "70--80",
journal = "IEEE Transactions on Neural Networks and Learning Systems",
issn = "2162-237X",
publisher = "IEEE Computational Intelligence Society",
number = "1",

}

TY - JOUR

T1 - Subspace information criterion for nonquadratic regularizers-model selection for sparse regressors

AU - Tsuda, Koji

AU - Sugiyama, Masashi

AU - Muller, Klaus

PY - 2002/1/1

Y1 - 2002/1/1

N2 - Nonquadratic regularizers, in particular the l1 norm regularizer can yield sparse solutions that generalize well. In this work we propose the generalized subspace information criterion (GSIC) that allows to predict the generalization error for this useful family of regularizers. We show that under some technical assumptions GSIC is an asymptotically unbiased estimator of the generalization error. GSIC is demonstrated to have a good performance in experiments with the l1 norm regularizer as we compare with the network information criterion (NIC) and cross- validation in relatively large sample cases. However in the small sample case, GSIC tends to fail to capture the optimal model due to its large variance. Therefore, also a biased version of GSIC is introduced, which achieves reliable model selection in the relevant and challenging scenario of high-dimensional data and few samples.

AB - Nonquadratic regularizers, in particular the l1 norm regularizer can yield sparse solutions that generalize well. In this work we propose the generalized subspace information criterion (GSIC) that allows to predict the generalization error for this useful family of regularizers. We show that under some technical assumptions GSIC is an asymptotically unbiased estimator of the generalization error. GSIC is demonstrated to have a good performance in experiments with the l1 norm regularizer as we compare with the network information criterion (NIC) and cross- validation in relatively large sample cases. However in the small sample case, GSIC tends to fail to capture the optimal model due to its large variance. Therefore, also a biased version of GSIC is introduced, which achieves reliable model selection in the relevant and challenging scenario of high-dimensional data and few samples.

KW - Kernel methods

KW - Model selection

KW - Regularization

KW - Sparse regressors

KW - Subspace information criterion (SIC)

UR - http://www.scopus.com/inward/record.url?scp=0036130853&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0036130853&partnerID=8YFLogxK

U2 - 10.1109/72.977272

DO - 10.1109/72.977272

M3 - Article

VL - 13

SP - 70

EP - 80

JO - IEEE Transactions on Neural Networks and Learning Systems

JF - IEEE Transactions on Neural Networks and Learning Systems

SN - 2162-237X

IS - 1

ER -