Model selection under covariate shift

Masashi Sugiyama, Klaus Muller

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Citations (Scopus)

Abstract

A common assumption in supervised learning is that the training and test input points follow the same probability distribution. However, this assumption is not fulfilled, e.g., in interpolation, extrapolation, or active learning scenarios. The violation of this assumption-known as the covariate shift-causes a heavy bias in standard generalization error estimation schemes such as cross-validation and thus they result in poor model selection. In this paper, we therefore propose an alternative estimator of the generalization error. Under covariate shift, the proposed generalization error estimator is unbiased if the learning target function is included in the model at hand and it is asymptotically unbiased in general. Experimental results show that model selection with the proposed generalization error estimator is compared favorably to cross-validation in extrapolation.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages235-240
Number of pages6
Volume3697 LNCS
Publication statusPublished - 2005 Dec 1
Externally publishedYes
Event15th International Conference on Artificial Neural Networks: Biological Inspirations - ICANN 2005 - Warsaw, Poland
Duration: 2005 Sep 112005 Sep 15

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume3697 LNCS
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other15th International Conference on Artificial Neural Networks: Biological Inspirations - ICANN 2005
CountryPoland
CityWarsaw
Period05/9/1105/9/15

Fingerprint

Generalization Error
Model Selection
Covariates
Extrapolation
Error Estimator
Cross-validation
Supervised learning
Learning
Error analysis
Probability distributions
Problem-Based Learning
Active Learning
Interpolation
Error Estimation
Supervised Learning
Standard error
Probability Distribution
Hand
Interpolate
Estimator

ASJC Scopus subject areas

  • Computer Science(all)
  • Biochemistry, Genetics and Molecular Biology(all)
  • Theoretical Computer Science

Cite this

Sugiyama, M., & Muller, K. (2005). Model selection under covariate shift. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 3697 LNCS, pp. 235-240). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 3697 LNCS).

Model selection under covariate shift. / Sugiyama, Masashi; Muller, Klaus.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 3697 LNCS 2005. p. 235-240 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 3697 LNCS).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Sugiyama, M & Muller, K 2005, Model selection under covariate shift. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 3697 LNCS, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 3697 LNCS, pp. 235-240, 15th International Conference on Artificial Neural Networks: Biological Inspirations - ICANN 2005, Warsaw, Poland, 05/9/11.
Sugiyama M, Muller K. Model selection under covariate shift. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 3697 LNCS. 2005. p. 235-240. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
Sugiyama, Masashi ; Muller, Klaus. / Model selection under covariate shift. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 3697 LNCS 2005. pp. 235-240 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{d6710b046b4046b294fc2663d7b2d64c,
title = "Model selection under covariate shift",
abstract = "A common assumption in supervised learning is that the training and test input points follow the same probability distribution. However, this assumption is not fulfilled, e.g., in interpolation, extrapolation, or active learning scenarios. The violation of this assumption-known as the covariate shift-causes a heavy bias in standard generalization error estimation schemes such as cross-validation and thus they result in poor model selection. In this paper, we therefore propose an alternative estimator of the generalization error. Under covariate shift, the proposed generalization error estimator is unbiased if the learning target function is included in the model at hand and it is asymptotically unbiased in general. Experimental results show that model selection with the proposed generalization error estimator is compared favorably to cross-validation in extrapolation.",
author = "Masashi Sugiyama and Klaus Muller",
year = "2005",
month = "12",
day = "1",
language = "English",
isbn = "3540287558",
volume = "3697 LNCS",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
pages = "235--240",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - Model selection under covariate shift

AU - Sugiyama, Masashi

AU - Muller, Klaus

PY - 2005/12/1

Y1 - 2005/12/1

N2 - A common assumption in supervised learning is that the training and test input points follow the same probability distribution. However, this assumption is not fulfilled, e.g., in interpolation, extrapolation, or active learning scenarios. The violation of this assumption-known as the covariate shift-causes a heavy bias in standard generalization error estimation schemes such as cross-validation and thus they result in poor model selection. In this paper, we therefore propose an alternative estimator of the generalization error. Under covariate shift, the proposed generalization error estimator is unbiased if the learning target function is included in the model at hand and it is asymptotically unbiased in general. Experimental results show that model selection with the proposed generalization error estimator is compared favorably to cross-validation in extrapolation.

AB - A common assumption in supervised learning is that the training and test input points follow the same probability distribution. However, this assumption is not fulfilled, e.g., in interpolation, extrapolation, or active learning scenarios. The violation of this assumption-known as the covariate shift-causes a heavy bias in standard generalization error estimation schemes such as cross-validation and thus they result in poor model selection. In this paper, we therefore propose an alternative estimator of the generalization error. Under covariate shift, the proposed generalization error estimator is unbiased if the learning target function is included in the model at hand and it is asymptotically unbiased in general. Experimental results show that model selection with the proposed generalization error estimator is compared favorably to cross-validation in extrapolation.

UR - http://www.scopus.com/inward/record.url?scp=33646240055&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33646240055&partnerID=8YFLogxK

M3 - Conference contribution

SN - 3540287558

SN - 9783540287551

VL - 3697 LNCS

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 235

EP - 240

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

ER -