Validation in Statistics and Machine Learning - Abstract

Bernau, Christoph

Correction for Tuning Bias in Resampling Based Error Rate Estimation

Most modern classification methods incorporate a tuning parameter which usually adjusts their complexity to the examined data set. These tuning parameters induce a bias in error rate estimation if they are not handled properly. By simply reporting the performance of the best tuning parameter, overly optimistic error rates have been published in the past. Varma and Simon (2006) recommend a sensible approach based on nested cross-validation for avoiding this tuning bias. However, the additional inner CV loop severely increases computation time. Tibshirani and Tibshirani (2009) introduce another approach which tries to directly correct the minimum error rate with regard to this tuning bias. Their method avoids the additional inner CV by exclusively using the results obtained on the test folds. However, our studies show that their procedure achieves a comparatively poor performance. We present a new method for tuning bias correction and compare it to the approaches introduced above. It is based on combining the CV errors of all candidate tuning parameters into a weighted mean. We motivate our new method within a decision theoretical framework which provides a suitable background for analyzing the tuning procedure. In this context, we also naturally extend our method from the choice of optimal tuning parameters to the choice of whole classifier algorithms. In our studies we find evidence that our approach often outperforms the approach proposed by Tibshirani and Tibshirani. Furthermore, our method produces error rate estimates quite close to the nested CV approach at significantly lower computation time. Additionally, we report some experiences with the nested CV method. For example, we sometimes encounter situations in which the nested CV approach produces error rates higher than the maximum CV error rate. Such problems often remain unnoticed in practice because in nested CV the test errors are computed for the tuned parameter only.

  1. R.J. Tibshirani and R. Tibshirani, 2009. A Bias correction for the minimum error rate in cross-validation. Annals of Applied Statistics, 3:822--829.
  2. S. Varma and R. Simon, 2006. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7:91.