Validation in Statistics and Machine Learning - Abstract
Machine learning research is either inspired by a particular application, or by a general desire to make technology more “inteligent”. In modern machine learning most methodological development is mathematically inspired and results in an algorithm for optimization or fitting of a model to data.
Design choices in implementation of an algorithm can have a significant effect on the quality of results. Decisions such as model initializaiton and data pre-processing are all part of the implementation. Necessarily, space constraints sometimes mean that such details are not included in the associated paper. It seems clear that the paper only tells part of the story. Implementations need to be made available at the time of submission of the paper, so that the full story may be followed. In our research group we have done this since 2001. In this talk I will make the arguments in favour of doing this universally and give personal experiences of the results.