Validation in Statistics and Machine Learning - Abstract
The ultimate goal of machine learning is to produce complex intelligent systems, able to deal with the versatility of real-world situations. However, no designing process or tools has been developed to facilitate the elaboration of such systems by large teams of designers.
I will present the MASH project, a European initiative investigating the design of very large sets of feature extractors for computer vision and goal-planning. This project necessitates in particular the careful design of meaningful summaries of experiments to help contributors identify the strengths and weaknesses of the global system, and of their own contributions.
Also, the resulting collaborative process can be seen as a large-scale learning system, in which contributors act as as many super-optimizers. The resulting learning may suffer critically from of over-fitting. I will present how we envision the experimental setup to be protected from it.