Variance in machine-learning benchmarks
Many sources of uncontroled variance in observed performance of machine-learning pipelines
How reliable is a empirical benchmark of machine pipelines? Solid benchmarks are harder than they seem.
Sources of variances in machine-learning benchmarks. [Bouthillier et al] systematically explore the variance of machine-learning benchmarks in 5 representative settings: 2 computer vision tasks, 2 natural language tasks, and a bioinformatics one.
Different sources of variation of the measured performance: across different case studies, as a fraction of the variance induced by bootstrapping the data. Different hyperparameter optimization are considered.
While most deep-learning benchmarks only study the influence of random initializations [1], the empirical study shows that the most important sources of arbitrary variation in the benchmarks are:
- The split between the train and the test set (or, in statistical terms, the data sampling);
- The hyper-parameter choice (even with a hyper-parameter optimizer, this choice is imperfect and thus has arbitrary aspects);
- The weights init only come after these two.
Comparing variance due to train-test split with theoretical uncertainty due the size of the test set.
The figure on the right shows that the dominant source of uncertainty (due to the arbitrary split between train and test set) is well explained by the sampling noise of the test set: as the test is of limited size, it can only characterize the quality of a prediction with a limited accuracy.
| [1] | There is a aweful trend of trying out a certain number of seeds and piking the best one. While such approach may select the best predictor for a given test set, it does not characterize well the learning procedure: the selected seed is still arbitrary and given new data there is no reason that it continues being a good seed. If “trying out 5 inits and selecting the best one” is part of the learning procedure, then this full learning procedure must be benchmarked, including the variation induced by the choice of 5 random inits rather than 5 others. |
Comparing benchmark variance to published improvements.
Benchmark variance is not negligeable. Comparing this benchmarking variance to typical published improvements in machine-learning research shows that many of the improvements can be explained be uncontrolled fluctuations.
Hence, it seems crucial that good practices account for this variance to decide whether or not an observed is meaningful.
Accounting for variance. Statistical control on machine-learning is not easy: often the computational cost of hyper-parameter selection is too high to repeat it multiple times, in addition cross-validations are not independent (don’t do a t-test on them). In addition, the number of repititions of the experiment is above all a function of the compute power: hence enough compute power can make any trivial difference statistically significant (a common problem of null-hypothesis testing).
Proposed testing on the probability of improvement (related to the Neyman-Pearson view on testing).
In [Bouthillier et al], we propose a simple procedure to account for the variance of benchmarks:
- set the hyper-parameters of all methods
- repeat the experiment varying all sources of uncontroled variation (data split, random seeds…)
- compute the probability that one pipeline outperforms the other
- accept the improvement if that probability is above 0.75 [2]
| [2] | Thresholding at 0.75 was chosen because on our case studies it accepted improvements with effect sizes that turned out to be historically meaningful. |
This procedure does not account well for the variance due to the imperfect choice of hyper-parameters. But in [Bouthillier et al], we characterize it and show that it is a surprisingly relibale and yet easy approach to better machine learning benchmarks.
Simpler variant
An even simpler variant consists in reporting the standard deviation of the performance scores after sampling all sources of variance (including data splits).
Imputation to predict with missing values
Classic (conditional expectation) imputation is not always the good choice for prediction
Many data-science applications must work with incomplete data: some measurements are missing, for instance in medicine where different patients recieve different examinations, or in social science with skipped survey questions. Most often, this missingness is not “at random”: there is a systematic aspect to missing values –whether or not a patient gets an exam is related to her suspect health conditions.
There is a long-running statistical framework to deal with missing values, for parameter inference in probabilistic models. It leads to the notion that when the data are missing at random, estimates of parameters of the fully-observed model can be obtained by imputing missing values by values that capture the conditional relations between missing and observed.
In Le Morvan NeurIPS 2020, we consider a different problem: that of finding the optimal prediction of an outcome y given data X with missing values (or, relatedly, estimating the conditional dependency of y given X). We show that, provided that we consider flexible enough (weakly parametric) estimators, this leads to completely different trade off.
Data with imputed missing values that form manifolds in the data cloud.
Namely, Le Morvan NeurIPS 2020 shows that given a flexible predictor, almost all imputation functions lead to optimal predictions asymptoticaly, for all missing mechanisms (even missing not at random). Indeed, the predictor can adapt to the manifolds of imputed values (the figure gives an intuition).