Understanding and diagnosing your machine-learning models

Achieving a good prediction is often only half of the job. Questions immediately arise: How to improve this prediction? What drives the prediction? Can we operate changes to the system based on the predictions? All these questions require understanding how good is the model prediction, and how do the model predict.

This tutorial assumes basic knowledge of scikit-learn. It will focus on statistics, tests, and interpretation rather than improving the prediction.

• 1. Measuring how well a model predicts
  • 1.1. Metrics to judge the sucess of a model
    • 1.1.1. Regression settings
      • 1.1.1.1. The Boston housing data
      • 1.1.1.2. Explained variance vs Mean Square Error
      • 1.1.1.3. Mean Squared Error versus Mean Absolute Error
      • 1.1.1.4. Summary
    • 1.1.2. Classification settings
      • 1.1.2.1. The digits data
      • 1.1.2.2. Accuracy and its shortcomings
      • 1.1.2.3. Precision, recall, and their shortcomings
      • 1.1.2.4. Area under the ROC curve
        • 1.1.2.4.1. Average precision
      • 1.1.2.5. Multiclass and multilabel settings
      • 1.1.2.6. Summary
  • 1.2. Cross-validation: some gotchas
    • 1.2.1. Uncertainty of measured accuracy
      • 1.2.1.1. Variations in cross_val_score: simple experiments
      • 1.2.1.2. A simple probabilistic model
      • 1.2.1.3. Empirical distribution of cross-validation scores
      • 1.2.1.4. Measuring baselines and chance
    • 1.2.2. Cross-validation with non iid data
      • 1.2.2.1. Stock market: time series
      • 1.2.2.2. School grades: repeated measures
      • 1.2.2.3. Summary
  • 1.3. Underfit vs overfit: do I need more data, or more complex models?
    • 1.3.1. A toy problem: fitting polynomes
      • 1.3.1.1. Data generation model
      • 1.3.1.2. Polynomial regression
    • 1.3.2. Train error versus test error
      • 1.3.2.1. Underfit: high bias
      • 1.3.2.2. Overfit: high variance
    • 1.3.3. Validation curve varying model complexity
      • 1.3.3.1. Generate a larger dataset
      • 1.3.3.2. The validation_curve function
    • 1.3.4. Learning curves: varying the amount of data
    • 1.3.5. Summary
• 2. Understanding why a classifier predicts
  • 2.1. Interpreting linear models
    • 2.1.1. Data on wages
    • 2.1.2. The challenge of correlated features
    • 2.1.3. Coefficients of a linear model
    • 2.1.4. The effect of regularization
    • 2.1.5. Stability to gauge significance
      • 2.1.5.1. With the lasso estimator
      • 2.1.5.2. With the ridge estimator
      • 2.1.5.3. Which is the truth?
        • 2.1.5.3.1. Conclusion on factors of wages?
  • 2.2. Interpreting random forests
    • 2.2.1. Data on wages
    • 2.2.2. Feature importance
    • 2.2.3. Meaning and Caveats
  • 2.3. Partial dependency plots
    • 2.3.1. All-in-one plotting function
    • 2.3.2. Lower-level partial_dependence function
  • 2.4. Black-box interpretation of models: LIME
    • 2.4.1. Regression on tabular data: factors of prices of houses
• 3. Appendix: auxiliary figures