1. Measuring how well a model predicts

• 1.1. Metrics to judge the sucess of a model
  • 1.1.1. Regression settings
    • 1.1.1.1. The Boston housing data
    • 1.1.1.2. Explained variance vs Mean Square Error
    • 1.1.1.3. Mean Squared Error versus Mean Absolute Error
    • 1.1.1.4. Summary
  • 1.1.2. Classification settings
    • 1.1.2.1. The digits data
    • 1.1.2.2. Accuracy and its shortcomings
    • 1.1.2.3. Precision, recall, and their shortcomings
    • 1.1.2.4. Area under the ROC curve
      • 1.1.2.4.1. Average precision
    • 1.1.2.5. Multiclass and multilabel settings
    • 1.1.2.6. Summary
• 1.2. Cross-validation: some gotchas
  • 1.2.1. Uncertainty of measured accuracy
    • 1.2.1.1. Variations in cross_val_score: simple experiments
    • 1.2.1.2. A simple probabilistic model
    • 1.2.1.3. Empirical distribution of cross-validation scores
    • 1.2.1.4. Measuring baselines and chance
  • 1.2.2. Cross-validation with non iid data
    • 1.2.2.1. Stock market: time series
    • 1.2.2.2. School grades: repeated measures
    • 1.2.2.3. Summary
• 1.3. Underfit vs overfit: do I need more data, or more complex models?
  • 1.3.1. A toy problem: fitting polynomes
    • 1.3.1.1. Data generation model
    • 1.3.1.2. Polynomial regression
  • 1.3.2. Train error versus test error
    • 1.3.2.1. Underfit: high bias
    • 1.3.2.2. Overfit: high variance
  • 1.3.3. Validation curve varying model complexity
    • 1.3.3.1. Generate a larger dataset
    • 1.3.3.2. The validation_curve function
  • 1.3.4. Learning curves: varying the amount of data
  • 1.3.5. Summary

---

Metrics to judge the sucess of a model

Cross-validation: some gotchas

Underfit vs overfit: do I need more data, or more complex models?