3. Model selection and evaluation

• 3.1. Cross-validation: evaluating estimator performance
  • 3.1.1. Computing cross-validated metrics
    • 3.1.1.1. The cross_validate function and multiple metric evaluation
    • 3.1.1.2. Obtaining predictions by cross-validation
  • 3.1.2. Cross validation iterators
    • 3.1.2.1. Cross-validation iterators for i.i.d. data
      • 3.1.2.1.1. K-fold
      • 3.1.2.1.2. Repeated K-Fold
      • 3.1.2.1.3. Leave One Out (LOO)
      • 3.1.2.1.4. Leave P Out (LPO)
      • 3.1.2.1.5. Random permutations cross-validation a.k.a. Shuffle & Split
    • 3.1.2.2. Cross-validation iterators with stratification based on class labels.
      • 3.1.2.2.1. Stratified k-fold
      • 3.1.2.2.2. Stratified Shuffle Split
    • 3.1.2.3. Cross-validation iterators for grouped data.
      • 3.1.2.3.1. Group k-fold
      • 3.1.2.3.2. Leave One Group Out
      • 3.1.2.3.3. Leave P Groups Out
      • 3.1.2.3.4. Group Shuffle Split
    • 3.1.2.4. Predefined Fold-Splits / Validation-Sets
    • 3.1.2.5. Cross validation of time series data
      • 3.1.2.5.1. Time Series Split
  • 3.1.3. A note on shuffling
  • 3.1.4. Cross validation and model selection
• 3.2. Tuning the hyper-parameters of an estimator
  • 3.2.1. Exhaustive Grid Search
  • 3.2.2. Randomized Parameter Optimization
  • 3.2.3. Tips for parameter search
    • 3.2.3.1. Specifying an objective metric
    • 3.2.3.2. Specifying multiple metrics for evaluation
    • 3.2.3.3. Composite estimators and parameter spaces
    • 3.2.3.4. Model selection: development and evaluation
    • 3.2.3.5. Parallelism
    • 3.2.3.6. Robustness to failure
  • 3.2.4. Alternatives to brute force parameter search
    • 3.2.4.1. Model specific cross-validation
      • 3.2.4.1.1. sklearn.linear_model.ElasticNetCV
      • 3.2.4.1.2. sklearn.linear_model.LarsCV
      • 3.2.4.1.3. sklearn.linear_model.LassoCV
        • 3.2.4.1.3.1. Examples using sklearn.linear_model.LassoCV
      • 3.2.4.1.4. sklearn.linear_model.LassoLarsCV
        • 3.2.4.1.4.1. Examples using sklearn.linear_model.LassoLarsCV
      • 3.2.4.1.5. sklearn.linear_model.LogisticRegressionCV
      • 3.2.4.1.6. sklearn.linear_model.MultiTaskElasticNetCV
      • 3.2.4.1.7. sklearn.linear_model.MultiTaskLassoCV
      • 3.2.4.1.8. sklearn.linear_model.OrthogonalMatchingPursuitCV
        • 3.2.4.1.8.1. Examples using sklearn.linear_model.OrthogonalMatchingPursuitCV
      • 3.2.4.1.9. sklearn.linear_model.RidgeCV
        • 3.2.4.1.9.1. Examples using sklearn.linear_model.RidgeCV
      • 3.2.4.1.10. sklearn.linear_model.RidgeClassifierCV
    • 3.2.4.2. Information Criterion
      • 3.2.4.2.1. sklearn.linear_model.LassoLarsIC
        • 3.2.4.2.1.1. Examples using sklearn.linear_model.LassoLarsIC
    • 3.2.4.3. Out of Bag Estimates
      • 3.2.4.3.1. sklearn.ensemble.RandomForestClassifier
        • 3.2.4.3.1.1. Examples using sklearn.ensemble.RandomForestClassifier
      • 3.2.4.3.2. sklearn.ensemble.RandomForestRegressor
        • 3.2.4.3.2.1. Examples using sklearn.ensemble.RandomForestRegressor
      • 3.2.4.3.3. sklearn.ensemble.ExtraTreesClassifier
        • 3.2.4.3.3.1. Examples using sklearn.ensemble.ExtraTreesClassifier
      • 3.2.4.3.4. sklearn.ensemble.ExtraTreesRegressor
        • 3.2.4.3.4.1. Examples using sklearn.ensemble.ExtraTreesRegressor
      • 3.2.4.3.5. sklearn.ensemble.GradientBoostingClassifier
        • 3.2.4.3.5.1. Examples using sklearn.ensemble.GradientBoostingClassifier
      • 3.2.4.3.6. sklearn.ensemble.GradientBoostingRegressor
        • 3.2.4.3.6.1. Examples using sklearn.ensemble.GradientBoostingRegressor
• 3.3. Model evaluation: quantifying the quality of predictions
  • 3.3.1. The scoring parameter: defining model evaluation rules
    • 3.3.1.1. Common cases: predefined values
    • 3.3.1.2. Defining your scoring strategy from metric functions
    • 3.3.1.3. Implementing your own scoring object
    • 3.3.1.4. Using multiple metric evaluation
  • 3.3.2. Classification metrics
    • 3.3.2.1. From binary to multiclass and multilabel
    • 3.3.2.2. Accuracy score
    • 3.3.2.3. Balanced accuracy score
    • 3.3.2.4. Cohen’s kappa
    • 3.3.2.5. Confusion matrix
    • 3.3.2.6. Classification report
    • 3.3.2.7. Hamming loss
    • 3.3.2.8. Precision, recall and F-measures
      • 3.3.2.8.1. Binary classification
      • 3.3.2.8.2. Multiclass and multilabel classification
    • 3.3.2.9. Jaccard similarity coefficient score
    • 3.3.2.10. Hinge loss
    • 3.3.2.11. Log loss
    • 3.3.2.12. Matthews correlation coefficient
    • 3.3.2.13. Multi-label confusion matrix
    • 3.3.2.14. Receiver operating characteristic (ROC)
    • 3.3.2.15. Zero one loss
    • 3.3.2.16. Brier score loss
  • 3.3.3. Multilabel ranking metrics
    • 3.3.3.1. Coverage error
    • 3.3.3.2. Label ranking average precision
    • 3.3.3.3. Ranking loss
  • 3.3.4. Regression metrics
    • 3.3.4.1. Explained variance score
    • 3.3.4.2. Max error
    • 3.3.4.3. Mean absolute error
    • 3.3.4.4. Mean squared error
    • 3.3.4.5. Mean squared logarithmic error
    • 3.3.4.6. Median absolute error
    • 3.3.4.7. R² score, the coefficient of determination
  • 3.3.5. Clustering metrics
  • 3.3.6. Dummy estimators
• 3.4. Model persistence
  • 3.4.1. Persistence example
  • 3.4.2. Security & maintainability limitations
• 3.5. Validation curves: plotting scores to evaluate models
  • 3.5.1. Validation curve
  • 3.5.2. Learning curve