API changes¶

• Reverted the addition of metrics.ndcg_score and metrics.dcg_score which had been merged into version 0.19.0 by error. The implementations were broken and undocumented.

• return_train_score which was added to model_selection.GridSearchCV, model_selection.RandomizedSearchCV and model_selection.cross_validate in version 0.19.0 will be changing its default value from True to False in version 0.21. We found that calculating training score could have a great effect on cross validation runtime in some cases. Users should explicitly set return_train_score to False if prediction or scoring functions are slow, resulting in a deleterious effect on CV runtime, or to True if they wish to use the calculated scores. #9677 by Kumar Ashutosh and Joel Nothman.

• correlation_models and regression_models from the legacy gaussian processes implementation have been belatedly deprecated. #9717 by Kumar Ashutosh.

Bug fixes¶

• Avoid integer overflows in metrics.matthews_corrcoef. #9693 by Sam Steingold.

• Fixed a bug in the objective function for manifold.TSNE (both exact and with the Barnes-Hut approximation) when n_components >= 3. #9711 by @goncalo-rodrigues.

• Fix regression in model_selection.cross_val_predict where it raised an error with method='predict_proba' for some probabilistic classifiers. #9641 by James Bourbeau.

• Fixed a bug where datasets.make_classification modified its input weights. #9865 by Sachin Kelkar.

• model_selection.StratifiedShuffleSplit now works with multioutput multiclass or multilabel data with more than 1000 columns. #9922 by Charlie Brummitt.

• Fixed a bug with nested and conditional parameter setting, e.g. setting a pipeline step and its parameter at the same time. #9945 by Andreas Müller and Joel Nothman.

Regressions in 0.19.0 fixed in 0.19.1:

• Fixed a bug where parallelised prediction in random forests was not thread-safe and could (rarely) result in arbitrary errors. #9830 by Joel Nothman.

• Fix regression in model_selection.cross_val_predict where it no longer accepted X as a list. #9600 by Rasul Kerimov.

• Fixed handling of cross_val_predict for binary classification with method='decision_function'. #9593 by Reiichiro Nakano and core devs.

• Fix regression in pipeline.Pipeline where it no longer accepted steps as a tuple. #9604 by Joris Van den Bossche.

• Fix bug where n_iter was not properly deprecated, leaving n_iter unavailable for interim use in linear_model.SGDClassifier, linear_model.SGDRegressor, linear_model.PassiveAggressiveClassifier, linear_model.PassiveAggressiveRegressor and linear_model.Perceptron. #9558 by Andreas Müller.

• Dataset fetchers make sure temporary files are closed before removing them, which caused errors on Windows. #9847 by Joan Massich.

• Fixed a regression in manifold.TSNE where it no longer supported metrics other than ‘euclidean’ and ‘precomputed’. #9623 by Oli Blum.

Enhancements¶

• Our test suite and utils.estimator_checks.check_estimators can now be run without Nose installed. #9697 by Joan Massich.

• To improve usability of version 0.19’s pipeline.Pipeline caching, memory now allows joblib.Memory instances. This make use of the new utils.validation.check_memory helper. issue:9584 by Kumar Ashutosh

• Some fixes to examples: #9750, #9788, #9815

• Made a FutureWarning in SGD-based estimators less verbose. #9802 by Vrishank Bhardwaj.

New features¶

Classifiers and regressors

• Added multioutput.ClassifierChain for multi-label classification. By Adam Kleczewski.

• Added solver 'saga' that implements the improved version of Stochastic Average Gradient, in linear_model.LogisticRegression and linear_model.Ridge. It allows the use of L1 penalty with multinomial logistic loss, and behaves marginally better than ‘sag’ during the first epochs of ridge and logistic regression. #8446 by Arthur Mensch.

Other estimators

• Added the neighbors.LocalOutlierFactor class for anomaly detection based on nearest neighbors. #5279 by Nicolas Goix and Alexandre Gramfort.

• Added preprocessing.QuantileTransformer class and preprocessing.quantile_transform function for features normalization based on quantiles. #8363 by Denis Engemann, Guillaume Lemaitre, Olivier Grisel, Raghav RV, Thierry Guillemot, and Gael Varoquaux.

• The new solver 'mu' implements a Multiplicate Update in decomposition.NMF, allowing the optimization of all beta-divergences, including the Frobenius norm, the generalized Kullback-Leibler divergence and the Itakura-Saito divergence. #5295 by Tom Dupre la Tour.

Model selection and evaluation

• model_selection.GridSearchCV and model_selection.RandomizedSearchCV now support simultaneous evaluation of multiple metrics. Refer to the Specifying multiple metrics for evaluation section of the user guide for more information. #7388 by Raghav RV

• Added the model_selection.cross_validate which allows evaluation of multiple metrics. This function returns a dict with more useful information from cross-validation such as the train scores, fit times and score times. Refer to The cross_validate function and multiple metric evaluation section of the userguide for more information. #7388 by Raghav RV

• Added metrics.mean_squared_log_error, which computes the mean square error of the logarithmic transformation of targets, particularly useful for targets with an exponential trend. #7655 by Karan Desai.

• Added metrics.dcg_score and metrics.ndcg_score, which compute Discounted cumulative gain (DCG) and Normalized discounted cumulative gain (NDCG). #7739 by David Gasquez.

• Added the model_selection.RepeatedKFold and model_selection.RepeatedStratifiedKFold. #8120 by Neeraj Gangwar.

Miscellaneous

• Validation that input data contains no NaN or inf can now be suppressed using config_context, at your own risk. This will save on runtime, and may be particularly useful for prediction time. #7548 by Joel Nothman.

• Added a test to ensure parameter listing in docstrings match the function/class signature. #9206 by Alexandre Gramfort and Raghav RV.

Enhancements¶

Trees and ensembles

• The min_weight_fraction_leaf constraint in tree construction is now more efficient, taking a fast path to declare a node a leaf if its weight is less than 2 * the minimum. Note that the constructed tree will be different from previous versions where min_weight_fraction_leaf is used. #7441 by Nelson Liu.

• ensemble.GradientBoostingClassifier and ensemble.GradientBoostingRegressor now support sparse input for prediction. #6101 by Ibraim Ganiev.

• ensemble.VotingClassifier now allows changing estimators by using ensemble.VotingClassifier.set_params. An estimator can also be removed by setting it to None. #7674 by Yichuan Liu.

• tree.export_graphviz now shows configurable number of decimal places. #8698 by Guillaume Lemaitre.

• Added flatten_transform parameter to ensemble.VotingClassifier to change output shape of transform method to 2 dimensional. #7794 by Ibraim Ganiev and Herilalaina Rakotoarison.

Linear, kernelized and related models

• linear_model.SGDClassifier, linear_model.SGDRegressor, linear_model.PassiveAggressiveClassifier, linear_model.PassiveAggressiveRegressor and linear_model.Perceptron now expose max_iter and tol parameters, to handle convergence more precisely. n_iter parameter is deprecated, and the fitted estimator exposes a n_iter_ attribute, with actual number of iterations before convergence. #5036 by Tom Dupre la Tour.

• Added average parameter to perform weight averaging in linear_model.PassiveAggressiveClassifier. #4939 by Andrea Esuli.

• linear_model.RANSACRegressor no longer throws an error when calling fit if no inliers are found in its first iteration. Furthermore, causes of skipped iterations are tracked in newly added attributes, n_skips_*. #7914 by Michael Horrell.

• In gaussian_process.GaussianProcessRegressor, method predict is a lot faster with return_std=True. #8591 by Hadrien Bertrand.

• Added return_std to predict method of linear_model.ARDRegression and linear_model.BayesianRidge. #7838 by Sergey Feldman.

• Memory usage enhancements: Prevent cast from float32 to float64 in: linear_model.MultiTaskElasticNet; linear_model.LogisticRegression when using newton-cg solver; and linear_model.Ridge when using svd, sparse_cg, cholesky or lsqr solvers. #8835, #8061 by Joan Massich and Nicolas Cordier and Thierry Guillemot.

Other predictors

• Custom metrics for the neighbors binary trees now have fewer constraints: they must take two 1d-arrays and return a float. #6288 by Jake Vanderplas.

• algorithm='auto in neighbors estimators now chooses the most appropriate algorithm for all input types and metrics. #9145 by Herilalaina Rakotoarison and Reddy Chinthala.

Decomposition, manifold learning and clustering

• cluster.MiniBatchKMeans and cluster.KMeans now use significantly less memory when assigning data points to their nearest cluster center. #7721 by Jon Crall.

• decomposition.PCA, decomposition.IncrementalPCA and decomposition.TruncatedSVD now expose the singular values from the underlying SVD. They are stored in the attribute singular_values_, like in decomposition.IncrementalPCA. #7685 by Tommy Löfstedt

• decomposition.NMF now faster when beta_loss=0. #9277 by @hongkahjun.

• Memory improvements for method barnes_hut in manifold.TSNE #7089 by Thomas Moreau and Olivier Grisel.

• Optimization schedule improvements for Barnes-Hut manifold.TSNE so the results are closer to the one from the reference implementation lvdmaaten/bhtsne by Thomas Moreau and Olivier Grisel.

• Memory usage enhancements: Prevent cast from float32 to float64 in decomposition.PCA and decomposition.randomized_svd_low_rank. #9067 by Raghav RV.

Preprocessing and feature selection

• Added norm_order parameter to feature_selection.SelectFromModel to enable selection of the norm order when coef_ is more than 1D. #6181 by Antoine Wendlinger.

• Added ability to use sparse matrices in feature_selection.f_regression with center=True. #8065 by Daniel LeJeune.

• Small performance improvement to n-gram creation in feature_extraction.text by binding methods for loops and special-casing unigrams. #7567 by Jaye Doepke

• Relax assumption on the data for the kernel_approximation.SkewedChi2Sampler. Since the Skewed-Chi2 kernel is defined on the open interval \((-skewedness; +\infty)^d\), the transform function should not check whether X < 0 but whether X < -self.skewedness. #7573 by Romain Brault.

• Made default kernel parameters kernel-dependent in kernel_approximation.Nystroem. #5229 by Saurabh Bansod and Andreas Müller.

Model evaluation and meta-estimators

• pipeline.Pipeline is now able to cache transformers within a pipeline by using the memory constructor parameter. #7990 by Guillaume Lemaitre.

• pipeline.Pipeline steps can now be accessed as attributes of its named_steps attribute. #8586 by Herilalaina Rakotoarison.

• Added sample_weight parameter to pipeline.Pipeline.score. #7723 by Mikhail Korobov.

• Added ability to set n_jobs parameter to pipeline.make_union. A TypeError will be raised for any other kwargs. #8028 by Alexander Booth.

• model_selection.GridSearchCV, model_selection.RandomizedSearchCV and model_selection.cross_val_score now allow estimators with callable kernels which were previously prohibited. #8005 by Andreas Müller .

• model_selection.cross_val_predict now returns output of the correct shape for all values of the argument method. #7863 by Aman Dalmia.

• Added shuffle and random_state parameters to shuffle training data before taking prefixes of it based on training sizes in model_selection.learning_curve. #7506 by Narine Kokhlikyan.

• model_selection.StratifiedShuffleSplit now works with multioutput multiclass (or multilabel) data. #9044 by Vlad Niculae.

• Speed improvements to model_selection.StratifiedShuffleSplit. #5991 by Arthur Mensch and Joel Nothman.

• Add shuffle parameter to model_selection.train_test_split. #8845 by themrmax

• multioutput.MultiOutputRegressor and multioutput.MultiOutputClassifier now support online learning using partial_fit. :issue: 8053 by Peng Yu.

• Add max_train_size parameter to model_selection.TimeSeriesSplit #8282 by Aman Dalmia.

• More clustering metrics are now available through metrics.get_scorer and scoring parameters. #8117 by Raghav RV.

• A scorer based on metrics.explained_variance_score is also available. #9259 by Hanmin Qin.

Metrics

• metrics.matthews_corrcoef now support multiclass classification. #8094 by Jon Crall.

• Add sample_weight parameter to metrics.cohen_kappa_score. #8335 by Victor Poughon.

Miscellaneous

• utils.check_estimator now attempts to ensure that methods transform, predict, etc. do not set attributes on the estimator. #7533 by Ekaterina Krivich.

• Added type checking to the accept_sparse parameter in utils.validation methods. This parameter now accepts only boolean, string, or list/tuple of strings. accept_sparse=None is deprecated and should be replaced by accept_sparse=False. #7880 by Josh Karnofsky.

• Make it possible to load a chunk of an svmlight formatted file by passing a range of bytes to datasets.load_svmlight_file. #935 by Olivier Grisel.

• dummy.DummyClassifier and dummy.DummyRegressor now accept non-finite features. #8931 by @Attractadore.

Bug fixes¶

Trees and ensembles

• Fixed a memory leak in trees when using trees with criterion='mae'. #8002 by Raghav RV.

• Fixed a bug where ensemble.IsolationForest uses an an incorrect formula for the average path length #8549 by Peter Wang.

• Fixed a bug where ensemble.AdaBoostClassifier throws ZeroDivisionError while fitting data with single class labels. #7501 by Dominik Krzeminski.

• Fixed a bug in ensemble.GradientBoostingClassifier and ensemble.GradientBoostingRegressor where a float being compared to 0.0 using == caused a divide by zero error. #7970 by He Chen.

• Fix a bug where ensemble.GradientBoostingClassifier and ensemble.GradientBoostingRegressor ignored the min_impurity_split parameter. #8006 by Sebastian Pölsterl.

• Fixed oob_score in ensemble.BaggingClassifier. #8936 by Michael Lewis

• Fixed excessive memory usage in prediction for random forests estimators. #8672 by Mike Benfield.

• Fixed a bug where sample_weight as a list broke random forests in Python 2 #8068 by @xor.

• Fixed a bug where ensemble.IsolationForest fails when max_features is less than 1. #5732 by Ishank Gulati.

• Fix a bug where gradient boosting with loss='quantile' computed negative errors for negative values of ytrue - ypred leading to wrong values when calling __call__. #8087 by Alexis Mignon

• Fix a bug where ensemble.VotingClassifier raises an error when a numpy array is passed in for weights. #7983 by Vincent Pham.

• Fixed a bug where tree.export_graphviz raised an error when the length of features_names does not match n_features in the decision tree. #8512 by Li Li.

Linear, kernelized and related models

• Fixed a bug where linear_model.RANSACRegressor.fit may run until max_iter if it finds a large inlier group early. #8251 by @aivision2020.

• Fixed a bug where naive_bayes.MultinomialNB and naive_bayes.BernoulliNB failed when alpha=0. #5814 by Yichuan Liu and Herilalaina Rakotoarison.

• Fixed a bug where linear_model.LassoLars does not give the same result as the LassoLars implementation available in R (lars library). #7849 by Jair Montoya Martinez.

• Fixed a bug in linear_model.RandomizedLasso, linear_model.Lars, linear_model.LassoLars, linear_model.LarsCV and linear_model.LassoLarsCV, where the parameter precompute was not used consistently across classes, and some values proposed in the docstring could raise errors. #5359 by Tom Dupre la Tour.

• Fix inconsistent results between linear_model.RidgeCV and linear_model.Ridge when using normalize=True. #9302 by Alexandre Gramfort.

• Fix a bug where linear_model.LassoLars.fit sometimes left coef_ as a list, rather than an ndarray. #8160 by CJ Carey.

• Fix linear_model.BayesianRidge.fit to return ridge parameter alpha_ and lambda_ consistent with calculated coefficients coef_ and intercept_. #8224 by Peter Gedeck.

• Fixed a bug in svm.OneClassSVM where it returned floats instead of integer classes. #8676 by Vathsala Achar.

• Fix AIC/BIC criterion computation in linear_model.LassoLarsIC. #9022 by Alexandre Gramfort and Mehmet Basbug.

• Fixed a memory leak in our LibLinear implementation. #9024 by Sergei Lebedev

• Fix bug where stratified CV splitters did not work with linear_model.LassoCV. #8973 by Paulo Haddad.

• Fixed a bug in gaussian_process.GaussianProcessRegressor when the standard deviation and covariance predicted without fit would fail with a unmeaningful error by default. #6573 by Quazi Marufur Rahman and Manoj Kumar.

Other predictors

• Fix semi_supervised.BaseLabelPropagation to correctly implement LabelPropagation and LabelSpreading as done in the referenced papers. #9239 by Andre Ambrosio Boechat, Utkarsh Upadhyay, and Joel Nothman.

Decomposition, manifold learning and clustering

• Fixed the implementation of manifold.TSNE:

• early_exageration parameter had no effect and is now used for the first 250 optimization iterations.

• Fixed the AssertionError: Tree consistency failed exception reported in #8992.

• Improve the learning schedule to match the one from the reference implementation lvdmaaten/bhtsne. by Thomas Moreau and Olivier Grisel.

• Fix a bug in decomposition.LatentDirichletAllocation where the perplexity method was returning incorrect results because the transform method returns normalized document topic distributions as of version 0.18. #7954 by Gary Foreman.

• Fix output shape and bugs with n_jobs > 1 in decomposition.SparseCoder transform and decomposition.sparse_encode for one-dimensional data and one component. This also impacts the output shape of decomposition.DictionaryLearning. #8086 by Andreas Müller.

• Fixed the implementation of explained_variance_ in decomposition.PCA, decomposition.RandomizedPCA and decomposition.IncrementalPCA. #9105 by Hanmin Qin.

• Fixed the implementation of noise_variance_ in decomposition.PCA. #9108 by Hanmin Qin.

• Fixed a bug where cluster.DBSCAN gives incorrect result when input is a precomputed sparse matrix with initial rows all zero. #8306 by Akshay Gupta

• Fix a bug regarding fitting cluster.KMeans with a sparse array X and initial centroids, where X’s means were unnecessarily being subtracted from the centroids. #7872 by Josh Karnofsky.

• Fixes to the input validation in covariance.EllipticEnvelope. #8086 by Andreas Müller.

• Fixed a bug in covariance.MinCovDet where inputting data that produced a singular covariance matrix would cause the helper method _c_step to throw an exception. #3367 by Jeremy Steward

• Fixed a bug in manifold.TSNE affecting convergence of the gradient descent. #8768 by David DeTomaso.

• Fixed a bug in manifold.TSNE where it stored the incorrect kl_divergence_. #6507 by Sebastian Saeger.

• Fixed improper scaling in cross_decomposition.PLSRegression with scale=True. #7819 by jayzed82.

• cluster.bicluster.SpectralCoclustering and cluster.bicluster.SpectralBiclustering fit method conforms with API by accepting y and returning the object. #6126, #7814 by Laurent Direr and Maniteja Nandana.

• Fix bug where mixture sample methods did not return as many samples as requested. #7702 by Levi John Wolf.

• Fixed the shrinkage implementation in neighbors.NearestCentroid. #9219 by Hanmin Qin.

Preprocessing and feature selection

• For sparse matrices, preprocessing.normalize with return_norm=True will now raise a NotImplementedError with ‘l1’ or ‘l2’ norm and with norm ‘max’ the norms returned will be the same as for dense matrices. #7771 by Ang Lu.

• Fix a bug where feature_selection.SelectFdr did not exactly implement Benjamini-Hochberg procedure. It formerly may have selected fewer features than it should. #7490 by Peng Meng.

• Fixed a bug where linear_model.RandomizedLasso and linear_model.RandomizedLogisticRegression breaks for sparse input. #8259 by Aman Dalmia.

• Fix a bug where feature_extraction.FeatureHasher mandatorily applied a sparse random projection to the hashed features, preventing the use of feature_extraction.text.HashingVectorizer in a pipeline with feature_extraction.text.TfidfTransformer. #7565 by Roman Yurchak.

• Fix a bug where feature_selection.mutual_info_regression did not correctly use n_neighbors. #8181 by Guillaume Lemaitre.

Model evaluation and meta-estimators

• Fixed a bug where model_selection.BaseSearchCV.inverse_transform returns self.best_estimator_.transform() instead of self.best_estimator_.inverse_transform(). #8344 by Akshay Gupta and Rasmus Eriksson.

• Added classes_ attribute to model_selection.GridSearchCV, model_selection.RandomizedSearchCV, grid_search.GridSearchCV, and grid_search.RandomizedSearchCV that matches the classes_ attribute of best_estimator_. #7661 and #8295 by Alyssa Batula, Dylan Werner-Meier, and Stephen Hoover.

• Fixed a bug where model_selection.validation_curve reused the same estimator for each parameter value. #7365 by Aleksandr Sandrovskii.

• model_selection.permutation_test_score now works with Pandas types. #5697 by Stijn Tonk.

• Several fixes to input validation in multiclass.OutputCodeClassifier #8086 by Andreas Müller.

• multiclass.OneVsOneClassifier’s partial_fit now ensures all classes are provided up-front. #6250 by Asish Panda.

• Fix multioutput.MultiOutputClassifier.predict_proba to return a list of 2d arrays, rather than a 3d array. In the case where different target columns had different numbers of classes, a ValueError would be raised on trying to stack matrices with different dimensions. #8093 by Peter Bull.

• Cross validation now works with Pandas datatypes that that have a read-only index. #9507 by Loic Esteve.

Metrics

• metrics.average_precision_score no longer linearly interpolates between operating points, and instead weighs precisions by the change in recall since the last operating point, as per the Wikipedia entry. (#7356). By Nick Dingwall and Gael Varoquaux.

• Fix a bug in metrics.classification._check_targets which would return 'binary' if y_true and y_pred were both 'binary' but the union of y_true and y_pred was 'multiclass'. #8377 by Loic Esteve.

• Fixed an integer overflow bug in metrics.confusion_matrix and hence metrics.cohen_kappa_score. #8354, #7929 by Joel Nothman and Jon Crall.

• Fixed passing of gamma parameter to the chi2 kernel in metrics.pairwise.pairwise_kernels #5211 by Nick Rhinehart, Saurabh Bansod and Andreas Müller.

Miscellaneous

• Fixed a bug when datasets.make_classification fails when generating more than 30 features. #8159 by Herilalaina Rakotoarison.

• Fixed a bug where datasets.make_moons gives an incorrect result when n_samples is odd. #8198 by Josh Levy.

• Some fetch_ functions in datasets were ignoring the download_if_missing keyword. #7944 by Ralf Gommers.

• Fix estimators to accept a sample_weight parameter of type pandas.Series in their fit function. #7825 by Kathleen Chen.

• Fix a bug in cases where numpy.cumsum may be numerically unstable, raising an exception if instability is identified. #7376 and #7331 by Joel Nothman and @yangarbiter.

• Fix a bug where base.BaseEstimator.__getstate__ obstructed pickling customizations of child-classes, when used in a multiple inheritance context. #8316 by Holger Peters.

• Update Sphinx-Gallery from 0.1.4 to 0.1.7 for resolving links in documentation build with Sphinx>1.5 #8010, #7986 by Oscar Najera

• Add data_home parameter to sklearn.datasets.fetch_kddcup99. #9289 by Loic Esteve.

• Fix dataset loaders using Python 3 version of makedirs to also work in Python 2. #9284 by Sebastin Santy.

• Several minor issues were fixed with thanks to the alerts of [lgtm.com](https://lgtm.com/). #9278 by Jean Helie, among others.