Glossary of Common Terms and API Elements¶

This glossary hopes to definitively represent the tacit and explicit conventions applied in Scikit-learn and its API, while providing a reference for users and contributors. It aims to describe the concepts and either detail their corresponding API or link to other relevant parts of the documentation which do so. By linking to glossary entries from the API Reference and User Guide, we may minimize redundancy and inconsistency.

We begin by listing general concepts (and any that didn’t fit elsewhere), but more specific sets of related terms are listed below: Class APIs and Estimator Types, Target Types, Methods, Parameters, Attributes, Data and sample properties.

General Concepts¶

1d¶

1d array¶

One-dimensional array. A NumPy array whose .shape has length 1. A vector.

2d¶

2d array¶

Two-dimensional array. A NumPy array whose .shape has length 2. Often represents a matrix.

API¶

Refers to both the specific interfaces for estimators implemented in Scikit-learn and the generalized conventions across types of estimators as described in this glossary and overviewed in the contributor documentation.

The specific interfaces that constitute Scikit-learn’s public API are largely documented in API Reference. However, we less formally consider anything as public API if none of the identifiers required to access it begins with _. We generally try to maintain backwards compatibility for all objects in the public API.

Private API, including functions, modules and methods beginning _ are not assured to be stable.

array-like¶

The most common data format for input to Scikit-learn estimators and functions, array-like is any type object for which numpy.asarray will produce an array of appropriate shape (usually 1 or 2-dimensional) of appropriate dtype (usually numeric).

This includes:

a numpy array
a list of numbers
a list of length-k lists of numbers for some fixed length k
a pandas.DataFrame with all columns numeric
a numeric pandas.Series

It excludes:

a sparse matrix
an iterator
a generator

Note that output from scikit-learn estimators and functions (e.g. predictions) should generally be arrays or sparse matrices, or lists thereof (as in multi-output tree.DecisionTreeClassifier’s predict_proba). An estimator where predict() returns a list or a pandas.Series is not valid.

attribute¶

attributes¶

We mostly use attribute to refer to how model information is stored on an estimator during fitting. Any public attribute stored on an estimator instance is required to begin with an alphabetic character and end in a single underscore if it is set in fit or partial_fit. These are what is documented under an estimator’s Attributes documentation. The information stored in attributes is usually either: sufficient statistics used for prediction or transformation; transductive outputs such as labels_ or embedding_; or diagnostic data, such as feature_importances_. Common attributes are listed below.

A public attribute may have the same name as a constructor parameter, with a _ appended. This is used to store a validated or estimated version of the user’s input. For example, decomposition.PCA is constructed with an n_components parameter. From this, together with other parameters and the data, PCA estimates the attribute n_components_.

Further private attributes used in prediction/transformation/etc. may also be set when fitting. These begin with a single underscore and are not assured to be stable for public access.

A public attribute on an estimator instance that does not end in an underscore should be the stored, unmodified value of an __init__ parameter of the same name. Because of this equivalence, these are documented under an estimator’s Parameters documentation.

backwards compatibility¶

We generally try to maintain backward compatibility (i.e. interfaces and behaviors may be extended but not changed or removed) from release to release but this comes with some exceptions:

Public API only: The behavior of objects accessed through private identifiers (those beginning _) may be changed arbitrarily between versions.
As documented: We will generally assume that the users have adhered to the documented parameter types and ranges. If the documentation asks for a list and the user gives a tuple, we do not assure consistent behavior from version to version.
Deprecation: Behaviors may change following a deprecation period (usually two releases long). Warnings are issued using Python’s warnings module.
Keyword arguments: We may sometimes assume that all optional parameters (other than X and y to fit and similar methods) are passed as keyword arguments only and may be positionally reordered.
Bug fixes and enhancements: Bug fixes and – less often – enhancements may change the behavior of estimators, including the predictions of an estimator trained on the same data and random_state. When this happens, we attempt to note it clearly in the changelog.
Serialization: We make no assurances that pickling an estimator in one version will allow it to be unpickled to an equivalent model in the subsequent version. (For estimators in the sklearn package, we issue a warning when this unpickling is attempted, even if it may happen to work.) See Security & maintainability limitations.
utils.estimator_checks.check_estimator: We provide limited backwards compatibility assurances for the estimator checks: we may add extra requirements on estimators tested with this function, usually when these were informally assumed but not formally tested.

Despite this informal contract with our users, the software is provided as is, as stated in the license. When a release inadvertently introduces changes that are not backward compatible, these are known as software regressions.

callable¶

A function, class or an object which implements the __call__ method; anything that returns True when the argument of callable().

categorical feature¶

A categorical or nominal feature is one that has a finite set of discrete values across the population of data. These are commonly represented as columns of integers or strings. Strings will be rejected by most scikit-learn estimators, and integers will be treated as ordinal or count-valued. For the use with most estimators, categorical variables should be one-hot encoded. Notable exceptions include tree-based models such as random forests and gradient boosting models that often work better and faster with integer-coded categorical variables. OrdinalEncoder helps encoding string-valued categorical features as ordinal integers, and OneHotEncoder can be used to one-hot encode categorical features. See also Encoding categorical features and the categorical-encoding package for tools related to encoding categorical features.

clone¶

cloned¶

To copy an estimator instance and create a new one with identical parameters, but without any fitted attributes, using clone.

When fit is called, a meta-estimator usually clones a wrapped estimator instance before fitting the cloned instance. (Exceptions, for legacy reasons, include Pipeline and FeatureUnion.)

If the estimator’s random_state parameter is an integer (or if the estimator doesn’t have a random_state parameter), an exact clone is returned: the clone and the original estimator will give the exact same results. Otherwise, statistical clone is returned: the clone might yield different results from the original estimator. More details can be found in Controlling randomness.

common tests¶

This refers to the tests run on almost every estimator class in Scikit-learn to check they comply with basic API conventions. They are available for external use through utils.estimator_checks.check_estimator, with most of the implementation in sklearn/utils/estimator_checks.py.

Note: Some exceptions to the common testing regime are currently hard-coded into the library, but we hope to replace this by marking exceptional behaviours on the estimator using semantic estimator tags.

cross-fitting¶

cross fitting¶

A resampling method that iteratively partitions data into mutually exclusive subsets to fit two stages. During the first stage, the mutually exclusive subsets enable predictions or transformations to be computed on data not seen during training. The computed data is then used in the second stage. The objective is to avoid having any overfitting in the first stage introduce bias into the input data distribution of the second stage. For examples of its use, see: TargetEncoder, StackingClassifier, StackingRegressor and CalibratedClassifierCV.

cross-validation¶

cross validation¶

A resampling method that iteratively partitions data into mutually exclusive ‘train’ and ‘test’ subsets so model performance can be evaluated on unseen data. This conserves data as avoids the need to hold out a ‘validation’ dataset and accounts for variability as multiple rounds of cross validation are generally performed. See User Guide for more details.

deprecation¶

We use deprecation to slowly violate our backwards compatibility assurances, usually to:

change the default value of a parameter; or
remove a parameter, attribute, method, class, etc.

We will ordinarily issue a warning when a deprecated element is used, although there may be limitations to this. For instance, we will raise a warning when someone sets a parameter that has been deprecated, but may not when they access that parameter’s attribute on the estimator instance.

See the Contributors’ Guide.

dimensionality¶

May be used to refer to the number of features (i.e. n_features), or columns in a 2d feature matrix. Dimensions are, however, also used to refer to the length of a NumPy array’s shape, distinguishing a 1d array from a 2d matrix.

docstring¶

The embedded documentation for a module, class, function, etc., usually in code as a string at the beginning of the object’s definition, and accessible as the object’s __doc__ attribute.

We try to adhere to PEP257, and follow NumpyDoc conventions.

double underscore¶

double underscore notation¶

When specifying parameter names for nested estimators, __ may be used to separate between parent and child in some contexts. The most common use is when setting parameters through a meta-estimator with set_params and hence in specifying a search grid in parameter search. See parameter. It is also used in pipeline.Pipeline.fit for passing sample properties to the fit methods of estimators in the pipeline.

dtype¶

data type¶

NumPy arrays assume a homogeneous data type throughout, available in the .dtype attribute of an array (or sparse matrix). We generally assume simple data types for scikit-learn data: float or integer. We may support object or string data types for arrays before encoding or vectorizing. Our estimators do not work with struct arrays, for instance.

Our documentation can sometimes give information about the dtype precision, e.g. np.int32, np.int64, etc. When the precision is provided, it refers to the NumPy dtype. If an arbitrary precision is used, the documentation will refer to dtype integer or floating. Note that in this case, the precision can be platform dependent. The numeric dtype refers to accepting both integer and floating.

TODO: Mention efficiency and precision issues; casting policy.

duck typing¶

We try to apply duck typing to determine how to handle some input values (e.g. checking whether a given estimator is a classifier). That is, we avoid using isinstance where possible, and rely on the presence or absence of attributes to determine an object’s behaviour. Some nuance is required when following this approach:

For some estimators, an attribute may only be available once it is fitted. For instance, we cannot a priori determine if predict_proba is available in a grid search where the grid includes alternating between a probabilistic and a non-probabilistic predictor in the final step of the pipeline. In the following, we can only determine if clf is probabilistic after fitting it on some data:
```
>>> from sklearn.model_selection import GridSearchCV
>>> from sklearn.linear_model import SGDClassifier
>>> clf = GridSearchCV(SGDClassifier(),
...                    param_grid={'loss': ['log_loss', 'hinge']})
```
This means that we can only check for duck-typed attributes after fitting, and that we must be careful to make meta-estimators only present attributes according to the state of the underlying estimator after fitting.
Checking if an attribute is present (using hasattr) is in general just as expensive as getting the attribute (getattr or dot notation). In some cases, getting the attribute may indeed be expensive (e.g. for some implementations of feature_importances_, which may suggest this is an API design flaw). So code which does hasattr followed by getattr should be avoided; getattr within a try-except block is preferred.
For determining some aspects of an estimator’s expectations or support for some feature, we use estimator tags instead of duck typing.

early stopping¶

This consists in stopping an iterative optimization method before the convergence of the training loss, to avoid over-fitting. This is generally done by monitoring the generalization score on a validation set. When available, it is activated through the parameter early_stopping or by setting a positive n_iter_no_change.

estimator instance¶

We sometimes use this terminology to distinguish an estimator class from a constructed instance. For example, in the following, cls is an estimator class, while est1 and est2 are instances:

cls = RandomForestClassifier
est1 = cls()
est2 = RandomForestClassifier()

examples¶

We try to give examples of basic usage for most functions and classes in the API:

as doctests in their docstrings (i.e. within the sklearn/ library code itself).
as examples in the example gallery rendered (using sphinx-gallery) from scripts in the examples/ directory, exemplifying key features or parameters of the estimator/function. These should also be referenced from the User Guide.
sometimes in the User Guide (built from doc/) alongside a technical description of the estimator.

experimental¶

An experimental tool is already usable but its public API, such as default parameter values or fitted attributes, is still subject to change in future versions without the usual deprecation warning policy.

evaluation metric¶

evaluation metrics¶

Evaluation metrics give a measure of how well a model performs. We may use this term specifically to refer to the functions in metrics (disregarding pairwise), as distinct from the score method and the scoring API used in cross validation. See Metrics and scoring: quantifying the quality of predictions.

These functions usually accept a ground truth (or the raw data where the metric evaluates clustering without a ground truth) and a prediction, be it the output of predict (y_pred), of predict_proba (y_proba), or of an arbitrary score function including decision_function (y_score). Functions are usually named to end with _score if a greater score indicates a better model, and _loss if a lesser score indicates a better model. This diversity of interface motivates the scoring API.

Note that some estimators can calculate metrics that are not included in metrics and are estimator-specific, notably model likelihoods.

estimator tags¶

A proposed feature (e.g. #8022) by which the capabilities of an estimator are described through a set of semantic tags. This would enable some runtime behaviors based on estimator inspection, but it also allows each estimator to be tested for appropriate invariances while being excepted from other common tests.

Some aspects of estimator tags are currently determined through the duck typing of methods like predict_proba and through some special attributes on estimator objects:

_estimator_type¶: This string-valued attribute identifies an estimator as being a classifier, regressor, etc. It is set by mixins such as base.ClassifierMixin, but needs to be more explicitly adopted on a meta-estimator. Its value should usually be checked by way of a helper such as base.is_classifier.

For more detailed info, see Estimator Tags.

feature¶

features¶

feature vector¶

In the abstract, a feature is a function (in its mathematical sense) mapping a sampled object to a numeric or categorical quantity. “Feature” is also commonly used to refer to these quantities, being the individual elements of a vector representing a sample. In a data matrix, features are represented as columns: each column contains the result of applying a feature function to a set of samples.

Elsewhere features are known as attributes, predictors, regressors, or independent variables.

Nearly all estimators in scikit-learn assume that features are numeric, finite and not missing, even when they have semantically distinct domains and distributions (categorical, ordinal, count-valued, real-valued, interval). See also categorical feature and missing values.

n_features indicates the number of features in a dataset.

fitting¶

Calling fit (or fit_transform, fit_predict, etc.) on an estimator.

fitted¶

The state of an estimator after fitting.

There is no conventional procedure for checking if an estimator is fitted. However, an estimator that is not fitted:

should raise exceptions.NotFittedError when a prediction method (predict, transform, etc.) is called. (utils.validation.check_is_fitted is used internally for this purpose.)
should not have any attributes beginning with an alphabetic character and ending with an underscore. (Note that a descriptor for the attribute may still be present on the class, but hasattr should return False)

function¶

We provide ad hoc function interfaces for many algorithms, while estimator classes provide a more consistent interface.

In particular, Scikit-learn may provide a function interface that fits a model to some data and returns the learnt model parameters, as in linear_model.enet_path. For transductive models, this also returns the embedding or cluster labels, as in manifold.spectral_embedding or cluster.dbscan. Many preprocessing transformers also provide a function interface, akin to calling fit_transform, as in preprocessing.maxabs_scale. Users should be careful to avoid data leakage when making use of these fit_transform-equivalent functions.

We do not have a strict policy about when to or when not to provide function forms of estimators, but maintainers should consider consistency with existing interfaces, and whether providing a function would lead users astray from best practices (as regards data leakage, etc.)

gallery¶

See examples.

hyperparameter¶

hyper-parameter¶

See parameter.

impute¶

imputation¶

Most machine learning algorithms require that their inputs have no missing values, and will not work if this requirement is violated. Algorithms that attempt to fill in (or impute) missing values are referred to as imputation algorithms.

indexable¶

An array-like, sparse matrix, pandas DataFrame or sequence (usually a list).

induction¶

inductive¶

Inductive (contrasted with transductive) machine learning builds a model of some data that can then be applied to new instances. Most estimators in Scikit-learn are inductive, having predict and/or transform methods.

joblib¶

A Python library (https://joblib.readthedocs.io) used in Scikit-learn to facilite simple parallelism and caching. Joblib is oriented towards efficiently working with numpy arrays, such as through use of memory mapping. See Parallelism for more information.

label indicator matrix¶

multilabel indicator matrix¶

multilabel indicator matrices¶

The format used to represent multilabel data, where each row of a 2d array or sparse matrix corresponds to a sample, each column corresponds to a class, and each element is 1 if the sample is labeled with the class and 0 if not.

leakage¶

data leakage¶

A problem in cross validation where generalization performance can be over-estimated since knowledge of the test data was inadvertently included in training a model. This is a risk, for instance, when applying a transformer to the entirety of a dataset rather than each training portion in a cross validation split.

We aim to provide interfaces (such as pipeline and model_selection) that shield the user from data leakage.

memmapping¶

memory map¶

memory mapping¶

A memory efficiency strategy that keeps data on disk rather than copying it into main memory. Memory maps can be created for arrays that can be read, written, or both, using numpy.memmap. When using joblib to parallelize operations in Scikit-learn, it may automatically memmap large arrays to reduce memory duplication overhead in multiprocessing.

missing values¶

Most Scikit-learn estimators do not work with missing values. When they do (e.g. in impute.SimpleImputer), NaN is the preferred representation of missing values in float arrays. If the array has integer dtype, NaN cannot be represented. For this reason, we support specifying another missing_values value when imputation or learning can be performed in integer space. Unlabeled data is a special case of missing values in the target.

n_features¶

The number of features.

n_outputs¶

The number of outputs in the target.

n_samples¶

The number of samples.

n_targets¶

Synonym for n_outputs.

narrative docs¶

narrative documentation¶

An alias for User Guide, i.e. documentation written in doc/modules/. Unlike the API reference provided through docstrings, the User Guide aims to:

group tools provided by Scikit-learn together thematically or in terms of usage;
motivate why someone would use each particular tool, often through comparison;
provide both intuitive and technical descriptions of tools;
provide or link to examples of using key features of a tool.

np¶

A shorthand for Numpy due to the conventional import statement:

import numpy as np

online learning¶

Where a model is iteratively updated by receiving each batch of ground truth targets soon after making predictions on corresponding batch of data. Intrinsically, the model must be usable for prediction after each batch. See partial_fit.

out-of-core¶

An efficiency strategy where not all the data is stored in main memory at once, usually by performing learning on batches of data. See partial_fit.

outputs¶

Individual scalar/categorical variables per sample in the target. For example, in multilabel classification each possible label corresponds to a binary output. Also called responses, tasks or targets. See multiclass multioutput and continuous multioutput.

pair¶

A tuple of length two.

parameter¶

parameters¶

param¶

params¶

We mostly use parameter to refer to the aspects of an estimator that can be specified in its construction. For example, max_depth and random_state are parameters of RandomForestClassifier. Parameters to an estimator’s constructor are stored unmodified as attributes on the estimator instance, and conventionally start with an alphabetic character and end with an alphanumeric character. Each estimator’s constructor parameters are described in the estimator’s docstring.

We do not use parameters in the statistical sense, where parameters are values that specify a model and can be estimated from data. What we call parameters might be what statisticians call hyperparameters to the model: aspects for configuring model structure that are often not directly learnt from data. However, our parameters are also used to prescribe modeling operations that do not affect the learnt model, such as n_jobs for controlling parallelism.

When talking about the parameters of a meta-estimator, we may also be including the parameters of the estimators wrapped by the meta-estimator. Ordinarily, these nested parameters are denoted by using a double underscore (__) to separate between the estimator-as-parameter and its parameter. Thus clf = BaggingClassifier(estimator=DecisionTreeClassifier(max_depth=3)) has a deep parameter estimator__max_depth with value 3, which is accessible with clf.estimator.max_depth or clf.get_params()['estimator__max_depth'].

The list of parameters and their current values can be retrieved from an estimator instance using its get_params method.

Between construction and fitting, parameters may be modified using set_params. To enable this, parameters are not ordinarily validated or altered when the estimator is constructed, or when each parameter is set. Parameter validation is performed when fit is called.

Common parameters are listed below.

pairwise metric¶

pairwise metrics¶

In its broad sense, a pairwise metric defines a function for measuring similarity or dissimilarity between two samples (with each ordinarily represented as a feature vector). We particularly provide implementations of distance metrics (as well as improper metrics like Cosine Distance) through metrics.pairwise_distances, and of kernel functions (a constrained class of similarity functions) in metrics.pairwise.pairwise_kernels. These can compute pairwise distance matrices that are symmetric and hence store data redundantly.

Class APIs and Estimator Types¶

classifier¶

classifiers¶

A supervised (or semi-supervised) predictor with a finite set of discrete possible output values.

A classifier supports modeling some of binary, multiclass, multilabel, or multiclass multioutput targets. Within scikit-learn, all classifiers support multi-class classification, defaulting to using a one-vs-rest strategy over the binary classification problem.

Classifiers must store a classes_ attribute after fitting, and usually inherit from base.ClassifierMixin, which sets their _estimator_type attribute.

A classifier can be distinguished from other estimators with is_classifier.

A classifier must implement:

fit
predict
score

It may also be appropriate to implement decision_function, predict_proba and predict_log_proba.

clusterer¶

clusterers¶

A unsupervised predictor with a finite set of discrete output values.

A clusterer usually stores labels_ after fitting, and must do so if it is transductive.

A clusterer must implement:

fit
fit_predict if transductive
predict if inductive

density estimator¶

An unsupervised estimation of input probability density function. Commonly used techniques are:

Kernel Density Estimation - uses a kernel function, controlled by the bandwidth parameter to represent density;
Gaussian mixture - uses mixture of Gaussian models to represent density.

estimator¶

estimators¶

An object which manages the estimation and decoding of a model. The model is estimated as a deterministic function of:

parameters provided in object construction or with set_params;
the global numpy.random random state if the estimator’s random_state parameter is set to None; and
any data or sample properties passed to the most recent call to fit, fit_transform or fit_predict, or data similarly passed in a sequence of calls to partial_fit.

The estimated model is stored in public and private attributes on the estimator instance, facilitating decoding through prediction and transformation methods.

Estimators must provide a fit method, and should provide set_params and get_params, although these are usually provided by inheritance from base.BaseEstimator.

The core functionality of some estimators may also be available as a function.

feature extractor¶

feature extractors¶

A transformer which takes input where each sample is not represented as an array-like object of fixed length, and produces an array-like object of features for each sample (and thus a 2-dimensional array-like for a set of samples). In other words, it (lossily) maps a non-rectangular data representation into rectangular data.

Feature extractors must implement at least:

fit
transform
get_feature_names_out

meta-estimator¶

meta-estimators¶

metaestimator¶

metaestimators¶

An estimator which takes another estimator as a parameter. Examples include pipeline.Pipeline, model_selection.GridSearchCV, feature_selection.SelectFromModel and ensemble.BaggingClassifier.

In a meta-estimator’s fit method, any contained estimators should be cloned before they are fit (although FIXME: Pipeline and FeatureUnion do not do this currently). An exception to this is that an estimator may explicitly document that it accepts a pre-fitted estimator (e.g. using prefit=True in feature_selection.SelectFromModel). One known issue with this is that the pre-fitted estimator will lose its model if the meta-estimator is cloned. A meta-estimator should have fit called before prediction, even if all contained estimators are pre-fitted.

In cases where a meta-estimator’s primary behaviors (e.g. predict or transform implementation) are functions of prediction/transformation methods of the provided base estimator (or multiple base estimators), a meta-estimator should provide at least the standard methods provided by the base estimator. It may not be possible to identify which methods are provided by the underlying estimator until the meta-estimator has been fitted (see also duck typing), for which utils.metaestimators.available_if may help. It should also provide (or modify) the estimator tags and classes_ attribute provided by the base estimator.

Meta-estimators should be careful to validate data as minimally as possible before passing it to an underlying estimator. This saves computation time, and may, for instance, allow the underlying estimator to easily work with data that is not rectangular.

outlier detector¶

outlier detectors¶

An unsupervised binary predictor which models the distinction between core and outlying samples.

Outlier detectors must implement:

fit
fit_predict if transductive
predict if inductive

Inductive outlier detectors may also implement decision_function to give a normalized inlier score where outliers have score below 0. score_samples may provide an unnormalized score per sample.

predictor¶

predictors¶

An estimator supporting predict and/or fit_predict. This encompasses classifier, regressor, outlier detector and clusterer.

In statistics, “predictors” refers to features.

regressor¶

regressors¶

A supervised (or semi-supervised) predictor with continuous output values.

Regressors usually inherit from base.RegressorMixin, which sets their _estimator_type attribute.

A regressor can be distinguished from other estimators with is_regressor.

A regressor must implement:

fit
predict
score

transformer¶

transformers¶

An estimator supporting transform and/or fit_transform. A purely transductive transformer, such as manifold.TSNE, may not implement transform.

vectorizer¶

vectorizers¶

See feature extractor.

There are further APIs specifically related to a small family of estimators, such as:

cross-validation splitter¶
CV splitter¶
cross-validation generator¶: A non-estimator family of classes used to split a dataset into a sequence of train and test portions (see Cross-validation: evaluating estimator performance), by providing split and get_n_splits methods. Note that unlike estimators, these do not have fit methods and do not provide set_params or get_params. Parameter validation may be performed in __init__.
cross-validation estimator¶: An estimator that has built-in cross-validation capabilities to automatically select the best hyper-parameters (see the User Guide). Some example of cross-validation estimators are ElasticNetCV and LogisticRegressionCV. Cross-validation estimators are named EstimatorCV and tend to be roughly equivalent to GridSearchCV(Estimator(), ...). The advantage of using a cross-validation estimator over the canonical estimator class along with grid search is that they can take advantage of warm-starting by reusing precomputed results in the previous steps of the cross-validation process. This generally leads to speed improvements. An exception is the RidgeCV class, which can instead perform efficient Leave-One-Out (LOO) CV. By default, all these estimators, apart from RidgeCV with an LOO-CV, will be refitted on the full training dataset after finding the best combination of hyper-parameters.
scorer¶: A non-estimator callable object which evaluates an estimator on given test data, returning a number. Unlike evaluation metrics, a greater returned number must correspond with a better score. See The scoring parameter: defining model evaluation rules.

Further examples:

Metadata Routing¶

consumer¶: An object which consumes metadata. This object is usually an estimator, a scorer, or a CV splitter. Consuming metadata means using it in calculations, e.g. using sample_weight to calculate a certain type of score. Being a consumer doesn’t mean that the object always receives a certain metadata, rather it means it can use it if it is provided.
metadata¶: Data which is related to the given X and y data, but is not directly a part of the data, e.g. sample_weight or groups, and is passed along to different objects and methods, e.g. to a scorer or a CV splitter.
router¶: An object which routes metadata to consumers. This object is usually a meta-estimator, e.g. Pipeline or GridSearchCV. Some routers can also be a consumer. This happens for example when a meta-estimator uses the given groups, and it also passes it along to some of its sub-objects, such as a CV splitter.

Please refer to Metadata Routing User Guide for more information.

Target Types¶

binary¶

A classification problem consisting of two classes. A binary target may be represented as for a multiclass problem but with only two labels. A binary decision function is represented as a 1d array.

Semantically, one class is often considered the “positive” class. Unless otherwise specified (e.g. using pos_label in evaluation metrics), we consider the class label with the greater value (numerically or lexicographically) as the positive class: of labels [0, 1], 1 is the positive class; of [1, 2], 2 is the positive class; of [‘no’, ‘yes’], ‘yes’ is the positive class; of [‘no’, ‘YES’], ‘no’ is the positive class. This affects the output of decision_function, for instance.

Note that a dataset sampled from a multiclass y or a continuous y may appear to be binary.

type_of_target will return ‘binary’ for binary input, or a similar array with only a single class present.

continuous¶

A regression problem where each sample’s target is a finite floating point number represented as a 1-dimensional array of floats (or sometimes ints).

type_of_target will return ‘continuous’ for continuous input, but if the data is all integers, it will be identified as ‘multiclass’.

continuous multioutput¶

continuous multi-output¶

multioutput continuous¶

multi-output continuous¶

A regression problem where each sample’s target consists of n_outputs outputs, each one a finite floating point number, for a fixed int n_outputs > 1 in a particular dataset.

Continuous multioutput targets are represented as multiple continuous targets, horizontally stacked into an array of shape (n_samples, n_outputs).

type_of_target will return ‘continuous-multioutput’ for continuous multioutput input, but if the data is all integers, it will be identified as ‘multiclass-multioutput’.

multiclass¶

multi-class¶

A classification problem consisting of more than two classes. A multiclass target may be represented as a 1-dimensional array of strings or integers. A 2d column vector of integers (i.e. a single output in multioutput terms) is also accepted.

We do not officially support other orderable, hashable objects as class labels, even if estimators may happen to work when given classification targets of such type.

For semi-supervised classification, unlabeled samples should have the special label -1 in y.

Within scikit-learn, all estimators supporting binary classification also support multiclass classification, using One-vs-Rest by default.

A preprocessing.LabelEncoder helps to canonicalize multiclass targets as integers.

type_of_target will return ‘multiclass’ for multiclass input. The user may also want to handle ‘binary’ input identically to ‘multiclass’.

multiclass multioutput¶

multi-class multi-output¶

multioutput multiclass¶

multi-output multi-class¶

A classification problem where each sample’s target consists of n_outputs outputs, each a class label, for a fixed int n_outputs > 1 in a particular dataset. Each output has a fixed set of available classes, and each sample is labeled with a class for each output. An output may be binary or multiclass, and in the case where all outputs are binary, the target is multilabel.

Multiclass multioutput targets are represented as multiple multiclass targets, horizontally stacked into an array of shape (n_samples, n_outputs).

XXX: For simplicity, we may not always support string class labels for multiclass multioutput, and integer class labels should be used.

multioutput provides estimators which estimate multi-output problems using multiple single-output estimators. This may not fully account for dependencies among the different outputs, which methods natively handling the multioutput case (e.g. decision trees, nearest neighbors, neural networks) may do better.

type_of_target will return ‘multiclass-multioutput’ for multiclass multioutput input.

multilabel¶

multi-label¶

A multiclass multioutput target where each output is binary. This may be represented as a 2d (dense) array or sparse matrix of integers, such that each column is a separate binary target, where positive labels are indicated with 1 and negative labels are usually -1 or 0. Sparse multilabel targets are not supported everywhere that dense multilabel targets are supported.

Semantically, a multilabel target can be thought of as a set of labels for each sample. While not used internally, preprocessing.MultiLabelBinarizer is provided as a utility to convert from a list of sets representation to a 2d array or sparse matrix. One-hot encoding a multiclass target with preprocessing.LabelBinarizer turns it into a multilabel problem.

type_of_target will return ‘multilabel-indicator’ for multilabel input, whether sparse or dense.

multioutput¶

multi-output¶

A target where each sample has multiple classification/regression labels. See multiclass multioutput and continuous multioutput. We do not currently support modelling mixed classification and regression targets.

Methods¶

decision_function¶

In a fitted classifier or outlier detector, predicts a “soft” score for each sample in relation to each class, rather than the “hard” categorical prediction produced by predict. Its input is usually only some observed data, X.

If the estimator was not already fitted, calling this method should raise a exceptions.NotFittedError.

Output conventions:

binary classification

A 1-dimensional array, where values strictly greater than zero indicate the positive class (i.e. the last class in classes_).

multiclass classification

A 2-dimensional array, where the row-wise arg-maximum is the predicted class. Columns are ordered according to classes_.

multilabel classification

Scikit-learn is inconsistent in its representation of multilabel decision functions. It may be represented one of two ways:

List of 2d arrays, each array of shape: (n_samples, 2), like in multiclass multioutput. List is of length n_labels.
Single 2d array of shape (n_samples, n_labels), with each ‘column’ in the array corresponding to the individual binary classification decisions. This is identical to the multiclass classification format, though its semantics differ: it should be interpreted, like in the binary case, by thresholding at 0.

multioutput classification

A list of 2d arrays, corresponding to each multiclass decision function.

outlier detection

A 1-dimensional array, where a value greater than or equal to zero indicates an inlier.

fit¶

The fit method is provided on every estimator. It usually takes some samples X, targets y if the model is supervised, and potentially other sample properties such as sample_weight. It should:

clear any prior attributes stored on the estimator, unless warm_start is used;
validate and interpret any parameters, ideally raising an error if invalid;
validate the input data;
estimate and store model attributes from the estimated parameters and provided data; and
return the now fitted estimator to facilitate method chaining.

Target Types describes possible formats for y.

fit_predict¶

Used especially for unsupervised, transductive estimators, this fits the model and returns the predictions (similar to predict) on the training data. In clusterers, these predictions are also stored in the labels_ attribute, and the output of .fit_predict(X) is usually equivalent to .fit(X).predict(X). The parameters to fit_predict are the same as those to fit.

fit_transform¶

A method on transformers which fits the estimator and returns the transformed training data. It takes parameters as in fit and its output should have the same shape as calling .fit(X, ...).transform(X). There are nonetheless rare cases where .fit_transform(X, ...) and .fit(X, ...).transform(X) do not return the same value, wherein training data needs to be handled differently (due to model blending in stacked ensembles, for instance; such cases should be clearly documented). Transductive transformers may also provide fit_transform but not transform.

One reason to implement fit_transform is that performing fit and transform separately would be less efficient than together. base.TransformerMixin provides a default implementation, providing a consistent interface across transformers where fit_transform is or is not specialized.

In inductive learning – where the goal is to learn a generalized model that can be applied to new data – users should be careful not to apply fit_transform to the entirety of a dataset (i.e. training and test data together) before further modelling, as this results in data leakage.

get_feature_names_out¶

Primarily for feature extractors, but also used for other transformers to provide string names for each column in the output of the estimator’s transform method. It outputs an array of strings and may take an array-like of strings as input, corresponding to the names of input columns from which output column names can be generated. If input_features is not passed in, then the feature_names_in_ attribute will be used. If the feature_names_in_ attribute is not defined, then the input names are named [x0, x1, ..., x(n_features_in_ - 1)].

get_n_splits¶

On a CV splitter (not an estimator), returns the number of elements one would get if iterating through the return value of split given the same parameters. Takes the same parameters as split.

get_params¶

Gets all parameters, and their values, that can be set using set_params. A parameter deep can be used, when set to False to only return those parameters not including __, i.e. not due to indirection via contained estimators.

Most estimators adopt the definition from base.BaseEstimator, which simply adopts the parameters defined for __init__. pipeline.Pipeline, among others, reimplements get_params to declare the estimators named in its steps parameters as themselves being parameters.

partial_fit¶

Facilitates fitting an estimator in an online fashion. Unlike fit, repeatedly calling partial_fit does not clear the model, but updates it with the data provided. The portion of data provided to partial_fit may be called a mini-batch. Each mini-batch must be of consistent shape, etc. In iterative estimators, partial_fit often only performs a single iteration.

partial_fit may also be used for out-of-core learning, although usually limited to the case where learning can be performed online, i.e. the model is usable after each partial_fit and there is no separate processing needed to finalize the model. cluster.Birch introduces the convention that calling partial_fit(X) will produce a model that is not finalized, but the model can be finalized by calling partial_fit() i.e. without passing a further mini-batch.

Generally, estimator parameters should not be modified between calls to partial_fit, although partial_fit should validate them as well as the new mini-batch of data. In contrast, warm_start is used to repeatedly fit the same estimator with the same data but varying parameters.

Like fit, partial_fit should return the estimator object.

To clear the model, a new estimator should be constructed, for instance with base.clone.

NOTE: Using partial_fit after fit results in undefined behavior.

predict¶

Makes a prediction for each sample, usually only taking X as input (but see under regressor output conventions below). In a classifier or regressor, this prediction is in the same target space used in fitting (e.g. one of {‘red’, ‘amber’, ‘green’} if the y in fitting consisted of these strings). Despite this, even when y passed to fit is a list or other array-like, the output of predict should always be an array or sparse matrix. In a clusterer or outlier detector the prediction is an integer.

If the estimator was not already fitted, calling this method should raise a exceptions.NotFittedError.

Output conventions:

classifier: An array of shape (n_samples,) (n_samples, n_outputs). Multilabel data may be represented as a sparse matrix if a sparse matrix was used in fitting. Each element should be one of the values in the classifier’s classes_ attribute.
clusterer: An array of shape (n_samples,) where each value is from 0 to n_clusters - 1 if the corresponding sample is clustered, and -1 if the sample is not clustered, as in cluster.dbscan.
outlier detector: An array of shape (n_samples,) where each value is -1 for an outlier and 1 otherwise.
regressor: A numeric array of shape (n_samples,), usually float64. Some regressors have extra options in their predict method, allowing them to return standard deviation (return_std=True) or covariance (return_cov=True) relative to the predicted value. In this case, the return value is a tuple of arrays corresponding to (prediction mean, std, cov) as required.

predict_log_proba¶

The natural logarithm of the output of predict_proba, provided to facilitate numerical stability.

predict_proba¶

A method in classifiers and clusterers that can return probability estimates for each class/cluster. Its input is usually only some observed data, X.

If the estimator was not already fitted, calling this method should raise a exceptions.NotFittedError.

Output conventions are like those for decision_function except in the binary classification case, where one column is output for each class (while decision_function outputs a 1d array). For binary and multiclass predictions, each row should add to 1.

Like other methods, predict_proba should only be present when the estimator can make probabilistic predictions (see duck typing). This means that the presence of the method may depend on estimator parameters (e.g. in linear_model.SGDClassifier) or training data (e.g. in model_selection.GridSearchCV) and may only appear after fitting.

score¶

A method on an estimator, usually a predictor, which evaluates its predictions on a given dataset, and returns a single numerical score. A greater return value should indicate better predictions; accuracy is used for classifiers and R^2 for regressors by default.

If the estimator was not already fitted, calling this method should raise a exceptions.NotFittedError.

Some estimators implement a custom, estimator-specific score function, often the likelihood of the data under the model.

score_samples¶

A method that returns a score for each given sample. The exact definition of score varies from one class to another. In the case of density estimation, it can be the log density model on the data, and in the case of outlier detection, it can be the opposite of the outlier factor of the data.

If the estimator was not already fitted, calling this method should raise a exceptions.NotFittedError.

set_params¶

Available in any estimator, takes keyword arguments corresponding to keys in get_params. Each is provided a new value to assign such that calling get_params after set_params will reflect the changed parameters. Most estimators use the implementation in base.BaseEstimator, which handles nested parameters and otherwise sets the parameter as an attribute on the estimator. The method is overridden in pipeline.Pipeline and related estimators.

split¶

On a CV splitter (not an estimator), this method accepts parameters (X, y, groups), where all may be optional, and returns an iterator over (train_idx, test_idx) pairs. Each of {train,test}_idx is a 1d integer array, with values from 0 from X.shape[0] - 1 of any length, such that no values appear in both some train_idx and its corresponding test_idx.

transform¶

In a transformer, transforms the input, usually only X, into some transformed space (conventionally notated as Xt). Output is an array or sparse matrix of length n_samples and with the number of columns fixed after fitting.

If the estimator was not already fitted, calling this method should raise a exceptions.NotFittedError.

Parameters¶

These common parameter names, specifically used in estimator construction (see concept parameter), sometimes also appear as parameters of functions or non-estimator constructors.

class_weight¶

Used to specify sample weights when fitting classifiers as a function of the target class. Where sample_weight is also supported and given, it is multiplied by the class_weight contribution. Similarly, where class_weight is used in a multioutput (including multilabel) tasks, the weights are multiplied across outputs (i.e. columns of y).

By default, all samples have equal weight such that classes are effectively weighted by their prevalence in the training data. This could be achieved explicitly with class_weight={label1: 1, label2: 1, ...} for all class labels.

More generally, class_weight is specified as a dict mapping class labels to weights ({class_label: weight}), such that each sample of the named class is given that weight.

class_weight='balanced' can be used to give all classes equal weight by giving each sample a weight inversely related to its class’s prevalence in the training data: n_samples / (n_classes * np.bincount(y)). Class weights will be used differently depending on the algorithm: for linear models (such as linear SVM or logistic regression), the class weights will alter the loss function by weighting the loss of each sample by its class weight. For tree-based algorithms, the class weights will be used for reweighting the splitting criterion. Note however that this rebalancing does not take the weight of samples in each class into account.

For multioutput classification, a list of dicts is used to specify weights for each output. For example, for four-class multilabel classification weights should be [{0: 1, 1: 1}, {0: 1, 1: 5}, {0: 1, 1: 1}, {0: 1, 1: 1}] instead of [{1:1}, {2:5}, {3:1}, {4:1}].

The class_weight parameter is validated and interpreted with utils.class_weight.compute_class_weight.

cv¶

Determines a cross validation splitting strategy, as used in cross-validation based routines. cv is also available in estimators such as multioutput.ClassifierChain or calibration.CalibratedClassifierCV which use the predictions of one estimator as training data for another, to not overfit the training supervision.

Possible inputs for cv are usually:

An integer, specifying the number of folds in K-fold cross validation. K-fold will be stratified over classes if the estimator is a classifier (determined by base.is_classifier) and the targets may represent a binary or multiclass (but not multioutput) classification problem (determined by utils.multiclass.type_of_target).
A cross-validation splitter instance. Refer to the User Guide for splitters available within Scikit-learn.
An iterable yielding train/test splits.

With some exceptions (especially where not using cross validation at all is an option), the default is 5-fold.

cv values are validated and interpreted with model_selection.check_cv.

kernel¶

Specifies the kernel function to be used by Kernel Method algorithms. For example, the estimators svm.SVC and gaussian_process.GaussianProcessClassifier both have a kernel parameter that takes the name of the kernel to use as string or a callable kernel function used to compute the kernel matrix. For more reference, see the Kernel Approximation and the Gaussian Processes user guides.

max_iter¶

For estimators involving iterative optimization, this determines the maximum number of iterations to be performed in fit. If max_iter iterations are run without convergence, a exceptions.ConvergenceWarning should be raised. Note that the interpretation of “a single iteration” is inconsistent across estimators: some, but not all, use it to mean a single epoch (i.e. a pass over every sample in the data).

FIXME perhaps we should have some common tests about the relationship between ConvergenceWarning and max_iter.

memory¶

Some estimators make use of joblib.Memory to store partial solutions during fitting. Thus when fit is called again, those partial solutions have been memoized and can be reused.

A memory parameter can be specified as a string with a path to a directory, or a joblib.Memory instance (or an object with a similar interface, i.e. a cache method) can be used.

memory values are validated and interpreted with utils.validation.check_memory.

metric¶

As a parameter, this is the scheme for determining the distance between two data points. See metrics.pairwise_distances. In practice, for some algorithms, an improper distance metric (one that does not obey the triangle inequality, such as Cosine Distance) may be used.

XXX: hierarchical clustering uses affinity with this meaning.

We also use metric to refer to evaluation metrics, but avoid using this sense as a parameter name.

n_components¶

The number of features which a transformer should transform the input into. See components_ for the special case of affine projection.

n_iter_no_change¶

Number of iterations with no improvement to wait before stopping the iterative procedure. This is also known as a patience parameter. It is typically used with early stopping to avoid stopping too early.

n_jobs¶

This parameter is used to specify how many concurrent processes or threads should be used for routines that are parallelized with joblib.

n_jobs is an integer, specifying the maximum number of concurrently running workers. If 1 is given, no joblib parallelism is used at all, which is useful for debugging. If set to -1, all CPUs are used. For n_jobs below -1, (n_cpus + 1 + n_jobs) are used. For example with n_jobs=-2, all CPUs but one are used.

n_jobs is None by default, which means unset; it will generally be interpreted as n_jobs=1, unless the current joblib.Parallel backend context specifies otherwise.

Note that even if n_jobs=1, low-level parallelism (via Numpy and OpenMP) might be used in some configuration.

For more details on the use of joblib and its interactions with scikit-learn, please refer to our parallelism notes.

pos_label¶

Value with which positive labels must be encoded in binary classification problems in which the positive class is not assumed. This value is typically required to compute asymmetric evaluation metrics such as precision and recall.

random_state¶

Whenever randomization is part of a Scikit-learn algorithm, a random_state parameter may be provided to control the random number generator used. Note that the mere presence of random_state doesn’t mean that randomization is always used, as it may be dependent on another parameter, e.g. shuffle, being set.

The passed value will have an effect on the reproducibility of the results returned by the function (fit, split, or any other function like k_means). random_state’s value may be:

None (default): Use the global random state instance from numpy.random. Calling the function multiple times will reuse the same instance, and will produce different results.
An integer: Use a new random number generator seeded by the given integer. Using an int will produce the same results across different calls. However, it may be worthwhile checking that your results are stable across a number of different distinct random seeds. Popular integer random seeds are 0 and 42. Integer values must be in the range [0, 2**32 - 1].
A numpy.random.RandomState instance: Use the provided random state, only affecting other users of that same random state instance. Calling the function multiple times will reuse the same instance, and will produce different results.

utils.check_random_state is used internally to validate the input random_state and return a RandomState instance.

For more details on how to control the randomness of scikit-learn objects and avoid common pitfalls, you may refer to Controlling randomness.

scoring¶

Specifies the score function to be maximized (usually by cross validation), or – in some cases – multiple score functions to be reported. The score function can be a string accepted by metrics.get_scorer or a callable scorer, not to be confused with an evaluation metric, as the latter have a more diverse API. scoring may also be set to None, in which case the estimator’s score method is used. See The scoring parameter: defining model evaluation rules in the User Guide.

Where multiple metrics can be evaluated, scoring may be given either as a list of unique strings, a dictionary with names as keys and callables as values or a callable that returns a dictionary. Note that this does not specify which score function is to be maximized, and another parameter such as refit maybe used for this purpose.

The scoring parameter is validated and interpreted using metrics.check_scoring.

verbose¶

Logging is not handled very consistently in Scikit-learn at present, but when it is provided as an option, the verbose parameter is usually available to choose no logging (set to False). Any True value should enable some logging, but larger integers (e.g. above 10) may be needed for full verbosity. Verbose logs are usually printed to Standard Output. Estimators should not produce any output on Standard Output with the default verbose setting.

warm_start¶

When fitting an estimator repeatedly on the same dataset, but for multiple parameter values (such as to find the value maximizing performance as in grid search), it may be possible to reuse aspects of the model learned from the previous parameter value, saving time. When warm_start is true, the existing fitted model attributes are used to initialize the new model in a subsequent call to fit.

Note that this is only applicable for some models and some parameters, and even some orders of parameter values. In general, there is an interaction between warm_start and the parameter controlling the number of iterations of the estimator.

For estimators imported from ensemble, warm_start will interact with n_estimators or max_iter. For these models, the number of iterations, reported via len(estimators_) or n_iter_, corresponds the total number of estimators/iterations learnt since the initialization of the model. Thus, if a model was already initialized with N estimators, and fit is called with n_estimators or max_iter set to M, the model will train M - N new estimators.

Other models, usually using gradient-based solvers, have a different behavior. They all expose a max_iter parameter. The reported n_iter_ corresponds to the number of iteration done during the last call to fit and will be at most max_iter. Thus, we do not consider the state of the estimator since the initialization.

partial_fit also retains the model between calls, but differs: with warm_start the parameters change and the data is (more-or-less) constant across calls to fit; with partial_fit, the mini-batch of data changes and model parameters stay fixed.

There are cases where you want to use warm_start to fit on different, but closely related data. For example, one may initially fit to a subset of the data, then fine-tune the parameter search on the full dataset. For classification, all data in a sequence of warm_start calls to fit must include samples from each class.

Attributes¶

See concept attribute.

classes_¶

A list of class labels known to the classifier, mapping each label to a numerical index used in the model representation our output. For instance, the array output from predict_proba has columns aligned with classes_. For multi-output classifiers, classes_ should be a list of lists, with one class listing for each output. For each output, the classes should be sorted (numerically, or lexicographically for strings).

classes_ and the mapping to indices is often managed with preprocessing.LabelEncoder.

components_¶

An affine transformation matrix of shape (n_components, n_features) used in many linear transformers where n_components is the number of output features and n_features is the number of input features.

See also components_ which is a similar attribute for linear predictors.

coef_¶

The weight/coefficient matrix of a generalized linear model predictor, of shape (n_features,) for binary classification and single-output regression, (n_classes, n_features) for multiclass classification and (n_targets, n_features) for multi-output regression. Note this does not include the intercept (or bias) term, which is stored in intercept_.

When available, feature_importances_ is not usually provided as well, but can be calculated as the norm of each feature’s entry in coef_.

See also components_ which is a similar attribute for linear transformers.

embedding_¶

An embedding of the training data in manifold learning estimators, with shape (n_samples, n_components), identical to the output of fit_transform. See also labels_.

n_iter_¶

The number of iterations actually performed when fitting an iterative estimator that may stop upon convergence. See also max_iter.

feature_importances_¶

A vector of shape (n_features,) available in some predictors to provide a relative measure of the importance of each feature in the predictions of the model.

labels_¶

A vector containing a cluster label for each sample of the training data in clusterers, identical to the output of fit_predict. See also embedding_.

Data and sample properties¶

See concept sample property.

groups¶

Used in cross-validation routines to identify samples that are correlated. Each value is an identifier such that, in a supporting CV splitter, samples from some groups value may not appear in both a training set and its corresponding test set. See Cross-validation iterators for grouped data.

sample_weight¶

A relative weight for each sample. Intuitively, if all weights are integers, a weighted model or score should be equivalent to that calculated when repeating the sample the number of times specified in the weight. Weights may be specified as floats, so that sample weights are usually equivalent up to a constant positive scaling factor.

FIXME Is this interpretation always the case in practice? We have no common tests.

Some estimators, such as decision trees, support negative weights. FIXME: This feature or its absence may not be tested or documented in many estimators.

This is not entirely the case where other parameters of the model consider the number of samples in a region, as with min_samples in cluster.DBSCAN. In this case, a count of samples becomes to a sum of their weights.

In classification, sample weights can also be specified as a function of class with the class_weight estimator parameter.

X¶

Denotes data that is observed at training and prediction time, used as independent variables in learning. The notation is uppercase to denote that it is ordinarily a matrix (see rectangular). When a matrix, each sample may be represented by a feature vector, or a vector of precomputed (dis)similarity with each training sample. X may also not be a matrix, and may require a feature extractor or a pairwise metric to turn it into one before learning a model.

Xt¶

Shorthand for “transformed X”.

y¶

Y¶

Denotes data that may be observed at training time as the dependent variable in learning, but which is unavailable at prediction time, and is usually the target of prediction. The notation may be uppercase to denote that it is a matrix, representing multi-output targets, for instance; but usually we use y and sometimes do so even when multiple outputs are assumed.