1.1. Generalized Linear Models¶
The following are a set of methods intended for regression in which the target value is expected to be a linear combination of the input variables. In mathematical notion, if is the predicted value.
Across the module, we designate the vector as
coef_ and as
To perform classification with generalized linear models, see Logistic regression.
1.1.1. Ordinary Least Squares¶
LinearRegression fits a linear model with coefficients
to minimize the residual sum
of squares between the observed responses in the dataset, and the
responses predicted by the linear approximation. Mathematically it
solves a problem of the form:
LinearRegression will take in its
fit method arrays X, y
and will store the coefficients of the linear model in its
>>> from sklearn import linear_model >>> clf = linear_model.LinearRegression() >>> clf.fit ([[0, 0], [1, 1], [2, 2]], [0, 1, 2]) LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False) >>> clf.coef_ array([ 0.5, 0.5])
However, coefficient estimates for Ordinary Least Squares rely on the independence of the model terms. When terms are correlated and the columns of the design matrix have an approximate linear dependence, the design matrix becomes close to singular and as a result, the least-squares estimate becomes highly sensitive to random errors in the observed response, producing a large variance. This situation of multicollinearity can arise, for example, when data are collected without an experimental design.
220.127.116.11. Ordinary Least Squares Complexity¶
This method computes the least squares solution using a singular value decomposition of X. If X is a matrix of size (n, p) this method has a cost of , assuming that .
1.1.2. Ridge Regression¶
Here, is a complexity parameter that controls the amount of shrinkage: the larger the value of , the greater the amount of shrinkage and thus the coefficients become more robust to collinearity.
As with other linear models,
Ridge will take in its
arrays X, y and will store the coefficients of the linear model in
>>> from sklearn import linear_model >>> clf = linear_model.Ridge (alpha = .5) >>> clf.fit ([[0, 0], [0, 0], [1, 1]], [0, .1, 1]) Ridge(alpha=0.5, copy_X=True, fit_intercept=True, max_iter=None, normalize=False, random_state=None, solver='auto', tol=0.001) >>> clf.coef_ array([ 0.34545455, 0.34545455]) >>> clf.intercept_ 0.13636...
18.104.22.168. Ridge Complexity¶
This method has the same order of complexity than an Ordinary Least Squares.
22.214.171.124. Setting the regularization parameter: generalized Cross-Validation¶
RidgeCV implements ridge regression with built-in
cross-validation of the alpha parameter. The object works in the same way
as GridSearchCV except that it defaults to Generalized Cross-Validation
(GCV), an efficient form of leave-one-out cross-validation:
>>> from sklearn import linear_model >>> clf = linear_model.RidgeCV(alphas=[0.1, 1.0, 10.0]) >>> clf.fit([[0, 0], [0, 0], [1, 1]], [0, .1, 1]) RidgeCV(alphas=[0.1, 1.0, 10.0], cv=None, fit_intercept=True, scoring=None, normalize=False) >>> clf.alpha_ 0.1
Lasso is a linear model that estimates sparse coefficients.
It is useful in some contexts due to its tendency to prefer solutions
with fewer parameter values, effectively reducing the number of variables
upon which the given solution is dependent. For this reason, the Lasso
and its variants are fundamental to the field of compressed sensing.
Under certain conditions, it can recover the exact set of non-zero
Compressive sensing: tomography reconstruction with L1 prior (Lasso)).
Mathematically, it consists of a linear model trained with prior as regularizer. The objective function to minimize is:
The lasso estimate thus solves the minimization of the least-squares penalty with added, where is a constant and is the -norm of the parameter vector.
>>> from sklearn import linear_model >>> clf = linear_model.Lasso(alpha = 0.1) >>> clf.fit([[0, 0], [1, 1]], [0, 1]) Lasso(alpha=0.1, copy_X=True, fit_intercept=True, max_iter=1000, normalize=False, positive=False, precompute=False, random_state=None, selection='cyclic', tol=0.0001, warm_start=False) >>> clf.predict([[1, 1]]) array([ 0.8])
Also useful for lower-level tasks is the function
computes the coefficients along the full path of possible values.
Feature selection with Lasso
As the Lasso regression yields sparse models, it can thus be used to perform feature selection, as detailed in L1-based feature selection.
For feature selection or sparse recovery, it may be interesting to use Randomized sparse models.
126.96.36.199. Setting regularization parameter¶
alpha parameter controls the degree of sparsity of the coefficients
188.8.131.52.1. Using cross-validation¶
For high-dimensional datasets with many collinear regressors,
LassoCV is most often preferable. However,
the advantage of exploring more relevant values of alpha parameter, and
if the number of samples is very small compared to the number of
observations, it is often faster than
184.108.40.206.2. Information-criteria based model selection¶
Alternatively, the estimator
LassoLarsIC proposes to use the
Akaike information criterion (AIC) and the Bayes Information criterion (BIC).
It is a computationally cheaper alternative to find the optimal value of alpha
as the regularization path is computed only once instead of k+1 times
when using k-fold cross-validation. However, such criteria needs a
proper estimation of the degrees of freedom of the solution, are
derived for large samples (asymptotic results) and assume the model
is correct, i.e. that the data are actually generated by this model.
They also tend to break when the problem is badly conditioned
(more features than samples).
1.1.4. Elastic Net¶
ElasticNet is a linear regression model trained with L1 and L2 prior
as regularizer. This combination allows for learning a sparse model where
few of the weights are non-zero like
Lasso, while still maintaining
the regularization properties of
Ridge. We control the convex
combination of L1 and L2 using the
Elastic-net is useful when there are multiple features which are correlated with one another. Lasso is likely to pick one of these at random, while elastic-net is likely to pick both.
A practical advantage of trading-off between Lasso and Ridge is it allows Elastic-Net to inherit some of Ridge’s stability under rotation.
The objective function to minimize is in this case
ElasticNetCV can be used to set the parameters
alpha () and
l1_ratio () by cross-validation.
1.1.5. Multi-task Lasso¶
MultiTaskLasso is a linear model that estimates sparse
coefficients for multiple regression problems jointly:
y is a 2D array,
of shape (n_samples, n_tasks). The constraint is that the selected
features are the same for all the regression problems, also called tasks.
The following figure compares the location of the non-zeros in W obtained with a simple Lasso or a MultiTaskLasso. The Lasso estimates yields scattered non-zeros while the non-zeros of the MultiTaskLasso are full columns.
Fitting a time-series model, imposing that any active feature be active at all times.
Mathematically, it consists of a linear model trained with a mixed prior as regularizer. The objective function to minimize is:
The implementation in the class
MultiTaskLasso uses coordinate descent as
the algorithm to fit the coefficients.
1.1.6. Least Angle Regression¶
Least-angle regression (LARS) is a regression algorithm for high-dimensional data, developed by Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani.
The advantages of LARS are:
- It is numerically efficient in contexts where p >> n (i.e., when the number of dimensions is significantly greater than the number of points)
- It is computationally just as fast as forward selection and has the same order of complexity as an ordinary least squares.
- It produces a full piecewise linear solution path, which is useful in cross-validation or similar attempts to tune the model.
- If two variables are almost equally correlated with the response, then their coefficients should increase at approximately the same rate. The algorithm thus behaves as intuition would expect, and also is more stable.
- It is easily modified to produce solutions for other estimators, like the Lasso.
The disadvantages of the LARS method include:
- Because LARS is based upon an iterative refitting of the residuals, it would appear to be especially sensitive to the effects of noise. This problem is discussed in detail by Weisberg in the discussion section of the Efron et al. (2004) Annals of Statistics article.
1.1.7. LARS Lasso¶
LassoLars is a lasso model implemented using the LARS
algorithm, and unlike the implementation based on coordinate_descent,
this yields the exact solution, which is piecewise linear as a
function of the norm of its coefficients.
>>> from sklearn import linear_model >>> clf = linear_model.LassoLars(alpha=.1) >>> clf.fit([[0, 0], [1, 1]], [0, 1]) LassoLars(alpha=0.1, copy_X=True, eps=..., fit_intercept=True, fit_path=True, max_iter=500, normalize=True, positive=False, precompute='auto', verbose=False) >>> clf.coef_ array([ 0.717157..., 0. ])
The Lars algorithm provides the full path of the coefficients along
the regularization parameter almost for free, thus a common operation
consist of retrieving the path with function
220.127.116.11. Mathematical formulation¶
The algorithm is similar to forward stepwise regression, but instead of including variables at each step, the estimated parameters are increased in a direction equiangular to each one’s correlations with the residual.
Instead of giving a vector result, the LARS solution consists of a
curve denoting the solution for each value of the L1 norm of the
parameter vector. The full coefficients path is stored in the array
coef_path_, which has size (n_features, max_features+1). The first
column is always zero.
- Original Algorithm is detailed in the paper Least Angle Regression by Hastie et al.
1.1.8. Orthogonal Matching Pursuit (OMP)¶
orthogonal_mp implements the OMP
algorithm for approximating the fit of a linear model with constraints imposed
on the number of non-zero coefficients (ie. the L 0 pseudo-norm).
Being a forward feature selection method like Least Angle Regression, orthogonal matching pursuit can approximate the optimum solution vector with a fixed number of non-zero elements:
Alternatively, orthogonal matching pursuit can target a specific error instead of a specific number of non-zero coefficients. This can be expressed as:
OMP is based on a greedy algorithm that includes at each step the atom most highly correlated with the current residual. It is similar to the simpler matching pursuit (MP) method, but better in that at each iteration, the residual is recomputed using an orthogonal projection on the space of the previously chosen dictionary elements.
1.1.9. Bayesian Regression¶
Bayesian regression techniques can be used to include regularization parameters in the estimation procedure: the regularization parameter is not set in a hard sense but tuned to the data at hand.
This can be done by introducing uninformative priors over the hyper parameters of the model. The regularization used in Ridge Regression is equivalent to finding a maximum a-postiori solution under a Gaussian prior over the parameters with precision . Instead of setting lambda manually, it is possible to treat it as a random variable to be estimated from the data.
To obtain a fully probabilistic model, the output is assumed to be Gaussian distributed around :
Alpha is again treated as a random variable that is to be estimated from the data.
The advantages of Bayesian Regression are:
- It adapts to the data at hand.
- It can be used to include regularization parameters in the estimation procedure.
The disadvantages of Bayesian regression include:
- Inference of the model can be time consuming.
- A good introduction to Bayesian methods is given in C. Bishop: Pattern Recognition and Machine learning
- Original Algorithm is detailed in the book Bayesian learning for neural networks by Radford M. Neal
18.104.22.168. Bayesian Ridge Regression¶
BayesianRidge estimates a probabilistic model of the
regression problem as described above.
The prior for the parameter is given by a spherical Gaussian:
The priors over and are chosen to be gamma distributions, the conjugate prior for the precision of the Gaussian.
The resulting model is called Bayesian Ridge Regression, and is similar to the
Ridge. The parameters , and
are estimated jointly during the fit of the model. The
remaining hyperparameters are the parameters of the gamma priors over
and . These are usually chosen to be
non-informative. The parameters are estimated by maximizing the marginal
By default .
Bayesian Ridge Regression is used for regression:
>>> from sklearn import linear_model >>> X = [[0., 0.], [1., 1.], [2., 2.], [3., 3.]] >>> Y = [0., 1., 2., 3.] >>> clf = linear_model.BayesianRidge() >>> clf.fit(X, Y) BayesianRidge(alpha_1=1e-06, alpha_2=1e-06, compute_score=False, copy_X=True, fit_intercept=True, lambda_1=1e-06, lambda_2=1e-06, n_iter=300, normalize=False, tol=0.001, verbose=False)
After being fitted, the model can then be used to predict new values:
>>> clf.predict ([[1, 0.]]) array([ 0.50000013])
The weights of the model can be access:
>>> clf.coef_ array([ 0.49999993, 0.49999993])
Due to the Bayesian framework, the weights found are slightly different to the ones found by Ordinary Least Squares. However, Bayesian Ridge Regression is more robust to ill-posed problem.
- More details can be found in the article Bayesian Interpolation by MacKay, David J. C.
22.214.171.124. Automatic Relevance Determination - ARD¶
ARDRegression is very similar to Bayesian Ridge Regression,
but can lead to sparser weights  .
ARDRegression poses a different prior over , by dropping the
assumption of the Gaussian being spherical.
Instead, the distribution over is assumed to be an axis-parallel, elliptical Gaussian distribution.
This means each weight is drawn from a Gaussian distribution, centered on zero and with a precision :
In contrast to Bayesian Ridge Regression, each coordinate of has its own standard deviation . The prior over all is chosen to be the same gamma distribution given by hyperparameters and .
1.1.10. Logistic regression¶
Logistic regression, despite its name, is a linear model for classification rather than regression. Logistic regression is also known in the literature as logit regression, maximum-entropy classification (MaxEnt) or the log-linear classifier. In this model, the probabilities describing the possible outcomes of a single trial are modeled using a logistic function.
The implementation of logistic regression in scikit-learn can be accessed from
implementation can fit a multiclass (one-vs-rest) logistic regression with optional
L2 or L1 regularization.
As an optimization problem, binary class L2 penalized logistic regression minimizes the following cost function:
Similarly, L1 regularized logistic regression solves the following optimization problem
The solvers implemented in the class
are “liblinear” (which is a wrapper around the C++ library,
LIBLINEAR), “newton-cg”, “lbfgs” and “sag”.
The “lbfgs” and “newton-cg” solvers only support L2 penalization and are found to converge faster for some high dimensional data. L1 penalization yields sparse predicting weights.
The solver “liblinear” uses a coordinate descent (CD) algorithm based on
Liblinear. For L1 penalization
sklearn.svm.l1_min_c allows to
calculate the lower bound for C in order to get a non “null” (all feature weights to
zero) model. This relies on the excellent
which is shipped with scikit-learn. However, the CD algorithm implemented in
liblinear cannot learn a true multinomial (multiclass) model;
instead, the optimization problem is decomposed in a “one-vs-rest” fashion
so separate binary classifiers are trained for all classes.
This happens under the hood, so
using this solver behave as multiclass classifiers.
Setting multi_class to “multinomial” with the “lbfgs” or “newton-cg” solver
LogisticRegression learns a true multinomial logistic
regression model, which means that its probability estimates should
be better calibrated than the default “one-vs-rest” setting.
“lbfgs”, “newton-cg” and “sag” solvers cannot optimize L1-penalized models, though, so the “multinomial” setting does not learn sparse models.
The solver “sag” uses a Stochastic Average Gradient descent . It does not handle “multinomial” case, and is limited to L2-penalized models, yet it is often faster than other solvers for large datasets, when both the number of samples and the number of features are large.
In a nutshell, one may choose the solver with the following rules:
|Small dataset or L1 penalty||“liblinear”|
|Multinomial loss||“lbfgs” or newton-cg”|
For large dataset, you may also consider using
SGDClassifier with ‘log’ loss.
Differences from liblinear:
There might be a difference in the scores obtained between
LinearSVC and the external liblinear library directly,
fit_intercept=False and the fit
coef_ (or) the data to
be predicted are zeroes. This is because for the sample(s) with
predict the negative class, while liblinear predicts the positive class.
Note that a model with
fit_intercept=False and having many samples with
decision_function zero, is likely to be a underfit, bad model and you are
advised to set
fit_intercept=True and increase the intercept_scaling.
Feature selection with sparse logistic regression
A logistic regression with L1 penalty yields sparse models, and can thus be used to perform feature selection, as detailed in L1-based feature selection.
LogisticRegressionCV implements Logistic Regression with
builtin cross-validation to find out the optimal C parameter.
“newton-cg”, “sag” and “lbfgs” solvers are found to be faster
for high-dimensional dense data, due to warm-starting.
For the multiclass case, if multi_class
option is set to “ovr”, an optimal C is obtained for each class and if
the multi_class option is set to “multinomial”, an optimal C is
obtained that minimizes the cross-entropy loss.
|||Mark Schmidt, Nicolas Le Roux, and Francis Bach: Minimizing Finite Sums with the Stochastic Average Gradient.|
1.1.11. Stochastic Gradient Descent - SGD¶
Stochastic gradient descent is a simple yet very efficient approach
to fit linear models. It is particularly useful when the number of samples
(and the number of features) is very large.
partial_fit method allows only/out-of-core learning.
functionality to fit linear models for classification and regression
using different (convex) loss functions and different penalties.
fits a logistic regression model,
loss="hinge" it fits a linear support vector machine (SVM).
Perceptron is another simple algorithm suitable for large scale
learning. By default:
- It does not require a learning rate.
- It is not regularized (penalized).
- It updates its model only on mistakes.
The last characteristic implies that the Perceptron is slightly faster to train than SGD with the hinge loss and that the resulting models are sparser.
1.1.13. Passive Aggressive Algorithms¶
The passive-aggressive algorithms are a family of algorithms for large-scale
learning. They are similar to the Perceptron in that they do not require a
learning rate. However, contrary to the Perceptron, they include a
PassiveAggressiveClassifier can be used with
loss='hinge' (PA-I) or
loss='squared_hinge' (PA-II). For regression,
PassiveAggressiveRegressor can be used with
loss='epsilon_insensitive' (PA-I) or
- “Online Passive-Aggressive Algorithms” K. Crammer, O. Dekel, J. Keshat, S. Shalev-Shwartz, Y. Singer - JMLR 7 (2006)
1.1.14. Robustness regression: outliers and modeling errors¶
Robust regression is interested in fitting a regression model in the presence of corrupt data: either outliers, or error in the model.
126.96.36.199. Different scenario and useful concepts¶
There are different things to keep in mind when dealing with data corrupted by outliers:
Outliers in X or in y?
Outliers in the y direction Outliers in the X direction
Fraction of outliers versus amplitude of error
The number of outlying points matters, but also how much they are outliers.
Small outliers Large outliers
An important notion of robust fitting is that of breakdown point: the fraction of data that can be outlying for the fit to start missing the inlying data.
Note that in general, robust fitting in high-dimensional setting (large n_features) is very hard. The robust models here will probably not work in these settings.
Trade-offs: which estimator?
- Theil Sen will cope better with medium-size outliers in the X direction, but this property will disappear in large dimensional settings.
When in doubt, use RANSAC
188.8.131.52. RANSAC: RANdom SAmple Consensus¶
RANSAC (RANdom SAmple Consensus) fits a model from random subsets of inliers from the complete data set.
RANSAC is a non-deterministic algorithm producing only a reasonable result with a certain probability, which is dependent on the number of iterations (see max_trials parameter). It is typically used for linear and non-linear regression problems and is especially popular in the fields of photogrammetric computer vision.
The algorithm splits the complete input sample data into a set of inliers, which may be subject to noise, and outliers, which are e.g. caused by erroneous measurements or invalid hypotheses about the data. The resulting model is then estimated only from the determined inliers.
184.108.40.206.1. Details of the algorithm¶
Each iteration performs the following steps:
min_samplesrandom samples from the original data and check whether the set of data is valid (see
- Fit a model to the random subset (
base_estimator.fit) and check whether the estimated model is valid (see
- Classify all data as inliers or outliers by calculating the residuals
to the estimated model (
base_estimator.predict(X) - y) - all data samples with absolute residuals smaller than the
residual_thresholdare considered as inliers.
- Save fitted model as best model if number of inlier samples is maximal. In case the current estimated model has the same number of inliers, it is only considered as the best model if it has better score.
These steps are performed either a maximum number of times (
until one of the special stop criteria are met (see
stop_score). The final model is estimated using all inlier samples (consensus
set) of the previously determined best model.
is_model_valid functions allow to identify and reject
degenerate combinations of random sub-samples. If the estimated model is not
needed for identifying degenerate cases,
is_data_valid should be used as it
is called prior to fitting the model and thus leading to better computational
- “Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography” Martin A. Fischler and Robert C. Bolles - SRI International (1981)
- “Performance Evaluation of RANSAC Family” Sunglok Choi, Taemin Kim and Wonpil Yu - BMVC (2009)
220.127.116.11. Theil-Sen estimator: generalized-median-based estimator¶
TheilSenRegressor estimator uses a generalization of the median in
multiple dimensions. It is thus robust to multivariate outliers. Note however
that the robustness of the estimator decreases quickly with the dimensionality
of the problem. It looses its robustness properties and becomes no
better than an ordinary least squares in high dimension.
18.104.22.168.1. Theoretical considerations¶
TheilSenRegressor is comparable to the Ordinary Least Squares
(OLS) in terms of asymptotic efficiency and as an
unbiased estimator. In contrast to OLS, Theil-Sen is a non-parametric
method which means it makes no assumption about the underlying
distribution of the data. Since Theil-Sen is a median-based estimator, it
is more robust against corrupted data aka outliers. In univariate
setting, Theil-Sen has a breakdown point of about 29.3% in case of a
simple linear regression which means that it can tolerate arbitrary
corrupted data of up to 29.3%.
The implementation of
TheilSenRegressor in scikit-learn follows a
generalization to a multivariate linear regression model  using the
spatial median which is a generalization of the median to multiple
In terms of time and space complexity, Theil-Sen scales according to
which makes it infeasible to be applied exhaustively to problems with a large number of samples and features. Therefore, the magnitude of a subpopulation can be chosen to limit the time and space complexity by considering only a random subset of all possible combinations.
|||Xin Dang, Hanxiang Peng, Xueqin Wang and Heping Zhang: Theil-Sen Estimators in a Multiple Linear Regression Model.|
1.1.15. Polynomial regression: extending linear models with basis functions¶
One common pattern within machine learning is to use linear models trained on nonlinear functions of the data. This approach maintains the generally fast performance of linear methods, while allowing them to fit a much wider range of data.
For example, a simple linear regression can be extended by constructing polynomial features from the coefficients. In the standard linear regression case, you might have a model that looks like this for two-dimensional data:
If we want to fit a paraboloid to the data instead of a plane, we can combine the features in second-order polynomials, so that the model looks like this:
The (sometimes surprising) observation is that this is still a linear model: to see this, imagine creating a new variable
With this re-labeling of the data, our problem can be written
We see that the resulting polynomial regression is in the same class of linear models we’d considered above (i.e. the model is linear in ) and can be solved by the same techniques. By considering linear fits within a higher-dimensional space built with these basis functions, the model has the flexibility to fit a much broader range of data.
Here is an example of applying this idea to one-dimensional data, using polynomial features of varying degrees:
This figure is created using the
This preprocessor transforms an input data matrix into a new data matrix
of a given degree. It can be used as follows:
>>> from sklearn.preprocessing import PolynomialFeatures >>> import numpy as np >>> X = np.arange(6).reshape(3, 2) >>> X array([[0, 1], [2, 3], [4, 5]]) >>> poly = PolynomialFeatures(degree=2) >>> poly.fit_transform(X) array([[ 1., 0., 1., 0., 0., 1.], [ 1., 2., 3., 4., 6., 9.], [ 1., 4., 5., 16., 20., 25.]])
The features of
X have been transformed from to
, and can now be used within
any linear model.
This sort of preprocessing can be streamlined with the Pipeline tools. A single object representing a simple polynomial regression can be created and used as follows:
>>> from sklearn.preprocessing import PolynomialFeatures >>> from sklearn.linear_model import LinearRegression >>> from sklearn.pipeline import Pipeline >>> import numpy as np >>> model = Pipeline([('poly', PolynomialFeatures(degree=3)), ... ('linear', LinearRegression(fit_intercept=False))]) >>> # fit to an order-3 polynomial data >>> x = np.arange(5) >>> y = 3 - 2 * x + x ** 2 - x ** 3 >>> model = model.fit(x[:, np.newaxis], y) >>> model.named_steps['linear'].coef_ array([ 3., -2., 1., -1.])
The linear model trained on polynomial features is able to exactly recover the input polynomial coefficients.
In some cases it’s not necessary to include higher powers of any single feature,
but only the so-called interaction features
that multiply together at most distinct features.
These can be gotten from
PolynomialFeatures with the setting
For example, when dealing with boolean features, for all and is therefore useless; but represents the conjunction of two booleans. This way, we can solve the XOR problem with a linear classifier:
>>> from sklearn.linear_model import Perceptron >>> from sklearn.preprocessing import PolynomialFeatures >>> import numpy as np >>> X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) >>> y = X[:, 0] ^ X[:, 1] >>> X = PolynomialFeatures(interaction_only=True).fit_transform(X) >>> X array([[ 1., 0., 0., 0.], [ 1., 0., 1., 0.], [ 1., 1., 0., 0.], [ 1., 1., 1., 1.]]) >>> clf = Perceptron(fit_intercept=False, n_iter=10, shuffle=False).fit(X, y) >>> clf.score(X, y) 1.0