In statistics and machine learning, lasso (least absolute shrinkage and selection operator; also Lasso or LASSO) is a regression analysis method that performs both variable selection and regularization in order to enhance the prediction accuracy and interpretability of the resulting statistical model. It was originally introduced in geophysics, and later by Robert Tibshirani, who coined the term.
Lasso was originally formulated for linear regression models. This simple case reveals a substantial amount about the estimator. These include its relationship to ridge regression and best subset selection and the connections between lasso coefficient estimates and so-called soft thresholding. It also reveals that (like standard linear regression) the coefficient estimates do not need to be unique if covariates are collinear.
Though originally defined for linear regression, lasso regularization is easily extended to other statistical models including generalized linear models, generalized estimating equations, proportional hazards models, and M-estimators. Lasso's ability to perform subset selection relies on the form of the constraint and has a variety of interpretations including in terms of geometry, Bayesian statistics and convex analysis.
The LASSO is closely related to basis pursuit denoising.
History
Lasso was introduced in order to improve the prediction accuracy and interpretability of regression models. It selects a reduced set of the known covariates for use in a model.
Lasso was developed independently in geophysics literature in 1986, based on prior work that used the penalty for both fitting and penalization of the coefficients. Statistician Robert Tibshirani independently rediscovered and popularized it in 1996, based on Breiman's nonnegative garrote.
Prior to lasso, the most widely used method for choosing covariates was stepwise selection. That approach only improves prediction accuracy in certain cases, such as when only a few covariates have a strong relationship with the outcome. However, in other cases, it can increase prediction error.
At the time, ridge regression was the most popular technique for improving prediction accuracy. Ridge regression improves prediction error by shrinking the sum of the squares of the regression coefficients to be less than a fixed value in order to reduce overfitting, but it does not perform covariate selection and therefore does not help to make the model more interpretable.
Lasso achieves both of these goals by forcing the sum of the absolute value of the regression coefficients to be less than a fixed value, which forces certain coefficients to zero, excluding them from impacting prediction. This idea is similar to ridge regression, which also shrinks the size of the coefficients; however, ridge regression does not set coefficients to zero (and, thus, does not perform variable selection).
Basic form
Least squares
Consider a sample consisting of N cases, each of which consists of p covariates and a single outcome. Let be the outcome and be the covariate vector for the i th case. Then the objective of lasso is to solve
Here is the constant coefficient, is the coefficient vector, and is a prespecified free parameter that determines the degree of regularization.
Letting be the covariate matrix, so that and is the i th row of , the expression can be written more compactly as
where is the standard norm.
Denoting the scalar mean of the data points by and the mean of the response variables by , the resulting estimate for is , so that
and therefore it is standard to work with variables that have been made zero-mean. Additionally, the covariates are typically standardized so that the solution does not depend on the measurement scale.
It can be helpful to rewrite
in the so-called Lagrangian form
where the exact relationship between and is data dependent.
Orthonormal covariates
Some basic properties of the lasso estimator can now be considered.
Assuming first that the covariates are orthonormal so that , where is the Kronecker delta, or, equivalently, , then using subgradient methods it can be shown that
is referred to as the soft thresholding operator, since it translates values towards zero (making them exactly zero if they are small enough) instead of setting smaller values to zero and leaving larger ones untouched as the hard thresholding operator, often denoted , would.
In ridge regression the objective is to minimize
Using and the ridge regression formula: , it yields:
Ridge regression shrinks all coefficients by a uniform factor of and does not set any coefficients to zero. .
