Regression Shrinkage Methods | 回归分析中的收缩方法

📚 Regression Shrinkage Methods | 回归分析中的收缩方法

Shrinkage methods are a family of regularized regression techniques that modify the ordinary least squares fitting process to reduce model complexity and improve prediction accuracy. They are especially useful when predictors are numerous, correlated, or when the risk of overfitting is high.

收缩方法是回归分析中的一类正则化技术，通过对普通最小二乘拟合过程进行修改，降低模型复杂度并提升预测准确性。当预测变量数量众多、彼此相关或过拟合风险较高时，这些方法尤其有用。

1. The Problem of Overfitting | 过拟合问题引言

Ordinary least squares (OLS) seeks the coefficient vector that minimizes the residual sum of squares. When the number of predictors p is large relative to the sample size n, OLS estimates become highly variable and may fit noise rather than signal.

普通最小二乘法（OLS）寻找使残差平方和最小的系数向量。当预测变量数量 p 相对样本量 n 较大时，OLS 估计会变得高度不稳定，可能拟合的是噪声而非真实信号。

Overfitting leads to excellent training performance but poor test performance. Shrinkage methods address this by constraining or penalizing the coefficient estimates toward zero.

过拟合会导致训练集表现极佳但测试集表现不佳。收缩方法通过将系数估计向零约束或惩罚，来解决这一问题。

2. Revisiting Ordinary Least Squares | 回顾普通最小二乘法

For a linear regression model with response y and predictors x₁, x₂, … , xₚ, the OLS objective is to minimize the residual sum of squares:

对于包含响应变量 y 和预测变量 x₁, x₂, … , xₚ 的线性回归模型，OLS 的目标是最小化残差平方和：

RSS(β) = ∑ᵢ (yᵢ − β₀ − ∑ⱼ βⱼxᵢⱼ)²

Here β₀ is the intercept and β₁, β₂, … , βₚ are the slope coefficients. The OLS solution is unbiased and has the smallest variance among all unbiased linear estimators under the Gauss-Markov assumptions.

其中 β₀ 是截距，β₁, β₂, … , βₚ 是斜率系数。在高斯-马尔可夫假设下，OLS 解是无偏的，并且在所有线性无偏估计量中方差最小。

However, unbiasedness is not always desirable. A small amount of bias can dramatically reduce variance, yielding a model with lower total prediction error.

然而，无偏性并非总是最佳选择。引入少量偏差可以大幅降低方差，从而得到总预测误差更低的模型。

3. What Are Shrinkage Methods? | 什么是收缩方法？

Shrinkage methods add a penalty term to the OLS objective function. The penalty becomes large when the coefficients are large, so the optimization balances between fitting the data and keeping coefficients small.

收缩方法在 OLS 目标函数中增加一个惩罚项。当系数较大时惩罚也随之增大，因此优化过程需要兼顾拟合数据与保持系数较小两者。

The general form of the penalized objective is:

带惩罚的目标函数一般形式为：

Minimize RSS(β) + λ × Penalty(β)

where λ is a non-negative tuning parameter. When λ = 0, the method reduces to OLS. As λ increases, coefficients are shrunk more aggressively toward zero.

其中 λ 是一个非负调优参数。当 λ = 0 时，该方法退化为 OLS。随着 λ 增大，系数会被更强烈地压缩向零。

Ridge regression uses an L₂ penalty, equal to the sum of squared coefficients.

岭回归使用 L₂ 惩罚，即系数平方和。
Lasso uses an L₁ penalty, equal to the sum of absolute values of coefficients.

套索回归使用 L₁ 惩罚，即系数绝对值之和。
Elastic Net combines both L₁ and L₂ penalties.

弹性网络同时结合 L₁ 与 L₂ 惩罚。

4. Ridge Regression | 岭回归

Ridge regression minimizes the residual sum of squares plus a penalty proportional to the sum of squared coefficients. The intercept is usually not penalized, because shrinking the intercept would make the model dependent on the origin of the response variable.

岭回归最小化残差平方和加上与系数平方和成正比的惩罚项。截距通常不参与惩罚，因为压缩截距会使模型依赖于响应变量的原点。

Minimize ∑ᵢ (yᵢ − β₀ − ∑ⱼ βⱼxᵢⱼ)² + λ ∑ⱼ βⱼ²

Ridge coefficients are shrunk continuously toward zero but are never exactly zero unless λ is infinite. Therefore ridge regression retains all predictors in the model, which is a form of continuous variable selection rather than discrete selection.

岭回归的系数会被连续压缩向零，但除非 λ 无穷大，否则不会真正变为零。因此岭回归保留模型中的所有预测变量，这是一种连续型变量选择而非离散选择。

In the case of highly correlated predictors, ridge regression tends to pull their coefficients toward each other, distributing the shared explanatory power among them.

当预测变量高度相关时，岭回归倾向于将它们的系数彼此拉近，在它们之间分配共同的解释能力。

5. Lasso Regression | 套索回归

Lasso, which stands for least absolute shrinkage and selection operator, uses the L₁ penalty. Its objective is:

套索回归（Lasso，即最小绝对收缩与选择算子）使用 L₁ 惩罚，其目标为：

Minimize ∑ᵢ (yᵢ − β₀ − ∑ⱼ βⱼxᵢⱼ)² + λ ∑ⱼ |βⱼ|

The absolute value penalty creates a constraint region with sharp corners. When the unconstrained OLS solution lies outside this region, the optimum occurs at a corner, causing some coefficients to be exactly zero.

绝对值惩罚产生带有尖锐棱角的约束区域。当无约束的 OLS 解位于该区域之外时，最优点落在棱角处，导致部分系数恰好为零。

This sparsity property makes Lasso a natural tool for variable selection. It automatically selects a subset of predictors while shrinking the remaining coefficients.

这种稀疏性使 Lasso 成为自然的变量选择工具。它自动选择部分预测变量，同时压缩其余系数。

However, if predictors are strongly correlated, Lasso tends to select only one of them and ignore the others, which can be unstable. This limitation motivates the elastic net.

然而，当预测变量强相关时，Lasso 倾向于只选择其中一个而忽略其他变量，这可能导致不稳定。这一局限催生了弹性网络。

6. Elastic Net | 弹性网络

Elastic net combines the L₁ and L₂ penalties. Its objective with a mixing parameter α is:

弹性网络将 L₁ 与 L₂ 惩罚结合起来。引入混合参数 α 后，其目标为：

Minimize ∑ᵢ (yᵢ − β₀ − ∑ⱼ βⱼxᵢⱼ)² + λ [ (1−α) ∑ⱼ βⱼ² + α ∑ⱼ |βⱼ| ]

When α = 0, elastic net reduces to ridge regression. When α = 1, it reduces to lasso. For values between 0 and 1, the method enjoys both the grouping property of ridge and the sparsity property of lasso.

当 α = 0 时，弹性网络退化为岭回归；当 α = 1 时，它退化为 Lasso。当 α 介于 0 与 1 之间时，该方法同时具备岭回归的群组效应和 Lasso 的稀疏性。

Elastic net is particularly effective when p is much larger than n and when predictors are grouped. It encourages groups of highly correlated variables to enter or leave the model together.

弹性网络在 p 远大于 n、且预测变量呈分组结构时尤为有效。它鼓励高度相关的变量组一同进入或离开模型。

Method	Penalty	Can set coefficients to zero
Ridge	λ ∑βⱼ²	No
Lasso	λ ∑\|βⱼ\|	Yes
Elastic Net	λ [(1−α)∑βⱼ² + α∑\|βⱼ\|]	Yes

7. Bias-Variance Tradeoff | 偏差与方差的权衡

Shrinkage methods work by trading a little bias for a large reduction in variance. The expected mean squared error of prediction can be written as:

收缩方法的核心思想是用少量偏差换取方差的大幅降低。预测的期望均方误差可以写成：

E[(ŷ − y)²] = Bias² + Variance + Irreducible Error

OLS estimates are unbiased, so their bias is zero, but their variance can be enormous when the design matrix is ill-conditioned or when p is close to n.

OLS 估计是无偏的，因此偏差为零，但当设计矩阵病态或 p 接近 n 时，其方差可能极其巨大。

As λ increases, the variance of the estimated coefficients decreases steadily. At the same time, the squared bias increases. The optimal λ minimizes the sum of these two components.

随着 λ 增加，系数估计的方差稳步下降，同时平方偏差逐渐增大。最优的 λ 使两者之和最小化。

Small λ: close to OLS, low bias, high variance.

较小的 λ：接近 OLS，偏差低，方差高。
Large λ: strong shrinkage, high bias, low variance.

较大的 λ：收缩强烈，偏差高，方差低。
Optimal λ: balanced tradeoff, minimized prediction error.

最优 λ：偏差与方差平衡，预测误差最小。

8. Choosing the Tuning Parameter λ | 选择调优参数 λ

In practice, λ is not fixed in advance. It is chosen by cross-validation, typically k-fold cross-validation. The data are split into k folds, and for each candidate λ, the model is trained on k−1 folds and evaluated on the remaining fold.

实践中，λ 并非预先固定，而是通过交叉验证来选择的，通常采用 k 折交叉验证。将数据分成 k 份，对每个候选 λ，模型在 k−1 份上训练，并在剩余的一份上评估。

The validation error is averaged over all folds. The value of λ that produces the smallest average error is selected. A common convention is to choose the largest λ within one standard error of the minimum, giving a simpler model.

验证误差在所有折上取平均。平均误差最小的 λ 被选中。常见惯例是选择最小误差一个标准误范围内的最大 λ，以获得更简洁的模型。

Cross-validation prevents the tuning process from merely minimizing training error. Using the test data to choose λ would lead to optimistic estimates of model performance.

交叉验证可以防止调参过程仅仅最小化训练误差。若使用测试数据来选择 λ，会对模型性能产生过于乐观的估计。

9. Scaling and Standardization | 缩放与标准化

Shrinkage penalties are not invariant to the scale of predictors. If one predictor is measured in kilograms and another in kilograms per squared meter, the penalty terms will unfairly penalize the variable with smaller numerical values.

收缩惩罚对预测变量的尺度不是不变的。如果一个预测变量以千克为单位，另一个以千克每平方米为单位，惩罚项将不公平地惩罚数值较小的变量。

Therefore predictors are usually standardized before applying ridge, lasso, or elastic net. Each predictor is centered by subtracting its mean and scaled by dividing by its standard deviation.

因此，在应用岭回归、Lasso 或弹性网络之前，通常对预测变量进行标准化。每个预测变量先减去均值进行中心化，再除以标准差进行缩放。

The intercept is estimated separately and is not penalized. After standardization, the penalty treats all predictors symmetrically.

截距单独估计且不受惩罚。标准化后，惩罚对所有预测变量对称处理。

When reporting final models, coefficients can be transformed back to the original scales, but the model selection process should be performed on standardized variables.

在报告最终模型时，可以将系数转换回原始尺度，但模型选择过程应在标准化变量上进行。

10. A Simple Numerical Illustration | 简单数值示例

Consider a simple linear regression with one predictor that has been centered and scaled so that the OLS slope estimate is β̂ = 4 and the residual sum of squares is minimized.

考虑一个简单线性回归，预测变量已经中心化并缩放，OLS 斜率估计为 β̂ = 4，且残差平方和达到最小。

Under ridge regression with an L₂ penalty, the new estimate satisfies approximately:

在 L₂ 惩罚的岭回归下，新估计近似满足：

β̂_ridge ≈ β̂_ols / (1 + λ)

If λ = 1, the ridge estimate becomes 4/2 = 2. The coefficient is pulled halfway toward zero.

如果 λ = 1，岭回归估计变为 4/2 = 2。系数被向零拉动一半。

Under lasso with an L₁ penalty, and assuming the design is orthonormal, the soft-thresholding rule gives:

在 L₁ 惩罚的 Lasso 下，假设设计正交，软阈值规则给出：

β̂_lasso = sign(β̂_ols) × max(|β̂_ols| − λ, 0)

With λ = 1, the lasso estimate is max(4 − 1, 0) = 3. If λ exceeds 4, the lasso coefficient becomes exactly zero.

当 λ = 1 时，Lasso 估计为 max(4 − 1, 0) = 3。如果 λ 大于 4，Lasso 系数将精确变为零。

This simple example illustrates the shrinkage property: ridge shrinks multiplicatively, while lasso subtracts a constant and can produce exact zeros.

这个简单示例说明了收缩的性质：岭回归按比例缩小，而 Lasso 减去一个常数并可能产生精确的零。

11. Shrinkage in Practice | 实践中的收缩方法

Shrinkage methods are widely used in modern data analysis. In fields such as genomics, economics, and marketing, the number of candidate predictors often exceeds the number of observations.

收缩方法在现代数据分析中应用广泛。在基因组学、经济学和市场营销等领域，候选预测变量的数量往往超过观测数量。

Ridge regression is preferred when most predictors are believed to be relevant and the goal is to improve prediction stability. Lasso is preferred when the goal is interpretability through variable selection.

当认为大多数预测变量都相关且目标是提升预测稳定性时，岭回归更合适；当目标是通过变量选择获得可解释性时，Lasso 更合适。

Elastic net is a strong default choice when the structure of the predictor correlation is unknown, because it combines the strengths of both methods.

当预测变量相关结构未知时，弹性网络是很好的默认选择，因为它结合了两种方法的优点。

Popular software implementations include the glmnet package in R and the Ridge, Lasso, and ElasticNet classes in Python’s scikit-learn library. All of these provide efficient algorithms for computing the entire regularization path.

常用的软件实现包括 R 中的 glmnet 包，以及 Python scikit-learn 库中的 Ridge、Lasso 和 ElasticNet 类。这些都提供了高效算法，可以计算完整的正则化路径。

12. Summary | 总结

Shrinkage methods are essential tools for regression analysis when data are high-dimensional or collinear. By adding a penalty to the OLS objective, they reduce variance, prevent overfitting, and often improve interpretability.

收缩方法是高维或共线性数据回归分析中的重要工具。通过在 OLS 目标中加入惩罚，它们能够降低方差、防止过拟合，并常常提升可解释性。

Ridge regression uses an L₂ penalty and shrinks coefficients without eliminating them. Lasso uses an L₁ penalty and performs automatic variable selection. Elastic net combines both penalties to handle correlated predictors robustly.

岭回归使用 L₂ 惩罚，压缩系数但不将其消除；Lasso 使用 L₁ 惩罚并进行自动变量选择；弹性网络结合两种惩罚，稳健处理相关预测变量。

The tuning parameter λ controls the strength of shrinkage and is best chosen by cross-validation. Predictors should be standardized before fitting any penalized regression model.

调优参数 λ 控制收缩强度，最好通过交叉验证选择。在拟合任何惩罚回归模型之前，应先将预测变量标准化。

A thorough understanding of shrinkage methods allows analysts to move beyond the limitations of ordinary least squares and build models that are both accurate and reliable.

深入理解收缩方法，能够让分析者超越普通最小二乘法的局限，构建既准确又可靠的模型。

Published by TutorHao | Mathematics Revision Series | aleveler.com

更多咨询请联系16621398022（同微信）

Regression Shrinkage Methods | 回归分析中的收缩方法