2.9 Model Selection and the Bias–Variance Tradeoff

结论

模型复杂度↑Bias↓Variance↓



例子

$y_i=f(x_i)+\epsilon_i,E(\epsilon_i)=0,Var(\epsilon_i)=\sigma^2$

使用knn做预测，在点$x_0$处的Excepted prediction error：

$EPE(x_0)=E\left[\left(y_0-\hat{f}(x_0)\right)^2|x_0\right]\\ \ \ =E\left[\left(y_0-E(y_0)\right)^2|x_0\right]+\left[E(\hat{f}(x_0))-E(y_0)|x_0\right]^2+E\left[\hat{f}(x_0)-E(\hat{f}(x_0))\right]^2\\ \ \ =\sigma^2+{Bias}^2(\hat{f}(x_0))+Var(\hat{f}(x_0))\\ \ \ =\sigma^2+\left[f(x_0)-\frac{1}{k}\sum_l^k y_l\right]^2+\frac{\sigma^2}{k}$

k↑在训练集上的表现(考虑k=1，k=2)↓模型复杂度(模型越复杂，在训练集上的表现越好)↓