[#1]Least square and Nearest neighbors
2015-12-18 05:25
411 查看
Least square and nearest neighbors
1 Least square in learn regression
2 Nearest neighbors
Rationality and difference of least square and nearest neighbors
rationality least square and nearest neighbors
extension of these simple procedures
Y=(y(1),y(2),⋯,y(N))
X=(X(1)T,X(2)T,⋯,X(N)T)T
Firstly, we need to choose a loss function. Here we choose the Least squares, which aims to minimize the quadratic function of parameter β,
RSSresidual sum of square(β)=∥Y−Xβ∥2F
which leads to the solution β^=(XTX)−1XTY if X is column full rank. Then prediction function at any X is given by y^=XTβ^.
y^(X)=1k∑X(i)∈Nk(X)y(i)
In classification, nearest neighbors method uses the maximum votes of labels of the k-nearest points of X as the class for X, which can be formulated as
g^(X)=argmaxg∑X(i)∈Nk(X)1{y(i)=g}
least square yields stable but possible inaccurate predictions, while predictions of nearest neighbors are often accurate but can be unstable
Note that:
stable means low variance
accurate means low bias
EPEepxect prediction error(f)=E∥Y−f(X)∥2F=EXEY|X[∥Y−f(X)∥2F|X]
It suffices to minimize EPE pointwise
f(x)=argminEY|X[∥Y−f(X)∥2F|X=x]
The solution is
f(x)=E[Y|X=x]
And least square and nearest neighbors both aim to approximate the expectation by averaging.
Least square assumes the linear structure and approximate the expectation in square loss function by averaging all training datas.
\mathrm{\hat{EPE}}(\beta) = \frac{1}{N}\sum_{i=1}^N\|y_i - X^{(i)}^t\beta\|_F^2
Nearest neighbors approximates the conditional expectation in solution by averaging the outputs near target x
Y^=ave(yi|X(i)∈Nk(x))
So, two things are happening in approximating of both least square and nearest neighbor.
Least square
model structure assumption
averaging over all training data in EPE
Nearest neighbors
1.condition on a small region of target point x instead of conditioning on it
2.averaging the outputs which are near to x
Kernel Methods use weights that decrease smoothly to zero with distance from the target point, rather that the effective 0/1 weights used by k-nearest neighbors.
In high dimensional spaces the distance kernels are modified to emphasize some variable more than others.
Local regression fits linear models by locally weighted least squares, rather than fitting constants locally.
Linear models fit to a basis expansion of the original inputs allow arbitrary complex models.
Projection pursuit and neural network models consist of sums of nonlinearly transformed linear models.
1 Least square in learn regression
2 Nearest neighbors
Rationality and difference of least square and nearest neighbors
rationality least square and nearest neighbors
extension of these simple procedures
1. Least square and nearest neighbors
1.1 Least square in learn regression
Assume we have a data set {(X(i),y(i))}Ni=1, and we will fit a linear regression y=XTβ+b on this data set. Notations to be used:Y=(y(1),y(2),⋯,y(N))
X=(X(1)T,X(2)T,⋯,X(N)T)T
Firstly, we need to choose a loss function. Here we choose the Least squares, which aims to minimize the quadratic function of parameter β,
RSSresidual sum of square(β)=∥Y−Xβ∥2F
which leads to the solution β^=(XTX)−1XTY if X is column full rank. Then prediction function at any X is given by y^=XTβ^.
1.2 Nearest neighbors
In regression, nearest neighbors method averages the outputs of the k-nearest points of X as prediction value at X, which can be formulated asy^(X)=1k∑X(i)∈Nk(X)y(i)
In classification, nearest neighbors method uses the maximum votes of labels of the k-nearest points of X as the class for X, which can be formulated as
g^(X)=argmaxg∑X(i)∈Nk(X)1{y(i)=g}
2. Rationality and difference of least square and nearest neighbors
least square makes huge assumptions about structure but nearest neighbors notleast square yields stable but possible inaccurate predictions, while predictions of nearest neighbors are often accurate but can be unstable
Note that:
stable means low variance
accurate means low bias
rationality least square and nearest neighbors
Suppose that we have random variables (X,Y) with jointly distribution Pr(X,Y). Then we want to find a function f(X) to approximate Y. If we use the suqare loss function as a criteria for choosing f(X),EPEepxect prediction error(f)=E∥Y−f(X)∥2F=EXEY|X[∥Y−f(X)∥2F|X]
It suffices to minimize EPE pointwise
f(x)=argminEY|X[∥Y−f(X)∥2F|X=x]
The solution is
f(x)=E[Y|X=x]
And least square and nearest neighbors both aim to approximate the expectation by averaging.
Least square assumes the linear structure and approximate the expectation in square loss function by averaging all training datas.
\mathrm{\hat{EPE}}(\beta) = \frac{1}{N}\sum_{i=1}^N\|y_i - X^{(i)}^t\beta\|_F^2
Nearest neighbors approximates the conditional expectation in solution by averaging the outputs near target x
Y^=ave(yi|X(i)∈Nk(x))
So, two things are happening in approximating of both least square and nearest neighbor.
Least square
model structure assumption
averaging over all training data in EPE
Nearest neighbors
1.condition on a small region of target point x instead of conditioning on it
2.averaging the outputs which are near to x
extension of these simple procedures
There are many complex algorithms are from these two,Kernel Methods use weights that decrease smoothly to zero with distance from the target point, rather that the effective 0/1 weights used by k-nearest neighbors.
In high dimensional spaces the distance kernels are modified to emphasize some variable more than others.
Local regression fits linear models by locally weighted least squares, rather than fitting constants locally.
Linear models fit to a basis expansion of the original inputs allow arbitrary complex models.
Projection pursuit and neural network models consist of sums of nonlinearly transformed linear models.
相关文章推荐
- 【jQuery】使用load()方法异步请求数据
- 【jQuery】调用delay()方法延时执行动画效果
- [leetcode] 30. Substring with Concatenation of All Words 解题报告
- 【jQuery】调用stop()方法停止当前所有动画效果
- 【jQuery】调用animate()方法制作移动位置的动画
- 【jQuery】调用animate()方法制作简单的动画效果
- Vba Json
- 【jQuery】使用fadeTo()方法设置淡入淡出效果的不透明度
- 利用VBA将excel数据表生成JSON文件(utf8)
- Excel2Unity
- [Python爬虫] 在Windows下安装PhantomJS和CasperJS及入门介绍(上)
- 【jQuery】使用fadeIn()与fadeOut()方法实现淡入淡出效果
- [Python爬虫] 在Windows下安装PIP+Phantomjs+Selenium
- 51Nod 1163 最高的奖励(贪心+优先队列 & 并差集)
- [Python爬虫] Selenium自动访问Firefox和Chrome并实现搜索截图
- [Python爬虫] Selenium实现自动登录163邮箱和Locating Elements介绍
- [Python爬虫] Selenium+Phantomjs动态获取CSDN下载资源信息和评论
- 【jQuery】使用slideToggle()方法实现图片“变脸”效果
- [Python爬虫] Selenium获取百度百科旅游景点的InfoBox消息盒
- [Python] 中文编码问题:raw_input输入、文件读取、变量比较等str、unicode、utf-8转换问题