机器学习2 梯度下降

梯度下降规则

θi:=θi−α∂∂θiJ(θ)∂∂θiJ(θ)=∂∂θi12(hθ(x)−y)2=(hθ(x)−y)∂∂θi(θ0x0+...+θixi+θnxn−y)=(hθ(x)−y)xiθi:=θi−α(hθ(x)−y)xi

批梯度下降

θi:=θi−α∑j=1n(hθ(x(j))−y(j))xi(j)

增量梯度下降

repeat j = 1 to m {

for all i {

θi:=θi−α(hθ(x(j))−y(j))xi(j)}

}

梯度

∇Af(A)=⎡⎣⎢⎢⎢⎢⎢∂f∂A11...∂f∂Am1.........∂f∂A1n...∂f∂Amn⎤⎦⎥⎥⎥⎥⎥

迹

trA=∑i=1nAii⇒trAB=trBAtrA=trATtr(A+B)=trA+trBtrαA=αtrA

∇AtrAB=BT∇ATf(A)=(∇Af(A))T∇AtrABATC=CAB+CTABT∇A|A|=|A|(A−1)T

最小二乘法

X=⎡⎣⎢⎢⎢⎢⎢⎢⎢⎢⎢(x(1))T(x(2))T...(x(m))T⎤⎦⎥⎥⎥⎥⎥⎥⎥⎥⎥y→=⎡⎣⎢⎢⎢⎢y(1)y(2)...y(m)⎤⎦⎥⎥⎥⎥Xθ=⎡⎣⎢⎢⎢⎢⎢⎢⎢⎢⎢(x(1))T(x(2))T...(x(m))T⎤⎦⎥⎥⎥⎥⎥⎥⎥⎥⎥θ=⎡⎣⎢⎢⎢⎢⎢⎢⎢⎢⎢(x(1))Tθ(x(2))Tθ...(x(m))Tθ⎤⎦⎥⎥⎥⎥⎥⎥⎥⎥⎥==⎡⎣⎢⎢⎢⎢⎢⎢⎢⎢hθ(x(1))hθ(x(2))...hθ(x(m))⎤⎦⎥⎥⎥⎥⎥⎥⎥⎥J(θ)=12∑i=1m(hθ(x(i))−y(i)))2=12(Xθ−y→)T(Xθ−y→)⇒∇θJ(θ)=∇θ12(Xθ−y→)T(Xθ−y→)=12∇θtr(θTXTXθ−θTXTy→−y→TXθ+y→Ty→)=12∇θ(trθTXTXθ−2try→TXθ)=12∇θ(trθTXTXθ−2try→TXθ)=12∇θ(trθθTXTX−2trθy→TX)=12(XTXθ+XTXθ−2XTy→)=XTXθ−XTy→⇒θ=(XTX)−1XTy→