机器学习3 logistic回归

局部权重似合

Fit θ to minimize

∑i=1mw(i)(y(i)−θTx(i))2w(i)=exp⎛⎝⎜⎜−(x(i)−x)22τ2⎞⎠⎟⎟

最大似然法

P(y(i)|x(i),θ)=12πσ−−−√exp⎛⎝−(y(i)−θTx(i))22σ2⎞⎠L(θ)=∏i=1m12πσ−−−√exp⎛⎝−(y(i)−θTx(i))22σ2⎞⎠log(L(θ))=−mlog(2πσ−−−√)−∑i=1m(y(i)−θTx(i))22σ2ifwantL(θ)maxthenjustmakeJ(θ)=∑i=1m(y(i)−θTx(i))22minend

sigmoid function

hθ(x)=g(θTx)=11+e−θTxwhereg(z)=11+e−z∂∂zg(z)=e−z(1+e−z)2=g(z)(1−g(z))∂hθ(x)∂θj=hθ(x)(1−hθ(x))xj

梯度上升

P(y(i)|x(i),θ)=hθ(x)y(1−hθ(x))1−yL(θ)=∏i=1mhθ(x(i))y(i)(1−hθ(x))1−y(i)l(θ)=log(L(θ))=∑i=1my(i)log(hθ(x(i)))+∑i=1m(1−y(i))log(1−hθ(x(i)))∂log(L(θ))∂θj=∑i=1my(i)(1−hθ(x))xj−∑i=1m(1−y(i))hθ(x)xj=∑i=1m(y(i)xj−y(i)xjhθ(x)−hθ(x)xj+hθ(x)xjy(i))=∑i=1m(y(i)−hθ(x))xj

θ:=θ+α∇θl(θ)=θ+α∑i=1m(y(i)−hθ(x))x