Exercise 4: Logistic Regressionand Newton's Method

Rawpage:http://openclassroom.stanford.edu/MainFolder/DocumentPage.php?course=MachineLearning&doc=exercises/ex4/ex4.html

Exercise4:LogisticRegressionandNewton'sMethod

Inthisexercise,youwilluseNewton'sMethodtoimplementlogisticregressiononaclassificationproblem.

Data

Tobegin,download
ex4Data.zipandextractthefilesfromthezipfile.

Forthisexercise,supposethatahighschoolhasadatasetrepresenting40studentswhowereadmittedtocollegeand40studentswhowerenotadmitted.Eachtrainingexamplecontainsastudent'sscoreontwostandardizedexamsandalabelofwhetherthestudent
wasadmitted.

Yourtaskistobuildabinaryclassificationmodelthatestimatescollegeadmissionchancesbasedonastudent'sscoresontwoexams.Inyourtrainingdata,

a.ThefirstcolumnofyourxarrayrepresentsallTest1scores,andthesecondcolumnrepresentsallTest2scores.

b.Theyvectoruses'1'tolabelastudentwhowasadmittedand'0'tolabelastudentwhowasnotadmitted.

Plotthedata

Loadthedataforthetrainingexamplesintoyourprogramandaddtheintercepttermintoyourxmatrix.

BeforebeginningNewton'sMethod,wewillfirstplotthedatausingdifferentsymbolstorepresentthetwoclasses.InMatlab/Octave,youcanseparatethepositiveclassandthenegativeclassusingthefindcommand:

%findreturnstheindicesofthe

%rowsmeetingthespecifiedcondition

pos=find(y==1);neg=find(y==0);

%Assumethefeaturesareinthe2ndand3rd

%columnsofx

plot(x(pos,2),x(pos,3),'+');holdon

plot(x(neg,2),x(neg,3),'o')

Yourplotshouldlooklikethefollowing:

Newton'sMethod

Recallthatinlogisticregression,thehypothesisfunctionis

Inourexample,thehypothesisisinterpretedastheprobabilitythatadriverwillbeaccident-free,giventhevaluesofthefeaturesinx.

Matlab/Octavedoesnothavealibraryfunctionforthesigmoid,soyouwillhavetodefineityourself.Theeasiestwaytodothisisthroughaninlineexpression:

g=inline('1.0./(1.0+exp(-z))');

%Usage:Tofindthevalueofthesigmoid

%evaluatedat2,callg(2)

Thecostfunctionisdefinedas

OurgoalistouseNewton'smethodtominimizethisfunction.RecallthattheupdateruleforNewton'smethodis

Inlogisticregression,thegradientandtheHessianare

Notethattheformulaspresentedabovearethevectorizedversions.Specifically,thismeansthat,,whileandarescalars.

Implementation

Now,implementNewton'sMethodinyourprogram,startingwiththeinitialvalueof.Todeterminehowmanyiterationstouse,calculateforeachiterationandplotyourresultsasyoudidinExercise2.Asmentionedinthelecturevideos,Newton'smethodoften
convergesin5-15iterations.Ifyoufindyourselfusingfarmoreiterations,youshouldcheckforerrorsinyourimplementation.

Afterconvergence,useyourvaluesofthetatofindthedecisionboundaryintheclassificationproblem.Thedecisionboundaryisdefinedasthelinewhere

whichcorrespondsto

Plottingthedecisionboundaryisequivalenttoplottingtheline.Whenyouarefinished,yourplotshouldappearlikethefigurebelow.

Questions

Finally,recordyouranswerstothesequestions.

1.Whatvaluesofdidyouget?Howmanyiterationswererequiredforconvergence?

2.Whatistheprobabilitythatastudentwithascoreof20onExam1andascoreof80onExam2willnotbeadmitted?

Solutions

Afteryouhavecompletedtheexercisesabove,pleaserefertothesolutionsbelowandcheckthatyourimplementationandyouranswersarecorrect.Inacasewhereyourimplementationdoesnotresultinthesameparameters/phenomenaasdescribedbelow,debug
yoursolutionuntilyoumanagetoreplicatethesameeffectasourimplementation.

Acompletem-fileimplementationofthesolutionscanbefound
here.

Newton'sMethod

1.Yourfinalvaluesofthetashouldbe

Plot.Yourplotofthecostfunctionshouldlooksimilartothepicturebelow:

Fromthisplot,youcaninferthatNewton'sMethodhasconvergedbyaround5iterations.Infact,bylookingataprintoutofthevaluesofJ,youwillseethatJchangesbylessthanbetweenthe4thand5thiterations.Recallthatintheprevioustwoexercises,
gradientdescenttookhundredsoreventhousandsofiterationstoconverge.Newton'sMethodismuchfasterincomparison.

2.Theprobabilitythatastudentwithascoreof20onExam1and80onExam2willnotbeadmittedtocollegeis0.668.