Everything you need to know about Neural Networks

Input layer — initial data for the neural network. · Hidden layers — intermediate layer between input and output layer and place where all the ... HomeNotificationsListsStoriesWritePublishedinTowardsDataScienceEverythingyouneedtoknowaboutNeuralNetworksandBackpropagation—MachineLearningEasyandFunNeuralNetworkexplanationfromthegroundincludingunderstandingthemathbehinditIfindithardtogetstepbystepanddetailedexplanationsaboutNeuralNetworksinoneplace.Alwayssomepartoftheexplanationwasmissingincoursesorinthevideos.SoItriedtogatheralltheinformationandexplanationsinoneblogpost(stepbystep).Iwouldseparatethisblogin8sectionsasIfinditmostrelevant.ModelRepresentationModelRepresentationMathematicsActivationFunctionsBiasNodeCostFunctionForwardPropagationCalculationBackpropagationAlgorithmCodeImplementationSolet’sstart…ModelRepresentationArtificialNeuralNetworkiscomputingsysteminspiredbybiologicalneuralnetworkthatconstituteanimalbrain.Suchsystems“learn”toperformtasksbyconsideringexamples,generallywithoutbeingprogrammedwithanytask-specificrules.Image1:NeuralNetworkArchitectureTheNeuralNetworkisconstructedfrom3typeoflayers:Inputlayer—initialdatafortheneuralnetwork.Hiddenlayers—intermediatelayerbetweeninputandoutputlayerandplacewhereallthecomputationisdone.Outputlayer—producetheresultforgiveninputs.Thereare3yellowcirclesontheimageabove.TheyrepresenttheinputlayerandusuallyarenotedasvectorX.Thereare4blueand4greencirclesthatrepresentthehiddenlayers.Thesecirclesrepresentthe“activation”nodesandusuallyarenotedasWorθ.Theredcircleistheoutputlayerorthepredictedvalue(orvaluesincaseofmultipleoutputclasses/types).Eachnodeisconnectedwitheachnodefromthenextlayerandeachconnection(blackarrow)hasparticularweight.Weightcanbeseenasimpactthatthatnodehasonthenodefromthenextlayer.SoifwetakealookononenodeitwouldlooklikethisImage2:NodefromNeuralNetworkLet’slookatthetopbluenode(“Image1”).Allthenodesfromthepreviouslayer(yellow)areconnectedwithit.Alltheseconnectionsrepresenttheweights(impact).Whenallthenodevaluesfromtheyellowlayeraremultipliedwiththeirweightandallthisissummarizeditgivessomevalueforthetopbluenode.Thebluenodehaspredefined“activation”function(unitstepfunctionon“Image2”)whichdefinesifthisnodewillbe“activated”orhow“active”itwillbe,basedonthesummarizedvalue.Theadditionalnodewithvalue1iscalled“bias”node.ModelRepresentationMathematicsInordertounderstandthemathematicalequationsIwilluseasimplerNeuralNetworkmodel.Thismodelwillhave4inputnodes(3+1“bias”).Onehiddenlayerwith4nodes(3+1“bias”)andoneoutputnode.Image3:SimpleNeuralNetworkWearegoingtomarkthe“bias”nodesasx₀anda₀respectively.So,theinputnodescanbeplacedinonevectorXandthenodesfromthehiddenlayerinvectorA.Image4:X(inputlayer)andA(hiddenlayer)vectorTheweights(arrows)areusuallynotedasθorW.InthiscaseIwillnotethemasθ.Theweightsbetweentheinputandhiddenlayerwillrepresent3x4matrix.Andtheweightsbetweenthehiddenlayerandtheoutputlayerwillrepresent1x4matrix.Ifnetworkhasaunitsinlayerjandb​unitsinlayerj+1,thenθⱼwillbeofdimensionb​×(a+1).Image5:Layer1WeightsMatrix(θ)Next,whatwewantistocomputethe“activation”nodesforthehiddenlayer.InordertodothatweneedtomultiplytheinputvectorXandweightsmatrixθ¹forthefirstlayer(X*θ¹)andthenapplytheactivationfunctiong.Whatwegetis:Image6:ComputeactivationnodesAndbymultiplyinghiddenlayervectorwithweightsmatrixθforthesecondlayer(A*θ)wegetoutputforthehypothesisfunction:Image7:Computeoutputnodevalue(hypothesis)Thisexampleiswithonlyonehiddenlayerand4nodesthere.IfwetrytogeneralizeforNeuralNetworkwithmultiplehiddenlayersandmultiplenodesineachofthelayerswewouldgetnextformula.Image8:GeneralizedComputenodevaluefunctionWherewehaveLlayerswithnnodesandL-1layerwithmnodes.ActivationFunctionsInNeuralNetworktheactivationfunctiondefinesifgivennodeshouldbe“activated”ornotbasedontheweightedsum.Let’sdefinethisweightedsumvalueasz.InthissectionIwouldexplainwhy“StepFunction”and“LinearFunction”won’tworkandtalkabout“SigmoidFunction”oneofthemostpopularactivationfunctions.TherearealsootherfunctionswhichIwillleaveasidefornow.StepFunctionOneofthefirstideaswouldbetousesocalled“StepFunction”(discreteoutputvalues)wherewedefinethresholdvalueand:if(z>threshold)—“activate”thenode(value1)if(z