7 Types of Neural Networks in Artificial Intelligence Explained

There are many types of neural networks like Perceptron, Hopfield, Self-organizing maps, Boltzmann machines, Deep belief networks, Auto encoders ... Programs DataScience DataScience|AllCourses M.ScinDataScience–LJMU&IIITBangalore ExecutivePGPinDataScience–IIITBangalore ACPinDataScience–IIITBangalore ExecutiveProgrammeinDataScience–IIITB PCPinDataScience–IIMKozhikode MasterDegreeinDataScience–IIITB&IUGermany M.ScinDataScience–UniversityofArizona Management PCPinHRMandAnalytics–IIMKozhikode ProductManagementCertification–DukeCE PGPinManagement–IMTGhaziabad Software&Technology SoftwareEngineering|AllCourses M.ScinCS–LJMU&IIITBangalore ExecutivePGPinSoftwareDevelopment ACPinCloudComputing ACPinDevOp ACPinCyberSecurity ACPinBigData ACPinBlockchainTechnology MasterinCyberSecurity–IIITB&IUGermany BusinessAnalyticsCertificationProgram 7TypesofNeuralNetworksinArtificialIntelligenceExplained byPavanVadapalli DirectorofEngineering@upGrad.Motivatedtoleveragetechnologytosolveproblems.Seasonedleaderforstartupsandfastmovingorgs.Workingonsolvingproblemsofscaleandlongtermtechnology… Dec29,2020 Home > ArtificialIntelligence > 7TypesofNeuralNetworksinArtificialIntelligenceExplained NeuralNetworksareasubsetofMachineLearningtechniqueswhichlearnthedataandpatternsinadifferentwayutilizingNeuronsandHiddenlayers.NeuralNetworksarewaymorepowerfulduetotheircomplexstructureandcanbeusedinapplicationswheretraditionalMachineLearningalgorithmsjustcannotsuffice. Bytheendofthistutorial,youwillhavetheknowledgeof: AbriefhistoryofNeuralNetworks WhatareNeuralNetworks TypesofNeuralNetworks Perceptron FeedForwardNetworks Multi-LayerPerceptron RadialBasedNetworks ConvolutionalNeuralNetworks RecurrentNeuralNetworks LongShort-TermMemoryNetworks TableofContents ABriefHistoryofNeuralNetworksWhatareNeuralNetworks?TypesofNeuralNetworksPerceptronFeedForwardNetworkMulti-LayerPerceptronRadialBasisNetworksConvolutionalNeuralNetworksRecurrentNeuralNetworksLongShort-TermMemoryNetworksBeforeyougoWhatareneuralnetworks?Whatare3majorcategoriesofneuralnetworks?Whatarethelimitationsofneuralnetworks? ABriefHistoryofNeuralNetworks Researchersfromthe60shavebeenresearchingandformulatingwaystoimitatethefunctioningofhumanneuronsandhowthebrainworks.Althoughitisextremelycomplextodecode,asimilarstructurewasproposedwhichcouldbeextremelyefficientinlearninghiddenpatternsinData.  Formostofthe20thcentury,NeuralNetworkswereconsideredincompetent.Theywerecomplexandtheirperformancewaspoor.Also,theyrequiredalotofcomputingpowerwhichwasnotavailableatthattime.However,whentheteamofSirGeoffreyHinton,alsodubbedas“TheFatherofDeepLearning”,publishedtheresearchpaperonBackpropagation,tablesturnedcompletely.NeuralNetworkscouldnowachievewhichwasnotthoughtof.  WhatareNeuralNetworks? NeuralNetworksusethearchitectureofhumanneuronswhichhavemultipleinputs,aprocessingunit,andsingle/multipleoutputs.Thereareweightsassociatedwitheachconnectionofneurons.Byadjustingtheseweights,aneuralnetworkarrivesatanequationwhichisusedforpredictingoutputsonnewunseendata.Thisprocessisdonebybackpropagationandupdatingoftheweights. TypesofNeuralNetworks Differenttypesofneuralnetworksareusedfordifferentdataandapplications.Thedifferentarchitecturesofneuralnetworksarespecificallydesignedtoworkonthoseparticulartypesofdataordomain.Let’sstartfromthemostbasiconesandgotowardsmorecomplexones. Perceptron ThePerceptronisthemostbasicandoldestformofneuralnetworks.Itconsistsofjust1neuronwhichtakestheinputandappliesactivationfunctiononittoproduceabinaryoutput.Itdoesn’tcontainanyhiddenlayersandcanonlybeusedforbinaryclassificationtasks. Theneurondoestheprocessingofadditionofinputvalueswiththeirweights.Theresultedsumisthenpassedtotheactivationfunctiontoproduceabinaryoutput. ImageSource Learnabout: DeepLearningvsNeuralNetworks FeedForwardNetwork TheFeedForward(FF)networksconsistofmultipleneuronsandhiddenlayerswhichareconnectedtoeachother.Thesearecalled“feed-forward”becausethedataflowintheforwarddirectiononly,andthereisnobackwardpropagation.Hiddenlayersmightnotbenecessarilypresentinthenetworkdependingupontheapplication. Morethenumberoflayersmorecanbethecustomizationoftheweights.Andhence,morewillbetheabilityofthenetworktolearn.Weightsarenotupdatedasthereisnobackpropagation.Theoutputofmultiplicationofweightswiththeinputsisfedtotheactivationfunctionwhichactsasathresholdvalue. FFnetworksareusedin: Classification Speechrecognition Facerecognition Patternrecognition ImageSource Multi-LayerPerceptron ThemainshortcomingoftheFeedForwardnetworkswasitsinabilitytolearnwithbackpropagation.Multi-layerPerceptronsaretheneuralnetworkswhichincorporatemultiplehiddenlayersandactivationfunctions.ThelearningtakesplaceinaSupervisedmannerwheretheweightsareupdatedbythemeansofGradientDescent. Multi-layerPerceptronisbi-directional,i.e.,Forwardpropagationoftheinputs,andthebackwardpropagationoftheweightupdates.Theactivationfunctionscanbechangeswithrespecttothetypeoftarget.Softmaxisusuallyusedformulti-classclassification,Sigmoidforbinaryclassificationandsoon.Thesearealsocalleddensenetworksbecausealltheneuronsinalayerareconnectedtoalltheneuronsinthenextlayer. TheyareusedinDeepLearningbasedapplicationsbutaregenerallyslowduetotheircomplexstructure. ImageSource RadialBasisNetworks RadialBasisNetworks(RBN)useacompletelydifferentwaytopredictthetargets.Itconsistsofaninputlayer,alayerwithRBFneuronsandanoutput.TheRBFneuronsstoretheactualclassesforeachofthetrainingdatainstances.TheRBNaredifferentfromtheusualMultilayerperceptronbecauseoftheRadialFunctionusedasanactivationfunction. Whenthenewdataisfedintotheneuralnetwork,theRBFneuronscomparetheEuclidiandistanceofthefeaturevalueswiththeactualclassesstoredintheneurons.Thisissimilartofindingwhichclustertodoestheparticularinstancebelong.Theclasswherethedistanceisminimumisassignedasthepredictedclass. TheRBNsareusedmostlyinfunctionapproximationapplicationslikePowerRestorationsystems. ImageSource Alsoread: NeuralNetworkApplicationsinRealWorld ConvolutionalNeuralNetworks Whenitcomestoimageclassification,themostusedneuralnetworksareConvolutionNeuralNetworks(CNN).CNNcontainmultipleconvolutionlayerswhichareresponsiblefortheextractionofimportantfeaturesfromtheimage.Theearlierlayersareresponsibleforlow-leveldetailsandthelaterlayersareresponsibleformorehigh-levelfeatures. TheConvolutionoperationusesacustommatrix,alsocalledasfilters,toconvoluteovertheinputimageandproducemaps.Thesefiltersareinitializedrandomlyandthenareupdatedviabackpropagation.OneexampleofsuchafilteristheCannyEdgeDetector,whichisusedtofindtheedgesinanyimage.  Aftertheconvolutionlayer,thereisapoolinglayerwhichisresponsiblefortheaggregationofthemapsproducedfromtheconvolutionallayer.ItcanbeMaxPooling,MinPooling,etc.Forregularization,CNNsalsoincludeanoptionforaddingdropoutlayerswhichdropormakecertainneuronsinactivetoreduceoverfittingandquickerconvergence. CNNsuseReLU(RectifiedLinearUnit)asactivationfunctionsinthehiddenlayers.Asthelastlayer,theCNNshaveafullyconnecteddenselayerandtheactivationfunctionmostlyasSoftmaxforclassification,andmostlyReLUforregression. ImageSource RecurrentNeuralNetworks RecurrentNeuralNetworkscomeintopicturewhenthere’saneedforpredictionsusingsequentialdata.Sequentialdatacanbeasequenceofimages,words,etc.TheRNNhaveasimilarstructuretothatofaFeed-ForwardNetwork,exceptthatthelayersalsoreceiveatime-delayedinputofthepreviousinstanceprediction.ThisinstancepredictionisstoredintheRNNcellwhichisasecondinputforeveryprediction. However,themaindisadvantageofRNNistheVanishingGradientproblemwhichmakesitverydifficulttorememberearlierlayers’weights. ImageSource  LongShort-TermMemoryNetworks LSTMneuralnetworksovercometheissueofVanishingGradientinRNNsbyaddingaspecialmemorycellthatcanstoreinformationforlongperiodsoftime.LSTMusesgatestodefinewhichoutputshouldbeusedorforgotten.Ituses3gates:Inputgate,OutputgateandaForgetgate.TheInputgatecontrolswhatalldatashouldbekeptinmemory.TheOutputgatecontrolsthedatagiventothenextlayerandtheforgetgatecontrolswhentodump/forgetthedatanotrequired.  LSTMsareusedinvariousapplicationssuchas: Gesturerecognition Speechrecognition Textprediction Beforeyougo NeuralNetworkscangetverycomplexwithinnotimesyoukeeponaddinglayersinthenetwork.Therearetimeswhenwherewecanleveragetheimmenseresearchinthisfieldbyusingpre-trainednetworksforouruse. ThisiscalledTransferLearning.Inthistutorial,wecoveredmostofthebasicneuralnetworksandtheirfunctioning.MakesuretotrytheseoutusingtheDeepLearningframeworkslikeKerasandTensorflow. Ifyou’reinterestedtolearnmoreaboutneuralnetwork,machinelearning&AI,checkoutIIIT-B&upGrad’s PGDiplomainMachineLearning&AI whichisdesignedforworkingprofessionalsandoffers450+hoursofrigoroustraining,30+casestudies&assignments,IIIT-BAlumnistatus,5+practicalhands-oncapstoneprojects&jobassistancewithtopfirms. Whatareneuralnetworks? Neuralnetworksareprobabilisticmodelsthatcanbeusedtoperformnonlinearclassificationandregression,meaningapproximatingamappingfrominputspacetooutputspace.Theinterestingthingaboutneuralnetworksisthattheycanbetrainedwithalotofdata,andtheycanbeusedtomodelcomplexnonlinearbehavior.Theycanbetrainedwithlotsofexamples,andtheycanbeusedtofindpatternswithoutanyguidance.Soneuralnetworksareusedinmanyapplicationswherethereisrandomnessandcomplexity. Whatare3majorcategoriesofneuralnetworks? Aneuralnetworkisacomputationalapproachtolearning,analogoustothebrain.Therearethreemajorcategoriesofneuralnetworks.Classification,SequencelearningandFunctionapproximationarethethreemajorcategoriesofneuralnetworks.TherearemanytypesofneuralnetworkslikePerceptron,Hopfield,Self-organizingmaps,Boltzmannmachines,Deepbeliefnetworks,Autoencoders,Convolutionalneuralnetworks,RestrictedBoltzmannmachines,Continuousvaluedneuralnetworks,RecurrentneuralnetworksandFunctionallinknetworks. Whatarethelimitationsofneuralnetworks? Neuralnetscansolveproblemswhichhavealargenumberofinputsandalargenumberofoutputs.Buttherearealsolimitsforneuralnets.Neuralnetsaremostlyusedforclassification.Theyperformverybadforregression.Andthisisaveryimportantpoint:Neuralnetsneedalotoftrainingdata.Ifthedatasetissmall,thenneuralnetswillnotbeabletolearntheunderlyingrules.Anotherlimitationforneuralnetsisthattheyareblackboxes.Theyarenottransparent.Theinternalstructureofaneuralnetworkisnoteasytounderstand. LeadtheAIDrivenTechnologicalRevolution PGDIPLOMAINMACHINELEARNINGANDARTIFICIALINTELLIGENCE LearnMore Leaveacomment CancelreplyYouremailaddresswillnotbepublished.CommentName Email Website Δ Postnavigation PrevNext RelatedArticles IoT:History,Present&Future byPavanVadapalli Mar3,2022 TensorFlowTutorialforBeginners byPavanVadapalli Feb24,2022 MachineLearningTutorial:LearnMLfromScratch bySumitShukla Feb17,2022 OurTrendingMachineLearningCourses AdvancedCertificationinMachineLearningandCloudfromIITMadras-Duration12Months MasterofScienceinMachineLearning&AIfromIIIT-B&LJMU-Duration18Months ExecutivePGPrograminMachineLearningandAIfromIIIT-B-Duration12Months OurPopularMachineLearningCourse RelatedArticles IoT:History,Present&Future byPavanVadapalli Mar3,2022 TensorFlowTutorialforBeginners byPavanVadapalli Feb24,2022 MachineLearningTutorial:LearnMLfromScratch bySumitShukla Feb17,2022 × Searchfor: RegisterforaDemoCourse Course PGDiplomainDataScience PGCertificationinDataScience PGPrograminBigData PGDiplomainDataScience(Alternative) PGDiplomainML&AI MastersinDataScience MastersinML&AI PGinML&DeepLearning PGCertificationinMachineLearningandNLP × RegisterforaDemoCourse Course PGCertificationinDigitalMarketingandCommunication ExecutivePrograminStrategicDigitalMarketing PGPrograminManagement,SpecialisationinSalesandDigitalMarketing EntrepreneurshipCertificateProgram ProductManagementCertificateProgram PostGraduatePrograminLifeinsurance(HDFC) × RegisterforaDemoCourse Course PGPrograminSoftwareDevelopment PGCertificateinBlockchainSpecialisation PGPrograminBlockchainTechnology × TalktoourCounselortofindabestcoursesuitabletoyourCareerGrowth Programs DataScience Management Technology Courses PGDiplomainDataScience PGCertificationinDataScience PGPrograminBigData PGDiplomainDataScience(Alternative) PGDiplomainML&AI MastersinDataScience MastersinML&AI PGinML&DeepLearning PGCertificationinMachineLearningandNLP PGCertificationinDigitalMarketingandCommunication ExecutivePrograminStrategicDigitalMarketing PGPrograminManagement,SpecialisationinSalesandDigitalMarketing EntrepreneurshipCertificateProgram ProductManagementCertificateProgram PostGraduatePrograminLifeinsurance(HDFC) PGPrograminSoftwareDevelopment PGCertificateinBlockchainSpecialisation PGPrograminBlockchainTechnology × Let’sdoit!No,thanks.