Deep Neural Network: The 3 Popular Types (MLP, CNN and ...

Multi-Layer Perceptrons (MLP); Convolutional Neural Networks (CNN); Recurrent Neural Networks (RNN). What Is a Deep Neural Network? Machine learning techniques ... BlogHomeCategoriesComputerVisionDeepLearningApplicationsEdgeAICompanyNewsMenuBlogHomeCategoriesComputerVisionDeepLearningApplicationsEdgeAICompanyNewsviso.aiSearchGetademoDeepLearningDeepNeuralNetwork:The3PopularTypes(MLP,CNNandRNN)ExploreNo-codeAIvisionGaudenzBoeschAboutVisoSuiteistheno-codecomputervisionplatformforteamstobuild,deployandoperatereal-worldapplications.ContentsNeedComputerVision?VisoSuiteisonlyall-in-onebusinessplatformtobuildanddelivercomputervisionwithoutcoding.Learnmore.ThisarticlewillexplainthedifferencesbetweenthethreetypesofneuralnetworksandcoverthebasicsofDeepNeuralNetworks.Suchdeepneuralnetworks(DNNs)haverecentlydemonstratedimpressiveperformanceincomplexmachinelearningtaskssuchasimageclassificationorspeechrecognition.Inparticular,wewillcoverthefollowingneuralnetworktypes:Multi-LayerPerceptrons(MLP)ConvolutionalNeuralNetworks(CNN)RecurrentNeuralNetworks(RNN)WhatIsaDeepNeuralNetwork?Machinelearningtechniqueshavebeenwidelyappliedinvariousareassuchaspatternrecognition,naturallanguageprocessing,andcomputationallearning.Duringthepastdecades,machinelearninghasbroughtenormousinfluence onourdailylifewithexamplesincludingefficientweb search,self-drivingsystems,computervision, andopticalcharacterrecognition.Especially,deepneuralnetworkmodelshavebecomeapowerfultoolofmachinelearningandartificialintelligence.Adeepneuralnetwork(DNN)isan artificialneuralnetwork(ANN) withmultiplelayersbetweentheinputandoutputlayers.Thesuccessofdeepneuralnetworkshasledtobreakthroughssuchasreducingworderrorratesinspeechrecognitionby30%overtraditionalapproaches(thebiggestgainin20years)ordrasticallycuttingtheerrorrateinanimagerecognitioncompetitionsince2011(from26%to3.5%whilehumansachieve5%).ConceptofDeepNeuralNetworksDeepneuralnetworkmodelswereoriginallyinspiredbyneurobiology.Onahighlevel,abiologicalneuronreceivesmultiplesignalsthroughthesynapsescontactingitsdendritesandsendsasinglestreamofactionpotentialsoutthroughitsaxon.Thecomplexityofmultipleinputsisreducedbycategorizingitsinputpatterns.Inspiredbythisintuition,artificialneuralnetworkmodelsarecomposedofunitsthatcombinemultipleinputsandproduceasingleoutput.Neuralnetworkstargetbrain-likefunctionalityandarebasedonasimpleartificialneuron:anonlinearfunction(suchasmax(0,value))ofaweightedsumoftheinputs.Thesepseudoneuronsarecollectedintolayers,andtheoutputsofonelayerbecomingtheinputsofthenextinthesequence.WhatmakesaNeuralNetwork“Deep”?Deepneuralnetworksemploydeeparchitecturesinneuralnetworks.“Deep”referstofunctionswithhighercomplexityinthenumberoflayersandunitsinasinglelayer.Thelargedatasetsinthecloudmadeitpossibletobuildmoreaccuratemodelsbyusingadditionalandlargerlayerstocapturehigherlevelsofpatterns.Thetwophasesofneuralnetworksarecalledtraining(orlearning)andinference(orprediction),andtheyrefertodevelopmentversusproduction.Thedeveloperchoosesthenumberoflayersandthetypeofneuralnetwork,andtrainingdeterminestheweights. 3TypesofDeepNeuralNetworksThreefollowingtypesofdeepneuralnetworksarepopularlyusedtoday:Multi-LayerPerceptrons(MLP)ConvolutionalNeuralNetworks(CNN)RecurrentNeuralNetworks(RNN)MultilayerPerceptrons(MLPs)Amultilayerperceptron(MLP)isaclassofafeedforwardartificialneuralnetwork(ANN).MLPsmodelsarethemostbasicdeepneuralnetwork,whichiscomposedofaseriesoffullyconnectedlayers.Today,MLPmachinelearningmethodscanbeusedtoovercometherequirementofhighcomputingpowerrequiredbymoderndeeplearningarchitectures.Eachnewlayerisasetofnonlinearfunctionsofaweightedsumofalloutputs(fullyconnected)fromthepriorone. ConceptofMultilayerPerceptrons(MLP) ConvolutionalNeuralNetwork(CNN)Aconvolutionalneuralnetwork(CNN,orConvNet)isanotherclassofdeepneuralnetworks. CNNsaremostcommonlyemployedincomputervision.Givenaseriesofimagesorvideosfromtherealworld,withtheutilizationofCNN,theAIsystemlearnstoautomaticallyextractthefeaturesoftheseinputstocompleteaspecifictask,e.g.,imageclassification,faceauthentication,andimagesemanticsegmentation.DifferentfromfullyconnectedlayersinMLPs,inCNNmodels,oneormultipleconvolutionlayersextractthesimplefeaturesfrominputbyexecutingconvolutionoperations.Eachlayerisasetofnonlinearfunctionsofweightedsumsatdifferentcoordinatesofspatiallynearbysubsetsofoutputsfromthepriorlayer,whichallowstheweightstobereused.ConceptofaConvolutionNeuralNetwork(CNN)Applyingvariousconvolutionalfilters,CNNmachinelearningmodelscancapturethehigh-levelrepresentationoftheinputdata,makingitmostpopularforcomputervisiontasks,suchasimageclassification(e.g.,AlexNet,VGGnetwork,ResNet,MobileNet)andobjectdetection(e.g.,FastR-CNN,MaskR-CNN,YOLO,SSD).AlexNet.Forimageclassification,asthefirstCNNneuralnetworktowintheImageNetChallengein2012,AlexNetconsistsoffiveconvolutionlayersandthreefullyconnectedlayers.Thus,AlexNetrequires61millionweightsand724millionMACs(multiply-addcomputation)toclassifytheimagewithasizeof227×227.VGG-16.Toachievehigheraccuracy,VGG-16istrainedtoadeeperstructureof16layersconsistingof13convolutionlayersandthreefullyconnectedlayers,requiring138millionweightsand15.5GMACstoclassifytheimagewithasizeof224×224.GoogleNet.ToimproveaccuracywhilereducingthecomputationofDNNinference,GoogleNetintroducesaninceptionmodulecomposedofdifferentsizedfilters.Asaresult,GoogleNetachievesabetteraccuracyperformancethanVGG-16whileonlyrequiringsevenmillionweightsand1.43GMACstoprocesstheimagewiththesamesize.ResNet.ResNet,thestate-of-the-arteffort,usesthe“shortcut”structuretoreachahuman-levelaccuracywithatop-5errorratebelow5%.Inaddition,the“shortcut”moduleisusedtosolvethegradientvanishingproblemduringthetrainingprocess,makingitpossibletotrainaDNNmodelwithadeeperstructure.TheperformanceofpopularCNNsappliedforAIvisiontasksgraduallyincreasedovertheyears,surpassinghumanvision(5%errorrateinthechartbelow). PerformanceofcurrentpopularDeepNeuralNetworksonImageNet.Humansachieveanerrorrateof5%.–SourceRecurrentNeuralNetwork(RNN)Arecurrentneuralnetwork(RNN)isanotherclassofartificialneuralnetworksthatusesequentialdatafeeding.RNNshavebeendevelopedtoaddressthetime-seriesproblemofsequentialinputdata.TheinputofRNNconsistsofthecurrentinputandtheprevioussamples.Therefore,theconnectionsbetweennodesformadirectedgraphalongatemporalsequence.Furthermore,eachneuroninanRNNownsaninternalmemorythatkeepstheinformationofthecomputationfromtheprevioussamples.ConceptofaRecurrentNeuralNetwork(RNN)RNNmodelsarewidelyusedinnaturallanguageprocessing(NLP)duetothesuperiorityofprocessingthedatawithaninputlengththatisnotfixed.ThetaskoftheAIhereistobuildasystemthatcancomprehendnaturallanguagespokenbyhumans,e.g.,naturallanguagemodeling,wordembedding,andmachinetranslation.InRNNs,eachsubsequentlayerisacollectionofnonlinearfunctionsofweightedsumsofoutputsandthepreviousstate.Thus,thebasicunitofRNNiscalled“cell”,andeachcellconsistsoflayersandaseriesofcellsthatenablesthesequentialprocessingofrecurrentneuralnetworkmodels.What’snextDeepneuralnetworksexcelatfindinghierarchicalrepresentationsthatsolvecomplextaskswithlargedatasets.Tolearnaboutusingdeepneuralnetworksinstate-of-the-artimagerecognition,checkoutourarticleImageRecognitionin2021:AComprehensiveGuide.AttheVisoComputerVisonBlogWealsocoverotherpopulartopicsrelatedtocomputervisionanddeeplearningtechnologies.Werecommendyouexplorethefollowingtopics:ReadaboutthedifferencebetweenCNNandANN.Aneasy-to-understandguidetoDeepReinforcementLearning.Readanintroductionto Self-SupervisedLearning.LearnaboutthedifferencebetweenDeepLearningvs.MachineLearning.RelatedArticlesShowmoreANNandCNN:AnalyzingDifferencesandSimilaritiesLearnaboutthemajordifferencebetweenatraditionalArtificialNeuralNetwork(ANN)andConvolutionalNeuralNetworks(CNN).ReadMore»Self-SupervisedLearning:WhatItIs,ExamplesandMethodsforComputerVisionLearnaboutself-supervisedlearning:Whatitis,examplesandmethodsforcomputervision.Fewerlabelsandsmallersamplesasnewstandard?ReadMore»All-in-oneplatformtobuildcomputervisionapplicationswithoutcodeShowmemoreviso.aiProductOverviewEvaluationGuideFeatureIndexAcademySecurityPrivacySolutionsPricingFeaturesComputerVisionVisualProgrammingCloudWorkspaceAnalyticsDashboardDeviceManagementIndustriesAgricultureHealthcareManufacturingRetailSecuritySmartCityTechnologyTransportationResourcesBlogLearnSupportDocumentationContributeAboutCompanyCareersTermsContactviso.aiFeaturesSolutionsCompanyBlogPricingRequestDemoPlayVideoJoin6,300+FellowAIEnthusiastsGetexpertAInews2xamonth.SubscribetothemostreadComputerVisionBlog.SubscribemeYoucanunsubscribeanytime.Seeourprivacypolicy.BuildanyComputerVisionApplication10xfasterNo-CodeComputerVisionPlatformforbusinessestobuild,deployandscaleonenterpriseinfrastructure.MoreRequestdemoSchedulealivedemoSelect...AfghanistanÅlandIslandsAlbaniaAlgeriaAmericanSamoaAndorraAngolaAnguillaAntarcticaAntiguaandBarbudaArgentinaArmeniaArubaAustraliaAustriaAzerbaijanBahrainBahamasBangladeshBarbadosBelarusBelgiumBelizeBeninBermudaBhutanBolivia,PlurinationalStateofBonaire,SintEustatiusandSabaBosniaandHerzegovinaBotswanaBouvetIslandBrazilBritishIndianOceanTerritoryBruneiDarussalamBulgariaBurkinaFasoBurundiCambodiaCameroonCanadaCapeVerdeCaymanIslandsCentralAfricanRepublicChadChileChinaChristmasIslandCocos(Keeling)IslandsColombiaComorosCongoCookIslandsCostaRicaCôted'IvoireCroatiaCubaCuraçaoCyprusCzechRepublicDenmarkDjiboutiDominicaDominicanRepublicEcuadorEgyptElSalvadorEquatorialGuineaEritreaEstoniaEthiopiaFalklandIslands(Malvinas)FaroeIslandsFijiFinlandFranceFrenchGuianaFrenchPolynesiaFrenchSouthernTerritoriesGabonGambiaGeorgiaGermanyGhanaGibraltarGreeceGreenlandGrenadaGuadeloupeGuamGuatemalaGuernseyGuineaGuinea-BissauGuyanaHaitiHolySee(VaticanCityState)HondurasHongKongHungaryIcelandIndiaIndonesiaIran,IslamicRepublicofIraqIrelandIsleofManIsraelItalyJamaicaJapanJerseyJordanKazakhstanKenyaKiribatiKoreaKuwaitKyrgyzstanLaoLatviaLebanonLesothoLiberiaLibyaLiechtensteinLithuaniaLuxembourgMacaoMacedoniaMadagascarMalawiMalaysiaMaldivesMaliMaltaMarshallIslandsMartiniqueMauritaniaMauritiusMayotteMexicoMicronesiaMoldovaMonacoMongoliaMontenegroMontserratMoroccoMozambiqueMyanmarNamibiaNauruNepalNetherlandsNewCaledoniaNewZealandNicaraguaNigerNigeriaNiueNorfolkIslandNorthernMarianaIslandsNorwayOmanPakistanPalauPalestine,StateofPanamaPapuaNewGuineaParaguayPeruPhilippinesPitcairnPolandPortugalPuertoRicoQatarRéunionRomaniaRussianFederationRwandaSaintBarthélemySaintHelenaSaintKittsandNevisSaintLuciaSaintMartin(Frenchpart)SaintPierreandMiquelonSaintVincentandtheGrenadinesSamoaSanMarinoSaoTomeandPrincipeSaudiArabiaSenegalSerbiaSeychellesSierraLeoneSingaporeSintMaarten(Dutchpart)SlovakiaSloveniaSolomonIslandsSomaliaSouthAfricaSouthSudanSpainSriLankaSudanSurinameSvalbardandJanMayenSwazilandSwedenSwitzerlandSyrianArabRepublicTaiwan,ProvinceofChinaTajikistanTanzania,UnitedRepublicofThailandTimor-LesteTogoTokelauTongaTrinidadandTobagoTunisiaTurkeyTurkmenistanTurksandCaicosIslandsTuvaluUgandaUkraineUnitedArabEmiratesUnitedKingdomUnitedStatesUruguayUzbekistanVanuatuVenezuela,BolivarianRepublicofVietNamVirginIslands,BritishVirginIslands,U.S.WallisandFutunaWesternSaharaYemenZambiaZimbabweIamlookingforVideoanalyticsEdgecomputervisionReal-timedeeplearningIaminterestedinPlatformoverviewFullsolutionservicesEvaluationaccessRequestDemoNotinterested?We’realwayslookingtoimprove,sopleaseletusknowwhyyouarenotinterestedinusingComputerVisionwithVisoSuite.IneedmoreinformationIthinkitstoodifficultIneedotherfeaturesIdon'thaveausecaseOtherFindsomehelpfulinformationorgetintouch:>Showmeallfeatures>Showmeusecases>Aboutthecompanyviso.ai>IhaveaquestionSubmitHowitworksBuildAppsDeployAppsOverviewMonitorAppsManageAppsCompanyAboutusJobsContactusAIVisionSolutionsPricingBlogSupportHelpCenterGetstarted