techniques of neural networksANN Deep LearningA mostly complete chart of neural networkstypes of convolutional neural networkCNN vs NNNeural network data sciencetypes of neural network pdfDeep learning - Wikipedia
文章推薦指數： 80 %
... because of artificial neural network's (ANN) computational cost and a lack of understanding of how the brain wires its biological networks. Deeplearning FromWikipedia,thefreeencyclopedia Jumptonavigation Jumptosearch Branchofmachinelearning Fordeepversusshallowlearningineducationalpsychology,seeStudentapproachestolearning.Formoreinformation,seeArtificialneuralnetwork. Representingimagesonmultiplelayersofabstractionindeeplearning[1] PartofaseriesonMachinelearninganddatamining Problems Classification Clustering Regression Anomalydetection DataCleaning AutoML Associationrules Reinforcementlearning Structuredprediction Featureengineering Featurelearning Onlinelearning Semi-supervisedlearning Unsupervisedlearning Learningtorank Grammarinduction Supervisedlearning(classification •regression) Decisiontrees Ensembles Bagging Boosting Randomforest k-NN Linearregression NaiveBayes Artificialneuralnetworks Logisticregression Perceptron Relevancevectormachine(RVM) Supportvectormachine(SVM) Clustering BIRCH CURE Hierarchical k-means Expectation–maximization(EM) DBSCAN OPTICS Meanshift Dimensionalityreduction Factoranalysis CCA ICA LDA NMF PCA PGD t-SNE Structuredprediction Graphicalmodels Bayesnet Conditionalrandomfield HiddenMarkov Anomalydetection k-NN Localoutlierfactor Artificialneuralnetwork Autoencoder Cognitivecomputing Deeplearning DeepDream Multilayerperceptron RNN LSTM GRU ESN RestrictedBoltzmannmachine GAN SOM Convolutionalneuralnetwork U-Net Transformer Vision Spikingneuralnetwork Memtransistor ElectrochemicalRAM(ECRAM) Reinforcementlearning Q-learning SARSA Temporaldifference(TD) Theory Kernelmachines Bias–variancetradeoff Computationallearningtheory Empiricalriskminimization Occamlearning PAClearning Statisticallearning VCtheory Machine-learningvenues NeurIPS ICML ML JMLR ArXiv:cs.LG Relatedarticles Glossaryofartificialintelligence Listofdatasetsformachine-learningresearch Outlineofmachinelearning vte PartofaseriesonArtificialintelligence Majorgoals Artificialgeneralintelligence Planning Computervision Generalgameplaying Knowledgereasoning Machinelearning Naturallanguageprocessing Robotics Approaches Symbolic Deeplearning Bayesiannetworks Evolutionaryalgorithms Philosophy Chineseroom FriendlyAI Controlproblem/Takeover Ethics Existentialrisk Turingtest History Timeline Progress AIwinter Technology Applications Projects Programminglanguages Glossary Glossary vte Deeplearning(alsoknownasdeepstructuredlearning)ispartofabroaderfamilyofmachinelearningmethodsbasedonartificialneuralnetworkswithrepresentationlearning.Learningcanbesupervised,semi-supervisedorunsupervised.[2] Deep-learningarchitecturessuchasdeepneuralnetworks,deepbeliefnetworks,deepreinforcementlearning,recurrentneuralnetworksandconvolutionalneuralnetworkshavebeenappliedtofieldsincludingcomputervision,speechrecognition,naturallanguageprocessing,machinetranslation,bioinformatics,drugdesign,medicalimageanalysis,climatescience,materialinspectionandboardgameprograms,wheretheyhaveproducedresultscomparabletoandinsomecasessurpassinghumanexpertperformance.[3][4][5] Artificialneuralnetworks(ANNs)wereinspiredbyinformationprocessinganddistributedcommunicationnodesinbiologicalsystems.ANNshavevariousdifferencesfrombiologicalbrains.Specifically,artificialneuralnetworkstendtobestaticandsymbolic,whilethebiologicalbrainofmostlivingorganismsisdynamic(plastic)andanalogue.[6][7] Theadjective"deep"indeeplearningreferstotheuseofmultiplelayersinthenetwork.Earlyworkshowedthatalinearperceptroncannotbeauniversalclassifier,butthatanetworkwithanonpolynomialactivationfunctionwithonehiddenlayerofunboundedwidthcan.Deeplearningisamodernvariationwhichisconcernedwithanunboundednumberoflayersofboundedsize,whichpermitspracticalapplicationandoptimizedimplementation,whileretainingtheoreticaluniversalityundermildconditions.Indeeplearningthelayersarealsopermittedtobeheterogeneousandtodeviatewidelyfrombiologicallyinformedconnectionistmodels,forthesakeofefficiency,trainabilityandunderstandability,whencethe"structured"part. Contents 1Definition 2Overview 3Interpretations 4History 4.1Deeplearningrevolution 5Neuralnetworks 5.1Artificialneuralnetworks 5.2Deepneuralnetworks 5.2.1Challenges 6Hardware 7Applications 7.1Automaticspeechrecognition 7.2Imagerecognition 7.3Visualartprocessing 7.4Naturallanguageprocessing 7.5Drugdiscoveryandtoxicology 7.6Customerrelationshipmanagement 7.7Recommendationsystems 7.8Bioinformatics 7.9Medicalimageanalysis 7.10Mobileadvertising 7.11Imagerestoration 7.12Financialfrauddetection 7.13Military 7.14Partialdifferentialequations 8Relationtohumancognitiveandbraindevelopment 9Commercialactivity 10Criticismandcomment 10.1Theory 10.2Errors 10.3Cyberthreat 10.4Relianceonhumanmicrowork 11Seealso 12References 13Furtherreading Definition[edit] Deeplearningisaclassofmachinelearningalgorithmsthat[8]: 199–200 usesmultiplelayerstoprogressivelyextracthigher-levelfeaturesfromtherawinput.Forexample,inimageprocessing,lowerlayersmayidentifyedges,whilehigherlayersmayidentifytheconceptsrelevanttoahumansuchasdigitsorlettersorfaces. Overview[edit] Mostmoderndeeplearningmodelsarebasedonartificialneuralnetworks,specificallyconvolutionalneuralnetworks(CNN)s,althoughtheycanalsoincludepropositionalformulasorlatentvariablesorganizedlayer-wiseindeepgenerativemodelssuchasthenodesindeepbeliefnetworksanddeepBoltzmannmachines.[9] Indeeplearning,eachlevellearnstotransformitsinputdataintoaslightlymoreabstractandcompositerepresentation.Inanimagerecognitionapplication,therawinputmaybeamatrixofpixels;thefirstrepresentationallayermayabstractthepixelsandencodeedges;thesecondlayermaycomposeandencodearrangementsofedges;thethirdlayermayencodeanoseandeyes;andthefourthlayermayrecognizethattheimagecontainsaface.Importantly,adeeplearningprocesscanlearnwhichfeaturestooptimallyplaceinwhichlevelonitsown.Thisdoesnoteliminatetheneedforhand-tuning;forexample,varyingnumbersoflayersandlayersizescanprovidedifferentdegreesofabstraction.[10][11] Theword"deep"in"deeplearning"referstothenumberoflayersthroughwhichthedataistransformed.Moreprecisely,deeplearningsystemshaveasubstantialcreditassignmentpath(CAP)depth.TheCAPisthechainoftransformationsfrominputtooutput.CAPsdescribepotentiallycausalconnectionsbetweeninputandoutput.Forafeedforwardneuralnetwork,thedepthoftheCAPsisthatofthenetworkandisthenumberofhiddenlayersplusone(astheoutputlayerisalsoparameterized).Forrecurrentneuralnetworks,inwhichasignalmaypropagatethroughalayermorethanonce,theCAPdepthispotentiallyunlimited.[12]Nouniversallyagreed-uponthresholdofdepthdividesshallowlearningfromdeeplearning,butmostresearchersagreethatdeeplearninginvolvesCAPdepthhigherthan2.CAPofdepth2hasbeenshowntobeauniversalapproximatorinthesensethatitcanemulateanyfunction.[13]Beyondthat,morelayersdonotaddtothefunctionapproximatorabilityofthenetwork.Deepmodels(CAP>2)areabletoextractbetterfeaturesthanshallowmodelsandhence,extralayershelpinlearningthefeatureseffectively. Deeplearningarchitecturescanbeconstructedwithagreedylayer-by-layermethod.[14]Deeplearninghelpstodisentangletheseabstractionsandpickoutwhichfeaturesimproveperformance.[10] Forsupervisedlearningtasks,deeplearningmethodseliminatefeatureengineering,bytranslatingthedataintocompactintermediaterepresentationsakintoprincipalcomponents,andderivelayeredstructuresthatremoveredundancyinrepresentation. Deeplearningalgorithmscanbeappliedtounsupervisedlearningtasks.Thisisanimportantbenefitbecauseunlabeleddataaremoreabundantthanthelabeleddata.Examplesofdeepstructuresthatcanbetrainedinanunsupervisedmanneraredeepbeliefnetworks.[10][15] Interpretations[edit] Deepneuralnetworksaregenerallyinterpretedintermsoftheuniversalapproximationtheorem[16][17][18][19][20]orprobabilisticinference.[8][9][10][12][21] Theclassicuniversalapproximationtheoremconcernsthecapacityoffeedforwardneuralnetworkswithasinglehiddenlayeroffinitesizetoapproximatecontinuousfunctions.[16][17][18][19]In1989,thefirstproofwaspublishedbyGeorgeCybenkoforsigmoidactivationfunctions[16]andwasgeneralisedtofeed-forwardmulti-layerarchitecturesin1991byKurtHornik.[17]Recentworkalsoshowedthatuniversalapproximationalsoholdsfornon-boundedactivationfunctionssuchastherectifiedlinearunit.[22] Theuniversalapproximationtheoremfordeepneuralnetworksconcernsthecapacityofnetworkswithboundedwidthbutthedepthisallowedtogrow.Luetal.[20]provedthatifthewidthofadeepneuralnetworkwithReLUactivationisstrictlylargerthantheinputdimension,thenthenetworkcanapproximateanyLebesgueintegrablefunction;Ifthewidthissmallerorequaltotheinputdimension,thenadeepneuralnetworkisnotauniversalapproximator. Theprobabilisticinterpretation[21]derivesfromthefieldofmachinelearning.Itfeaturesinference,[8][9][10][12][15][21]aswellastheoptimizationconceptsoftrainingandtesting,relatedtofittingandgeneralization,respectively.Morespecifically,theprobabilisticinterpretationconsiderstheactivationnonlinearityasacumulativedistributionfunction.[21]Theprobabilisticinterpretationledtotheintroductionofdropoutasregularizerinneuralnetworks.[23]TheprobabilisticinterpretationwasintroducedbyresearchersincludingHopfield,WidrowandNarendraandpopularizedinsurveyssuchastheonebyBishop.[24] History[edit] SomesourcespointoutthatFrankRosenblattdevelopedandexploredallofthebasicingredientsofthedeeplearningsystemsoftoday.[25]Hedescribeditinhisbook"PrinciplesofNeurodynamics:PerceptronsandtheTheoryofBrainMechanisms",publishedbyCornellAeronauticalLaboratory,Inc.,CornellUniversityin1962. Thefirstgeneral,workinglearningalgorithmforsupervised,deep,feedforward,multilayerperceptronswaspublishedbyAlexeyIvakhnenkoandLapain1967.[26]A1971paperdescribedadeepnetworkwitheightlayerstrainedbythegroupmethodofdatahandling.[27]Otherdeeplearningworkingarchitectures,specificallythosebuiltforcomputervision,beganwiththeNeocognitronintroducedbyKunihikoFukushimain1980.[28] ThetermDeepLearningwasintroducedtothemachinelearningcommunitybyRinaDechterin1986,[29]andtoartificialneuralnetworksbyIgorAizenbergandcolleaguesin2000,inthecontextofBooleanthresholdneurons.[30][31] In1989,YannLeCunetal.appliedthestandardbackpropagationalgorithm,whichhadbeenaroundasthereversemodeofautomaticdifferentiationsince1970,[32][33][34][35]toadeepneuralnetworkwiththepurposeofrecognizinghandwrittenZIPcodesonmail.Whilethealgorithmworked,trainingrequired3days.[36] In1994,AndrédeCarvalho,togetherwithMikeFairhurstandDavidBisset,publishedexperimentalresultsofamulti-layerbooleanneuralnetwork,alsoknownasaweightlessneuralnetwork,composedofa3-layersself-organisingfeatureextractionneuralnetworkmodule(SOFT)followedbyamulti-layerclassificationneuralnetworkmodule(GSN),whichwereindependentlytrained.Eachlayerinthefeatureextractionmoduleextractedfeatureswithgrowingcomplexityregardingthepreviouslayer.[37] In1995,BrendanFreydemonstratedthatitwaspossibletotrain(overtwodays)anetworkcontainingsixfullyconnectedlayersandseveralhundredhiddenunitsusingthewake-sleepalgorithm,co-developedwithPeterDayanandHinton.[38]Manyfactorscontributetotheslowspeed,includingthevanishinggradientproblemanalyzedin1991bySeppHochreiter.[39][40] Since1997,SvenBehnkeextendedthefeed-forwardhierarchicalconvolutionalapproachintheNeuralAbstractionPyramid[41]bylateralandbackwardconnectionsinordertoflexiblyincorporatecontextintodecisionsanditerativelyresolvelocalambiguities. Simplermodelsthatusetask-specifichandcraftedfeaturessuchasGaborfiltersandsupportvectormachines(SVMs)wereapopularchoiceinthe1990sand2000s,becauseofartificialneuralnetwork's(ANN)computationalcostandalackofunderstandingofhowthebrainwiresitsbiologicalnetworks. Bothshallowanddeeplearning(e.g.,recurrentnets)ofANNshavebeenexploredformanyyears.[42][43][44]Thesemethodsneveroutperformednon-uniforminternal-handcraftingGaussianmixturemodel/HiddenMarkovmodel(GMM-HMM)technologybasedongenerativemodelsofspeechtraineddiscriminatively.[45]Keydifficultieshavebeenanalyzed,includinggradientdiminishing[39]andweaktemporalcorrelationstructureinneuralpredictivemodels.[46][47]Additionaldifficultieswerethelackoftrainingdataandlimitedcomputingpower. Mostspeechrecognitionresearchersmovedawayfromneuralnetstopursuegenerativemodeling.AnexceptionwasatSRIInternationalinthelate1990s.FundedbytheUSgovernment'sNSAandDARPA,SRIstudieddeepneuralnetworksinspeechandspeakerrecognition.ThespeakerrecognitionteamledbyLarryHeckreportedsignificantsuccesswithdeepneuralnetworksinspeechprocessinginthe1998NationalInstituteofStandardsandTechnologySpeakerRecognitionevaluation.[48]TheSRIdeepneuralnetworkwasthendeployedintheNuanceVerifier,representingthefirstmajorindustrialapplicationofdeeplearning.[49] Theprincipleofelevating"raw"featuresoverhand-craftedoptimizationwasfirstexploredsuccessfullyinthearchitectureofdeepautoencoderonthe"raw"spectrogramorlinearfilter-bankfeaturesinthelate1990s,[49]showingitssuperiorityovertheMel-Cepstralfeaturesthatcontainstagesoffixedtransformationfromspectrograms.Therawfeaturesofspeech,waveforms,laterproducedexcellentlarger-scaleresults.[50] Manyaspectsofspeechrecognitionweretakenoverbyadeeplearningmethodcalledlongshort-termmemory(LSTM),arecurrentneuralnetworkpublishedbyHochreiterandSchmidhuberin1997.[51]LSTMRNNsavoidthevanishinggradientproblemandcanlearn"VeryDeepLearning"tasks[12]thatrequirememoriesofeventsthathappenedthousandsofdiscretetimestepsbefore,whichisimportantforspeech.In2003,LSTMstartedtobecomecompetitivewithtraditionalspeechrecognizersoncertaintasks.[52]Lateritwascombinedwithconnectionisttemporalclassification(CTC)[53]instacksofLSTMRNNs.[54]In2015,Google'sspeechrecognitionreportedlyexperiencedadramaticperformancejumpof49%throughCTC-trainedLSTM,whichtheymadeavailablethroughGoogleVoiceSearch.[55] In2006,publicationsbyGeoffHinton,RuslanSalakhutdinov,OsinderoandTeh[56] [57][58]showedhowamany-layeredfeedforwardneuralnetworkcouldbeeffectivelypre-trainedonelayeratatime,treatingeachlayerinturnasanunsupervisedrestrictedBoltzmannmachine,thenfine-tuningitusingsupervisedbackpropagation.[59]Thepapersreferredtolearningfordeepbeliefnets. Deeplearningispartofstate-of-the-artsystemsinvariousdisciplines,particularlycomputervisionandautomaticspeechrecognition(ASR).ResultsoncommonlyusedevaluationsetssuchasTIMIT(ASR)andMNIST(imageclassification),aswellasarangeoflarge-vocabularyspeechrecognitiontaskshavesteadilyimproved.[60][61][62][63]Convolutionalneuralnetworks(CNNs)weresupersededforASRbyCTC[53]forLSTM.[51][55][64][65][66][67][68]butaremoresuccessfulincomputervision. Theimpactofdeeplearninginindustrybeganintheearly2000s,whenCNNsalreadyprocessedanestimated10%to20%ofallthecheckswrittenintheUS,accordingtoYannLeCun.[69]Industrialapplicationsofdeeplearningtolarge-scalespeechrecognitionstartedaround2010. The2009NIPSWorkshoponDeepLearningforSpeechRecognition[70]wasmotivatedbythelimitationsofdeepgenerativemodelsofspeech,andthepossibilitythatgivenmorecapablehardwareandlarge-scaledatasetsthatdeepneuralnets(DNN)mightbecomepractical.Itwasbelievedthatpre-trainingDNNsusinggenerativemodelsofdeepbeliefnets(DBN)wouldovercomethemaindifficultiesofneuralnets.[71]However,itwasdiscoveredthatreplacingpre-trainingwithlargeamountsoftrainingdataforstraightforwardbackpropagationwhenusingDNNswithlarge,context-dependentoutputlayersproducederrorratesdramaticallylowerthanthen-state-of-the-artGaussianmixturemodel(GMM)/HiddenMarkovModel(HMM)andalsothanmore-advancedgenerativemodel-basedsystems.[60][72]Thenatureoftherecognitionerrorsproducedbythetwotypesofsystemswascharacteristicallydifferent,[73][70]offeringtechnicalinsightsintohowtointegratedeeplearningintotheexistinghighlyefficient,run-timespeechdecodingsystemdeployedbyallmajorspeechrecognitionsystems.[8][74][75]Analysisaround2009–2010,contrastingtheGMM(andothergenerativespeechmodels)vs.DNNmodels,stimulatedearlyindustrialinvestmentindeeplearningforspeechrecognition,[73][70]eventuallyleadingtopervasiveanddominantuseinthatindustry.Thatanalysiswasdonewithcomparableperformance(lessthan1.5%inerrorrate)betweendiscriminativeDNNsandgenerativemodels.[60][73][71][76] In2010,researchersextendeddeeplearningfromTIMITtolargevocabularyspeechrecognition,byadoptinglargeoutputlayersoftheDNNbasedoncontext-dependentHMMstatesconstructedbydecisiontrees.[77][78][79][74] Advancesinhardwarehavedrivenrenewedinterestindeeplearning.In2009,Nvidiawasinvolvedinwhatwascalledthe“bigbang”ofdeeplearning,“asdeep-learningneuralnetworksweretrainedwithNvidiagraphicsprocessingunits(GPUs).”[80]Thatyear,AndrewNgdeterminedthatGPUscouldincreasethespeedofdeep-learningsystemsbyabout100times.[81]Inparticular,GPUsarewell-suitedforthematrix/vectorcomputationsinvolvedinmachinelearning.[82][83][84]GPUsspeeduptrainingalgorithmsbyordersofmagnitude,reducingrunningtimesfromweekstodays.[85][86]Further,specializedhardwareandalgorithmoptimizationscanbeusedforefficientprocessingofdeeplearningmodels.[87] Deeplearningrevolution[edit] Howdeeplearningisasubsetofmachinelearningandhowmachinelearningisasubsetofartificialintelligence(AI). In2012,ateamledbyGeorgeE.Dahlwonthe"MerckMolecularActivityChallenge"usingmulti-taskdeepneuralnetworkstopredictthebiomoleculartargetofonedrug.[88][89]In2014,Hochreiter'sgroupuseddeeplearningtodetectoff-targetandtoxiceffectsofenvironmentalchemicalsinnutrients,householdproductsanddrugsandwonthe"Tox21DataChallenge"ofNIH,FDAandNCATS.[90][91][92] Significantadditionalimpactsinimageorobjectrecognitionwerefeltfrom2011to2012.AlthoughCNNstrainedbybackpropagationhadbeenaroundfordecades,andGPUimplementationsofNNsforyears,includingCNNs,fastimplementationsofCNNsonGPUswereneededtoprogressoncomputervision.[82][84][36][93][12]In2011,thisapproachachievedforthefirsttimesuperhumanperformanceinavisualpatternrecognitioncontest.Alsoin2011,itwontheICDARChinesehandwritingcontest,andinMay2012,itwontheISBIimagesegmentationcontest.[94]Until2011,CNNsdidnotplayamajorroleatcomputervisionconferences,butinJune2012,apaperbyCiresanetal.attheleadingconferenceCVPR[3]showedhowmax-poolingCNNsonGPUcandramaticallyimprovemanyvisionbenchmarkrecords.InOctober2012,asimilarsystembyKrizhevskyetal.[4]wonthelarge-scaleImageNetcompetitionbyasignificantmarginovershallowmachinelearningmethods.InNovember2012,Ciresanetal.'ssystemalsowontheICPRcontestonanalysisoflargemedicalimagesforcancerdetection,andinthefollowingyearalsotheMICCAIGrandChallengeonthesametopic.[95]In2013and2014,theerrorrateontheImageNettaskusingdeeplearningwasfurtherreduced,followingasimilartrendinlarge-scalespeechrecognition. Imageclassificationwasthenextendedtothemorechallengingtaskofgeneratingdescriptions(captions)forimages,oftenasacombinationofCNNsandLSTMs.[96][97][98][99] SomeresearchersstatethattheOctober2012ImageNetvictoryanchoredthestartofa"deeplearningrevolution"thathastransformedtheAIindustry.[100] InMarch2019,YoshuaBengio,GeoffreyHintonandYannLeCunwereawardedtheTuringAwardforconceptualandengineeringbreakthroughsthathavemadedeepneuralnetworksacriticalcomponentofcomputing. Neuralnetworks[edit] Artificialneuralnetworks[edit] Mainarticle:Artificialneuralnetwork Artificialneuralnetworks(ANNs)orconnectionistsystemsarecomputingsystemsinspiredbythebiologicalneuralnetworksthatconstituteanimalbrains.Suchsystemslearn(progressivelyimprovetheirability)todotasksbyconsideringexamples,generallywithouttask-specificprogramming.Forexample,inimagerecognition,theymightlearntoidentifyimagesthatcontaincatsbyanalyzingexampleimagesthathavebeenmanuallylabeledas"cat"or"nocat"andusingtheanalyticresultstoidentifycatsinotherimages.Theyhavefoundmostuseinapplicationsdifficulttoexpresswithatraditionalcomputeralgorithmusingrule-basedprogramming. AnANNisbasedonacollectionofconnectedunitscalledartificialneurons,(analogoustobiologicalneuronsinabiologicalbrain).Eachconnection(synapse)betweenneuronscantransmitasignaltoanotherneuron.Thereceiving(postsynaptic)neuroncanprocessthesignal(s)andthensignaldownstreamneuronsconnectedtoit.Neuronsmayhavestate,generallyrepresentedbyrealnumbers,typicallybetween0and1.Neuronsandsynapsesmayalsohaveaweightthatvariesaslearningproceeds,whichcanincreaseordecreasethestrengthofthesignalthatitsendsdownstream. Typically,neuronsareorganizedinlayers.Differentlayersmayperformdifferentkindsoftransformationsontheirinputs.Signalstravelfromthefirst(input),tothelast(output)layer,possiblyaftertraversingthelayersmultipletimes. Theoriginalgoaloftheneuralnetworkapproachwastosolveproblemsinthesamewaythatahumanbrainwould.Overtime,attentionfocusedonmatchingspecificmentalabilities,leadingtodeviationsfrombiologysuchasbackpropagation,orpassinginformationinthereversedirectionandadjustingthenetworktoreflectthatinformation. Neuralnetworkshavebeenusedonavarietyoftasks,includingcomputervision,speechrecognition,machinetranslation,socialnetworkfiltering,playingboardandvideogamesandmedicaldiagnosis. Asof2017,neuralnetworkstypicallyhaveafewthousandtoafewmillionunitsandmillionsofconnections.Despitethisnumberbeingseveralorderofmagnitudelessthanthenumberofneuronsonahumanbrain,thesenetworkscanperformmanytasksatalevelbeyondthatofhumans(e.g.,recognizingfaces,playing"Go"[101]). Deepneuralnetworks[edit] Thissectionmaybetootechnicalformostreaderstounderstand.Pleasehelpimproveittomakeitunderstandabletonon-experts,withoutremovingthetechnicaldetails.(July2016)(Learnhowandwhentoremovethistemplatemessage) Adeepneuralnetwork(DNN)isanartificialneuralnetwork(ANN)withmultiplelayersbetweentheinputandoutputlayers.[9][12]Therearedifferenttypesofneuralnetworksbuttheyalwaysconsistofthesamecomponents:neurons,synapses,weights,biases,andfunctions.[102]ThesecomponentsfunctioningsimilartothehumanbrainsandcanbetrainedlikeanyotherMLalgorithm.[citationneeded] Forexample,aDNNthatistrainedtorecognizedogbreedswillgooverthegivenimageandcalculatetheprobabilitythatthedogintheimageisacertainbreed.Theusercanreviewtheresultsandselectwhichprobabilitiesthenetworkshoulddisplay(aboveacertainthreshold,etc.)andreturntheproposedlabel.Eachmathematicalmanipulationassuchisconsideredalayer,andcomplexDNNhavemanylayers,hencethename"deep"networks. DNNscanmodelcomplexnon-linearrelationships.DNNarchitecturesgeneratecompositionalmodelswheretheobjectisexpressedasalayeredcompositionofprimitives.[103]Theextralayersenablecompositionoffeaturesfromlowerlayers,potentiallymodelingcomplexdatawithfewerunitsthanasimilarlyperformingshallownetwork.[9]Forinstance,itwasprovedthatsparsemultivariatepolynomialsareexponentiallyeasiertoapproximatewithDNNsthanwithshallownetworks.[104] Deeparchitecturesincludemanyvariantsofafewbasicapproaches.Eacharchitecturehasfoundsuccessinspecificdomains.Itisnotalwayspossibletocomparetheperformanceofmultiplearchitectures,unlesstheyhavebeenevaluatedonthesamedatasets. DNNsaretypicallyfeedforwardnetworksinwhichdataflowsfromtheinputlayertotheoutputlayerwithoutloopingback.Atfirst,theDNNcreatesamapofvirtualneuronsandassignsrandomnumericalvalues,or"weights",toconnectionsbetweenthem.Theweightsandinputsaremultipliedandreturnanoutputbetween0and1.Ifthenetworkdidnotaccuratelyrecognizeaparticularpattern,analgorithmwouldadjusttheweights.[105]Thatwaythealgorithmcanmakecertainparametersmoreinfluential,untilitdeterminesthecorrectmathematicalmanipulationtofullyprocessthedata. Recurrentneuralnetworks(RNNs),inwhichdatacanflowinanydirection,areusedforapplicationssuchaslanguagemodeling.[106][107][108][109][110]Longshort-termmemoryisparticularlyeffectiveforthisuse.[51][111] Convolutionaldeepneuralnetworks(CNNs)areusedincomputervision.[112]CNNsalsohavebeenappliedtoacousticmodelingforautomaticspeechrecognition(ASR).[68] Challenges[edit] AswithANNs,manyissuescanarisewithnaivelytrainedDNNs.Twocommonissuesareoverfittingandcomputationtime. DNNsarepronetooverfittingbecauseoftheaddedlayersofabstraction,whichallowthemtomodelraredependenciesinthetrainingdata.RegularizationmethodssuchasIvakhnenko'sunitpruning[27]orweightdecay( ℓ 2 {\displaystyle\ell_{2}} -regularization)orsparsity( ℓ 1 {\displaystyle\ell_{1}} -regularization)canbeappliedduringtrainingtocombatoverfitting.[113]Alternativelydropoutregularizationrandomlyomitsunitsfromthehiddenlayersduringtraining.Thishelpstoexcluderaredependencies.[114]Finally,datacanbeaugmentedviamethodssuchascroppingandrotatingsuchthatsmallertrainingsetscanbeincreasedinsizetoreducethechancesofoverfitting.[115] DNNsmustconsidermanytrainingparameters,suchasthesize(numberoflayersandnumberofunitsperlayer),thelearningrate,andinitialweights.Sweepingthroughtheparameterspaceforoptimalparametersmaynotbefeasibleduetothecostintimeandcomputationalresources.Varioustricks,suchasbatching(computingthegradientonseveraltrainingexamplesatonceratherthanindividualexamples)[116]speedupcomputation.Largeprocessingcapabilitiesofmany-corearchitectures(suchasGPUsortheIntelXeonPhi)haveproducedsignificantspeedupsintraining,becauseofthesuitabilityofsuchprocessingarchitecturesforthematrixandvectorcomputations.[117][118] Alternatively,engineersmaylookforothertypesofneuralnetworkswithmorestraightforwardandconvergenttrainingalgorithms.CMAC(cerebellarmodelarticulationcontroller)isonesuchkindofneuralnetwork.Itdoesn'trequirelearningratesorrandomizedinitialweightsforCMAC.Thetrainingprocesscanbeguaranteedtoconvergeinonestepwithanewbatchofdata,andthecomputationalcomplexityofthetrainingalgorithmislinearwithrespecttothenumberofneuronsinvolved.[119][120] Hardware[edit] Sincethe2010s,advancesinbothmachinelearningalgorithmsandcomputerhardwarehaveledtomoreefficientmethodsfortrainingdeepneuralnetworksthatcontainmanylayersofnon-linearhiddenunitsandaverylargeoutputlayer.[121]By2019,graphicprocessingunits(GPUs),oftenwithAI-specificenhancements,haddisplacedCPUsasthedominantmethodoftraininglarge-scalecommercialcloudAI.[122]OpenAIestimatedthehardwarecomputationusedinthelargestdeeplearningprojectsfromAlexNet(2012)toAlphaZero(2017),andfounda300,000-foldincreaseintheamountofcomputationrequired,withadoubling-timetrendlineof3.4months.[123][124] Applications[edit] Automaticspeechrecognition[edit] Mainarticle:Speechrecognition Large-scaleautomaticspeechrecognitionisthefirstandmostconvincingsuccessfulcaseofdeeplearning.LSTMRNNscanlearn"VeryDeepLearning"tasks[12]thatinvolvemulti-secondintervalscontainingspeecheventsseparatedbythousandsofdiscretetimesteps,whereonetimestepcorrespondstoabout10ms.LSTMwithforgetgates[111]iscompetitivewithtraditionalspeechrecognizersoncertaintasks.[52] Theinitialsuccessinspeechrecognitionwasbasedonsmall-scalerecognitiontasksbasedonTIMIT.Thedatasetcontains630speakersfromeightmajordialectsofAmericanEnglish,whereeachspeakerreads10sentences.[125]Itssmallsizeletsmanyconfigurationsbetried.Moreimportantly,theTIMITtaskconcernsphone-sequencerecognition,which,unlikeword-sequencerecognition,allowsweakphonebigramlanguagemodels.Thisletsthestrengthoftheacousticmodelingaspectsofspeechrecognitionbemoreeasilyanalyzed.Theerrorrateslistedbelow,includingtheseearlyresultsandmeasuredaspercentphoneerrorrates(PER),havebeensummarizedsince1991. Method Percentphoneerrorrate(PER)(%) RandomlyInitializedRNN[126] 26.1 BayesianTriphoneGMM-HMM 25.6 HiddenTrajectory(Generative)Model 24.8 MonophoneRandomlyInitializedDNN 23.4 MonophoneDBN-DNN 22.4 TriphoneGMM-HMMwithBMMITraining 21.7 MonophoneDBN-DNNonfbank 20.7 ConvolutionalDNN[127] 20.0 ConvolutionalDNNw.HeterogeneousPooling 18.7 EnsembleDNN/CNN/RNN[128] 18.3 BidirectionalLSTM 17.8 HierarchicalConvolutionalDeepMaxoutNetwork[129] 16.5 ThedebutofDNNsforspeakerrecognitioninthelate1990sandspeechrecognitionaround2009-2011andofLSTMaround2003–2007,acceleratedprogressineightmajorareas:[8][76][74] Scale-up/outandacceleratedDNNtraininganddecoding Sequencediscriminativetraining Featureprocessingbydeepmodelswithsolidunderstandingoftheunderlyingmechanisms AdaptationofDNNsandrelateddeepmodels Multi-taskandtransferlearningbyDNNsandrelateddeepmodels CNNsandhowtodesignthemtobestexploitdomainknowledgeofspeech RNNanditsrichLSTMvariants Othertypesofdeepmodelsincludingtensor-basedmodelsandintegrateddeepgenerative/discriminativemodels. Allmajorcommercialspeechrecognitionsystems(e.g.,MicrosoftCortana,Xbox,SkypeTranslator,AmazonAlexa,GoogleNow,AppleSiri,BaiduandiFlyTekvoicesearch,andarangeofNuancespeechproducts,etc.)arebasedondeeplearning.[8][130][131] Imagerecognition[edit] Mainarticle:Computervision AcommonevaluationsetforimageclassificationistheMNISTdatabasedataset.MNISTiscomposedofhandwrittendigitsandincludes60,000trainingexamplesand10,000testexamples.AswithTIMIT,itssmallsizeletsuserstestmultipleconfigurations.Acomprehensivelistofresultsonthissetisavailable.[132] Deeplearning-basedimagerecognitionhasbecome"superhuman",producingmoreaccurateresultsthanhumancontestants.Thisfirstoccurredin2011inrecognitionoftrafficsigns,andin2014,withrecognitionofhumanfaces.[133]SurpassingHumanLevelFaceRecognition Deeplearning-trainedvehiclesnowinterpret360°cameraviews.[134]AnotherexampleisFacialDysmorphologyNovelAnalysis(FDNA)usedtoanalyzecasesofhumanmalformationconnectedtoalargedatabaseofgeneticsyndromes. Visualartprocessing[edit] Closelyrelatedtotheprogressthathasbeenmadeinimagerecognitionistheincreasingapplicationofdeeplearningtechniquestovariousvisualarttasks.DNNshaveproventhemselvescapable,forexample,ofa)identifyingthestyleperiodofagivenpainting,b)NeuralStyleTransfer –capturingthestyleofagivenartworkandapplyingitinavisuallypleasingmannertoanarbitraryphotographorvideo,andc)generatingstrikingimagerybasedonrandomvisualinputfields.[135][136] Naturallanguageprocessing[edit] Mainarticle:Naturallanguageprocessing Neuralnetworkshavebeenusedforimplementinglanguagemodelssincetheearly2000s.[106]LSTMhelpedtoimprovemachinetranslationandlanguagemodeling.[107][108][109] Otherkeytechniquesinthisfieldarenegativesampling[137]andwordembedding.Wordembedding,suchasword2vec,canbethoughtofasarepresentationallayerinadeeplearningarchitecturethattransformsanatomicwordintoapositionalrepresentationofthewordrelativetootherwordsinthedataset;thepositionisrepresentedasapointinavectorspace.UsingwordembeddingasanRNNinputlayerallowsthenetworktoparsesentencesandphrasesusinganeffectivecompositionalvectorgrammar.Acompositionalvectorgrammarcanbethoughtofasprobabilisticcontextfreegrammar(PCFG)implementedbyanRNN.[138]Recursiveauto-encodersbuiltatopwordembeddingscanassesssentencesimilarityanddetectparaphrasing.[138]Deepneuralarchitecturesprovidethebestresultsforconstituencyparsing,[139]sentimentanalysis,[140]informationretrieval,[141][142]spokenlanguageunderstanding,[143]machinetranslation,[107][144]contextualentitylinking,[144]writingstylerecognition,[145]Textclassificationandothers.[146] Recentdevelopmentsgeneralizewordembeddingtosentenceembedding. GoogleTranslate(GT)usesalargeend-to-endlongshort-termmemory(LSTM)network.[147][148][149][150][151][152]GoogleNeuralMachineTranslation(GNMT)usesanexample-basedmachinetranslationmethodinwhichthesystem"learnsfrommillionsofexamples."[148]Ittranslates"wholesentencesatatime,ratherthanpieces.GoogleTranslatesupportsoveronehundredlanguages.[148]Thenetworkencodesthe"semanticsofthesentenceratherthansimplymemorizingphrase-to-phrasetranslations".[148][153]GTusesEnglishasanintermediatebetweenmostlanguagepairs.[153] Drugdiscoveryandtoxicology[edit] Formoreinformation,seeDrugdiscoveryandToxicology. Alargepercentageofcandidatedrugsfailtowinregulatoryapproval.Thesefailuresarecausedbyinsufficientefficacy(on-targeteffect),undesiredinteractions(off-targeteffects),orunanticipatedtoxiceffects.[154][155]Researchhasexploreduseofdeeplearningtopredictthebiomoleculartargets,[88][89]off-targets,andtoxiceffectsofenvironmentalchemicalsinnutrients,householdproductsanddrugs.[90][91][92] AtomNetisadeeplearningsystemforstructure-basedrationaldrugdesign.[156]AtomNetwasusedtopredictnovelcandidatebiomoleculesfordiseasetargetssuchastheEbolavirus[157]andmultiplesclerosis.[158][159] In2017graphneuralnetworkswereusedforthefirsttimetopredictvariouspropertiesofmoleculesinalargetoxicologydataset.[160]In2019,generativeneuralnetworkswereusedtoproducemoleculesthatwerevalidatedexperimentallyallthewayintomice.[161][162] Customerrelationshipmanagement[edit] Mainarticle:Customerrelationshipmanagement Deepreinforcementlearninghasbeenusedtoapproximatethevalueofpossibledirectmarketingactions,definedintermsofRFMvariables.Theestimatedvaluefunctionwasshowntohaveanaturalinterpretationascustomerlifetimevalue.[163] Recommendationsystems[edit] Mainarticle:Recommendersystem Recommendationsystemshaveuseddeeplearningtoextractmeaningfulfeaturesforalatentfactormodelforcontent-basedmusicandjournalrecommendations.[164][165]Multi-viewdeeplearninghasbeenappliedforlearninguserpreferencesfrommultipledomains.[166]Themodelusesahybridcollaborativeandcontent-basedapproachandenhancesrecommendationsinmultipletasks. Bioinformatics[edit] Mainarticle:Bioinformatics AnautoencoderANNwasusedinbioinformatics,topredictgeneontologyannotationsandgene-functionrelationships.[167] Inmedicalinformatics,deeplearningwasusedtopredictsleepqualitybasedondatafromwearables[168]andpredictionsofhealthcomplicationsfromelectronichealthrecorddata.[169] Medicalimageanalysis[edit] Deeplearninghasbeenshowntoproducecompetitiveresultsinmedicalapplicationsuchascancercellclassification,lesiondetection,organsegmentationandimageenhancement.[170][171] Mobileadvertising[edit] Findingtheappropriatemobileaudienceformobileadvertisingisalwayschallenging,sincemanydatapointsmustbeconsideredandanalyzedbeforeatargetsegmentcanbecreatedandusedinadservingbyanyadserver.[172]Deeplearninghasbeenusedtointerpretlarge,many-dimensionedadvertisingdatasets.Manydatapointsarecollectedduringtherequest/serve/clickinternetadvertisingcycle.Thisinformationcanformthebasisofmachinelearningtoimproveadselection. Imagerestoration[edit] Deeplearninghasbeensuccessfullyappliedtoinverseproblemssuchasdenoising,super-resolution,inpainting,andfilmcolorization.[173]Theseapplicationsincludelearningmethodssuchas"ShrinkageFieldsforEffectiveImageRestoration"[174]whichtrainsonanimagedataset,andDeepImagePrior,whichtrainsontheimagethatneedsrestoration. Financialfrauddetection[edit] Deeplearningisbeingsuccessfullyappliedtofinancialfrauddetection,taxevasiondetection,[175]andanti-moneylaundering.[176] Military[edit] TheUnitedStatesDepartmentofDefenseapplieddeeplearningtotrainrobotsinnewtasksthroughobservation.[177] Partialdifferentialequations[edit] Physicsinformedneuralnetworkshavebeenusedtosolvepartialdifferentialequationsinbothforwardandinverseproblemsinadatadrivenmanner.[178]OneexampleisthereconstructingfluidflowgovernedbytheNavier-Stokesequations.UsingphysicsinformedneuralnetworksdoesnotrequiretheoftenexpensivemeshgenerationthatconventionalCFDmethodsrelieson.[179][180] Relationtohumancognitiveandbraindevelopment[edit] Deeplearningiscloselyrelatedtoaclassoftheoriesofbraindevelopment(specifically,neocorticaldevelopment)proposedbycognitiveneuroscientistsintheearly1990s.[181][182][183][184]Thesedevelopmentaltheorieswereinstantiatedincomputationalmodels,makingthempredecessorsofdeeplearningsystems.Thesedevelopmentalmodelssharethepropertythatvariousproposedlearningdynamicsinthebrain(e.g.,awaveofnervegrowthfactor)supporttheself-organizationsomewhatanalogoustotheneuralnetworksutilizedindeeplearningmodels.Liketheneocortex,neuralnetworksemployahierarchyoflayeredfiltersinwhicheachlayerconsidersinformationfromapriorlayer(ortheoperatingenvironment),andthenpassesitsoutput(andpossiblytheoriginalinput),tootherlayers.Thisprocessyieldsaself-organizingstackoftransducers,well-tunedtotheiroperatingenvironment.A1995descriptionstated,"...theinfant'sbrainseemstoorganizeitselfundertheinfluenceofwavesofso-calledtrophic-factors...differentregionsofthebrainbecomeconnectedsequentially,withonelayeroftissuematuringbeforeanotherandsoonuntilthewholebrainismature."[185] Avarietyofapproacheshavebeenusedtoinvestigatetheplausibilityofdeeplearningmodelsfromaneurobiologicalperspective.Ontheonehand,severalvariantsofthebackpropagationalgorithmhavebeenproposedinordertoincreaseitsprocessingrealism.[186][187]Otherresearchershavearguedthatunsupervisedformsofdeeplearning,suchasthosebasedonhierarchicalgenerativemodelsanddeepbeliefnetworks,maybeclosertobiologicalreality.[188][189]Inthisrespect,generativeneuralnetworkmodelshavebeenrelatedtoneurobiologicalevidenceaboutsampling-basedprocessinginthecerebralcortex.[190] Althoughasystematiccomparisonbetweenthehumanbrainorganizationandtheneuronalencodingindeepnetworkshasnotyetbeenestablished,severalanalogieshavebeenreported.Forexample,thecomputationsperformedbydeeplearningunitscouldbesimilartothoseofactualneurons[191][192]andneuralpopulations.[193]Similarly,therepresentationsdevelopedbydeeplearningmodelsaresimilartothosemeasuredintheprimatevisualsystem[194]bothatthesingle-unit[195]andatthepopulation[196]levels. Commercialactivity[edit] Facebook'sAIlabperformstaskssuchasautomaticallytagginguploadedpictureswiththenamesofthepeopleinthem.[197] Google'sDeepMindTechnologiesdevelopedasystemcapableoflearninghowtoplayAtarivideogamesusingonlypixelsasdatainput.In2015theydemonstratedtheirAlphaGosystem,whichlearnedthegameofGowellenoughtobeataprofessionalGoplayer.[198][199][200]GoogleTranslateusesaneuralnetworktotranslatebetweenmorethan100languages. In2015,Blippardemonstratedamobileaugmentedrealityapplicationthatusesdeeplearningtorecognizeobjectsinrealtime.[201] In2017,Covariant.aiwaslaunched,whichfocusesonintegratingdeeplearningintofactories.[202] Asof2008,[203]researchersatTheUniversityofTexasatAustin(UT)developedamachinelearningframeworkcalledTraininganAgentManuallyviaEvaluativeReinforcement,orTAMER,whichproposednewmethodsforrobotsorcomputerprogramstolearnhowtoperformtasksbyinteractingwithahumaninstructor.[177]FirstdevelopedasTAMER,anewalgorithmcalledDeepTAMERwaslaterintroducedin2018duringacollaborationbetweenU.S.ArmyResearchLaboratory(ARL)andUTresearchers.DeepTAMERuseddeeplearningtoprovidearobottheabilitytolearnnewtasksthroughobservation.[177]UsingDeepTAMER,arobotlearnedataskwithahumantrainer,watchingvideostreamsorobservingahumanperformataskin-person.Therobotlaterpracticedthetaskwiththehelpofsomecoachingfromthetrainer,whoprovidedfeedbacksuchas“goodjob”and“badjob.”[204] Criticismandcomment[edit] Deeplearninghasattractedbothcriticismandcomment,insomecasesfromoutsidethefieldofcomputerscience. Theory[edit] Seealso:ExplainableAI Amaincriticismconcernsthelackoftheorysurroundingsomemethods.[205]Learninginthemostcommondeeparchitecturesisimplementedusingwell-understoodgradientdescent.However,thetheorysurroundingotheralgorithms,suchascontrastivedivergenceislessclear.[citationneeded](e.g.,Doesitconverge?Ifso,howfast?Whatisitapproximating?)Deeplearningmethodsareoftenlookedatasablackbox,withmostconfirmationsdoneempirically,ratherthantheoretically.[206] OtherspointoutthatdeeplearningshouldbelookedatasasteptowardsrealizingstrongAI,notasanall-encompassingsolution.Despitethepowerofdeeplearningmethods,theystilllackmuchofthefunctionalityneededforrealizingthisgoalentirely.ResearchpsychologistGaryMarcusnoted:"Realistically,deeplearningisonlypartofthelargerchallengeofbuildingintelligentmachines.Suchtechniqueslackwaysofrepresentingcausalrelationships(...)havenoobviouswaysofperforminglogicalinferences,andtheyarealsostillalongwayfromintegratingabstractknowledge,suchasinformationaboutwhatobjectsare,whattheyarefor,andhowtheyaretypicallyused.ThemostpowerfulA.I.systems,likeWatson(...)usetechniqueslikedeeplearningasjustoneelementinaverycomplicatedensembleoftechniques,rangingfromthestatisticaltechniqueofBayesianinferencetodeductivereasoning."[207] Infurtherreferencetotheideathatartisticsensitivitymightbeinherentinrelativelylowlevelsofthecognitivehierarchy,apublishedseriesofgraphicrepresentationsoftheinternalstatesofdeep(20-30layers)neuralnetworksattemptingtodiscernwithinessentiallyrandomdatatheimagesonwhichtheyweretrained[208]demonstrateavisualappeal:theoriginalresearchnoticereceivedwellover1,000comments,andwasthesubjectofwhatwasforatimethemostfrequentlyaccessedarticleonTheGuardian's[209]website. Errors[edit] Somedeeplearningarchitecturesdisplayproblematicbehaviors,[210]suchasconfidentlyclassifyingunrecognizableimagesasbelongingtoafamiliarcategoryofordinaryimages(2014)[211]andmisclassifyingminusculeperturbationsofcorrectlyclassifiedimages(2013).[212]Goertzelhypothesizedthatthesebehaviorsareduetolimitationsintheirinternalrepresentationsandthattheselimitationswouldinhibitintegrationintoheterogeneousmulti-componentartificialgeneralintelligence(AGI)architectures.[210]Theseissuesmaypossiblybeaddressedbydeeplearningarchitecturesthatinternallyformstateshomologoustoimage-grammar[213]decompositionsofobservedentitiesandevents.[210]Learningagrammar(visualorlinguistic)fromtrainingdatawouldbeequivalenttorestrictingthesystemtocommonsensereasoningthatoperatesonconceptsintermsofgrammaticalproductionrulesandisabasicgoalofbothhumanlanguageacquisition[214]andartificialintelligence(AI).[215] Cyberthreat[edit] Asdeeplearningmovesfromthelabintotheworld,researchandexperienceshowthatartificialneuralnetworksarevulnerabletohacksanddeception.[216]Byidentifyingpatternsthatthesesystemsusetofunction,attackerscanmodifyinputstoANNsinsuchawaythattheANNfindsamatchthathumanobserverswouldnotrecognize.Forexample,anattackercanmakesubtlechangestoanimagesuchthattheANNfindsamatcheventhoughtheimagelookstoahumannothinglikethesearchtarget.Suchmanipulationistermedan“adversarialattack.”[217] In2016researchersusedoneANNtodoctorimagesintrialanderrorfashion,identifyanother'sfocalpointsandtherebygenerateimagesthatdeceivedit.Themodifiedimageslookednodifferenttohumaneyes.Anothergroupshowedthatprintoutsofdoctoredimagesthenphotographedsuccessfullytrickedanimageclassificationsystem.[218]Onedefenseisreverseimagesearch,inwhichapossiblefakeimageissubmittedtoasitesuchasTinEyethatcanthenfindotherinstancesofit.Arefinementistosearchusingonlypartsoftheimage,toidentifyimagesfromwhichthatpiecemayhavebeentaken.[219] Anothergroupshowedthatcertainpsychedelicspectaclescouldfoolafacialrecognitionsystemintothinkingordinarypeoplewerecelebrities,potentiallyallowingonepersontoimpersonateanother.In2017researchersaddedstickerstostopsignsandcausedanANNtomisclassifythem.[218] ANNscanhoweverbefurthertrainedtodetectattemptsatdeception,potentiallyleadingattackersanddefendersintoanarmsracesimilartothekindthatalreadydefinesthemalwaredefenseindustry.ANNshavebeentrainedtodefeatANN-basedanti-malwaresoftwarebyrepeatedlyattackingadefensewithmalwarethatwascontinuallyalteredbyageneticalgorithmuntilittrickedtheanti-malwarewhileretainingitsabilitytodamagethetarget.[218] In2016,anothergroupdemonstratedthatcertainsoundscouldmaketheGoogleNowvoicecommandsystemopenaparticularwebaddress,andhypothesizedthatthiscould"serveasasteppingstoneforfurtherattacks(e.g.,openingawebpagehostingdrive-bymalware)."[218] In“datapoisoning,”falsedataiscontinuallysmuggledintoamachinelearningsystem'strainingsettopreventitfromachievingmastery.[218] Relianceonhumanmicrowork[edit] Thissectionneedsadditionalcitationsforverification.Pleasehelpimprovethisarticlebyaddingcitationstoreliablesources.Unsourcedmaterialmaybechallengedandremoved.Findsources: "Deeplearning" – news ·newspapers ·books ·scholar ·JSTOR(April2021)(Learnhowandwhentoremovethistemplatemessage) MostDeepLearningsystemsrelyontrainingandverificationdatathatisgeneratedand/orannotatedbyhumans.Ithasbeenarguedinmediaphilosophythatnotonlylow-paidclickwork(e.g.onAmazonMechanicalTurk)isregularlydeployedforthispurpose,butalsoimplicitformsofhumanmicroworkthatareoftennotrecognizedassuch.[220]ThephilosopherRainerMühlhoffdistinguishesfivetypesof"machiniccapture"ofhumanmicroworktogeneratetrainingdata:(1)gamification(theembeddingofannotationorcomputationtasksintheflowofagame),(2)"trappingandtracking"(e.g.CAPTCHAsforimagerecognitionorclick-trackingonGooglesearchresultspages),(3)exploitationofsocialmotivations(e.g.taggingfacesonFacebooktoobtainlabeledfacialimages),(4)informationmining(e.g.byleveragingquantified-selfdevicessuchasactivitytrackers)and(5)clickwork.[220] Mühlhoffarguesthatinmostcommercialend-userapplicationsofDeepLearningsuchasFacebook'sfacerecognitionsystem,theneedfortrainingdatadoesnotstoponceanANNistrained.Rather,thereisacontinueddemandforhuman-generatedverificationdata toconstantlycalibrateandupdatetheANN.ForthispurposeFacebookintroducedthefeaturethatonceauserisautomaticallyrecognizedinanimage,theyreceiveanotification.Theycanchoosewhetherofnottheyliketobepubliclylabeledontheimage,ortellFacebookthatitisnottheminthepicture.[221]Thisuserinterfaceisamechanismtogenerate"aconstantstreamofverificationdata"[220]tofurthertrainthenetworkinreal-time.AsMühlhoffargues,involvementofhumanuserstogeneratetrainingandverificationdataissotypicalformostcommercialend-userapplicationsofDeepLearningthatsuchsystemsmaybereferredtoas"human-aidedartificialintelligence".[220] Seealso[edit] Applicationsofartificialintelligence Comparisonofdeeplearningsoftware Compressedsensing Differentiableprogramming Echostatenetwork Listofartificialintelligenceprojects Liquidstatemachine Listofdatasetsformachinelearningresearch Reservoircomputing Sparsecoding References[edit] ^Schulz,Hannes;Behnke,Sven(1November2012)."DeepLearning".KI-KünstlicheIntelligenz.26(4):357–363.doi:10.1007/s13218-012-0198-z.ISSN 1610-1987.S2CID 220523562. ^Bengio,Yoshua;LeCun,Yann;Hinton,Geoffrey(2015)."DeepLearning".Nature.521(7553):436–444.Bibcode:2015Natur.521..436L.doi:10.1038/nature14539.PMID 26017442.S2CID 3074096. ^abCiresan,D.;Meier,U.;Schmidhuber,J.(2012)."Multi-columndeepneuralnetworksforimageclassification".2012IEEEConferenceonComputerVisionandPatternRecognition.pp. 3642–3649.arXiv:1202.2745.doi:10.1109/cvpr.2012.6248110.ISBN 978-1-4673-1228-8.S2CID 2161592. ^abKrizhevsky,Alex;Sutskever,Ilya;Hinton,Geoffry(2012)."ImageNetClassificationwithDeepConvolutionalNeuralNetworks"(PDF).NIPS2012:NeuralInformationProcessingSystems,LakeTahoe,Nevada.Archived(PDF)fromtheoriginalon2017-01-10.Retrieved2017-05-24. ^"Google'sAlphaGoAIwinsthree-matchseriesagainsttheworld'sbestGoplayer".TechCrunch.25May2017.Archivedfromtheoriginalon17June2018.Retrieved17June2018. ^Marblestone,AdamH.;Wayne,Greg;Kording,KonradP.(2016)."TowardanIntegrationofDeepLearningandNeuroscience".FrontiersinComputationalNeuroscience.10:94.arXiv:1606.03813.Bibcode:2016arXiv160603813M.doi:10.3389/fncom.2016.00094.PMC 5021692.PMID 27683554.S2CID 1994856. ^Bengio,Yoshua;Lee,Dong-Hyun;Bornschein,Jorg;Mesnard,Thomas;Lin,Zhouhan(13February2015)."TowardsBiologicallyPlausibleDeepLearning".arXiv:1502.04156[cs.LG]. ^abcdefDeng,L.;Yu,D.(2014)."DeepLearning:MethodsandApplications"(PDF).FoundationsandTrendsinSignalProcessing.7(3–4):1–199.doi:10.1561/2000000039.Archived(PDF)fromtheoriginalon2016-03-14.Retrieved2014-10-18. ^abcdeBengio,Yoshua(2009)."LearningDeepArchitecturesforAI"(PDF).FoundationsandTrendsinMachineLearning.2(1):1–127.CiteSeerX 10.1.1.701.9550.doi:10.1561/2200000006.Archivedfromtheoriginal(PDF)on4March2016.Retrieved3September2015. ^abcdeBengio,Y.;Courville,A.;Vincent,P.(2013)."RepresentationLearning:AReviewandNewPerspectives".IEEETransactionsonPatternAnalysisandMachineIntelligence.35(8):1798–1828.arXiv:1206.5538.doi:10.1109/tpami.2013.50.PMID 23787338.S2CID 393948. ^LeCun,Yann;Bengio,Yoshua;Hinton,Geoffrey(28May2015)."Deeplearning".Nature.521(7553):436–444.Bibcode:2015Natur.521..436L.doi:10.1038/nature14539.PMID 26017442.S2CID 3074096. ^abcdefgSchmidhuber,J.(2015)."DeepLearninginNeuralNetworks:AnOverview".NeuralNetworks.61:85–117.arXiv:1404.7828.doi:10.1016/j.neunet.2014.09.003.PMID 25462637.S2CID 11715509. ^Shigeki,Sugiyama(12April2019).HumanBehaviorandAnotherKindinConsciousness:EmergingResearchandOpportunities:EmergingResearchandOpportunities.IGIGlobal.ISBN 978-1-5225-8218-2. ^Bengio,Yoshua;Lamblin,Pascal;Popovici,Dan;Larochelle,Hugo(2007).Greedylayer-wisetrainingofdeepnetworks(PDF).Advancesinneuralinformationprocessingsystems.pp. 153–160.Archived(PDF)fromtheoriginalon2019-10-20.Retrieved2019-10-06. ^abHinton,G.E.(2009)."Deepbeliefnetworks".Scholarpedia.4(5):5947.Bibcode:2009SchpJ...4.5947H.doi:10.4249/scholarpedia.5947. ^abcCybenko(1989)."Approximationsbysuperpositionsofsigmoidalfunctions"(PDF).MathematicsofControl,Signals,andSystems.2(4):303–314.doi:10.1007/bf02551274.S2CID 3958369.Archivedfromtheoriginal(PDF)on10October2015. ^abcHornik,Kurt(1991)."ApproximationCapabilitiesofMultilayerFeedforwardNetworks".NeuralNetworks.4(2):251–257.doi:10.1016/0893-6080(91)90009-t. ^abHaykin,SimonS.(1999).NeuralNetworks:AComprehensiveFoundation.PrenticeHall.ISBN 978-0-13-273350-2. ^abHassoun,MohamadH.(1995).FundamentalsofArtificialNeuralNetworks.MITPress.p. 48.ISBN 978-0-262-08239-6. ^abLu,Z.,Pu,H.,Wang,F.,Hu,Z.,&Wang,L.(2017).TheExpressivePowerofNeuralNetworks:AViewfromtheWidthArchived2019-02-13attheWaybackMachine.NeuralInformationProcessingSystems,6231-6239. ^abcdMurphy,KevinP.(24August2012).MachineLearning:AProbabilisticPerspective.MITPress.ISBN 978-0-262-01802-9. ^Sonoda,Sho;Murata,Noboru(2017)."Neuralnetworkwithunboundedactivationfunctionsisuniversalapproximator".AppliedandComputationalHarmonicAnalysis.43(2):233–268.arXiv:1505.03654.doi:10.1016/j.acha.2015.12.005.S2CID 12149203. ^Hinton,G.E.;Srivastava,N.;Krizhevsky,A.;Sutskever,I.;Salakhutdinov,R.R.(2012)."Improvingneuralnetworksbypreventingco-adaptationoffeaturedetectors".arXiv:1207.0580[math.LG]. ^Bishop,ChristopherM.(2006).PatternRecognitionandMachineLearning(PDF).Springer.ISBN 978-0-387-31073-2.Archived(PDF)fromtheoriginalon2017-01-11.Retrieved2017-08-06. ^Tappert,CharlesC.(2019)."WhoIstheFatherofDeepLearning?".2019InternationalConferenceonComputationalScienceandComputationalIntelligence(CSCI).IEEE.pp. 343–348.doi:10.1109/CSCI49370.2019.00067.ISBN 978-1-7281-5584-5.S2CID 216043128.Retrieved31May2021. ^Ivakhnenko,A.G.;Lapa,V.G.(1967).CyberneticsandForecastingTechniques.AmericanElsevierPublishingCo.ISBN 978-0-444-00020-0. ^abIvakhnenko,Alexey(1971)."Polynomialtheoryofcomplexsystems"(PDF).IEEETransactionsonSystems,ManandCybernetics.SMC-1(4):364–378.doi:10.1109/TSMC.1971.4308320.Archived(PDF)fromtheoriginalon2017-08-29.Retrieved2019-11-05. ^Fukushima,K.(1980)."Neocognitron:Aself-organizingneuralnetworkmodelforamechanismofpatternrecognitionunaffectedbyshiftinposition".Biol.Cybern.36(4):193–202.doi:10.1007/bf00344251.PMID 7370364.S2CID 206775608. ^RinaDechter(1986).Learningwhilesearchinginconstraint-satisfactionproblems.UniversityofCalifornia,ComputerScienceDepartment,CognitiveSystemsLaboratory.OnlineArchived2016-04-19attheWaybackMachine ^IgorAizenberg,NaumN.Aizenberg,JoosP.L.Vandewalle(2000).Multi-ValuedandUniversalBinaryNeurons:Theory,LearningandApplications.SpringerScience&BusinessMedia. ^Co-evolvingrecurrentneuronslearndeepmemoryPOMDPs.Proc.GECCO,Washington,D.C.,pp.1795-1802,ACMPress,NewYork,NY,USA,2005. ^SeppoLinnainmaa(1970).TherepresentationofthecumulativeroundingerrorofanalgorithmasaTaylorexpansionofthelocalroundingerrors.Master'sThesis(inFinnish),Univ.Helsinki,6-7. ^Griewank,Andreas(2012)."WhoInventedtheReverseModeofDifferentiation?"(PDF).DocumentaMathematica(ExtraVolumeISMP):389–400.Archivedfromtheoriginal(PDF)on21July2017.Retrieved11June2017. ^Werbos,P.(1974)."BeyondRegression:NewToolsforPredictionandAnalysisintheBehavioralSciences".HarvardUniversity.Retrieved12June2017. ^Werbos,Paul(1982)."Applicationsofadvancesinnonlinearsensitivityanalysis"(PDF).Systemmodelingandoptimization.Springer.pp. 762–770. ^abLeCunetal.,"BackpropagationAppliedtoHandwrittenZipCodeRecognition,"NeuralComputation,1,pp.541–551,1989. ^deCarvalho,AndreC.L.F.;Fairhurst,MikeC.;Bisset,David(8August1994)."AnintegratedBooleanneuralnetworkforpatternclassification".PatternRecognitionLetters.15(8):807–813.Bibcode:1994PaReL..15..807D.doi:10.1016/0167-8655(94)90009-4. ^Hinton,GeoffreyE.;Dayan,Peter;Frey,BrendanJ.;Neal,Radford(26May1995)."Thewake-sleepalgorithmforunsupervisedneuralnetworks".Science.268(5214):1158–1161.Bibcode:1995Sci...268.1158H.doi:10.1126/science.7761831.PMID 7761831. ^abS.Hochreiter.,"UntersuchungenzudynamischenneuronalenNetzenArchived2015-03-06attheWaybackMachine,"Diplomathesis.Institutf.Informatik,TechnischeUniv.Munich.Advisor:J.Schmidhuber,1991. ^Hochreiter,S.;et al.(15January2001)."Gradientflowinrecurrentnets:thedifficultyoflearninglong-termdependencies".InKolen,JohnF.;Kremer,StefanC.(eds.).AFieldGuidetoDynamicalRecurrentNetworks.JohnWiley&Sons.ISBN 978-0-7803-5369-5. ^Behnke,Sven(2003)."HierarchicalNeuralNetworksforImageInterpretation".LectureNotesinComputerScience.2766.doi:10.1007/b11963.ISBN 978-3-540-40722-5.ISSN 0302-9743.S2CID 1304548.Archivedfromtheoriginalon2021-05-09.Retrieved2020-11-28. ^Morgan,Nelson;Bourlard,Hervé;Renals,Steve;Cohen,Michael;Franco,Horacio(1August1993)."Hybridneuralnetwork/hiddenmarkovmodelsystemsforcontinuousspeechrecognition".InternationalJournalofPatternRecognitionandArtificialIntelligence.07(4):899–916.doi:10.1142/s0218001493000455.ISSN 0218-0014. ^Robinson,T.(1992)."Areal-timerecurrenterrorpropagationnetworkwordrecognitionsystem".ICASSP.Icassp'92:617–620.ISBN 9780780305328.Archivedfromtheoriginalon2021-05-09.Retrieved2017-06-12. ^Waibel,A.;Hanazawa,T.;Hinton,G.;Shikano,K.;Lang,K.J.(March1989)."Phonemerecognitionusingtime-delayneuralnetworks"(PDF).IEEETransactionsonAcoustics,Speech,andSignalProcessing.37(3):328–339.doi:10.1109/29.21701.hdl:10338.dmlcz/135496.ISSN 0096-3518.Archived(PDF)fromtheoriginalon2021-04-27.Retrieved2019-09-24. ^Baker,J.;Deng,Li;Glass,Jim;Khudanpur,S.;Lee,C.-H.;Morgan,N.;O'Shaughnessy,D.(2009)."ResearchDevelopmentsandDirectionsinSpeechRecognitionandUnderstanding,Part1".IEEESignalProcessingMagazine.26(3):75–80.Bibcode:2009ISPM...26...75B.doi:10.1109/msp.2009.932166.hdl:1721.1/51891.S2CID 357467. ^Bengio,Y.(1991)."ArtificialNeuralNetworksandtheirApplicationtoSpeech/SequenceRecognition".McGillUniversityPh.D.thesis.Archivedfromtheoriginalon2021-05-09.Retrieved2017-06-12. ^Deng,L.;Hassanein,K.;Elmasry,M.(1994)."Analysisofcorrelationstructureforaneuralpredictivemodelwithapplicationstospeechrecognition".NeuralNetworks.7(2):331–339.doi:10.1016/0893-6080(94)90027-2. ^Doddington,G.;Przybocki,M.;Martin,A.;Reynolds,D.(2000)."TheNISTspeakerrecognitionevaluation±Overview,methodology,systems,results,perspective".SpeechCommunication.31(2):225–254.doi:10.1016/S0167-6393(99)00080-1. ^abHeck,L.;Konig,Y.;Sonmez,M.;Weintraub,M.(2000)."RobustnesstoTelephoneHandsetDistortioninSpeakerRecognitionbyDiscriminativeFeatureDesign".SpeechCommunication.31(2):181–192.doi:10.1016/s0167-6393(99)00077-1. ^"AcousticModelingwithDeepNeuralNetworksUsingRawTimeSignalforLVCSR(PDFDownloadAvailable)".ResearchGate.Archivedfromtheoriginalon9May2021.Retrieved14June2017. ^abcHochreiter,Sepp;Schmidhuber,Jürgen(1November1997)."LongShort-TermMemory".NeuralComputation.9(8):1735–1780.doi:10.1162/neco.1997.9.8.1735.ISSN 0899-7667.PMID 9377276.S2CID 1915014. ^abGraves,Alex;Eck,Douglas;Beringer,Nicole;Schmidhuber,Jürgen(2003)."BiologicallyPlausibleSpeechRecognitionwithLSTMNeuralNets"(PDF).1stIntl.WorkshoponBiologicallyInspiredApproachestoAdvancedInformationTechnology,Bio-ADIT2004,Lausanne,Switzerland.pp. 175–184.Archived(PDF)fromtheoriginalon2021-05-09.Retrieved2016-04-09. ^abGraves,Alex;Fernández,Santiago;Gomez,Faustino(2006)."Connectionisttemporalclassification:Labellingunsegmentedsequencedatawithrecurrentneuralnetworks".ProceedingsoftheInternationalConferenceonMachineLearning,ICML2006:369–376.CiteSeerX 10.1.1.75.6306. ^SantiagoFernandez,AlexGraves,andJürgenSchmidhuber(2007).AnapplicationofrecurrentneuralnetworkstodiscriminativekeywordspottingArchived2018-11-18attheWaybackMachine.ProceedingsofICANN(2),pp.220–229. ^abSak,Haşim;Senior,Andrew;Rao,Kanishka;Beaufays,Françoise;Schalkwyk,Johan(September2015)."Googlevoicesearch:fasterandmoreaccurate".Archivedfromtheoriginalon2016-03-09.Retrieved2016-04-09. ^Hinton,GeoffreyE.(1October2007)."Learningmultiplelayersofrepresentation".TrendsinCognitiveSciences.11(10):428–434.doi:10.1016/j.tics.2007.09.004.ISSN 1364-6613.PMID 17921042.S2CID 15066318.Archivedfromtheoriginalon11October2013.Retrieved12June2017. ^Hinton,G.E.;Osindero,S.;Teh,Y.W.(2006)."AFastLearningAlgorithmforDeepBeliefNets"(PDF).NeuralComputation.18(7):1527–1554.doi:10.1162/neco.2006.18.7.1527.PMID 16764513.S2CID 2309950.Archived(PDF)fromtheoriginalon2015-12-23.Retrieved2011-07-20. ^Bengio,Yoshua(2012)."Practicalrecommendationsforgradient-basedtrainingofdeeparchitectures".arXiv:1206.5533[cs.LG]. ^G.E.Hinton.,"LearningmultiplelayersofrepresentationArchived2018-05-22attheWaybackMachine,"TrendsinCognitiveSciences,11,pp.428–434,2007. ^abcHinton,G.;Deng,L.;Yu,D.;Dahl,G.;Mohamed,A.;Jaitly,N.;Senior,A.;Vanhoucke,V.;Nguyen,P.;Sainath,T.;Kingsbury,B.(2012)."DeepNeuralNetworksforAcousticModelinginSpeechRecognition:TheSharedViewsofFourResearchGroups".IEEESignalProcessingMagazine.29(6):82–97.Bibcode:2012ISPM...29...82H.doi:10.1109/msp.2012.2205597.S2CID 206485943. ^Deng,Li;Hinton,Geoffrey;Kingsbury,Brian(1May2013)."Newtypesofdeepneuralnetworklearningforspeechrecognitionandrelatedapplications:Anoverview".MicrosoftResearch.CiteSeerX 10.1.1.368.1123.Archivedfromtheoriginalon14February2017.Retrieved13February2017–viaresearch.microsoft.com. ^Deng,Li;Li,Jinyu;Huang,Jui-Ting;Yao,Kaisheng;Yu,Dong;Seide,Frank;Seltzer,Michael;Zweig,Geoff;He,Xiaodong;Williams,Jason;Gong,Yifan;Acero,Alex(2013)."RecentadvancesindeeplearningforspeechresearchatMicrosoft".2013IEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing.pp. 8604–8608.doi:10.1109/icassp.2013.6639345.ISBN 978-1-4799-0356-6.S2CID 13412186. ^Singh,Premjeet;Saha,Goutam;Sahidullah,Md(2021)."Non-linearfrequencywarpingusingconstant-Qtransformationforspeechemotionrecognition".2021InternationalConferenceonComputerCommunicationandInformatics(ICCCI).pp. 1–4.arXiv:2102.04029.doi:10.1109/ICCCI50826.2021.9402569.ISBN 978-1-7281-5875-4.S2CID 231846518. ^Sak,Hasim;Senior,Andrew;Beaufays,Francoise(2014)."LongShort-TermMemoryrecurrentneuralnetworkarchitecturesforlargescaleacousticmodeling"(PDF).Archivedfromtheoriginal(PDF)on24April2018. ^Li,Xiangang;Wu,Xihong(2014)."ConstructingLongShort-TermMemorybasedDeepRecurrentNeuralNetworksforLargeVocabularySpeechRecognition".arXiv:1410.4281[cs.CL]. ^Zen,Heiga;Sak,Hasim(2015)."UnidirectionalLongShort-TermMemoryRecurrentNeuralNetworkwithRecurrentOutputLayerforLow-LatencySpeechSynthesis"(PDF).Google.com.ICASSP.pp. 4470–4474.Archived(PDF)fromtheoriginalon2021-05-09.Retrieved2017-06-13. ^Deng,L.;Abdel-Hamid,O.;Yu,D.(2013)."Adeepconvolutionalneuralnetworkusingheterogeneouspoolingfortradingacousticinvariancewithphoneticconfusion"(PDF).Google.com.ICASSP.Archived(PDF)fromtheoriginalon2021-05-09.Retrieved2017-06-13. ^abSainath,TaraN.;Mohamed,Abdel-Rahman;Kingsbury,Brian;Ramabhadran,Bhuvana(2013)."DeepconvolutionalneuralnetworksforLVCSR".2013IEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing.pp. 8614–8618.doi:10.1109/icassp.2013.6639347.ISBN 978-1-4799-0356-6.S2CID 13816461. ^YannLeCun(2016).SlidesonDeepLearningOnlineArchived2016-04-23attheWaybackMachine ^abcNIPSWorkshop:DeepLearningforSpeechRecognitionandRelatedApplications,Whistler,BC,Canada,Dec.2009(Organizers:LiDeng,GeoffHinton,D.Yu). ^abKeynotetalk:RecentDevelopmentsinDeepNeuralNetworks.ICASSP,2013(byGeoffHinton). ^D.Yu,L.Deng,G.Li,andF.Seide(2011)."Discriminativepretrainingofdeepneuralnetworks,"U.S.PatentFiling. ^abcDeng,L.;Hinton,G.;Kingsbury,B.(2013)."Newtypesofdeepneuralnetworklearningforspeechrecognitionandrelatedapplications:Anoverview(ICASSP)"(PDF).Archived(PDF)fromtheoriginalon2017-09-26.Retrieved2017-06-12.{{citejournal}}:Citejournalrequires|journal=(help) ^abcYu,D.;Deng,L.(2014).AutomaticSpeechRecognition:ADeepLearningApproach(Publisher:Springer).ISBN 978-1-4471-5779-3. ^"DengreceivesprestigiousIEEETechnicalAchievementAward-MicrosoftResearch".MicrosoftResearch.3December2015.Archivedfromtheoriginalon16March2018.Retrieved16March2018. ^abLi,Deng(September2014)."Keynotetalk:'AchievementsandChallengesofDeepLearning-FromSpeechAnalysisandRecognitionToLanguageandMultimodalProcessing'".Interspeech.Archivedfromtheoriginalon2017-09-26.Retrieved2017-06-12. ^Yu,D.;Deng,L.(2010)."RolesofPre-TrainingandFine-TuninginContext-DependentDBN-HMMsforReal-WorldSpeechRecognition".NIPSWorkshoponDeepLearningandUnsupervisedFeatureLearning.Archivedfromtheoriginalon2017-10-12.Retrieved2017-06-14. ^Seide,F.;Li,G.;Yu,D.(2011)."Conversationalspeechtranscriptionusingcontext-dependentdeepneuralnetworks".Interspeech:437–440.doi:10.21437/Interspeech.2011-169.Archivedfromtheoriginalon2017-10-12.Retrieved2017-06-14. ^Deng,Li;Li,Jinyu;Huang,Jui-Ting;Yao,Kaisheng;Yu,Dong;Seide,Frank;Seltzer,Mike;Zweig,Geoff;He,Xiaodong(1May2013)."RecentAdvancesinDeepLearningforSpeechResearchatMicrosoft".MicrosoftResearch.Archivedfromtheoriginalon12October2017.Retrieved14June2017. ^"NvidiaCEObetsbigondeeplearningandVR".VentureBeat.5April2016.Archivedfromtheoriginalon25November2020.Retrieved21April2017. ^"Fromnotworkingtoneuralnetworking".TheEconomist.Archivedfromtheoriginalon2016-12-31.Retrieved2017-08-26. ^abOh,K.-S.;Jung,K.(2004)."GPUimplementationofneuralnetworks".PatternRecognition.37(6):1311–1314.Bibcode:2004PatRe..37.1311O.doi:10.1016/j.patcog.2004.01.013. ^"ASurveyofTechniquesforOptimizingDeepLearningonGPUsArchived2021-05-09attheWaybackMachine",S.MittalandS.Vaishay,JournalofSystemsArchitecture,2019 ^abChellapilla,Kumar;Puri,Sidd;Simard,Patrice(2006),Highperformanceconvolutionalneuralnetworksfordocumentprocessing,archivedfromtheoriginalon2020-05-18,retrieved2021-02-14 ^Cireşan,DanClaudiu;Meier,Ueli;Gambardella,LucaMaria;Schmidhuber,Jürgen(21September2010)."Deep,Big,SimpleNeuralNetsforHandwrittenDigitRecognition".NeuralComputation.22(12):3207–3220.arXiv:1003.0358.doi:10.1162/neco_a_00052.ISSN 0899-7667.PMID 20858131.S2CID 1918673. ^Raina,Rajat;Madhavan,Anand;Ng,AndrewY.(2009)."Large-scaleDeepUnsupervisedLearningUsingGraphicsProcessors".Proceedingsofthe26thAnnualInternationalConferenceonMachineLearning.ICML'09.NewYork,NY,USA:ACM:873–880.CiteSeerX 10.1.1.154.372.doi:10.1145/1553374.1553486.ISBN 9781605585161.S2CID 392458. ^Sze,Vivienne;Chen,Yu-Hsin;Yang,Tien-Ju;Emer,Joel(2017)."EfficientProcessingofDeepNeuralNetworks:ATutorialandSurvey".arXiv:1703.09039[cs.CV]. ^ab"MerckMolecularActivityChallenge".kaggle.com.Archivedfromtheoriginalon2020-07-16.Retrieved2020-07-16. ^ab"Multi-taskNeuralNetworksforQSARPredictions|DataScienceAssociation".www.datascienceassn.org.Archivedfromtheoriginalon30April2017.Retrieved14June2017. ^ab"Toxicologyinthe21stcenturyDataChallenge" ^ab"NCATSAnnouncesTox21DataChallengeWinners".Archivedfromtheoriginalon2015-09-08.Retrieved2015-03-05. ^ab"Archivedcopy".Archivedfromtheoriginalon28February2015.Retrieved5March2015.{{citeweb}}:CS1maint:archivedcopyastitle(link) ^Ciresan,D.C.;Meier,U.;Masci,J.;Gambardella,L.M.;Schmidhuber,J.(2011)."Flexible,HighPerformanceConvolutionalNeuralNetworksforImageClassification"(PDF).InternationalJointConferenceonArtificialIntelligence.doi:10.5591/978-1-57735-516-8/ijcai11-210.Archived(PDF)fromtheoriginalon2014-09-29.Retrieved2017-06-13. ^Ciresan,Dan;Giusti,Alessandro;Gambardella,LucaM.;Schmidhuber,Juergen(2012).Pereira,F.;Burges,C.J.C.;Bottou,L.;Weinberger,K.Q.(eds.).AdvancesinNeuralInformationProcessingSystems25(PDF).CurranAssociates,Inc.pp. 2843–2851.Archived(PDF)fromtheoriginalon2017-08-09.Retrieved2017-06-13. ^Ciresan,D.;Giusti,A.;Gambardella,L.M.;Schmidhuber,J.(2013)."MitosisDetectioninBreastCancerHistologyImagesusingDeepNeuralNetworks".ProceedingsMICCAI.LectureNotesinComputerScience.7908(Pt2):411–418.doi:10.1007/978-3-642-40763-5_51.ISBN 978-3-642-38708-1.PMID 24579167. ^Vinyals,Oriol;Toshev,Alexander;Bengio,Samy;Erhan,Dumitru(2014)."ShowandTell:ANeuralImageCaptionGenerator".arXiv:1411.4555[cs.CV].. ^Fang,Hao;Gupta,Saurabh;Iandola,Forrest;Srivastava,Rupesh;Deng,Li;Dollár,Piotr;Gao,Jianfeng;He,Xiaodong;Mitchell,Margaret;Platt,JohnC;LawrenceZitnick,C;Zweig,Geoffrey(2014)."FromCaptionstoVisualConceptsandBack".arXiv:1411.4952[cs.CV].. ^Kiros,Ryan;Salakhutdinov,Ruslan;Zemel,RichardS(2014)."UnifyingVisual-SemanticEmbeddingswithMultimodalNeuralLanguageModels".arXiv:1411.2539[cs.LG].. ^Zhong,Sheng-hua;Liu,Yan;Liu,Yang(2011)."BilinearDeepLearningforImageClassification".Proceedingsofthe19thACMInternationalConferenceonMultimedia.MM'11.NewYork,NY,USA:ACM:343–352.doi:10.1145/2072298.2072344.hdl:10397/23574.ISBN 9781450306164.S2CID 11922007. ^"WhyDeepLearningIsSuddenlyChangingYourLife".Fortune.2016.Archivedfromtheoriginalon14April2018.Retrieved13April2018. ^Silver,David;Huang,Aja;Maddison,ChrisJ.;Guez,Arthur;Sifre,Laurent;Driessche,Georgevanden;Schrittwieser,Julian;Antonoglou,Ioannis;Panneershelvam,Veda(January2016)."MasteringthegameofGowithdeepneuralnetworksandtreesearch".Nature.529(7587):484–489.Bibcode:2016Natur.529..484S.doi:10.1038/nature16961.ISSN 1476-4687.PMID 26819042.S2CID 515925. ^AGuidetoDeepLearningandNeuralNetworks,archivedfromtheoriginalon2020-11-02,retrieved2020-11-16 ^Szegedy,Christian;Toshev,Alexander;Erhan,Dumitru(2013)."Deepneuralnetworksforobjectdetection".AdvancesinNeuralInformationProcessingSystems:2553–2561.Archivedfromtheoriginalon2017-06-29.Retrieved2017-06-13. ^Rolnick,David;Tegmark,Max(2018)."Thepowerofdeepernetworksforexpressingnaturalfunctions".InternationalConferenceonLearningRepresentations.ICLR2018.Archivedfromtheoriginalon2021-01-07.Retrieved2021-01-05. ^Hof,RobertD."IsArtificialIntelligenceFinallyComingintoItsOwn?".MITTechnologyReview.Archivedfromtheoriginalon31March2019.Retrieved10July2018. ^abGers,FelixA.;Schmidhuber,Jürgen(2001)."LSTMRecurrentNetworksLearnSimpleContextFreeandContextSensitiveLanguages".IEEETransactionsonNeuralNetworks.12(6):1333–1340.doi:10.1109/72.963769.PMID 18249962.Archivedfromtheoriginalon2020-01-26.Retrieved2020-02-25. ^abcSutskever,L.;Vinyals,O.;Le,Q.(2014)."SequencetoSequenceLearningwithNeuralNetworks"(PDF).Proc.NIPS.arXiv:1409.3215.Bibcode:2014arXiv1409.3215S.Archived(PDF)fromtheoriginalon2021-05-09.Retrieved2017-06-13. ^abJozefowicz,Rafal;Vinyals,Oriol;Schuster,Mike;Shazeer,Noam;Wu,Yonghui(2016)."ExploringtheLimitsofLanguageModeling".arXiv:1602.02410[cs.CL]. ^abGillick,Dan;Brunk,Cliff;Vinyals,Oriol;Subramanya,Amarnag(2015)."MultilingualLanguageProcessingfromBytes".arXiv:1512.00103[cs.CL]. ^Mikolov,T.;et al.(2010)."Recurrentneuralnetworkbasedlanguagemodel"(PDF).Interspeech:1045–1048.doi:10.21437/Interspeech.2010-343.Archived(PDF)fromtheoriginalon2017-05-16.Retrieved2017-06-13. ^ab"LearningPreciseTimingwithLSTMRecurrentNetworks(PDFDownloadAvailable)".ResearchGate.Archivedfromtheoriginalon9May2021.Retrieved13June2017. ^LeCun,Y.;et al.(1998)."Gradient-basedlearningappliedtodocumentrecognition".ProceedingsoftheIEEE.86(11):2278–2324.doi:10.1109/5.726791. ^Bengio,Yoshua;Boulanger-Lewandowski,Nicolas;Pascanu,Razvan(2013)."Advancesinoptimizingrecurrentnetworks".2013IEEEInternationalConferenceonAcoustics,SpeechandSignalProcessing.pp. 8624–8628.arXiv:1212.0901.CiteSeerX 10.1.1.752.9151.doi:10.1109/icassp.2013.6639349.ISBN 978-1-4799-0356-6.S2CID 12485056. ^Dahl,G.;et al.(2013)."ImprovingDNNsforLVCSRusingrectifiedlinearunitsanddropout"(PDF).ICASSP.Archived(PDF)fromtheoriginalon2017-08-12.Retrieved2017-06-13. ^"DataAugmentation-deeplearning.ai|Coursera".Coursera.Archivedfromtheoriginalon1December2017.Retrieved30November2017. ^Hinton,G.E.(2010)."APracticalGuidetoTrainingRestrictedBoltzmannMachines".Tech.Rep.UTMLTR2010-003.Archivedfromtheoriginalon2021-05-09.Retrieved2017-06-13. ^You,Yang;Buluç,Aydın;Demmel,James(November2017)."ScalingdeeplearningonGPUandknightslandingclusters".ProceedingsoftheInternationalConferenceforHighPerformanceComputing,Networking,StorageandAnalysison-SC'17.SC'17,ACM.pp. 1–12.doi:10.1145/3126908.3126912.ISBN 9781450351140.S2CID 8869270.Archivedfromtheoriginalon29July2020.Retrieved5March2018. ^Viebke,André;Memeti,Suejb;Pllana,Sabri;Abraham,Ajith(2019)."CHAOS:aparallelizationschemefortrainingconvolutionalneuralnetworksonIntelXeonPhi".TheJournalofSupercomputing.75:197–227.arXiv:1702.07908.Bibcode:2017arXiv170207908V.doi:10.1007/s11227-017-1994-x.S2CID 14135321. ^TingQin,etal."AlearningalgorithmofCMACbasedonRLS."NeuralProcessingLetters19.1(2004):49-61. ^TingQin,etal."ContinuousCMAC-QRLSanditssystolicarrayArchived2018-11-18attheWaybackMachine."NeuralProcessingLetters22.1(2005):1-16. ^Research,AI(23October2015)."DeepNeuralNetworksforAcousticModelinginSpeechRecognition".airesearch.com.Archivedfromtheoriginalon1February2016.Retrieved23October2015. ^"GPUsContinuetoDominatetheAIAcceleratorMarketforNow".InformationWeek.December2019.Archivedfromtheoriginalon10June2020.Retrieved11June2020. ^Ray,Tiernan(2019)."AIischangingtheentirenatureofcomputation".ZDNet.Archivedfromtheoriginalon25May2020.Retrieved11June2020. ^"AIandCompute".OpenAI.16May2018.Archivedfromtheoriginalon17June2020.Retrieved11June2020. ^TIMITAcoustic-PhoneticContinuousSpeechCorpusLinguisticDataConsortium,Philadelphia. ^Robinson,Tony(30September1991)."SeveralImprovementstoaRecurrentErrorPropagationNetworkPhoneRecognitionSystem".CambridgeUniversityEngineeringDepartmentTechnicalReport.CUED/F-INFENG/TR82.doi:10.13140/RG.2.2.15418.90567. ^Abdel-Hamid,O.;et al.(2014)."ConvolutionalNeuralNetworksforSpeechRecognition".IEEE/ACMTransactionsonAudio,Speech,andLanguageProcessing.22(10):1533–1545.doi:10.1109/taslp.2014.2339736.S2CID 206602362.Archivedfromtheoriginalon2020-09-22.Retrieved2018-04-20. ^Deng,L.;Platt,J.(2014)."EnsembleDeepLearningforSpeechRecognition".Proc.Interspeech.S2CID 15641618. ^Tóth,Laszló(2015)."PhoneRecognitionwithHierarchicalConvolutionalDeepMaxoutNetworks"(PDF).EURASIPJournalonAudio,Speech,andMusicProcessing.2015.doi:10.1186/s13636-015-0068-3.S2CID 217950236.Archived(PDF)fromtheoriginalon2020-09-24.Retrieved2019-04-01. ^McMillan,Robert(17December2014)."HowSkypeUsedAItoBuildItsAmazingNewLanguageTranslator|WIRED".Wired.Archivedfromtheoriginalon8June2017.Retrieved14June2017. ^Hannun,Awni;Case,Carl;Casper,Jared;Catanzaro,Bryan;Diamos,Greg;Elsen,Erich;Prenger,Ryan;Satheesh,Sanjeev;Sengupta,Shubho;Coates,Adam;Ng,AndrewY(2014)."DeepSpeech:Scalingupend-to-endspeechrecognition".arXiv:1412.5567[cs.CL]. ^"MNISThandwrittendigitdatabase,YannLeCun,CorinnaCortesandChrisBurges".yann.lecun.com.Archivedfromtheoriginalon2014-01-13.Retrieved2014-01-28. ^Cireşan,Dan;Meier,Ueli;Masci,Jonathan;Schmidhuber,Jürgen(August2012)."Multi-columndeepneuralnetworkfortrafficsignclassification".NeuralNetworks.SelectedPapersfromIJCNN2011.32:333–338.CiteSeerX 10.1.1.226.8219.doi:10.1016/j.neunet.2012.02.023.PMID 22386783. ^NvidiaDemosaCarComputerTrainedwith"DeepLearning"(6January2015),DavidTalbot,MITTechnologyReview ^G.W.Smith;FredericFolLeymarie(10April2017)."TheMachineasArtist:AnIntroduction".Arts.6(4):5.doi:10.3390/arts6020005. ^BlaiseAgüerayArcas(29September2017)."ArtintheAgeofMachineIntelligence".Arts.6(4):18.doi:10.3390/arts6040018. ^Goldberg,Yoav;Levy,Omar(2014)."word2vecExplained:DerivingMikolovetal.'sNegative-SamplingWord-EmbeddingMethod".arXiv:1402.3722[cs.CL]. ^abSocher,Richard;Manning,Christopher."DeepLearningforNLP"(PDF).Archived(PDF)fromtheoriginalon6July2014.Retrieved26October2014. ^Socher,Richard;Bauer,John;Manning,Christopher;Ng,Andrew(2013)."ParsingWithCompositionalVectorGrammars"(PDF).ProceedingsoftheACL2013Conference.Archived(PDF)fromtheoriginalon2014-11-27.Retrieved2014-09-03. ^Socher,Richard(2013)."RecursiveDeepModelsforSemanticCompositionalityOveraSentimentTreebank"(PDF).Archived(PDF)fromtheoriginalon2016-12-28.Retrieved2014-09-03.{{citejournal}}:Citejournalrequires|journal=(help) ^Shen,Yelong;He,Xiaodong;Gao,Jianfeng;Deng,Li;Mesnil,Gregoire(1November2014)."ALatentSemanticModelwithConvolutional-PoolingStructureforInformationRetrieval".MicrosoftResearch.Archivedfromtheoriginalon27October2017.Retrieved14June2017. ^Huang,Po-Sen;He,Xiaodong;Gao,Jianfeng;Deng,Li;Acero,Alex;Heck,Larry(1October2013)."LearningDeepStructuredSemanticModelsforWebSearchusingClickthroughData".MicrosoftResearch.Archivedfromtheoriginalon27October2017.Retrieved14June2017. ^Mesnil,G.;Dauphin,Y.;Yao,K.;Bengio,Y.;Deng,L.;Hakkani-Tur,D.;He,X.;Heck,L.;Tur,G.;Yu,D.;Zweig,G.(2015)."Usingrecurrentneuralnetworksforslotfillinginspokenlanguageunderstanding".IEEETransactionsonAudio,Speech,andLanguageProcessing.23(3):530–539.doi:10.1109/taslp.2014.2383614.S2CID 1317136. ^abGao,Jianfeng;He,Xiaodong;Yih,ScottWen-tau;Deng,Li(1June2014)."LearningContinuousPhraseRepresentationsforTranslationModeling".MicrosoftResearch.Archivedfromtheoriginalon27October2017.Retrieved14June2017. ^Brocardo,MarceloLuiz;Traore,Issa;Woungang,Isaac;Obaidat,MohammadS.(2017)."Authorshipverificationusingdeepbeliefnetworksystems".InternationalJournalofCommunicationSystems.30(12):e3259.doi:10.1002/dac.3259. ^"DeepLearningforNaturalLanguageProcessing:TheoryandPractice(CIKM2014Tutorial)-MicrosoftResearch".MicrosoftResearch.Archivedfromtheoriginalon13March2017.Retrieved14June2017. ^Turovsky,Barak(15November2016)."Foundintranslation:Moreaccurate,fluentsentencesinGoogleTranslate".TheKeywordGoogleBlog.Archivedfromtheoriginalon7April2017.Retrieved23March2017. ^abcdSchuster,Mike;Johnson,Melvin;Thorat,Nikhil(22November2016)."Zero-ShotTranslationwithGoogle'sMultilingualNeuralMachineTranslationSystem".GoogleResearchBlog.Archivedfromtheoriginalon10July2017.Retrieved23March2017. ^SeppHochreiter;JürgenSchmidhuber(1997)."Longshort-termmemory".NeuralComputation.9(8):1735–1780.doi:10.1162/neco.1997.9.8.1735.PMID 9377276.S2CID 1915014.Archivedfromtheoriginalon2021-01-22.Retrieved2017-10-12. ^FelixA.Gers;JürgenSchmidhuber;FredCummins(2000)."LearningtoForget:ContinualPredictionwithLSTM".NeuralComputation.12(10):2451–2471.CiteSeerX 10.1.1.55.5709.doi:10.1162/089976600300015015.PMID 11032042.S2CID 11598600. ^Wu,Yonghui;Schuster,Mike;Chen,Zhifeng;Le,QuocV;Norouzi,Mohammad;Macherey,Wolfgang;Krikun,Maxim;Cao,Yuan;Gao,Qin;Macherey,Klaus;Klingner,Jeff;Shah,Apurva;Johnson,Melvin;Liu,Xiaobing;Kaiser,Łukasz;Gouws,Stephan;Kato,Yoshikiyo;Kudo,Taku;Kazawa,Hideto;Stevens,Keith;Kurian,George;Patil,Nishant;Wang,Wei;Young,Cliff;Smith,Jason;Riesa,Jason;Rudnick,Alex;Vinyals,Oriol;Corrado,Greg;et al.(2016)."Google'sNeuralMachineTranslationSystem:BridgingtheGapbetweenHumanandMachineTranslation".arXiv:1609.08144[cs.CL]. ^Metz,Cade(27September2016)."AnInfusionofAIMakesGoogleTranslateMorePowerfulThanEver".Wired.Archivedfromtheoriginalon8November2020.Retrieved12October2017. ^abBoitet,Christian;Blanchon,Hervé;Seligman,Mark;Bellynck,Valérie(2010)."MTonandfortheWeb"(PDF).Archivedfromtheoriginal(PDF)on29March2017.Retrieved1December2016. ^Arrowsmith,J;Miller,P(2013)."Trialwatch:PhaseIIandphaseIIIattritionrates2011-2012".NatureReviewsDrugDiscovery.12(8):569.doi:10.1038/nrd4090.PMID 23903212.S2CID 20246434. ^Verbist,B;Klambauer,G;Vervoort,L;Talloen,W;TheQstar,Consortium;Shkedy,Z;Thas,O;Bender,A;Göhlmann,H.W.;Hochreiter,S(2015)."Usingtranscriptomicstoguideleadoptimizationindrugdiscoveryprojects:LessonslearnedfromtheQSTARproject".DrugDiscoveryToday.20(5):505–513.doi:10.1016/j.drudis.2014.12.014.PMID 25582842. ^Wallach,Izhar;Dzamba,Michael;Heifets,Abraham(9October2015)."AtomNet:ADeepConvolutionalNeuralNetworkforBioactivityPredictioninStructure-basedDrugDiscovery".arXiv:1510.02855[cs.LG]. ^"Torontostartuphasafasterwaytodiscovereffectivemedicines".TheGlobeandMail.Archivedfromtheoriginalon20October2015.Retrieved9November2015. ^"StartupHarnessesSupercomputerstoSeekCures".KQEDFutureofYou.Archivedfromtheoriginalon24December2015.Retrieved9November2015. ^"Torontostartuphasafasterwaytodiscovereffectivemedicines".TheGlobeandMail.Archivedfromtheoriginalon2015-12-25.Retrieved2017-08-26. ^Gilmer,Justin;Schoenholz,SamuelS.;Riley,PatrickF.;Vinyals,Oriol;Dahl,GeorgeE.(2017-06-12)."NeuralMessagePassingforQuantumChemistry".arXiv:1704.01212[cs.LG]. ^Zhavoronkov,Alex(2019)."DeeplearningenablesrapididentificationofpotentDDR1kinaseinhibitors".NatureBiotechnology.37(9):1038–1040.doi:10.1038/s41587-019-0224-x.PMID 31477924.S2CID 201716327. ^Gregory,Barber."AMoleculeDesignedByAIExhibits'Druglike'Qualities".Wired.Archivedfromtheoriginalon2020-04-30.Retrieved2019-09-05. ^Tkachenko,Yegor(8April2015)."AutonomousCRMControlviaCLVApproximationwithDeepReinforcementLearninginDiscreteandContinuousActionSpace".arXiv:1504.01840[cs.LG]. ^vandenOord,Aaron;Dieleman,Sander;Schrauwen,Benjamin(2013).Burges,C.J.C.;Bottou,L.;Welling,M.;Ghahramani,Z.;Weinberger,K.Q.(eds.).AdvancesinNeuralInformationProcessingSystems26(PDF).CurranAssociates,Inc.pp. 2643–2651.Archived(PDF)fromtheoriginalon2017-05-16.Retrieved2017-06-14. ^Feng,X.Y.;Zhang,H.;Ren,Y.J.;Shang,P.H.;Zhu,Y.;Liang,Y.C.;Guan,R.C.;Xu,D.(2019)."TheDeepLearning–BasedRecommenderSystem"Pubmender"forChoosingaBiomedicalPublicationVenue:DevelopmentandValidationStudy".JournalofMedicalInternetResearch.21(5):e12957.doi:10.2196/12957.PMC 6555124.PMID 31127715. ^Elkahky,AliMamdouh;Song,Yang;He,Xiaodong(1May2015)."AMulti-ViewDeepLearningApproachforCrossDomainUserModelinginRecommendationSystems".MicrosoftResearch.Archivedfromtheoriginalon25January2018.Retrieved14June2017. ^Chicco,Davide;Sadowski,Peter;Baldi,Pierre(1January2014).DeepAutoencoderNeuralNetworksforGeneOntologyAnnotationPredictions.Proceedingsofthe5thACMConferenceonBioinformatics,ComputationalBiology,andHealthInformatics-BCB'14.ACM.pp. 533–540.doi:10.1145/2649387.2649442.hdl:11311/964622.ISBN 9781450328944.S2CID 207217210.Archivedfromtheoriginalon9May2021.Retrieved23November2015. ^Sathyanarayana,Aarti(1January2016)."SleepQualityPredictionFromWearableDataUsingDeepLearning".JMIRmHealthanduHealth.4(4):e125.doi:10.2196/mhealth.6562.PMC 5116102.PMID 27815231.S2CID 3821594. ^Choi,Edward;Schuetz,Andy;Stewart,WalterF.;Sun,Jimeng(13August2016)."Usingrecurrentneuralnetworkmodelsforearlydetectionofheartfailureonset".JournaloftheAmericanMedicalInformaticsAssociation.24(2):361–370.doi:10.1093/jamia/ocw112.ISSN 1067-5027.PMC 5391725.PMID 27521897. ^Litjens,Geert;Kooi,Thijs;Bejnordi,BabakEhteshami;Setio,ArnaudArindraAdiyoso;Ciompi,Francesco;Ghafoorian,Mohsen;vanderLaak,JeroenA.W.M.;vanGinneken,Bram;Sánchez,ClaraI.(December2017)."Asurveyondeeplearninginmedicalimageanalysis".MedicalImageAnalysis.42:60–88.arXiv:1702.05747.Bibcode:2017arXiv170205747L.doi:10.1016/j.media.2017.07.005.PMID 28778026.S2CID 2088679. ^Forslid,Gustav;Wieslander,Hakan;Bengtsson,Ewert;Wahlby,Carolina;Hirsch,Jan-Michael;Stark,ChristinaRunow;Sadanandan,SajithKecheril(2017)."DeepConvolutionalNeuralNetworksforDetectingCellularChangesDuetoMalignancy".2017IEEEInternationalConferenceonComputerVisionWorkshops(ICCVW).pp. 82–89.doi:10.1109/ICCVW.2017.18.ISBN 9781538610343.S2CID 4728736.Archivedfromtheoriginalon2021-05-09.Retrieved2019-11-12. ^De,Shaunak;Maity,Abhishek;Goel,Vritti;Shitole,Sanjay;Bhattacharya,Avik(2017)."Predictingthepopularityofinstagrampostsforalifestylemagazineusingdeeplearning".20172ndInternationalConferenceonCommunicationSystems,ComputingandITApplications(CSCITA).pp. 174–177.doi:10.1109/CSCITA.2017.8066548.ISBN 978-1-5090-4381-1.S2CID 35350962. ^"ColorizingandRestoringOldImageswithDeepLearning".FloydHubBlog.13November2018.Archivedfromtheoriginalon11October2019.Retrieved11October2019. ^Schmidt,Uwe;Roth,Stefan.ShrinkageFieldsforEffectiveImageRestoration(PDF).ComputerVisionandPatternRecognition(CVPR),2014IEEEConferenceon.Archived(PDF)fromtheoriginalon2018-01-02.Retrieved2018-01-01. ^Kleanthous,Christos;Chatzis,Sotirios(2020)."GatedMixtureVariationalAutoencodersforValueAddedTaxauditcaseselection".Knowledge-BasedSystems.188:105048.doi:10.1016/j.knosys.2019.105048.S2CID 204092079. ^Czech,Tomasz."Deeplearning:thenextfrontierformoneylaunderingdetection".GlobalBankingandFinanceReview.Archivedfromtheoriginalon2018-11-16.Retrieved2018-07-15. ^abc"Armyresearchersdevelopnewalgorithmstotrainrobots".EurekAlert!.Archivedfromtheoriginalon28August2018.Retrieved29August2018. ^Raissi,M.;Perdikaris,P.;Karniadakis,G.E.(2019-02-01)."Physics-informedneuralnetworks:Adeeplearningframeworkforsolvingforwardandinverseproblemsinvolvingnonlinearpartialdifferentialequations".JournalofComputationalPhysics.378:686–707.Bibcode:2019JCoPh.378..686R.doi:10.1016/j.jcp.2018.10.045.ISSN 0021-9991.OSTI 1595805.S2CID 57379996. ^Mao,Zhiping;Jagtap,AmeyaD.;Karniadakis,GeorgeEm(2020-03-01)."Physics-informedneuralnetworksforhigh-speedflows".ComputerMethodsinAppliedMechanicsandEngineering.360:112789.Bibcode:2020CMAME.360k2789M.doi:10.1016/j.cma.2019.112789.ISSN 0045-7825.S2CID 212755458. ^Raissi,Maziar;Yazdani,Alireza;Karniadakis,GeorgeEm(2020-02-28)."Hiddenfluidmechanics:Learningvelocityandpressurefieldsfromflowvisualizations".Science.367(6481):1026–1030.Bibcode:2020Sci...367.1026R.doi:10.1126/science.aaw4741.PMC 7219083.PMID 32001523. ^Utgoff,P.E.;Stracuzzi,D.J.(2002)."Many-layeredlearning".NeuralComputation.14(10):2497–2529.doi:10.1162/08997660260293319.PMID 12396572.S2CID 1119517. ^Elman,JeffreyL.(1998).RethinkingInnateness:AConnectionistPerspectiveonDevelopment.MITPress.ISBN 978-0-262-55030-7. ^Shrager,J.;Johnson,MH(1996)."Dynamicplasticityinfluencestheemergenceoffunctioninasimplecorticalarray".NeuralNetworks.9(7):1119–1129.doi:10.1016/0893-6080(96)00033-0.PMID 12662587. ^Quartz,SR;Sejnowski,TJ(1997)."Theneuralbasisofcognitivedevelopment:Aconstructivistmanifesto".BehavioralandBrainSciences.20(4):537–556.CiteSeerX 10.1.1.41.7854.doi:10.1017/s0140525x97001581.PMID 10097006.S2CID 5818342. ^S.Blakeslee.,"Inbrain'searlygrowth,timetablemaybecritical,"TheNewYorkTimes,ScienceSection,pp.B5–B6,1995. ^Mazzoni,P.;Andersen,R.A.;Jordan,M.I.(15May1991)."Amorebiologicallyplausiblelearningruleforneuralnetworks".ProceedingsoftheNationalAcademyofSciences.88(10):4433–4437.Bibcode:1991PNAS...88.4433M.doi:10.1073/pnas.88.10.4433.ISSN 0027-8424.PMC 51674.PMID 1903542. ^O'Reilly,RandallC.(1July1996)."BiologicallyPlausibleError-DrivenLearningUsingLocalActivationDifferences:TheGeneralizedRecirculationAlgorithm".NeuralComputation.8(5):895–938.doi:10.1162/neco.1996.8.5.895.ISSN 0899-7667.S2CID 2376781. ^Testolin,Alberto;Zorzi,Marco(2016)."ProbabilisticModelsandGenerativeNeuralNetworks:TowardsanUnifiedFrameworkforModelingNormalandImpairedNeurocognitiveFunctions".FrontiersinComputationalNeuroscience.10:73.doi:10.3389/fncom.2016.00073.ISSN 1662-5188.PMC 4943066.PMID 27468262.S2CID 9868901. ^Testolin,Alberto;Stoianov,Ivilin;Zorzi,Marco(September2017)."Letterperceptionemergesfromunsuperviseddeeplearningandrecyclingofnaturalimagefeatures".NatureHumanBehaviour.1(9):657–664.doi:10.1038/s41562-017-0186-2.ISSN 2397-3374.PMID 31024135.S2CID 24504018. ^Buesing,Lars;Bill,Johannes;Nessler,Bernhard;Maass,Wolfgang(3November2011)."NeuralDynamicsasSampling:AModelforStochasticComputationinRecurrentNetworksofSpikingNeurons".PLOSComputationalBiology.7(11):e1002211.Bibcode:2011PLSCB...7E2211B.doi:10.1371/journal.pcbi.1002211.ISSN 1553-7358.PMC 3207943.PMID 22096452.S2CID 7504633. ^Morel,Danielle;Singh,Chandan;Levy,WilliamB.(25January2018)."Linearizationofexcitatorysynapticintegrationatnoextracost".JournalofComputationalNeuroscience.44(2):173–188.doi:10.1007/s10827-017-0673-5.ISSN 0929-5313.PMID 29372434.S2CID 3831587. ^Cash,S.;Yuste,R.(February1999)."LinearsummationofexcitatoryinputsbyCA1pyramidalneurons".Neuron.22(2):383–394.doi:10.1016/s0896-6273(00)81098-3.ISSN 0896-6273.PMID 10069343.S2CID 14663106. ^Olshausen,B;Field,D(1August2004)."Sparsecodingofsensoryinputs".CurrentOpinioninNeurobiology.14(4):481–487.doi:10.1016/j.conb.2004.07.007.ISSN 0959-4388.PMID 15321069.S2CID 16560320. ^Yamins,DanielLK;DiCarlo,JamesJ(March2016)."Usinggoal-drivendeeplearningmodelstounderstandsensorycortex".NatureNeuroscience.19(3):356–365.doi:10.1038/nn.4244.ISSN 1546-1726.PMID 26906502.S2CID 16970545. ^Zorzi,Marco;Testolin,Alberto(19February2018)."Anemergentistperspectiveontheoriginofnumbersense".Phil.Trans.R.Soc.B.373(1740):20170043.doi:10.1098/rstb.2017.0043.ISSN 0962-8436.PMC 5784047.PMID 29292348.S2CID 39281431. ^Güçlü,Umut;vanGerven,MarcelA.J.(8July2015)."DeepNeuralNetworksRevealaGradientintheComplexityofNeuralRepresentationsacrosstheVentralStream".JournalofNeuroscience.35(27):10005–10014.arXiv:1411.6422.doi:10.1523/jneurosci.5023-14.2015.PMC 6605414.PMID 26157000. ^Metz,C.(12December2013)."Facebook's'DeepLearning'GuruRevealstheFutureofAI".Wired.Archivedfromtheoriginalon28March2014.Retrieved26August2017. ^Gibney,Elizabeth(2016)."GoogleAIalgorithmmastersancientgameofGo".Nature.529(7587):445–446.Bibcode:2016Natur.529..445G.doi:10.1038/529445a.PMID 26819021.S2CID 4460235.Archivedfromtheoriginalon2May2019.Retrieved30January2016. ^Silver,David;Huang,Aja;Maddison,ChrisJ.;Guez,Arthur;Sifre,Laurent;Driessche,Georgevanden;Schrittwieser,Julian;Antonoglou,Ioannis;Panneershelvam,Veda;Lanctot,Marc;Dieleman,Sander;Grewe,Dominik;Nham,John;Kalchbrenner,Nal;Sutskever,Ilya;Lillicrap,Timothy;Leach,Madeleine;Kavukcuoglu,Koray;Graepel,Thore;Hassabis,Demis(28January2016)."MasteringthegameofGowithdeepneuralnetworksandtreesearch".Nature.529(7587):484–489.Bibcode:2016Natur.529..484S.doi:10.1038/nature16961.ISSN 0028-0836.PMID 26819042.S2CID 515925. ^"AGoogleDeepMindAlgorithmUsesDeepLearningandMoretoMastertheGameofGo|MITTechnologyReview".MITTechnologyReview.Retrieved30January2016. ^"BlipparDemonstratesNewReal-TimeAugmentedRealityApp".TechCrunch.Archivedfromtheoriginalon2017-07-05.Retrieved2017-06-25. ^Metz,Cade(6November2017)."A.I.ResearchersLeaveElonMuskLabtoBeginRoboticsStart-Up".TheNewYorkTimes.Archivedfromtheoriginalon7July2019.Retrieved5July2019. ^BradleyKnox,W.;Stone,Peter(2008)."TAMER:TraininganAgentManuallyviaEvaluativeReinforcement".20087thIEEEInternationalConferenceonDevelopmentandLearning:292–297.doi:10.1109/devlrn.2008.4640845.ISBN 978-1-4244-2661-4.S2CID 5613334. ^"TalktotheAlgorithms:AIBecomesaFasterLearner".governmentciomedia.com.Archivedfromtheoriginalon28August2018.Retrieved29August2018. ^Marcus,Gary(14January2018)."Indefenseofskepticismaboutdeeplearning".GaryMarcus.Archivedfromtheoriginalon12October2018.Retrieved11October2018. ^Knight,Will(14March2017)."DARPAisfundingprojectsthatwilltrytoopenupAI'sblackboxes".MITTechnologyReview.Archivedfromtheoriginalon4November2019.Retrieved2November2017. ^Marcus,Gary(November25,2012)."Is"DeepLearning"aRevolutioninArtificialIntelligence?".TheNewYorker.Archivedfromtheoriginalon2009-11-27.Retrieved2017-06-14. ^AlexanderMordvintsev;ChristopherOlah;MikeTyka(17June2015)."Inceptionism:GoingDeeperintoNeuralNetworks".GoogleResearchBlog.Archivedfromtheoriginalon3July2015.Retrieved20June2015. ^AlexHern(18June2015)."Yes,androidsdodreamofelectricsheep".TheGuardian.Archivedfromtheoriginalon19June2015.Retrieved20June2015. ^abcGoertzel,Ben(2015)."ArethereDeepReasonsUnderlyingthePathologiesofToday'sDeepLearningAlgorithms?"(PDF).Archived(PDF)fromtheoriginalon2015-05-13.Retrieved2015-05-10. ^Nguyen,Anh;Yosinski,Jason;Clune,Jeff(2014)."DeepNeuralNetworksareEasilyFooled:HighConfidencePredictionsforUnrecognizableImages".arXiv:1412.1897[cs.CV]. ^Szegedy,Christian;Zaremba,Wojciech;Sutskever,Ilya;Bruna,Joan;Erhan,Dumitru;Goodfellow,Ian;Fergus,Rob(2013)."Intriguingpropertiesofneuralnetworks".arXiv:1312.6199[cs.CV]. ^Zhu,S.C.;Mumford,D.(2006)."Astochasticgrammarofimages".Found.TrendsComput.Graph.Vis.2(4):259–362.CiteSeerX 10.1.1.681.2190.doi:10.1561/0600000018. ^Miller,G.A.,andN.Chomsky."Patternconception."PaperforConferenceonpatterndetection,UniversityofMichigan.1957. ^Eisner,Jason."DeepLearningofRecursiveStructure:GrammarInduction".Archivedfromtheoriginalon2017-12-30.Retrieved2015-05-10. ^"HackersHaveAlreadyStartedtoWeaponizeArtificialIntelligence".Gizmodo.Archivedfromtheoriginalon11October2019.Retrieved11October2019. ^"HowhackerscanforceAItomakedumbmistakes".TheDailyDot.18June2018.Archivedfromtheoriginalon11October2019.Retrieved11October2019. ^abcde"AIIsEasytoFool—WhyThatNeedstoChange".SingularityHub.10October2017.Archivedfromtheoriginalon11October2017.Retrieved11October2017. ^Gibney,Elizabeth(2017)."Thescientistwhospotsfakevideos".Nature.doi:10.1038/nature.2017.22784.Archivedfromtheoriginalon2017-10-10.Retrieved2017-10-11. ^abcdMühlhoff,Rainer(6November2019)."Human-aidedartificialintelligence:Or,howtorunlargecomputationsinhumanbrains?Towardamediasociologyofmachinelearning".NewMedia&Society.22(10):1868–1884.doi:10.1177/1461444819885334.ISSN 1461-4448.S2CID 209363848. ^"FacebookCanNowFindYourFace,EvenWhenIt'sNotTagged".Wired.ISSN 1059-1028.Archivedfromtheoriginalon10August2019.Retrieved22November2019. Furtherreading[edit] Goodfellow,Ian;Bengio,Yoshua;Courville,Aaron(2016).DeepLearning.MITPress.ISBN 978-0-26203561-3.Archivedfromtheoriginalon2016-04-16.Retrieved2021-05-09,introductorytextbook.{{citebook}}:CS1maint:postscript(link) vteDifferentiablecomputingGeneral Differentiableprogramming NeuralTuringmachine Differentiableneuralcomputer Automaticdifferentiation Neuromorphicengineering Cabletheory Patternrecognition Computationallearningtheory Tensorcalculus Concepts Gradientdescent SGD Clustering Regression Overfitting Adversary Attention Convolution Lossfunctions Backpropagation Normalization Activation Softmax Sigmoid Rectifier Regularization Datasets Augmentation Programminglanguages Python Julia Application Machinelearning Artificialneuralnetwork Deeplearning Scientificcomputing ArtificialIntelligence Hardware IPU TPU VPU Memristor SpiNNaker Softwarelibrary TensorFlow PyTorch Keras Theano ImplementationAudio-visual AlexNet WaveNet Humanimagesynthesis HWR OCR Speechsynthesis Speechrecognition Facialrecognition AlphaFold DALL-E Verbal Word2vec Transformer BERT NMT ProjectDebater Watson GPT-2 GPT-3 Decisional AlphaGo AlphaZero Q-learning SARSA OpenAIFive Self-drivingcar MuZero Actionselection Robotcontrol People AlexGraves IanGoodfellow YoshuaBengio GeoffreyHinton YannLeCun AndrewNg DemisHassabis DavidSilver Fei-FeiLi Organizations DeepMind OpenAI MITCSAIL Mila GoogleBrain FAIR Portals Computerprogramming Technology Category Artificialneuralnetworks Machinelearning Retrievedfrom"https://en.wikipedia.org/w/index.php?title=Deep_learning&oldid=1075970859" Categories:DeeplearningArtificialneuralnetworksArtificialintelligenceEmergingtechnologiesHiddencategories:WebarchivetemplatewaybacklinksCS1:longvolumevalueCS1errors:missingperiodicalCS1maint:archivedcopyastitleArticleswithshortdescriptionShortdescriptionmatchesWikidataWikipediaarticlesthataretootechnicalfromJuly2016AllarticlesthataretootechnicalAllarticleswithunsourcedstatementsArticleswithunsourcedstatementsfromNovember2020ArticleswithunsourcedstatementsfromJuly2016ArticlesneedingadditionalreferencesfromApril2021AllarticlesneedingadditionalreferencesCS1maint:postscriptArticlespronetospamfromJune2015 Navigationmenu Personaltools NotloggedinTalkContributionsCreateaccountLogin Namespaces ArticleTalk English expanded collapsed Views ReadEditViewhistory More expanded collapsed Search Navigation MainpageContentsCurrenteventsRandomarticleAboutWikipediaContactusDonate Contribute HelpLearntoeditCommunityportalRecentchangesUploadfile Tools WhatlinkshereRelatedchangesUploadfileSpecialpagesPermanentlinkPageinformationCitethispageWikidataitem Print/export DownloadasPDFPrintableversion Inotherprojects WikimediaCommons Languages العربيةবাংলাBân-lâm-gúБългарскиCatalàČeštinaDanskDeutschEestiEspañolEuskaraفارسیFrançais한국어ՀայերենBahasaIndonesiaItalianoעבריתമലയാളംBahasaMelayuМонголNederlands日本語NorskbokmålOccitanPolskiPortuguêsRomânăРусскийShqipSimpleEnglishSlovenščinaکوردیСрпски/srpskiSuomiSvenskaதமிழ்ไทยTürkçeУкраїнськаTiếngViệt粵語中文 Editlinks
延伸文章資訊
- 1Neural Networks - What are they and why do they matter? | SAS
- 2Types of artificial neural networks - Wikipedia
It uses a deep multilayer perceptron with eight layers. It is a supervised learning network that ...
- 3Types of Neural Networks and Definition of ... - Great Learning
Definition & Types of Neural Networks: There are 7 types of Neural Networks, know the advantages ...
- 4Artificial Neural Networks Applications and Algorithms - XenonStack
- 56 Types of Artificial Neural Networks Currently Being Used in ML
6 Types of Artificial Neural Networks Currently Being Used in Machine Learning · 1. Feedforward N...
