Artificial neural network - Wikipedia

Stochastic neural network Artificialneuralnetwork FromWikipedia,thefreeencyclopedia Jumptonavigation Jumptosearch Computationalmodelusedinmachinelearning,basedonconnected,hierarchicalfunctions PartofaseriesonMachinelearninganddatamining Problems Classification Clustering Regression Anomalydetection DataCleaning AutoML Associationrules Reinforcementlearning Structuredprediction Featureengineering Featurelearning Onlinelearning Semi-supervisedlearning Unsupervisedlearning Learningtorank Grammarinduction Supervisedlearning(classification •regression) Decisiontrees Ensembles Bagging Boosting Randomforest k-NN Linearregression NaiveBayes Artificialneuralnetworks Logisticregression Perceptron Relevancevectormachine(RVM) Supportvectormachine(SVM) Clustering BIRCH CURE Hierarchical k-means Expectation–maximization(EM) DBSCAN OPTICS Meanshift Dimensionalityreduction Factoranalysis CCA ICA LDA NMF PCA PGD t-SNE Structuredprediction Graphicalmodels Bayesnet Conditionalrandomfield HiddenMarkov Anomalydetection k-NN Localoutlierfactor Artificialneuralnetwork Autoencoder Cognitivecomputing Deeplearning DeepDream Multilayerperceptron RNN LSTM GRU ESN RestrictedBoltzmannmachine GAN SOM Convolutionalneuralnetwork U-Net Transformer Vision Spikingneuralnetwork Memtransistor ElectrochemicalRAM(ECRAM) Reinforcementlearning Q-learning SARSA Temporaldifference(TD) Theory Kernelmachines Bias–variancetradeoff Computationallearningtheory Empiricalriskminimization Occamlearning PAClearning Statisticallearning VCtheory Machine-learningvenues NeurIPS ICML ML JMLR ArXiv:cs.LG Relatedarticles Glossaryofartificialintelligence Listofdatasetsformachine-learningresearch Outlineofmachinelearning vte PartofaseriesonArtificialintelligence Majorgoals Artificialgeneralintelligence Planning Computervision Generalgameplaying Knowledgereasoning Machinelearning Naturallanguageprocessing Robotics Approaches Symbolic Deeplearning Bayesiannetworks Evolutionaryalgorithms Philosophy Chineseroom FriendlyAI Controlproblem/Takeover Ethics Existentialrisk Turingtest History Timeline Progress AIwinter Technology Applications Projects Programminglanguages Glossary Glossary vte Complexsystems Topics Self-organizationEmergence CollectivebehaviorSocialdynamics Collectiveintelligence Collectiveaction Self-organizedcriticality Herdmentality Phasetransition Agent-basedmodelling Synchronization Antcolonyoptimization Particleswarmoptimization Swarmbehaviour Collectiveconsciousness NetworksScale-freenetworks Socialnetworkanalysis Small-worldnetworks Centrality Motifs Graphtheory Scaling Robustness Systemsbiology Dynamicnetworks Adaptivenetworks EvolutionandadaptationArtificialneuralnetwork Evolutionarycomputation Geneticalgorithms Geneticprogramming Artificiallife Machinelearning Evolutionarydevelopmentalbiology Artificialintelligence Evolutionaryrobotics Evolvability PatternformationFractals Reaction–diffusionsystems Partialdifferentialequations Dissipativestructures Percolation Cellularautomata Spatialecology Self-replication Geomorphology SystemstheoryandcyberneticsAutopoiesis Informationtheory Entropy Feedback Goal-oriented Homeostasis Operationalization Second-ordercybernetics Self-reference Systemdynamics Systemsscience Systemsthinking Sensemaking Variety Theoryofcomputation NonlineardynamicsTimeseriesanalysis Ordinarydifferentialequations Phasespace Attractors Populationdynamics Chaos Multistability Bifurcation Coupledmaplattices GametheoryPrisoner'sdilemma Rationalchoicetheory Boundedrationality Evolutionarygametheory vte Networkscience Theory Graph Complexnetwork Contagion Small-world Scale-free Communitystructure Percolation Evolution Controllability Graphdrawing Socialcapital Linkanalysis Optimization Reciprocity Closure Homophily Transitivity Preferentialattachment Balancetheory Networkeffect Socialinfluence Networktypes Informational(computing) Telecommunication Transport Social Scientificcollaboration Biological Artificialneural Interdependent Semantic Spatial Dependency Flow on-Chip Graphs Features Clique Component Cut Cycle Datastructure Edge Loop Neighborhood Path Vertex Adjacencylist /matrix Incidencelist /matrix Types Bipartite Complete Directed Hyper Multi Random Weighted MetricsAlgorithms Centrality Degree Motif Clustering Degreedistribution Assortativity Distance Modularity Efficiency Models Topology Randomgraph Erdős–Rényi Barabási–Albert Bianconi–Barabási Fitnessmodel Watts–Strogatz Exponentialrandom(ERGM) Randomgeometric(RGG) Hyperbolic(HGN) Hierarchical Stochasticblock Blockmodeling Maximumentropy Softconfiguration LFRBenchmark Dynamics Booleannetwork agentbased Epidemic/SIR ListsCategories Topics Software Networkscientists Category:Networktheory Category:Graphtheory vte Anartificialneuralnetworkisaninterconnectedgroupofnodes,inspiredbyasimplificationofneuronsinabrain.Here,eachcircularnoderepresentsanartificialneuronandanarrowrepresentsaconnectionfromtheoutputofoneartificialneurontotheinputofanother. Artificialneuralnetworks(ANNs),usuallysimplycalledneuralnetworks(NNs),arecomputingsystemsinspiredbythebiologicalneuralnetworksthatconstituteanimalbrains. AnANNisbasedonacollectionofconnectedunitsornodescalledartificialneurons,whichlooselymodeltheneuronsinabiologicalbrain.Eachconnection,likethesynapsesinabiologicalbrain,cantransmitasignaltootherneurons.Anartificialneuronreceivesasignalthenprocessesitandcansignalneuronsconnectedtoit.The"signal"ataconnectionisarealnumber,andtheoutputofeachneuroniscomputedbysomenon-linearfunctionofthesumofitsinputs.Theconnectionsarecallededges.Neuronsandedgestypicallyhaveaweightthatadjustsaslearningproceeds.Theweightincreasesordecreasesthestrengthofthesignalataconnection.Neuronsmayhaveathresholdsuchthatasignalissentonlyiftheaggregatesignalcrossesthatthreshold.Typically,neuronsareaggregatedintolayers.Differentlayersmayperformdifferenttransformationsontheirinputs.Signalstravelfromthefirstlayer(theinputlayer),tothelastlayer(theoutputlayer),possiblyaftertraversingthelayersmultipletimes.Contents 1Training 2History 3Models 3.1Artificialneurons 3.2Organization 3.3Hyperparameter 3.4Learning 3.4.1Learningrate 3.4.2Costfunction 3.4.3Backpropagation 3.5Learningparadigms 3.5.1Supervisedlearning 3.5.2Unsupervisedlearning 3.5.3Reinforcementlearning 3.5.4Self-learning 3.5.5Neuroevolution 3.6Stochasticneuralnetwork 3.7Other 3.7.1Modes 4Types 5Networkdesign 6Use 7Applications 8Theoreticalproperties 8.1Computationalpower 8.2Capacity 8.3Convergence 8.4Generalizationandstatistics 9Criticism 9.1Training 9.2Theory 9.3Hardware 9.4Practicalcounterexamples 9.5Hybridapproaches 10Gallery 11Seealso 12Notes 13References 14Bibliography 15Externallinks Training[edit] Neuralnetworkslearn(oraretrained)byprocessingexamples,eachofwhichcontainsaknown"input"and"result,"formingprobability-weightedassociationsbetweenthetwo,whicharestoredwithinthedatastructureofthenetitself.Thetrainingofaneuralnetworkfromagivenexampleisusuallyconductedbydeterminingthedifferencebetweentheprocessedoutputofthenetwork(oftenaprediction)andatargetoutput.Thisdifferenceistheerror.Thenetworkthenadjustsitsweightedassociationsaccordingtoalearningruleandusingthiserrorvalue.Successiveadjustmentswillcausetheneuralnetworktoproduceoutputwhichisincreasinglysimilartothetargetoutput.Afterasufficientnumberoftheseadjustmentsthetrainingcanbeterminatedbaseduponcertaincriteria.Thisisknownassupervisedlearning. Suchsystems"learn"toperformtasksbyconsideringexamples,generallywithoutbeingprogrammedwithtask-specificrules.Forexample,inimagerecognition,theymightlearntoidentifyimagesthatcontaincatsbyanalyzingexampleimagesthathavebeenmanuallylabeledas"cat"or"nocat"andusingtheresultstoidentifycatsinotherimages.Theydothiswithoutanypriorknowledgeofcats,forexample,thattheyhavefur,tails,whiskers,andcat-likefaces.Instead,theyautomaticallygenerateidentifyingcharacteristicsfromtheexamplesthattheyprocess. History[edit] Mainarticle:Historyofartificialneuralnetworks WarrenMcCullochandWalterPitts[1](1943)openedthesubjectbycreatingacomputationalmodelforneuralnetworks.[2]Inthelate1940s,D.O.Hebb[3]createdalearninghypothesisbasedonthemechanismofneuralplasticitythatbecameknownasHebbianlearning.FarleyandWesleyA.Clark[4](1954)firstusedcomputationalmachines,thencalled"calculators",tosimulateaHebbiannetwork.In1958,psychologistFrankRosenblattinventedtheperceptron,thefirstartificialneuralnetwork,[5][6][7][8]fundedbytheUnitedStatesOfficeofNavalResearch.[9]ThefirstfunctionalnetworkswithmanylayerswerepublishedbyIvakhnenkoandLapain1965,astheGroupMethodofDataHandling.[10][11][12]Thebasicsofcontinuousbackpropagation[10][13][14][15]werederivedinthecontextofcontroltheorybyKelley[16]in1960andbyBrysonin1961,[17]usingprinciplesofdynamicprogramming.ThereafterresearchstagnatedfollowingMinskyandPapert(1969),[18]whodiscoveredthatbasicperceptronswereincapableofprocessingtheexclusive-orcircuitandthatcomputerslackedsufficientpowertoprocessusefulneuralnetworks. In1970,SeppoLinnainmaapublishedthegeneralmethodforautomaticdifferentiation(AD)ofdiscreteconnectednetworksofnesteddifferentiablefunctions.[19][20]In1973,Dreyfususedbackpropagationtoadaptparametersofcontrollersinproportiontoerrorgradients.[21]Werbos's(1975)backpropagationalgorithmenabledpracticaltrainingofmulti-layernetworks.In1982,heappliedLinnainmaa'sADmethodtoneuralnetworksinthewaythatbecamewidelyused.[13][22] Thedevelopmentofmetal–oxide–semiconductor(MOS)very-large-scaleintegration(VLSI),intheformofcomplementaryMOS(CMOS)technology,enabledincreasingMOStransistorcountsindigitalelectronics.Thisprovidedmoreprocessingpowerforthedevelopmentofpracticalartificialneuralnetworksinthe1980s.[23] In1986Rumelhart,HintonandWilliamsshowedthatbackpropagationlearnedinterestinginternalrepresentationsofwordsasfeaturevectorswhentrainedtopredictthenextwordinasequence.[24] In1992,max-poolingwasintroducedtohelpwithleast-shiftinvarianceandtolerancetodeformationtoaid3Dobjectrecognition.[25][26][27]Schmidhuberadoptedamulti-levelhierarchyofnetworks(1992)pre-trainedonelevelatatimebyunsupervisedlearningandfine-tunedbybackpropagation.[28] Neuralnetworks'earlysuccessesincludedpredictingthestockmarketandin1995a(mostly)self-drivingcar.[a][29] GeoffreyHintonetal.(2006)proposedlearningahigh-levelrepresentationusingsuccessivelayersofbinaryorreal-valuedlatentvariableswitharestrictedBoltzmannmachine[30]tomodeleachlayer.In2012,NgandDeancreatedanetworkthatlearnedtorecognizehigher-levelconcepts,suchascats,onlyfromwatchingunlabeledimages.[31]Unsupervisedpre-trainingandincreasedcomputingpowerfromGPUsanddistributedcomputingallowedtheuseoflargernetworks,particularlyinimageandvisualrecognitionproblems,whichbecameknownas"deeplearning".[32] Ciresanandcolleagues(2010)[33]showedthatdespitethevanishinggradientproblem,GPUsmakebackpropagationfeasibleformany-layeredfeedforwardneuralnetworks.[34]Between2009and2012,ANNsbeganwinningprizesinimagerecognitioncontests,approachinghumanlevelperformanceonvarioustasks,initiallyinpatternrecognitionandhandwritingrecognition.[35][36]Forexample,thebi-directionalandmulti-dimensionallongshort-termmemory(LSTM)[37][38][39][40]ofGravesetal.wonthreecompetitionsinconnectedhandwritingrecognitionin2009withoutanypriorknowledgeaboutthethreelanguagestobelearned.[39][38] Ciresanandcolleaguesbuiltthefirstpatternrecognizerstoachievehuman-competitive/superhumanperformance[41]onbenchmarkssuchastrafficsignrecognition(IJCNN2012). Models[edit] Thissectionmaybeconfusingoruncleartoreaders.Pleasehelpclarifythesection.Theremightbeadiscussionaboutthisonthetalkpage.(April2017)(Learnhowandwhentoremovethistemplatemessage)Furtherinformation:MathematicsofartificialneuralnetworksNeuronandmyelinatedaxon,withsignalflowfrominputsatdendritestooutputsataxonterminals ANNsbeganasanattempttoexploitthearchitectureofthehumanbraintoperformtasksthatconventionalalgorithmshadlittlesuccesswith.Theysoonreorientedtowardsimprovingempiricalresults,mostlyabandoningattemptstoremaintruetotheirbiologicalprecursors.Neuronsareconnectedtoeachotherinvariouspatterns,toallowtheoutputofsomeneuronstobecometheinputofothers.Thenetworkformsadirected,weightedgraph.[42] Anartificialneuralnetworkconsistsofacollectionofsimulatedneurons.Eachneuronisanodewhichisconnectedtoothernodesvialinksthatcorrespondtobiologicalaxon-synapse-dendriteconnections.Eachlinkhasaweight,whichdeterminesthestrengthofonenode'sinfluenceonanother.[43] Artificialneurons[edit] ANNsarecomposedofartificialneuronswhichareconceptuallyderivedfrombiologicalneurons.Eachartificialneuronhasinputsandproducesasingleoutputwhichcanbesenttomultipleotherneurons.[44]Theinputscanbethefeaturevaluesofasampleofexternaldata,suchasimagesordocuments,ortheycanbetheoutputsofotherneurons.Theoutputsofthefinaloutputneuronsoftheneuralnetaccomplishthetask,suchasrecognizinganobjectinanimage. Tofindtheoutputoftheneuron,Firstwemusttaketheweightedsumofalltheinputs,weightedbytheweightsoftheconnectionsfromtheinputstotheneuron.Weaddabiastermtothissum.[45]Thisweightedsumissometimescalledtheactivation.Thisweightedsumisthenpassedthrougha(usuallynonlinear)activationfunctiontoproducetheoutput.Theinitialinputsareexternaldata,suchasimagesanddocuments.Theultimateoutputsaccomplishthetask,suchasrecognizinganobjectinanimage.[46] Organization[edit] Theneuronsaretypicallyorganizedintomultiplelayers,especiallyindeeplearning.Neuronsofonelayerconnectonlytoneuronsoftheimmediatelyprecedingandimmediatelyfollowinglayers.Thelayerthatreceivesexternaldataistheinputlayer.Thelayerthatproducestheultimateresultistheoutputlayer.Inbetweenthemarezeroormorehiddenlayers.Singlelayerandunlayerednetworksarealsoused.Betweentwolayers,multipleconnectionpatternsarepossible.Theycanbe'fullyconnected',witheveryneuroninonelayerconnectingtoeveryneuroninthenextlayer.Theycanbepooling,whereagroupofneuronsinonelayerconnecttoasingleneuroninthenextlayer,therebyreducingthenumberofneuronsinthatlayer.[47]Neuronswithonlysuchconnectionsformadirectedacyclicgraphandareknownasfeedforwardnetworks.[48]Alternatively,networksthatallowconnectionsbetweenneuronsinthesameorpreviouslayersareknownasrecurrentnetworks.[49] Hyperparameter[edit] Mainarticle:Hyperparameter(machinelearning) Ahyperparameterisaconstantparameterwhosevalueissetbeforethelearningprocessbegins.Thevaluesofparametersarederivedvialearning.Examplesofhyperparametersincludelearningrate,thenumberofhiddenlayersandbatchsize.[50]Thevaluesofsomehyperparameterscanbedependentonthoseofotherhyperparameters.Forexample,thesizeofsomelayerscandependontheoverallnumberoflayers. Learning[edit] Thissectionincludesalistofreferences,relatedreadingorexternallinks,butitssourcesremainunclearbecauseitlacksinlinecitations.Pleasehelptoimprovethissectionbyintroducingmoreprecisecitations.(August2019)(Learnhowandwhentoremovethistemplatemessage)Seealso:Mathematicaloptimization,Estimationtheory,andMachinelearning Learningistheadaptationofthenetworktobetterhandleataskbyconsideringsampleobservations.Learninginvolvesadjustingtheweights(andoptionalthresholds)ofthenetworktoimprovetheaccuracyoftheresult.Thisisdonebyminimizingtheobservederrors.Learningiscompletewhenexaminingadditionalobservationsdoesnotusefullyreducetheerrorrate.Evenafterlearning,theerrorratetypicallydoesnotreach0.Ifafterlearning,theerrorrateistoohigh,thenetworktypicallymustberedesigned.Practicallythisisdonebydefiningacostfunctionthatisevaluatedperiodicallyduringlearning.Aslongasitsoutputcontinuestodecline,learningcontinues.Thecostisfrequentlydefinedasastatisticwhosevaluecanonlybeapproximated.Theoutputsareactuallynumbers,sowhentheerrorislow,thedifferencebetweentheoutput(almostcertainlyacat)andthecorrectanswer(cat)issmall.Learningattemptstoreducethetotalofthedifferencesacrosstheobservations.Mostlearningmodelscanbeviewedasastraightforwardapplicationofoptimizationtheoryandstatisticalestimation.[51][42] Learningrate[edit] Thelearningratedefinesthesizeofthecorrectivestepsthatthemodeltakestoadjustforerrorsineachobservation.[52]Ahighlearningrateshortensthetrainingtime,butwithlowerultimateaccuracy,whilealowerlearningratetakeslonger,butwiththepotentialforgreateraccuracy.OptimizationssuchasQuickpropareprimarilyaimedatspeedinguperrorminimization,whileotherimprovementsmainlytrytoincreasereliability.Inordertoavoidoscillationinsidethenetworksuchasalternatingconnectionweights,andtoimprovetherateofconvergence,refinementsuseanadaptivelearningratethatincreasesordecreasesasappropriate.[53]Theconceptofmomentumallowsthebalancebetweenthegradientandthepreviouschangetobeweightedsuchthattheweightadjustmentdependstosomedegreeonthepreviouschange.Amomentumcloseto0emphasizesthegradient,whileavaluecloseto1emphasizesthelastchange. Costfunction[edit] Whileitispossibletodefineacostfunctionadhoc,frequentlythechoiceisdeterminedbythefunction'sdesirableproperties(suchasconvexity)orbecauseitarisesfromthemodel(e.g.inaprobabilisticmodelthemodel'sposteriorprobabilitycanbeusedasaninversecost). Backpropagation[edit] Mainarticle:Backpropagation Backpropagationisamethodusedtoadjusttheconnectionweightstocompensateforeacherrorfoundduringlearning.Theerroramountiseffectivelydividedamongtheconnections.Technically,backpropcalculatesthegradient(thederivative)ofthecostfunctionassociatedwithagivenstatewithrespecttotheweights.Theweightupdatescanbedoneviastochasticgradientdescentorothermethods,suchasExtremeLearningMachines,[54]"No-prop"networks,[55]trainingwithoutbacktracking,[56]"weightless"networks,[57][58]andnon-connectionistneuralnetworks. Learningparadigms[edit] Thissectionincludesalistofreferences,relatedreadingorexternallinks,butitssourcesremainunclearbecauseitlacksinlinecitations.Pleasehelptoimprovethissectionbyintroducingmoreprecisecitations.(August2019)(Learnhowandwhentoremovethistemplatemessage) Thethreemajorlearningparadigmsaresupervisedlearning,unsupervisedlearningandreinforcementlearning.Theyeachcorrespondtoaparticularlearningtask Supervisedlearning[edit] Supervisedlearningusesasetofpairedinputsanddesiredoutputs.Thelearningtaskistoproducethedesiredoutputforeachinput.Inthiscasethecostfunctionisrelatedtoeliminatingincorrectdeductions.[59]Acommonlyusedcostisthemean-squarederror,whichtriestominimizetheaveragesquarederrorbetweenthenetwork'soutputandthedesiredoutput.Taskssuitedforsupervisedlearningarepatternrecognition(alsoknownasclassification)andregression(alsoknownasfunctionapproximation).Supervisedlearningisalsoapplicabletosequentialdata(e.g.,forhandwriting,speechandgesturerecognition).Thiscanbethoughtofaslearningwitha"teacher",intheformofafunctionthatprovidescontinuousfeedbackonthequalityofsolutionsobtainedthusfar. Unsupervisedlearning[edit] Inunsupervisedlearning,inputdataisgivenalongwiththecostfunction,somefunctionofthedata x {\displaystyle\textstylex} andthenetwork'soutput.Thecostfunctionisdependentonthetask(themodeldomain)andanyaprioriassumptions(theimplicitpropertiesofthemodel,itsparametersandtheobservedvariables).Asatrivialexample,considerthemodel f ( x ) = a {\displaystyle\textstylef(x)=a} where a {\displaystyle\textstylea} isaconstantandthecost C = E [ ( x − f ( x ) ) 2 ] {\displaystyle\textstyleC=E[(x-f(x))^{2}]} .Minimizingthiscostproducesavalueof a {\displaystyle\textstylea} thatisequaltothemeanofthedata.Thecostfunctioncanbemuchmorecomplicated.Itsformdependsontheapplication:forexample,incompressionitcouldberelatedtothemutualinformationbetween x {\displaystyle\textstylex} and f ( x ) {\displaystyle\textstylef(x)} ,whereasinstatisticalmodeling,itcouldberelatedtotheposteriorprobabilityofthemodelgiventhedata(notethatinbothofthoseexamplesthosequantitieswouldbemaximizedratherthanminimized).Tasksthatfallwithintheparadigmofunsupervisedlearningareingeneralestimationproblems;theapplicationsincludeclustering,theestimationofstatisticaldistributions,compressionandfiltering. Reinforcementlearning[edit] Mainarticle:Reinforcementlearning Seealso:Stochasticcontrol Inapplicationssuchasplayingvideogames,anactortakesastringofactions,receivingagenerallyunpredictableresponsefromtheenvironmentaftereachone.Thegoalistowinthegame,i.e.,generatethemostpositive(lowestcost)responses.Inreinforcementlearning,theaimistoweightthenetwork(deviseapolicy)toperformactionsthatminimizelong-term(expectedcumulative)cost.Ateachpointintimetheagentperformsanactionandtheenvironmentgeneratesanobservationandaninstantaneouscost,accordingtosome(usuallyunknown)rules.Therulesandthelong-termcostusuallyonlycanbeestimated.Atanyjuncture,theagentdecideswhethertoexplorenewactionstouncovertheircostsortoexploitpriorlearningtoproceedmorequickly. FormallytheenvironmentismodeledasaMarkovdecisionprocess(MDP)withstates s 1 , . . . , s n ∈ S {\displaystyle\textstyle{s_{1},...,s_{n}}\inS} andactions a 1 , . . . , a m ∈ A {\displaystyle\textstyle{a_{1},...,a_{m}}\inA} .Becausethestatetransitionsarenotknown,probabilitydistributionsareusedinstead:theinstantaneouscostdistribution P ( c t | s t ) {\displaystyle\textstyleP(c_{t}|s_{t})} ,theobservationdistribution P ( x t | s t ) {\displaystyle\textstyleP(x_{t}|s_{t})} andthetransitiondistribution P ( s t + 1 | s t , a t ) {\displaystyle\textstyleP(s_{t+1}|s_{t},a_{t})} ,whileapolicyisdefinedastheconditionaldistributionoveractionsgiventheobservations.Takentogether,thetwodefineaMarkovchain(MC).Theaimistodiscoverthelowest-costMC. ANNsserveasthelearningcomponentinsuchapplications.[60][61]DynamicprogrammingcoupledwithANNs(givingneurodynamicprogramming)[62]hasbeenappliedtoproblemssuchasthoseinvolvedinvehiclerouting,[63]videogames,naturalresourcemanagement[64][65]andmedicine[66]becauseofANNsabilitytomitigatelossesofaccuracyevenwhenreducingthediscretizationgriddensityfornumericallyapproximatingthesolutionofcontrolproblems.Tasksthatfallwithintheparadigmofreinforcementlearningarecontrolproblems,gamesandothersequentialdecisionmakingtasks. Self-learning[edit] Self-learninginneuralnetworkswasintroducedin1982alongwithaneuralnetworkcapableofself-learningnamedCrossbarAdaptiveArray(CAA).[67]Itisasystemwithonlyoneinput,situations,andonlyoneoutput,action(orbehavior)a.Ithasneitherexternaladviceinputnorexternalreinforcementinputfromtheenvironment.TheCAAcomputes,inacrossbarfashion,bothdecisionsaboutactionsandemotions(feelings)aboutencounteredsituations.Thesystemisdrivenbytheinteractionbetweencognitionandemotion.[68]Giventhememorymatrix,W=||w(a,s)||,thecrossbarself-learningalgorithmineachiterationperformsthefollowingcomputation: Insituationsperformactiona; Receiveconsequencesituations'; Computeemotionofbeinginconsequencesituationv(s'); Updatecrossbarmemoryw'(a,s)=w(a,s)+v(s'). Thebackpropagatedvalue(secondaryreinforcement)istheemotiontowardtheconsequencesituation.TheCAAexistsintwoenvironments,oneisbehavioralenvironmentwhereitbehaves,andtheotherisgeneticenvironment,wherefromitinitiallyandonlyoncereceivesinitialemotionsabouttobeencounteredsituationsinthebehavioralenvironment.Havingreceivedthegenomevector(speciesvector)fromthegeneticenvironment,theCAAwilllearnagoal-seekingbehavior,inthebehavioralenvironmentthatcontainsbothdesirableandundesirablesituations.[69] Neuroevolution[edit] Mainarticle:Neuroevolution Neuroevolutioncancreateneuralnetworktopologiesandweightsusingevolutionarycomputation.Itiscompetitivewithsophisticatedgradientdescentapproaches[citationneeded].Oneadvantageofneuroevolutionisthatitmaybelesspronetogetcaughtin"deadends".[70] Stochasticneuralnetwork[edit] StochasticneuralnetworksoriginatingfromSherrington–Kirkpatrickmodelsareatypeofartificialneuralnetworkbuiltbyintroducingrandomvariationsintothenetwork,eitherbygivingthenetwork'sartificialneuronsstochastictransferfunctions,orbygivingthemstochasticweights.Thismakesthemusefultoolsforoptimizationproblems,sincetherandomfluctuationshelpthenetworkescapefromlocalminima.[71] Other[edit] InaBayesianframework,adistributionoverthesetofallowedmodelsischosentominimizethecost.Evolutionarymethods,[72]geneexpressionprogramming,[73]simulatedannealing,[74]expectation-maximization,non-parametricmethodsandparticleswarmoptimization[75]areotherlearningalgorithms.Convergentrecursionisalearningalgorithmforcerebellarmodelarticulationcontroller(CMAC)neuralnetworks.[76][77] Modes[edit] Thissectionincludesalistofreferences,relatedreadingorexternallinks,butitssourcesremainunclearbecauseitlacksinlinecitations.Pleasehelptoimprovethissectionbyintroducingmoreprecisecitations.(August2019)(Learnhowandwhentoremovethistemplatemessage) Twomodesoflearningareavailable:stochasticandbatch.Instochasticlearning,eachinputcreatesaweightadjustment.Inbatchlearningweightsareadjustedbasedonabatchofinputs,accumulatingerrorsoverthebatch.Stochasticlearningintroduces"noise"intotheprocess,usingthelocalgradientcalculatedfromonedatapoint;thisreducesthechanceofthenetworkgettingstuckinlocalminima.However,batchlearningtypicallyyieldsafaster,morestabledescenttoalocalminimum,sinceeachupdateisperformedinthedirectionofthebatch'saverageerror.Acommoncompromiseistouse"mini-batches",smallbatcheswithsamplesineachbatchselectedstochasticallyfromtheentiredataset. Types[edit] Mainarticle:Typesofartificialneuralnetworks ANNshaveevolvedintoabroadfamilyoftechniquesthathaveadvancedthestateoftheartacrossmultipledomains.Thesimplesttypeshaveoneormorestaticcomponents,includingnumberofunits,numberoflayers,unitweightsandtopology.Dynamictypesallowoneormoreofthesetoevolvevialearning.Thelatteraremuchmorecomplicated,butcanshortenlearningperiodsandproducebetterresults.Sometypesallow/requirelearningtobe"supervised"bytheoperator,whileothersoperateindependently.Sometypesoperatepurelyinhardware,whileothersarepurelysoftwareandrunongeneralpurposecomputers. Someofthemainbreakthroughsinclude:convolutionalneuralnetworksthathaveprovenparticularlysuccessfulinprocessingvisualandothertwo-dimensionaldata;[78][79]longshort-termmemoryavoidthevanishinggradientproblem[80]andcanhandlesignalsthathaveamixoflowandhighfrequencycomponentsaidinglarge-vocabularyspeechrecognition,[81][82]text-to-speechsynthesis,[83][13][84]andphoto-realtalkingheads;[85]competitivenetworkssuchasgenerativeadversarialnetworksinwhichmultiplenetworks(ofvaryingstructure)competewitheachother,ontaskssuchaswinningagame[86]orondeceivingtheopponentabouttheauthenticityofaninput.[87] Networkdesign[edit] Mainarticle:Neuralarchitecturesearch Neuralarchitecturesearch(NAS)usesmachinelearningtoautomateANNdesign.VariousapproachestoNAShavedesignednetworksthatcomparewellwithhand-designedsystems.Thebasicsearchalgorithmistoproposeacandidatemodel,evaluateitagainstadatasetandusetheresultsasfeedbacktoteachtheNASnetwork.[88]AvailablesystemsincludeAutoMLandAutoKeras.[89] Designissuesincludedecidingthenumber,typeandconnectednessofnetworklayers,aswellasthesizeofeachandtheconnectiontype(full,pooling,...). Hyperparametersmustalsobedefinedaspartofthedesign(theyarenotlearned),governingmatterssuchashowmanyneuronsareineachlayer,learningrate,step,stride,depth,receptivefieldandpadding(forCNNs),etc.[90] Use[edit] Thissectiondoesnotciteanysources.Pleasehelpimprovethissectionbyaddingcitationstoreliablesources.Unsourcedmaterialmaybechallengedandremoved.(November2020)(Learnhowandwhentoremovethistemplatemessage) UsingArtificialneuralnetworksrequiresanunderstandingoftheircharacteristics. Choiceofmodel:Thisdependsonthedatarepresentationandtheapplication.Overlycomplexmodelsareslowlearning. Learningalgorithm:Numeroustrade-offsexistbetweenlearningalgorithms.Almostanyalgorithmwillworkwellwiththecorrecthyperparametersfortrainingonaparticulardataset.However,selectingandtuninganalgorithmfortrainingonunseendatarequiressignificantexperimentation. Robustness:Ifthemodel,costfunctionandlearningalgorithmareselectedappropriately,theresultingANNcanbecomerobust. ANNcapabilitiesfallwithinthefollowingbroadcategories:[citationneeded] Functionapproximation,orregressionanalysis,includingtimeseriesprediction,fitnessapproximationandmodeling. Classification,includingpatternandsequencerecognition,noveltydetectionandsequentialdecisionmaking.[91] Dataprocessing,includingfiltering,clustering,blindsourceseparationandcompression. Robotics,includingdirectingmanipulatorsandprostheses. Applications[edit] Becauseoftheirabilitytoreproduceandmodelnonlinearprocesses,artificialneuralnetworkshavefoundapplicationsinmanydisciplines.Applicationareasincludesystemidentificationandcontrol(vehiclecontrol,trajectoryprediction,[92]processcontrol,naturalresourcemanagement),quantumchemistry,[93]generalgameplaying,[94]patternrecognition(radarsystems,faceidentification,signalclassification,[95]3Dreconstruction,[96]objectrecognitionandmore),sensordataanalysis,[97]sequencerecognition(gesture,speech,handwrittenandprintedtextrecognition[98]),medicaldiagnosis,finance[99](e.g.automatedtradingsystems),datamining,visualization,machinetranslation,socialnetworkfiltering[100]ande-mailspamfiltering.ANNshavebeenusedtodiagnoseseveraltypesofcancers[101][102]andtodistinguishhighlyinvasivecancercelllinesfromlessinvasivelinesusingonlycellshapeinformation.[103][104] ANNshavebeenusedtoacceleratereliabilityanalysisofinfrastructuressubjecttonaturaldisasters[105][106]andtopredictfoundationsettlements.[107]ANNshavealsobeenusedforbuildingblack-boxmodelsingeoscience:hydrology,[108][109]oceanmodellingandcoastalengineering,[110][111]andgeomorphology.[112]ANNshavebeenemployedincybersecurity,withtheobjectivetodiscriminatebetweenlegitimateactivitiesandmaliciousones.Forexample,machinelearninghasbeenusedforclassifyingAndroidmalware,[113]foridentifyingdomainsbelongingtothreatactorsandfordetectingURLsposingasecurityrisk.[114]ResearchisunderwayonANNsystemsdesignedforpenetrationtesting,fordetectingbotnets,[115]creditcardsfrauds[116]andnetworkintrusions. ANNshavebeenproposedasatooltosolvepartialdifferentialequationsinphysics[117][118][119]andsimulatethepropertiesofmany-bodyopenquantumsystems.[120][121][122][123]InbrainresearchANNshavestudiedshort-termbehaviorofindividualneurons,[124]thedynamicsofneuralcircuitryarisefrominteractionsbetweenindividualneuronsandhowbehaviorcanarisefromabstractneuralmodulesthatrepresentcompletesubsystems.Studiesconsideredlong-andshort-termplasticityofneuralsystemsandtheirrelationtolearningandmemoryfromtheindividualneurontothesystemlevel. Theoreticalproperties[edit] Computationalpower[edit] Themultilayerperceptronisauniversalfunctionapproximator,asprovenbytheuniversalapproximationtheorem.However,theproofisnotconstructiveregardingthenumberofneuronsrequired,thenetworktopology,theweightsandthelearningparameters. Aspecificrecurrentarchitecturewithrational-valuedweights(asopposedtofullprecisionrealnumber-valuedweights)hasthepowerofauniversalTuringmachine,[125]usingafinitenumberofneuronsandstandardlinearconnections.Further,theuseofirrationalvaluesforweightsresultsinamachinewithsuper-Turingpower.[126] Capacity[edit] Amodel's"capacity"propertycorrespondstoitsabilitytomodelanygivenfunction.Itisrelatedtotheamountofinformationthatcanbestoredinthenetworkandtothenotionofcomplexity. Twonotionsofcapacityareknownbythecommunity.TheinformationcapacityandtheVCDimension.TheinformationcapacityofaperceptronisintensivelydiscussedinSirDavidMacKay'sbook[127]whichsummarizesworkbyThomasCover.[128]Thecapacityofanetworkofstandardneurons(notconvolutional)canbederivedbyfourrules[129]thatderivefromunderstandinganeuronasanelectricalelement.Theinformationcapacitycapturesthefunctionsmodelablebythenetworkgivenanydataasinput.Thesecondnotion,istheVCdimension.VCDimensionusestheprinciplesofmeasuretheoryandfindsthemaximumcapacityunderthebestpossiblecircumstances.Thisis,giveninputdatainaspecificform.Asnotedin,[127]theVCDimensionforarbitraryinputsishalftheinformationcapacityofaPerceptron.TheVCDimensionforarbitrarypointsissometimesreferredtoasMemoryCapacity.[130] Convergence[edit] Modelsmaynotconsistentlyconvergeonasinglesolution,firstlybecauselocalminimamayexist,dependingonthecostfunctionandthemodel.Secondly,theoptimizationmethodusedmightnotguaranteetoconvergewhenitbeginsfarfromanylocalminimum.Thirdly,forsufficientlylargedataorparameters,somemethodsbecomeimpractical. AnotherissueworthytomentionisthattrainingmaycrosssomeSaddlepointwhichmayleadtheconvergencetothewrongdirection. TheconvergencebehaviorofcertaintypesofANNarchitecturesaremoreunderstoodthanothers.Whenthewidthofnetworkapproachestoinfinity,theANNiswelldescribedbyitsfirstorderTaylorexpansionthroughouttraining,andsoinheritstheconvergencebehaviorofaffinemodels.[131][132]Anotherexampleiswhenparametersaresmall,itisobservedthatANNsoftenfitstargetfunctionsfromlowtohighfrequencies.Thisbehaviorisreferredtoasthespectralbias,orfrequencyprinciple,ofneuralnetworks.[133][134][135][136]ThisphenomenonistheoppositetothebehaviorofsomewellstudiediterativenumericalschemessuchasJacobimethod.Deeperneuralnetworkshavebeenobservedtobemorebiasedtowardslowfrequencyfunctions.[137] Generalizationandstatistics[edit] Thissectionincludesalistofreferences,relatedreadingorexternallinks,butitssourcesremainunclearbecauseitlacksinlinecitations.Pleasehelptoimprovethissectionbyintroducingmoreprecisecitations.(August2019)(Learnhowandwhentoremovethistemplatemessage) Applicationswhosegoalistocreateasystemthatgeneralizeswelltounseenexamples,facethepossibilityofover-training.Thisarisesinconvolutedorover-specifiedsystemswhenthenetworkcapacitysignificantlyexceedstheneededfreeparameters.Twoapproachesaddressover-training.Thefirstistousecross-validationandsimilartechniquestocheckforthepresenceofover-trainingandtoselecthyperparameterstominimizethegeneralizationerror. Thesecondistousesomeformofregularization.Thisconceptemergesinaprobabilistic(Bayesian)framework,whereregularizationcanbeperformedbyselectingalargerpriorprobabilityoversimplermodels;butalsoinstatisticallearningtheory,wherethegoalistominimizeovertwoquantities:the'empiricalrisk'andthe'structuralrisk',whichroughlycorrespondstotheerroroverthetrainingsetandthepredictederrorinunseendataduetooverfitting. Confidenceanalysisofaneuralnetwork Supervisedneuralnetworksthatuseameansquarederror(MSE)costfunctioncanuseformalstatisticalmethodstodeterminetheconfidenceofthetrainedmodel.TheMSEonavalidationsetcanbeusedasanestimateforvariance.Thisvaluecanthenbeusedtocalculatetheconfidenceintervalofnetworkoutput,assuminganormaldistribution.Aconfidenceanalysismadethiswayisstatisticallyvalidaslongastheoutputprobabilitydistributionstaysthesameandthenetworkisnotmodified. Byassigningasoftmaxactivationfunction,ageneralizationofthelogisticfunction,ontheoutputlayeroftheneuralnetwork(orasoftmaxcomponentinacomponent-basednetwork)forcategoricaltargetvariables,theoutputscanbeinterpretedasposteriorprobabilities.Thisisusefulinclassificationasitgivesacertaintymeasureonclassifications. Thesoftmaxactivationfunctionis: y i = e x i ∑ j = 1 c e x j {\displaystyley_{i}={\frac{e^{x_{i}}}{\sum_{j=1}^{c}e^{x_{j}}}}} Criticism[edit] Training[edit] Acommoncriticismofneuralnetworks,particularlyinrobotics,isthattheyrequiretoomuchtrainingforreal-worldoperation.[citationneeded]Potentialsolutionsincluderandomlyshufflingtrainingexamples,byusinganumericaloptimizationalgorithmthatdoesnottaketoolargestepswhenchangingthenetworkconnectionsfollowinganexample,groupingexamplesinso-calledmini-batchesand/orintroducingarecursiveleastsquaresalgorithmforCMAC.[76] Theory[edit] AfundamentalobjectionisthatANNsdonotsufficientlyreflectneuronalfunction.Backpropagationisacriticalstep,althoughnosuchmechanismexistsinbiologicalneuralnetworks.[138]Howinformationiscodedbyrealneuronsisnotknown.Sensorneuronsfireactionpotentialsmorefrequentlywithsensoractivationandmusclecellspullmorestronglywhentheirassociatedmotorneuronsreceiveactionpotentialsmorefrequently.[139]Otherthanthecaseofrelayinginformationfromasensorneurontoamotorneuron,almostnothingoftheprinciplesofhowinformationishandledbybiologicalneuralnetworksisknown. AcentralclaimofANNsisthattheyembodynewandpowerfulgeneralprinciplesforprocessinginformation.Theseprinciplesareill-defined.Itisoftenclaimedthattheyareemergentfromthenetworkitself.Thisallowssimplestatisticalassociation(thebasicfunctionofartificialneuralnetworks)tobedescribedaslearningorrecognition.In1997,AlexanderDewdneycommentedthat,asaresult,artificialneuralnetworkshavea"something-for-nothingquality,onethatimpartsapeculiarauraoflazinessandadistinctlackofcuriosityaboutjusthowgoodthesecomputingsystemsare.Nohumanhand(ormind)intervenes;solutionsarefoundasifbymagic;andnoone,itseems,haslearnedanything".[140]OneresponsetoDewdneyisthatneuralnetworkshandlemanycomplexanddiversetasks,rangingfromautonomouslyflyingaircraft[141]todetectingcreditcardfraudtomasteringthegameofGo. TechnologywriterRogerBridgmancommented: Neuralnetworks,forinstance,areinthedocknotonlybecausetheyhavebeenhypedtohighheaven,(whathasn't?)butalsobecauseyoucouldcreateasuccessfulnetwithoutunderstandinghowitworked:thebunchofnumbersthatcapturesitsbehaviourwouldinallprobabilitybe"anopaque,unreadabletable...valuelessasascientificresource". Inspiteofhisemphaticdeclarationthatscienceisnottechnology,Dewdneyseemsheretopilloryneuralnetsasbadsciencewhenmostofthosedevisingthemarejusttryingtobegoodengineers.Anunreadabletablethatausefulmachinecouldreadwouldstillbewellworthhaving.[142] Biologicalbrainsusebothshallowanddeepcircuitsasreportedbybrainanatomy,[143]displayingawidevarietyofinvariance.Weng[144]arguedthatthebrainself-wireslargelyaccordingtosignalstatisticsandtherefore,aserialcascadecannotcatchallmajorstatisticaldependencies. Hardware[edit] Largeandeffectiveneuralnetworksrequireconsiderablecomputingresources.[145]Whilethebrainhashardwaretailoredtothetaskofprocessingsignalsthroughagraphofneurons,simulatingevenasimplifiedneurononvonNeumannarchitecturemayconsumevastamountsofmemoryandstorage.Furthermore,thedesigneroftenneedstotransmitsignalsthroughmanyoftheseconnectionsandtheirassociatedneurons –whichrequireenormousCPUpowerandtime. Schmidhubernotedthattheresurgenceofneuralnetworksinthetwenty-firstcenturyislargelyattributabletoadvancesinhardware:from1991to2015,computingpower,especiallyasdeliveredbyGPGPUs(onGPUs),hasincreasedaroundamillion-fold,makingthestandardbackpropagationalgorithmfeasiblefortrainingnetworksthatareseverallayersdeeperthanbefore.[10]TheuseofacceleratorssuchasFPGAsandGPUscanreducetrainingtimesfrommonthstodays.[145] Neuromorphicengineeringoraphysicalneuralnetworkaddressesthehardwaredifficultydirectly,byconstructingnon-von-Neumannchipstodirectlyimplementneuralnetworksincircuitry.AnothertypeofchipoptimizedforneuralnetworkprocessingiscalledaTensorProcessingUnit,orTPU.[146] Practicalcounterexamples[edit] AnalyzingwhathasbeenlearnedbyanANNismucheasierthananalyzingwhathasbeenlearnedbyabiologicalneuralnetwork.Furthermore,researchersinvolvedinexploringlearningalgorithmsforneuralnetworksaregraduallyuncoveringgeneralprinciplesthatallowalearningmachinetobesuccessful.Forexample,localvs.non-locallearningandshallowvs.deeparchitecture.[147] Hybridapproaches[edit] Advocatesofhybridmodels(combiningneuralnetworksandsymbolicapproaches),claimthatsuchamixturecanbettercapturethemechanismsofthehumanmind.[148][149] Gallery[edit] Asingle-layerfeedforwardartificialneuralnetwork.Arrowsoriginatingfrom x 2 {\displaystyle\scriptstylex_{2}} areomittedforclarity.Therearepinputstothisnetworkandqoutputs.Inthissystem,thevalueoftheqthoutput, y q {\displaystyle\scriptstyley_{q}} wouldbecalculatedas y q = K ∗ ( ∑ ( x i ∗ w i q ) − b q ) {\displaystyle\scriptstyley_{q}=K*(\sum(x_{i}*w_{iq})-b_{q})} Atwo-layerfeedforwardartificialneuralnetwork. Anartificialneuralnetwork. AnANNdependencygraph. Asingle-layerfeedforwardartificialneuralnetworkwith4inputs,6hiddenand2outputs.Givenpositionstateanddirectionoutputswheelbasedcontrolvalues. Atwo-layerfeedforwardartificialneuralnetworkwith8inputs,2x8hiddenand2outputs.Givenpositionstate,directionandotherenvironmentvaluesoutputsthrusterbasedcontrolvalues. ParallelpipelinestructureofCMACneuralnetwork.Thislearningalgorithmcanconvergeinonestep. Seealso[edit] ADALINE Autoencoder Biologicallyinspiredcomputing BlueBrainProject Catastrophicinterference Cognitivearchitecture Connectionistexpertsystem Connectomics Largewidthlimitsofneuralnetworks Machinelearningconcepts Neuralgas Neuralnetworksoftware Opticalneuralnetwork Paralleldistributedprocessing Recurrentneuralnetworks Spikingneuralnetwork Tensorproductnetwork Notes[edit] ^Steeringforthe1995"NoHandsAcrossAmerica"required"onlyafewhumanassists". References[edit] ^McCulloch,Warren;WalterPitts(1943)."ALogicalCalculusofIdeasImmanentinNervousActivity".BulletinofMathematicalBiophysics.5(4):115–133.doi:10.1007/BF02478259. ^Kleene,S.C.(1956)."RepresentationofEventsinNerveNetsandFiniteAutomata".AnnalsofMathematicsStudies.No. 34.PrincetonUniversityPress.pp. 3–41.Retrieved17June2017. ^Hebb,Donald(1949).TheOrganizationofBehavior.NewYork:Wiley.ISBN 978-1-135-63190-1. ^Farley,B.G.;W.A.Clark(1954)."SimulationofSelf-OrganizingSystemsbyDigitalComputer".IRETransactionsonInformationTheory.4(4):76–84.doi:10.1109/TIT.1954.1057468. ^Haykin(2008)NeuralNetworksandLearningMachines,3rdedition ^Rosenblatt,F.(1958)."ThePerceptron:AProbabilisticModelForInformationStorageAndOrganizationintheBrain".PsychologicalReview.65(6):386–408.CiteSeerX 10.1.1.588.3775.doi:10.1037/h0042519.PMID 13602029. ^Werbos,P.J.(1975).BeyondRegression:NewToolsforPredictionandAnalysisintheBehavioralSciences. ^Rosenblatt,Frank(1957)."ThePerceptron—aperceivingandrecognizingautomaton".Report85-460-1.CornellAeronauticalLaboratory. ^Olazaran,Mikel(1996)."ASociologicalStudyoftheOfficialHistoryofthePerceptronsControversy".SocialStudiesofScience.26(3):611–659.doi:10.1177/030631296026003005.JSTOR 285702.S2CID 16786738. ^abcSchmidhuber,J.(2015)."DeepLearninginNeuralNetworks:AnOverview".NeuralNetworks.61:85–117.arXiv:1404.7828.doi:10.1016/j.neunet.2014.09.003.PMID 25462637.S2CID 11715509. ^Ivakhnenko,A.G.(1973).CyberneticPredictingDevices.CCMInformationCorporation. ^Ivakhnenko,A.G.;GrigorʹevichLapa,Valentin(1967).Cyberneticsandforecastingtechniques.AmericanElsevierPub.Co. ^abcSchmidhuber,Jürgen(2015)."DeepLearning".Scholarpedia.10(11):85–117.Bibcode:2015SchpJ..1032832S.doi:10.4249/scholarpedia.32832. ^Dreyfus,StuartE.(1September1990)."Artificialneuralnetworks,backpropagation,andtheKelley-Brysongradientprocedure".JournalofGuidance,Control,andDynamics.13(5):926–928.Bibcode:1990JGCD...13..926D.doi:10.2514/3.25422.ISSN 0731-5090. ^Mizutani,E.;Dreyfus,S.E.;Nishio,K.(2000)."OnderivationofMLPbackpropagationfromtheKelley-Brysonoptimal-controlgradientformulaanditsapplication".ProceedingsoftheIEEE-INNS-ENNSInternationalJointConferenceonNeuralNetworks.IJCNN2000.NeuralComputing:NewChallengesandPerspectivesfortheNewMillennium.IEEE:167–172vol.2.doi:10.1109/ijcnn.2000.857892.ISBN 0-7695-0619-4.S2CID 351146. ^Kelley,HenryJ.(1960)."Gradienttheoryofoptimalflightpaths".ARSJournal.30(10):947–954.doi:10.2514/8.5282. ^"Agradientmethodforoptimizingmulti-stageallocationprocesses".ProceedingsoftheHarvardUniv.Symposiumondigitalcomputersandtheirapplications.April1961. ^Minsky,Marvin;Papert,Seymour(1969).Perceptrons:AnIntroductiontoComputationalGeometry.MITPress.ISBN 978-0-262-63022-1. ^Linnainmaa,Seppo(1970).TherepresentationofthecumulativeroundingerrorofanalgorithmasaTaylorexpansionofthelocalroundingerrors(Masters)(inFinnish).UniversityofHelsinki.pp. 6–7. ^Linnainmaa,Seppo(1976)."Taylorexpansionoftheaccumulatedroundingerror".BITNumericalMathematics.16(2):146–160.doi:10.1007/bf01931367.S2CID 122357351. ^Dreyfus,Stuart(1973)."Thecomputationalsolutionofoptimalcontrolproblemswithtimelag".IEEETransactionsonAutomaticControl.18(4):383–385.doi:10.1109/tac.1973.1100330. ^Werbos,Paul(1982)."Applicationsofadvancesinnonlinearsensitivityanalysis"(PDF).Systemmodelingandoptimization.Springer.pp. 762–770. ^Mead,CarverA.;Ismail,Mohammed(8May1989).AnalogVLSIImplementationofNeuralSystems(PDF).TheKluwerInternationalSeriesinEngineeringandComputerScience.Vol. 80.Norwell,MA:KluwerAcademicPublishers.doi:10.1007/978-1-4613-1639-8.ISBN 978-1-4613-1639-8. ^DavidE.Rumelhart,GeoffreyE.Hinton&RonaldJ.Williams,"Learningrepresentationsbyback-propagatingerrors,"Nature',323,pages533–5361986. ^J.Weng,N.AhujaandT.S.Huang,"Cresceptron:aself-organizingneuralnetworkwhichgrowsadaptively,"Proc.InternationalJointConferenceonNeuralNetworks,Baltimore,Maryland,volI,pp.576–581,June1992. ^J.Weng,N.AhujaandT.S.Huang,"Learningrecognitionandsegmentationof3-Dobjectsfrom2-Dimages,"Proc.4thInternationalConf.ComputerVision,Berlin,Germany,pp.121–128,May1993. ^J.Weng,N.AhujaandT.S.Huang,"LearningrecognitionandsegmentationusingtheCresceptron,"InternationalJournalofComputerVision,vol.25,no.2,pp.105–139,Nov.1997. ^J.Schmidhuber.,"Learningcomplex,extendedsequencesusingtheprincipleofhistorycompression,"NeuralComputation,4,pp.234–242,1992. ^Domingos,Pedro(22September2015).TheMasterAlgorithm:HowtheQuestfortheUltimateLearningMachineWillRemakeOurWorld.chapter4:BasicBooks.ISBN 978-0465065707.{{citebook}}:CS1maint:location(link) ^Smolensky,P.(1986)."Informationprocessingindynamicalsystems:Foundationsofharmonytheory.".InD.E.Rumelhart;J.L.McClelland;PDPResearchGroup(eds.).ParallelDistributedProcessing:ExplorationsintheMicrostructureofCognition.Vol. 1.pp. 194–281.ISBN 978-0-262-68053-0. ^Ng,Andrew;Dean,Jeff(2012)."BuildingHigh-levelFeaturesUsingLargeScaleUnsupervisedLearning".arXiv:1112.6209[cs.LG]. ^IanGoodfellowandYoshuaBengioandAaronCourville(2016).DeepLearning.MITPress. ^Cireşan,DanClaudiu;Meier,Ueli;Gambardella,LucaMaria;Schmidhuber,Jürgen(21September2010)."Deep,Big,SimpleNeuralNetsforHandwrittenDigitRecognition".NeuralComputation.22(12):3207–3220.arXiv:1003.0358.doi:10.1162/neco_a_00052.ISSN 0899-7667.PMID 20858131.S2CID 1918673. ^DominikScherer,AndreasC.Müller,andSvenBehnke:"EvaluationofPoolingOperationsinConvolutionalArchitecturesforObjectRecognition,"In20thInternationalConferenceArtificialNeuralNetworks(ICANN),pp.92–101,2010.doi:10.1007/978-3-642-15825-4_10. ^2012KurzweilAIInterviewArchived31August2018attheWaybackMachinewithJürgenSchmidhuberontheeightcompetitionswonbyhisDeepLearningteam2009–2012 ^"Howbio-inspireddeeplearningkeepswinningcompetitions|KurzweilAI".www.kurzweilai.net.Archivedfromtheoriginalon31August2018.Retrieved16June2017. ^Graves,Alex;andSchmidhuber,Jürgen;OfflineHandwritingRecognitionwithMultidimensionalRecurrentNeuralNetworks,inBengio,Yoshua;Schuurmans,Dale;Lafferty,John;Williams,ChrisK.I.;andCulotta,Aron(eds.),AdvancesinNeuralInformationProcessingSystems22(NIPS'22),7–10December2009,Vancouver,BC,NeuralInformationProcessingSystems(NIPS)Foundation,2009,pp.545–552. ^abGraves,A.;Liwicki,M.;Fernandez,S.;Bertolami,R.;Bunke,H.;Schmidhuber,J.(2009)."ANovelConnectionistSystemforImprovedUnconstrainedHandwritingRecognition"(PDF).IEEETransactionsonPatternAnalysisandMachineIntelligence.31(5):855–868.CiteSeerX 10.1.1.139.4502.doi:10.1109/tpami.2008.137.PMID 19299860.S2CID 14635907. ^abGraves,Alex;Schmidhuber,Jürgen(2009).Bengio,Yoshua;Schuurmans,Dale;Lafferty,John;Williams,Chriseditor-K.I.;Culotta,Aron(eds.)."OfflineHandwritingRecognitionwithMultidimensionalRecurrentNeuralNetworks".NeuralInformationProcessingSystems(NIPS)Foundation.CurranAssociates,Inc.21:545–552.{{citejournal}}:|editor-first4=hasgenericname(help) ^Graves,A.;Liwicki,M.;Fernández,S.;Bertolami,R.;Bunke,H.;Schmidhuber,J.(May2009)."ANovelConnectionistSystemforUnconstrainedHandwritingRecognition".IEEETransactionsonPatternAnalysisandMachineIntelligence.31(5):855–868.CiteSeerX 10.1.1.139.4502.doi:10.1109/tpami.2008.137.ISSN 0162-8828.PMID 19299860.S2CID 14635907. ^Ciresan,Dan;Meier,U.;Schmidhuber,J.(June2012).Multi-columndeepneuralnetworksforimageclassification.2012IEEEConferenceonComputerVisionandPatternRecognition.pp. 3642–3649.arXiv:1202.2745.Bibcode:2012arXiv1202.2745C.CiteSeerX 10.1.1.300.3283.doi:10.1109/cvpr.2012.6248110.ISBN 978-1-4673-1228-8.S2CID 2161592. ^abZell,Andreas(2003)."chapter5.2".SimulationneuronalerNetze[SimulationofNeuralNetworks](inGerman)(1st ed.).Addison-Wesley.ISBN 978-3-89319-554-1.OCLC 249017987. ^Artificialintelligence(3rd ed.).Addison-WesleyPub.Co.1992.ISBN 0-201-53377-4. ^Abbod,MaysamF(2007)."ApplicationofArtificialIntelligencetotheManagementofUrologicalCancer".TheJournalofUrology.178(4):1150–1156.doi:10.1016/j.juro.2007.05.122.PMID 17698099. ^DAWSON,CHRISTIANW(1998)."Anartificialneuralnetworkapproachtorainfall-runoffmodelling".HydrologicalSciencesJournal.43(1):47–66.doi:10.1080/02626669809492102. ^"TheMachineLearningDictionary".www.cse.unsw.edu.au.Archivedfromtheoriginalon26August2018.Retrieved4November2009. ^Ciresan,Dan;UeliMeier;JonathanMasci;LucaM.Gambardella;JurgenSchmidhuber(2011)."Flexible,HighPerformanceConvolutionalNeuralNetworksforImageClassification"(PDF).ProceedingsoftheTwenty-SecondInternationalJointConferenceonArtificialIntelligence-VolumeVolumeTwo.2:1237–1242.Retrieved17November2013. ^Zell,Andreas(1994).SimulationNeuronalerNetze[SimulationofNeuralNetworks](inGerman)(1st ed.).Addison-Wesley.p. 73.ISBN 3-89319-554-8. ^Miljanovic,Milos(February–March2012)."ComparativeanalysisofRecurrentandFiniteImpulseResponseNeuralNetworksinTimeSeriesPrediction"(PDF).IndianJournalofComputerandEngineering.3(1). ^Lau,Suki(10July2017)."AWalkthroughofConvolutionalNeuralNetwork–HyperparameterTuning".Medium.Retrieved23August2019. ^Kelleher,JohnD.(2020)."7-8".Fundamentalsofmachinelearningforpredictivedataanalytics :algorithms,workedexamples,andcasestudies.BrianMacNamee,AoifeD'Arcy(2 ed.).Cambridge,Massachusetts.ISBN 978-0-262-36110-1.OCLC 1162184998. ^Wei,Jiakai(26April2019)."ForgettheLearningRate,DecayLoss".arXiv:1905.00094[cs.LG]. ^Li,Y.;Fu,Y.;Li,H.;Zhang,S.W.(1June2009).TheImprovedTrainingAlgorithmofBackPropagationNeuralNetworkwithSelf-adaptiveLearningRate.2009InternationalConferenceonComputationalIntelligenceandNaturalComputing.Vol. 1.pp. 73–76.doi:10.1109/CINC.2009.111.ISBN 978-0-7695-3645-3.S2CID 10557754. ^Huang,Guang-Bin;Zhu,Qin-Yu;Siew,Chee-Kheong(2006)."Extremelearningmachine:theoryandapplications".Neurocomputing.70(1):489–501.CiteSeerX 10.1.1.217.3692.doi:10.1016/j.neucom.2005.12.126. ^Widrow,Bernard;et al.(2013)."Theno-propalgorithm:Anewlearningalgorithmformultilayerneuralnetworks".NeuralNetworks.37:182–188.doi:10.1016/j.neunet.2012.09.020.PMID 23140797. ^Ollivier,Yann;Charpiat,Guillaume(2015)."Trainingrecurrentnetworkswithoutbacktracking".arXiv:1507.07680[cs.NE]. ^ESANN.2009 ^Hinton,G.E.(2010)."APracticalGuidetoTrainingRestrictedBoltzmannMachines".Tech.Rep.UTMLTR2010-003. ^Ojha,VarunKumar;Abraham,Ajith;Snášel,Václav(1April2017)."Metaheuristicdesignoffeedforwardneuralnetworks:Areviewoftwodecadesofresearch".EngineeringApplicationsofArtificialIntelligence.60:97–116.arXiv:1705.05584.Bibcode:2017arXiv170505584O.doi:10.1016/j.engappai.2017.01.013.S2CID 27910748. ^Dominic,S.;Das,R.;Whitley,D.;Anderson,C.(July1991)."Geneticreinforcementlearningforneuralnetworks".IJCNN-91-SeattleInternationalJointConferenceonNeuralNetworks.IJCNN-91-SeattleInternationalJointConferenceonNeuralNetworks.Seattle,Washington,USA:IEEE.doi:10.1109/IJCNN.1991.155315.ISBN 0-7803-0164-1. ^Hoskins,J.C.;Himmelblau,D.M.(1992)."Processcontrolviaartificialneuralnetworksandreinforcementlearning".Computers&ChemicalEngineering.16(4):241–251.doi:10.1016/0098-1354(92)80045-B. ^Bertsekas,D.P.;Tsitsiklis,J.N.(1996).Neuro-dynamicprogramming.AthenaScientific.p. 512.ISBN 978-1-886529-10-6. ^Secomandi,Nicola(2000)."Comparingneuro-dynamicprogrammingalgorithmsforthevehicleroutingproblemwithstochasticdemands".Computers&OperationsResearch.27(11–12):1201–1225.CiteSeerX 10.1.1.392.4034.doi:10.1016/S0305-0548(99)00146-X. ^deRigo,D.;Rizzoli,A.E.;Soncini-Sessa,R.;Weber,E.;Zenesi,P.(2001)."Neuro-dynamicprogrammingfortheefficientmanagementofreservoirnetworks".ProceedingsofMODSIM2001,InternationalCongressonModellingandSimulation.MODSIM2001,InternationalCongressonModellingandSimulation.Canberra,Australia:ModellingandSimulationSocietyofAustraliaandNewZealand.doi:10.5281/zenodo.7481.ISBN 0-86740-525-2. ^Damas,M.;Salmeron,M.;Diaz,A.;Ortega,J.;Prieto,A.;Olivares,G.(2000)."Geneticalgorithmsandneuro-dynamicprogramming:applicationtowatersupplynetworks".Proceedingsof2000CongressonEvolutionaryComputation.2000CongressonEvolutionaryComputation.LaJolla,California,USA:IEEE.doi:10.1109/CEC.2000.870269.ISBN 0-7803-6375-2. ^Deng,Geng;Ferris,M.C.(2008).Neuro-dynamicprogrammingforfractionatedradiotherapyplanning.SpringerOptimizationandItsApplications.Vol. 12.pp. 47–70.CiteSeerX 10.1.1.137.8288.doi:10.1007/978-0-387-73299-2_3.ISBN 978-0-387-73298-5. ^Bozinovski,S.(1982)."Aself-learningsystemusingsecondaryreinforcement".InR.Trappl(ed.)CyberneticsandSystemsResearch:ProceedingsoftheSixthEuropeanMeetingonCyberneticsandSystemsResearch.NorthHolland.pp.397–402.ISBN 978-0-444-86488-8. ^Bozinovski,S.(2014)"Modelingmechanismsofcognition-emotioninteractioninartificialneuralnetworks,since1981."ProcediaComputerSciencep.255-263 ^Bozinovski,Stevo;Bozinovska,Liljana(2001)."Self-learningagents:Aconnectionisttheoryofemotionbasedoncrossbarvaluejudgment".CyberneticsandSystems.32(6):637–667.doi:10.1080/01969720118145.S2CID 8944741. ^"Artificialintelligencecan'evolve'tosolveproblems".Science|AAAS.10January2018.Retrieved7February2018. ^Turchetti,Claudio(2004),StochasticModelsofNeuralNetworks,Frontiersinartificialintelligenceandapplications:Knowledge-basedintelligentengineeringsystems,vol. 102,IOSPress,ISBN 9781586033880 ^deRigo,D.;Castelletti,A.;Rizzoli,A.E.;Soncini-Sessa,R.;Weber,E.(January2005)."AselectiveimprovementtechniqueforfasteningNeuro-DynamicProgramminginWaterResourcesNetworkManagement".InPavelZítek(ed.).Proceedingsofthe16thIFACWorldCongress–IFAC-PapersOnLine.16thIFACWorldCongress.Vol. 16.Prague,CzechRepublic:IFAC.doi:10.3182/20050703-6-CZ-1902.02172.hdl:11311/255236.ISBN 978-3-902661-75-3.Retrieved30December2011. ^Ferreira,C.(2006)."DesigningNeuralNetworksUsingGeneExpressionProgramming".InA.Abraham;B.deBaets;M.Köppen;B.Nickolay(eds.).AppliedSoftComputingTechnologies:TheChallengeofComplexity(PDF).Springer-Verlag.pp. 517–536. ^Da,Y.;Xiurun,G.(July2005)."AnimprovedPSO-basedANNwithsimulatedannealingtechnique".InT.Villmann(ed.).NewAspectsinNeurocomputing:11thEuropeanSymposiumonArtificialNeuralNetworks.Elsevier.doi:10.1016/j.neucom.2004.07.002.Archivedfromtheoriginalon25April2012.Retrieved30December2011. ^Wu,J.;Chen,E.(May2009)."ANovelNonparametricRegressionEnsembleforRainfallForecastingUsingParticleSwarmOptimizationTechniqueCoupledwithArtificialNeuralNetwork".InWang,H.;Shen,Y.;Huang,T.;Zeng,Z.(eds.).6thInternationalSymposiumonNeuralNetworks,ISNN2009.Springer.doi:10.1007/978-3-642-01513-7_6.ISBN 978-3-642-01215-0.Archivedfromtheoriginalon31December2014.Retrieved1January2012. ^abTingQin,etal."AlearningalgorithmofCMACbasedonRLS."NeuralProcessingLetters19.1(2004):49–61. ^TingQin,etal."ContinuousCMAC-QRLSanditssystolicarray."NeuralProcessingLetters22.1(2005):1–16. ^LeCunetal.,"BackpropagationAppliedtoHandwrittenZipCodeRecognition,"NeuralComputation,1,pp.541–551,1989. ^YannLeCun(2016).SlidesonDeepLearningOnline ^Hochreiter,Sepp;Schmidhuber,Jürgen(1November1997)."LongShort-TermMemory".NeuralComputation.9(8):1735–1780.doi:10.1162/neco.1997.9.8.1735.ISSN 0899-7667.PMID 9377276.S2CID 1915014. ^Sak,Hasim;Senior,Andrew;Beaufays,Francoise(2014)."LongShort-TermMemoryrecurrentneuralnetworkarchitecturesforlargescaleacousticmodeling"(PDF).Archivedfromtheoriginal(PDF)on24April2018. ^Li,Xiangang;Wu,Xihong(15October2014)."ConstructingLongShort-TermMemorybasedDeepRecurrentNeuralNetworksforLargeVocabularySpeechRecognition".arXiv:1410.4281[cs.CL]. ^Fan,Y.;Qian,Y.;Xie,F.;Soong,F.K.(2014)."TTSsynthesiswithbidirectionalLSTMbasedRecurrentNeuralNetworks".ProceedingsoftheAnnualConferenceoftheInternationalSpeechCommunicationAssociation,Interspeech:1964–1968.Retrieved13June2017. ^Zen,Heiga;Sak,Hasim(2015)."UnidirectionalLongShort-TermMemoryRecurrentNeuralNetworkwithRecurrentOutputLayerforLow-LatencySpeechSynthesis"(PDF).Google.com.ICASSP.pp. 4470–4474. ^Fan,Bo;Wang,Lijuan;Soong,FrankK.;Xie,Lei(2015)."Photo-RealTalkingHeadwithDeepBidirectionalLSTM"(PDF).ProceedingsofICASSP. ^Silver,David;Hubert,Thomas;Schrittwieser,Julian;Antonoglou,Ioannis;Lai,Matthew;Guez,Arthur;Lanctot,Marc;Sifre,Laurent;Kumaran,Dharshan;Graepel,Thore;Lillicrap,Timothy;Simonyan,Karen;Hassabis,Demis(5December2017)."MasteringChessandShogibySelf-PlaywithaGeneralReinforcementLearningAlgorithm".arXiv:1712.01815[cs.AI]. ^Goodfellow,Ian;Pouget-Abadie,Jean;Mirza,Mehdi;Xu,Bing;Warde-Farley,David;Ozair,Sherjil;Courville,Aaron;Bengio,Yoshua(2014).GenerativeAdversarialNetworks(PDF).ProceedingsoftheInternationalConferenceonNeuralInformationProcessingSystems(NIPS2014).pp. 2672–2680. ^Zoph,Barret;Le,QuocV.(4November2016)."NeuralArchitectureSearchwithReinforcementLearning".arXiv:1611.01578[cs.LG]. ^"AutoKeras".autokeras.com.Retrieved21August2019. ^"Claesen,Marc,andBartDeMoor."HyperparameterSearchinMachineLearning."arXivpreprintarXiv:1502.02127(2015)".arXiv:1502.02127.Bibcode:2015arXiv150202127C. ^Turek,FredD.(March2007)."IntroductiontoNeuralNetMachineVision".VisionSystemsDesign.12(3).Retrieved5March2013. ^Zissis,Dimitrios(October2015)."Acloudbasedarchitecturecapableofperceivingandpredictingmultiplevesselbehaviour".AppliedSoftComputing.35:652–661.doi:10.1016/j.asoc.2015.07.002. ^RomanM.Balabin;EkaterinaI.Lomakina(2009)."Neuralnetworkapproachtoquantum-chemistrydata:Accuratepredictionofdensityfunctionaltheoryenergies".J.Chem.Phys.131(7):074104.Bibcode:2009JChPh.131g4104B.doi:10.1063/1.3206326.PMID 19708729. ^Silver,David;et al.(2016)."MasteringthegameofGowithdeepneuralnetworksandtreesearch"(PDF).Nature.529(7587):484–9.Bibcode:2016Natur.529..484S.doi:10.1038/nature16961.PMID 26819042.S2CID 515925. ^Sengupta,Nandini;Sahidullah,Md;Saha,Goutam(August2016)."Lungsoundclassificationusingcepstral-basedstatisticalfeatures".ComputersinBiologyandMedicine.75(1):118–129.doi:10.1016/j.compbiomed.2016.05.013.PMID 27286184. ^Choy,ChristopherB.,etal."3d-r2n2:Aunifiedapproachforsingleandmulti-view3dobjectreconstruction."Europeanconferenceoncomputervision.Springer,Cham,2016. ^Gessler,Josef(August2021)."Sensorforfoodanalysisapplyingimpedancespectroscopyandartificialneuralnetworks".RiuNetUPV(1):8–12. ^Maitra,D.S.;Bhattacharya,U.;Parui,S.K.(August2015)."CNNbasedcommonapproachtohandwrittencharacterrecognitionofmultiplescripts".201513thInternationalConferenceonDocumentAnalysisandRecognition(ICDAR):1021–1025.doi:10.1109/ICDAR.2015.7333916.ISBN 978-1-4799-1805-8.S2CID 25739012. ^French,Jordan(2016)."Thetimetraveller'sCAPM".InvestmentAnalystsJournal.46(2):81–96.doi:10.1080/10293523.2016.1255469.S2CID 157962452. ^Schechner,Sam(15June2017)."FacebookBoostsA.I.toBlockTerroristPropaganda".WallStreetJournal.ISSN 0099-9660.Retrieved16June2017. ^Ganesan,N(2010)."ApplicationofNeuralNetworksinDiagnosingCancerDiseaseUsingDemographicData".InternationalJournalofComputerApplications.1(26):81–97.Bibcode:2010IJCA....1z..81G.doi:10.5120/476-783. ^Bottaci,Leonardo(1997)."ArtificialNeuralNetworksAppliedtoOutcomePredictionforColorectalCancerPatientsinSeparateInstitutions"(PDF).Lancet.TheLancet.350(9076):469–72.doi:10.1016/S0140-6736(96)11196-X.PMID 9274582.S2CID 18182063.Archivedfromtheoriginal(PDF)on23November2018.Retrieved2May2012. ^Alizadeh,Elaheh;Lyons,SamantheM;Castle,JordanM;Prasad,Ashok(2016)."MeasuringsystematicchangesininvasivecancercellshapeusingZernikemoments".IntegrativeBiology.8(11):1183–1193.doi:10.1039/C6IB00100A.PMID 27735002. ^Lyons,Samanthe(2016)."Changesincellshapearecorrelatedwithmetastaticpotentialinmurine".BiologyOpen.5(3):289–299.doi:10.1242/bio.013409.PMC 4810736.PMID 26873952. ^Nabian,MohammadAmin;Meidani,Hadi(28August2017)."DeepLearningforAcceleratedReliabilityAnalysisofInfrastructureNetworks".Computer-AidedCivilandInfrastructureEngineering.33(6):443–458.arXiv:1708.08551.Bibcode:2017arXiv170808551N.doi:10.1111/mice.12359.S2CID 36661983. ^Nabian,MohammadAmin;Meidani,Hadi(2018)."AcceleratingStochasticAssessmentofPost-EarthquakeTransportationNetworkConnectivityviaMachine-Learning-BasedSurrogates".TransportationResearchBoard97thAnnualMeeting. ^Díaz,E.;Brotons,V.;Tomás,R.(September2018)."Useofartificialneuralnetworkstopredict3-Delasticsettlementoffoundationsonsoilswithinclinedbedrock".SoilsandFoundations.58(6):1414–1422.doi:10.1016/j.sandf.2018.08.001.hdl:10045/81208.ISSN 0038-0806. ^Govindaraju,RaoS.(1April2000)."ArtificialNeuralNetworksinHydrology.I:PreliminaryConcepts".JournalofHydrologicEngineering.5(2):115–123.doi:10.1061/(ASCE)1084-0699(2000)5:2(115). ^Govindaraju,RaoS.(1April2000)."ArtificialNeuralNetworksinHydrology.II:HydrologicApplications".JournalofHydrologicEngineering.5(2):124–137.doi:10.1061/(ASCE)1084-0699(2000)5:2(124). ^Peres,D.J.;Iuppa,C.;Cavallaro,L.;Cancelliere,A.;Foti,E.(1October2015)."Significantwaveheightrecordextensionbyneuralnetworksandreanalysiswinddata".OceanModelling.94:128–140.Bibcode:2015OcMod..94..128P.doi:10.1016/j.ocemod.2015.08.002. ^Dwarakish,G.S.;Rakshith,Shetty;Natesan,Usha(2013)."ReviewonApplicationsofNeuralNetworkinCoastalEngineering".ArtificialIntelligentSystemsandMachineLearning.5(7):324–331. ^Ermini,Leonardo;Catani,Filippo;Casagli,Nicola(1March2005)."ArtificialNeuralNetworksappliedtolandslidesusceptibilityassessment".Geomorphology.Geomorphologicalhazardandhumanimpactinmountainenvironments.66(1):327–343.Bibcode:2005Geomo..66..327E.doi:10.1016/j.geomorph.2004.09.025. ^Nix,R.;Zhang,J.(May2017)."ClassificationofAndroidappsandmalwareusingdeepneuralnetworks".2017InternationalJointConferenceonNeuralNetworks(IJCNN):1871–1878.doi:10.1109/IJCNN.2017.7966078.ISBN 978-1-5090-6182-2.S2CID 8838479. ^"DetectingMaliciousURLs".ThesystemsandnetworkinggroupatUCSD.Archivedfromtheoriginalon14July2019.Retrieved15February2019. ^Homayoun,Sajad;Ahmadzadeh,Marzieh;Hashemi,Sattar;Dehghantanha,Ali;Khayami,Raouf(2018),Dehghantanha,Ali;Conti,Mauro;Dargahi,Tooska(eds.),"BoTShark:ADeepLearningApproachforBotnetTrafficDetection",CyberThreatIntelligence,AdvancesinInformationSecurity,SpringerInternationalPublishing,pp. 137–153,doi:10.1007/978-3-319-73951-9_7,ISBN 978-3-319-73951-9 ^and(January1994)."Creditcardfrauddetectionwithaneural-network".1994ProceedingsoftheTwenty-SeventhHawaiiInternationalConferenceonSystemSciences.3:621–630.doi:10.1109/HICSS.1994.323314.ISBN 978-0-8186-5090-1.S2CID 13260377. ^Ananthaswamy,Anil(19April2021)."LatestNeuralNetsSolveWorld'sHardestEquationsFasterThanEverBefore".QuantaMagazine.Retrieved12May2021. ^"AIhascrackedakeymathematicalpuzzleforunderstandingourworld".MITTechnologyReview.Retrieved19November2020. ^"CaltechOpen-SourcesAIforSolvingPartialDifferentialEquations".InfoQ.Retrieved20January2021. ^Nagy,Alexandra(28June2019)."VariationalQuantumMonteCarloMethodwithaNeural-NetworkAnsatzforOpenQuantumSystems".PhysicalReviewLetters.122(25):250501.arXiv:1902.09483.Bibcode:2019PhRvL.122y0501N.doi:10.1103/PhysRevLett.122.250501.PMID 31347886.S2CID 119074378. ^Yoshioka,Nobuyuki;Hamazaki,Ryusuke(28June2019)."Constructingneuralstationarystatesforopenquantummany-bodysystems".PhysicalReviewB.99(21):214306.arXiv:1902.07006.Bibcode:2019arXiv190207006Y.doi:10.1103/PhysRevB.99.214306.S2CID 119470636. ^Hartmann,MichaelJ.;Carleo,Giuseppe(28June2019)."Neural-NetworkApproachtoDissipativeQuantumMany-BodyDynamics".PhysicalReviewLetters.122(25):250502.arXiv:1902.05131.Bibcode:2019arXiv190205131H.doi:10.1103/PhysRevLett.122.250502.PMID 31347862.S2CID 119357494. ^Vicentini,Filippo;Biella,Alberto;Regnault,Nicolas;Ciuti,Cristiano(28June2019)."VariationalNeural-NetworkAnsatzforSteadyStatesinOpenQuantumSystems".PhysicalReviewLetters.122(25):250503.arXiv:1902.10104.Bibcode:2019arXiv190210104V.doi:10.1103/PhysRevLett.122.250503.PMID 31347877.S2CID 119504484. ^ForrestMD(April2015)."SimulationofalcoholactionuponadetailedPurkinjeneuronmodelandasimplersurrogatemodelthatruns>400timesfaster".BMCNeuroscience.16(27):27.doi:10.1186/s12868-015-0162-6.PMC 4417229.PMID 25928094. ^Siegelmann,H.T.;Sontag,E.D.(1991)."Turingcomputabilitywithneuralnets"(PDF).Appl.Math.Lett.4(6):77–80.doi:10.1016/0893-9659(91)90080-F. ^Balcázar,José(July1997)."ComputationalPowerofNeuralNetworks:AKolmogorovComplexityCharacterization".IEEETransactionsonInformationTheory.43(4):1175–1183.CiteSeerX 10.1.1.411.7782.doi:10.1109/18.605580. ^abMacKay,David,J.C.(2003).InformationTheory,Inference,andLearningAlgorithms(PDF).CambridgeUniversityPress.ISBN 978-0-521-64298-9. ^Cover,Thomas(1965)."GeometricalandStatisticalPropertiesofSystemsofLinearInequalitieswithApplicationsinPatternRecognition"(PDF).IEEETransactionsonElectronicComputers.IEEE.EC-14(3):326–334.doi:10.1109/PGEC.1965.264137. ^Gerald,Friedland(2019)."ReproducibilityandExperimentalDesignforMachineLearningonAudioandMultimediaData".MM'19:Proceedingsofthe27thACMInternationalConferenceonMultimedia.ACM:2709–2710.doi:10.1145/3343031.3350545.ISBN 978-1-4503-6889-6.S2CID 204837170. ^"TheTensorflowMeter". ^Lee,Jaehoon;Xiao,Lechao;Schoenholz,SamuelS.;Bahri,Yasaman;Novak,Roman;Sohl-Dickstein,Jascha;Pennington,Jeffrey(2020)."Wideneuralnetworksofanydepthevolveaslinearmodelsundergradientdescent".JournalofStatisticalMechanics:TheoryandExperiment.2020(12):124002.arXiv:1902.06720.Bibcode:2020JSMTE2020l4002L.doi:10.1088/1742-5468/abc62b.S2CID 62841516. ^[1],NeuralTangentKernel:ConvergenceandGeneralizationinNeuralNetworks. ^[2],TrainingBehaviorofDeepNeuralNetworkinFrequencyDomain. ^[3],OntheSpectralBiasofNeuralNetworks. ^[4],FrequencyPrinciple:FourierAnalysisShedsLightonDeepNeuralNetworks. ^[5],TheoryoftheFrequencyPrincipleforGeneralDeepNeuralNetworks. ^Xu,ZhiqinJohn;Zhou,Hanxu(18May2021)."DeepFrequencyPrincipleTowardsUnderstandingWhyDeeperLearningIsFaster".ProceedingsoftheAAAIConferenceonArtificialIntelligence.35(12):10541–10550.arXiv:2007.14313.ISSN 2374-3468. ^Crick,Francis(1989)."Therecentexcitementaboutneuralnetworks".Nature.337(6203):129–132.Bibcode:1989Natur.337..129C.doi:10.1038/337129a0.PMID 2911347.S2CID 5892527. ^Adrian,EdwardD.(1926)."Theimpulsesproducedbysensorynerveendings".TheJournalofPhysiology.61(1):49–72.doi:10.1113/jphysiol.1926.sp002273.PMC 1514809.PMID 16993776. ^Dewdney,A.K.(1April1997).Yes,wehavenoneutrons:aneye-openingtourthroughthetwistsandturnsofbadscience.Wiley.p. 82.ISBN 978-0-471-10806-1. ^NASA–DrydenFlightResearchCenter–NewsRoom:NewsReleases:NASANEURALNETWORKPROJECTPASSESMILESTONE.Nasa.gov.Retrievedon20November2013. ^"RogerBridgman'sdefenceofneuralnetworks".Archivedfromtheoriginalon19March2012.Retrieved12July2010. ^D.J.FellemanandD.C.VanEssen,"Distributedhierarchicalprocessingintheprimatecerebralcortex,"CerebralCortex,1,pp.1–47,1991. ^J.Weng,"NaturalandArtificialIntelligence:IntroductiontoComputationalBrain-Mind,"BMIPress,ISBN 978-0-9858757-2-5,2012. ^abEdwards,Chris(25June2015)."Growingpainsfordeeplearning".CommunicationsoftheACM.58(7):14–16.doi:10.1145/2771283.S2CID 11026540. ^CadeMetz(18May2016)."GoogleBuiltItsVeryOwnChipstoPowerItsAIBots".Wired. ^"ScalingLearningAlgorithmstowards{AI}–LISA–Publications–Aigaion2.0".www.iro.umontreal.ca. ^SunandBookman(1990) ^Tahmasebi;Hezarkhani(2012)."Ahybridneuralnetworks-fuzzylogic-geneticalgorithmforgradeestimation".Computers&Geosciences.42:18–27.Bibcode:2012CG.....42...18T.doi:10.1016/j.cageo.2012.02.004.PMC 4268588.PMID 25540468. Bibliography[edit] BhadeshiaH.K.D.H.(1999)."NeuralNetworksinMaterialsScience"(PDF).ISIJInternational.39(10):966–979.doi:10.2355/isijinternational.39.966. Bishop,ChristopherM.(1995).Neuralnetworksforpatternrecognition.ClarendonPress.ISBN 978-0-19-853849-3.OCLC 33101074. Borgelt,Christian(2003).Neuro-Fuzzy-Systeme :vondenGrundlagenkünstlicherNeuronalerNetzezurKopplungmitFuzzy-Systemen.Vieweg.ISBN 978-3-528-25265-6.OCLC 76538146. Cybenko,G.V.(2006)."ApproximationbySuperpositionsofaSigmoidalfunction".InvanSchuppen,JanH.(ed.).MathematicsofControl,Signals,andSystems.SpringerInternational.pp. 303–314.PDF Dewdney,A.K.(1997).Yes,wehavenoneutrons :aneye-openingtourthroughthetwistsandturnsofbadscience.NewYork:Wiley.ISBN 978-0-471-10806-1.OCLC 35558945. Duda,RichardO.;Hart,PeterElliot;Stork,DavidG.(2001).Patternclassification(2 ed.).Wiley.ISBN 978-0-471-05669-0.OCLC 41347061. Egmont-Petersen,M.;deRidder,D.;Handels,H.(2002)."Imageprocessingwithneuralnetworks–areview".PatternRecognition.35(10):2279–2301.CiteSeerX 10.1.1.21.5444.doi:10.1016/S0031-3203(01)00178-9. Fahlman,S.;Lebiere,C(1991)."TheCascade-CorrelationLearningArchitecture"(PDF). createdforNationalScienceFoundation,ContractNumberEET-8716324,andDefenseAdvancedResearchProjectsAgency(DOD),ARPAOrderNo.4976underContractF33615-87-C-1499. Gurney,Kevin(1997).Anintroductiontoneuralnetworks.UCLPress.ISBN 978-1-85728-673-1.OCLC 37875698. Haykin,SimonS.(1999).Neuralnetworks :acomprehensivefoundation.PrenticeHall.ISBN 978-0-13-273350-2.OCLC 38908586. Hertz,J.;Palmer,RichardG.;Krogh,AndersS.(1991).Introductiontothetheoryofneuralcomputation.Addison-Wesley.ISBN 978-0-201-51560-2.OCLC 21522159. Informationtheory,inference,andlearningalgorithms.CambridgeUniversityPress.25September2003.Bibcode:2003itil.book.....M.ISBN 978-0-521-64298-9.OCLC 52377690. Kruse,Rudolf;Borgelt,Christian;Klawonn,F.;Moewes,Christian;Steinbrecher,Matthias;Held,Pascal(2013).Computationalintelligence :amethodologicalintroduction.Springer.ISBN 978-1-4471-5012-1.OCLC 837524179. Lawrence,Jeanette(1994).Introductiontoneuralnetworks :design,theoryandapplications.CaliforniaScientificSoftware.ISBN 978-1-883157-00-5.OCLC 32179420. MacKay,David,J.C.(2003).InformationTheory,Inference,andLearningAlgorithms(PDF).CambridgeUniversityPress.ISBN 978-0-521-64298-9. Masters,Timothy(1994).Signalandimageprocessingwithneuralnetworks :aC++sourcebook.J.Wiley.ISBN 978-0-471-04963-0.OCLC 29877717. Ripley,BrianD.(2007).PatternRecognitionandNeuralNetworks.CambridgeUniversityPress.ISBN 978-0-521-71770-0. Siegelmann,H.T.;Sontag,EduardoD.(1994)."Analogcomputationvianeuralnetworks".TheoreticalComputerScience.131(2):331–360.doi:10.1016/0304-3975(94)90178-3.S2CID 2456483. Smith,Murray(1993).Neuralnetworksforstatisticalmodeling.VanNostrandReinhold.ISBN 978-0-442-01310-3.OCLC 27145760. Wasserman,PhilipD.(1993).Advancedmethodsinneuralcomputing.VanNostrandReinhold.ISBN 978-0-442-00461-3.OCLC 27429729. Wilson,Halsey(2018).Artificialintelligence.GreyHousePublishing.ISBN 978-1-68217-867-6. Externallinks[edit] TheNeuralNetworkZoo–acompilationofneuralnetworktypes TheStilwellBrain–aMindFieldepisodefeaturinganexperimentinwhichhumansactasindividualneuronsinaneuralnetworkthatclassifieshandwrittendigits vteComplexsystemsBackground Emergence Self-organization Collectivebehaviour Socialdynamics Collectiveintelligence Collectiveaction Collectiveconsciousness Self-organizedcriticality Herdmentality Phasetransition Agent-basedmodelling Synchronization Antcolonyoptimization Particleswarmoptimization Swarmbehaviour Evolutionandadaptation Artificialneuralnetwork Evolutionarycomputation Geneticalgorithms Geneticprogramming Artificiallife Machinelearning Evolutionarydevelopmentalbiology Artificialintelligence Evolutionaryrobotics Evolvability Gametheory Prisoner'sdilemma Rationalchoicetheory Boundedrationality Irrationalbehaviour Evolutionarygametheory Networks Socialnetworkanalysis Small-worldnetworks Communityidentification Centrality Motifs GraphTheory Scaling Robustness Systemsbiology Dynamicnetworks Adaptivenetworks Nonlineardynamics Timeseriesanalysis Ordinarydifferentialequations Iterativemaps Phasespace Attractor Stabilityanalysis Populationdynamics Chaos Multistability Bifurcation Coupledmaplattices Patternformation Spatialfractals Reaction-diffusionsystems Partialdifferentialequations Dissipativestructures Percolation Cellularautomata Spatialecology Self-replication Spatialevolutionarybiology Geomorphology Systemstheory Homeostasis Operationalization Feedback Self-reference Goal-oriented Systemdynamics Sensemaking Entropy Cybernetics Autopoiesis Informationtheory Computationtheory Complexitymeasurement vteControltheoryBranches Adaptivecontrol ControlTheory Digitalcontrol Energy-shapingcontrol Fuzzycontrol Hybridcontrol Intelligentcontrol Modelpredictivecontrol Multivariablecontrol Neuralcontrol Nonlinearcontrol Optimalcontrol Real-timecontrol Robustcontrol Stochasticcontrol SystemProperties Bodeplot Blockdiagram Closed-looptransferfunction Controllability Fouriertransform Frequencyresponse Laplacetransform Negativefeedback Observability Performance Positivefeedback RootLocusMethod Servomechanism Signal-flowgraph Statespacerepresentation StabilityTheory SteadyStateAnalysis&design SystemDynamics Transferfunction DigitalControl Discrete-timesignal Digitalsignalprocessing Quantization RealTimeSoftware SampledData Systemidentification ZTransform AdvancedTechniques Artificialneuralnetwork Coefficientdiagrammethod Controlreconfiguration Distributedparametersystems Fractional-ordercontrol Fuzzylogic H-infinityloop-shaping Hankelsingularvalue Kalmanfilter Krener'stheorem Leastsquares Lyapunovstability Minorloopfeedback Perceptualcontroltheory Stateobserver Vectorcontrol Controllers Embeddedcontroller Closed-loopcontroller Lead-lagcompensator Numericalcontrol PIDcontroller Programmablelogiccontroller ControlApplications AutomationandRemoteControl DistributedControlSystem Electricmotors IndustrialControlSystems Mechatronics Motioncontrol ProcessControl Robotics Supervisorycontrol(SCADA) vteDifferentiablecomputingGeneral Differentiableprogramming NeuralTuringmachine Differentiableneuralcomputer Automaticdifferentiation Neuromorphicengineering Cabletheory Patternrecognition Computationallearningtheory Tensorcalculus Concepts Gradientdescent SGD Clustering Regression Overfitting Adversary Attention Convolution Lossfunctions Backpropagation Normalization Activation Softmax Sigmoid Rectifier Regularization Datasets Augmentation Programminglanguages Python Julia Application Machinelearning Artificialneuralnetwork Deeplearning Scientificcomputing ArtificialIntelligence Hardware IPU TPU VPU Memristor SpiNNaker Softwarelibrary TensorFlow PyTorch Keras Theano ImplementationAudio-visual AlexNet WaveNet Humanimagesynthesis HWR OCR Speechsynthesis Speechrecognition Facialrecognition AlphaFold DALL-E Verbal Word2vec Transformer BERT NMT ProjectDebater Watson GPT-2 GPT-3 Decisional AlphaGo AlphaZero Q-learning SARSA OpenAIFive Self-drivingcar MuZero Actionselection Robotcontrol People AlexGraves IanGoodfellow YoshuaBengio GeoffreyHinton YannLeCun AndrewNg DemisHassabis DavidSilver Fei-FeiLi Organizations DeepMind OpenAI MITCSAIL Mila GoogleBrain FAIR Portals Computerprogramming Technology Category Artificialneuralnetworks Machinelearning vteNeuroscience Outlineofneuroscience Historyofneuroscience Basicscience Behavioralepigenetics Behavioralgenetics Brainmapping Brain-reading Cellularneuroscience Computationalneuroscience Connectomics Imaginggenetics Integrativeneuroscience Molecularneuroscience Neuraldecoding Neuralengineering Neuroanatomy Neurochemistry Neuroendocrinology Neurogenetics Neuroinformatics Neurometrics Neuromorphology Neurophysics Neurophysiology Systemsneuroscience Clinicalneuroscience Behavioralneurology Clinicalneurophysiology Neurocardiology Neuroepidemiology Neurogastroenterology Neuroimmunology Neurointensivecare Neurology Neurooncology Neuro-ophthalmology Neuropathology Neuropharmacology Neuroprosthetics Neuropsychiatry Neuroradiology Neurorehabilitation Neurosurgery Neurotology Neurovirology Nutritionalneuroscience Psychiatry Cognitiveneuroscience Affectiveneuroscience Behavioralneuroscience Chronobiology Molecularcellularcognition Motorcontrol Neurolinguistics Neuropsychology Sensoryneuroscience Socialcognitiveneuroscience Interdisciplinaryfields Consumerneuroscience Culturalneuroscience Educationalneuroscience Evolutionaryneuroscience Neuroanthropology Neurobioengineering Neurobiotics Neurocriminology Neuroeconomics Neuroepistemology Neuroesthetics Neuroethics Neuroethology Neurohistory Neurolaw Neuromarketing Neuromorphics Neurophenomenology Neurophilosophy Neuropolitics Neurorobotics Neurotheology Paleoneurobiology Socialneuroscience Concepts Brain–computerinterface Neuraldevelopment Neuralnetwork(artificial) Neuralnetwork(biological) Detectiontheory Intraoperativeneurophysiologicalmonitoring Neurochip Neurodegenerativedisease Neurodevelopmentaldisorder Neurodiversity Neurogenesis Neuroimaging Neuroimmunesystem Neuromanagement Neuromodulation Neuroplasticity Neurotechnology Neurotoxin Category vteSelf-drivingcarsandenablingtechnologiesOverviewandcontext Historyofself-drivingcars Intelligenttransportationsystem Context-awarepervasivesystems Mobilecomputing Smart,connectedproducts Ubiquitouscomputing Ambientintelligence Internetofthings SAELevelsHumandrivermonitorsthedrivingenvironment(Levels0,1,2) Lanedeparturewarningsystem Automaticparking Collisionavoidancesystem Cruisecontrol Adaptivecruisecontrol Advanceddriver-assistancesystems Driverdrowsinessdetection Intelligentspeedadaptation Blindspotmonitor Systemmonitorsthedrivingenvironment(Levels3,4,5) Vehicularadhocnetwork(V2V) Connectedcar Automotivenavigationsystem VehiclesCars VaMP(1994) SpiritofBerlin(2007) GeneralMotorsEN-V(2010) MadeInGermany(2011) Waymo,formerlyGoogleCar(2012) TeslaModelSwithAutopilot(2015) LUTZPathfinder(2015) Yandexself-drivingcar(2017) HondaLegend(2021) Busesandcommercialvehicles Automatedguidewaytransit ParkShuttle Naviashuttle NuTonomytaxi FreightlinerInspiration Driverlesstractor Mobilityasaservice Regulation Legislation IEEE802.11p Safespeedautomotivecommonlaw Automatedlanekeepingsystem(uneceregulation157) Regulation(EU)2019/2144 Enablingtechnologies Radar Laser LIDAR Artificialneuralnetwork Computerstereovision Imagerecognition Dedicatedshort-rangecommunications Real-timeControlSystem rFpro Eyetracking Radio-frequencyidentification Automotivenavigationsystem Organizations,Projects&PeopleOrganizations,projectsandevents AmericanCenterforMobility DAVI EuropeanLand-RobotTrial Navlab DARPAGrandChallenge VisLabIntercontinentalAutonomousChallenge EurekaPrometheusProject IEEEIntelligentTransportationSystemsSociety People HaroldGoddijn AlbertoBroggi AnthonyLevandowski Authoritycontrol:Nationallibraries Germany Israel UnitedStates Japan Retrievedfrom"https://en.wikipedia.org/w/index.php?title=Artificial_neural_network&oldid=1076558128" Categories:ComputationalstatisticsArtificialneuralnetworksClassificationalgorithmsComputationalneuroscienceMarketresearchMathematicalpsychologyMathematicalandquantitativemethods(economics)Hiddencategories:CS1Finnish-languagesources(fi)CS1maint:locationWebarchivetemplatewaybacklinksCS1errors:genericnameCS1German-languagesources(de)CS1:longvolumevalueArticleswithshortdescriptionShortdescriptionmatchesWikidataUsedmydatesfromJuly2021WikipediaarticlesneedingclarificationfromApril2017AllWikipediaarticlesneedingclarificationArticleslackingin-textcitationsfromAugust2019Allarticleslackingin-textcitationsAllarticleswithunsourcedstatementsArticleswithunsourcedstatementsfromJuly2019ArticlesneedingadditionalreferencesfromNovember2020AllarticlesneedingadditionalreferencesArticleswithunsourcedstatementsfromJune2017ArticleswithunsourcedstatementsfromNovember2014ArticleswithGNDidentifiersArticleswithJ9UidentifiersArticleswithLCCNidentifiersArticleswithNDLidentifiers Navigationmenu Personaltools NotloggedinTalkContributionsCreateaccountLogin Namespaces ArticleTalk English expanded collapsed Views ReadEditViewhistory More expanded collapsed Search Navigation MainpageContentsCurrenteventsRandomarticleAboutWikipediaContactusDonate Contribute HelpLearntoeditCommunityportalRecentchangesUploadfile Tools WhatlinkshereRelatedchangesUploadfileSpecialpagesPermanentlinkPageinformationCitethispageWikidataitem Print/export DownloadasPDFPrintableversion Inotherprojects WikimediaCommonsWikibooksWikiversity Languages العربيةAzərbaycancaবাংলাBân-lâm-gúБългарскиBosanskiCatalàČeštinaDanskDeutschEestiΕλληνικάEspañolEsperantoEuskaraفارسیFrançaisGaeilgeGalego한국어Հայերենहिन्दीHrvatskiBahasaIndonesiaÍslenskaItalianoעבריתქართულიLatinaLatviešuLietuviųMagyarМакедонскиMalagasyമലയാളംBahasaMelayuNederlands日本語NorskbokmålNorsknynorskଓଡ଼ିଆPolskiPortuguêsRomânăRunaSimiРусскийSimpleEnglishSlovenčinaСрпски/srpskiSuomiSvenskaதமிழ்ไทยTürkçeУкраїнськаاردوTiếngViệt吴语粵語中文 Editlinks