2020:Audio Classification (Train/Test) Tasks - MIREX Wiki

Audio Latin Music Genre Classification 2020:AudioClassification(Train/Test)TasksFromMIREXWikiJumpto:navigation,searchContents 1AudioClassification(Test/Train)tasks 1.1Description 1.1.1Taskspecificmailinglist 1.2Data 1.2.1AudioClassicalComposerIdentification 1.2.2AudioUSPopMusicGenreClassification 1.2.3AudioLatinMusicGenreClassification 1.2.4AudioMoodClassification 1.3AudioFormats 1.4Evaluation 1.5SubmissionFormat 1.5.1FileI/OFormat 1.5.1.1Featureextractionlistfile 1.5.1.2Traininglistfile 1.5.1.3Test(classification)listfile 1.5.1.4Classificationoutputfile 1.5.2Submissioncallingformats 1.5.2.1Examplesubmissioncallingformats 1.5.3Packagingsubmissions 1.5.4Timeandhardwarelimits 1.5.5PotentialParticipants AudioClassification(Test/Train)tasks Description Manytasksinmusicclassificationcanbecharacterizedintoatwo-stageprocess:trainingclassificationmodelsusinglabeleddataandtestingthemodelsusingnew/unseendata.Therefore,weproposethis"meta"taskwhichincludesvariousaudioclassificationtasksthatfollowthisTrain/Testprocess.ForMIREX2019,sixclassificationsub-tasksareincluded: AudioClassicalComposerIdentification AudioUSPopMusicGenreClassification AudioLatinMusicGenreClassification AudioMoodClassification AudioK-POPMoodClassification AudioK-POPGenreClassification AllsixclassificationtaskswereconductedinpreviousMIREXruns(pleaseseeMIREX2019Train-TestTaskResults).Thispagepresentstheevaluationresultsofthesetaskslastyear. Taskspecificmailinglist Inthepastwehaveuseaspecificmailinglistforthediscussionofthistaskandrelatedtasks.Thisyear,however,weareaskingthatalldiscussionstakeplaceontheMIREX"EvalFest"list.Ifyouhaveanquestionorcomment,simplyincludethetasknameinthesubjectheading. Data AudioClassicalComposerIdentification Thisdatasetrequiresalgorithmstoclassifymusicaudioaccordingtothecomposerofthetrack(drawnfromacollectionofperformancesofavarietyofclassicalmusicgenres).ThecollectionusedatMIREX2009willbere-used. Collectionstatistics: 277230-second22.05kHzmonowavclips 11"classical"composers(252clipspercomposer),including: Bach Beethoven Brahms Chopin Dvorak Handel Haydn Mendelssohnn Mozart Schubert Vivaldi AudioUSPopMusicGenreClassification Thisdatasetrequiresalgorithmstoclassifymusicaudioaccordingtothegenreofthetrack(drawnfromacollectionofUSPopmusictracks).TheMIREX2007Genredatasetwillbere-used,whichwasdrawnfromtheUSPOP2002andUSCRAPcollections. Collectionstatistics: 700030-secondaudioclipsin22.05kHzmonoWAVformat 10genres(700clipsfromeachgenre),including: Blues Jazz Country/Western Baroque Classical Romantic Electronica Hip-Hop Rock HardRock/Metal AudioLatinMusicGenreClassification Thisdatasetrequiresalgorithmstoclassifymusicaudioaccordingtothegenreofthetrack(drawnfromacollectionofLatinpopularanddancemusic,sourcedfromBrazilandhandlabeledbymusicexperts).CarlosSilla's(cns2(at)kent(dot)ac(dot)uk)Latinpopularanddancemusicdataset[1]willbere-used.ThiscollectionislikelytocontainagreaternumberofstylesofmusicthatwillbedifferentiatedbyrhythmiccharacteristicsthantheMIREX2007dataset. Collectionstatistics: 3,227audiofilesin22.05kHzmonoWAVformat 10Latinmusicgenres,including: Axe Bachata Bolero Forro Gaucha Merengue Pagode Sertaneja Tango AudioMoodClassification Thisdatasetrequiresalgorithmstoclassifymusicaudioaccordingtothemoodofthetrack(drawnfromacollectionofproductionmsuicsourcedfromtheAPMcollection[2]).TheMIREX2007MoodClassificationdataset[3]willbere-used. Collectionstatistics: 60030secondaudioclipsin22.05kHzmonoWAVformatselectedfromtheAPMcollection[4],andlabeledbyhumanjudgesusingtheEvalutron6000system. 5moodcategories[5]eachofwhichcontains120clips: Cluster_1:passionate,rousing,confident,boisterous,rowdy Cluster_2:rollicking,cheerful,fun,sweet,amiable/goodnatured Cluster_3:literate,poignant,wistful,bittersweet,autumnal,brooding Cluster_4:humorous,silly,campy,quirky,whimsical,witty,wry Cluster_5:aggressive,fiery,tense/anxious,intense,volatile,visceral AudioFormats Foralldatasets,participatingalgorithmswillhavetoreadaudiointhefollowingformat: Samplerate:22KHz Samplesize:16bit Numberofchannels:1(mono) Encoding:WAV Evaluation Thissectionfirstdescribesevaluationmethodscommontoallthedatasets,thenspecifiessettingsuniquetoeachofthetasks. Participatingalgorithmswillbeevaluatedwith3-foldcrossvalidation.ForArtistIdentificationandClassicalComposerClassification,albumfilteringwillbeusedthetestandtrainingsplits,i.e.trainingandtestsetswillcontaintracksfromdifferentalbums;forUSPopGenreClassificationandLatinGenreClassification,artistfilteringwillbeusedthetestandtrainingsplits,i.e.trainingandtestsetswillcontaindifferentartists. Therawclassification(identification)accuracy,standarddeviationandaconfusionmatrixforeachalgorithmwillbecomputed. ClassificationaccuracieswillbetestedforstatisticallysignificantdifferencesusingFriedman'sAnovawithTukey-Kramerhonestlysignificantdifference(HSD)testsformultiplecomparisons.Thistestwillbeusedtorankthealgorithmsandtogroupthemintosetsofequivalentperformance. Inadditioncomputationtimesforfeatureextractionandtraining/classificationwillbemeasured. SubmissionFormat FileI/OFormat TheaudiofilestobeusedinthesetaskswillbespecifiedinasimpleASCIIlistfile.Theformatsforthelistfilesarespecifiedbelow: Featureextractionlistfile ThelistfilepassedforfeatureextractionwillbeasimpleASCIIlistfile.Thisfilewillcontainonepathperlinewithnoheaderline. I.e. E.g. /path/to/track1.wav /path/to/track2.wav ... Traininglistfile ThelistfilepassedformodeltrainingwillbeasimpleASCIIlistfile.Thisfilewillcontainonepathperline,followedbyatabcharacterandtheclass(artist,genreormood)label,againwithnoheaderline. I.e. \t E.g. /path/to/track1.wav rock /path/to/track2.wav blues ... Test(classification)listfile ThelistfilepassedfortestingclassificationwillbeasimpleASCIIlistfileidenticalinformattotheFeatureextractionlistfile.Thisfilewillcontainonepathperlinewithnoheaderline. I.e. E.g. /path/to/track1.wav /path/to/track2.wav ... Classificationoutputfile ParticipatingalgorithmsshouldproduceasimpleASCIIlistfileidenticalinformattotheTraininglistfile.Thisfilewillcontainonepathperline,followedbyatabcharacterandtheartistlabel,againwithnoheaderline. I.e. \t E.g. /path/to/track1.wav classical /path/to/track2.wav blues ... Submissioncallingformats Algorithmsshoulddividetheirfeatureextractionandtraining/classificationintoseparateruns.Thiswillfacilitateasinglefeatureextractionstepforthetask,whiletrainingandclassificationcanberunforeachcross-validationfold. Hence,participantsshouldprovidetwoexecutablesorcommandlineparametersforasingleexecutabletorunthetwoseparateprocesses. Executableswillhavetoacceptthepathstotheaforementionedlistfilesascommandlineparameters. Scratchfolderswillbeprovidedforallsubmissionsforthestorageoffeaturefilesandanymodelfilestobeproduced.Executableswillhavetoacceptthepathtotheirscratchfolderasacommandlineparameter.Executableswillalsohavetotrackwhichfeaturefilescorrespondtowhichaudiofilesinternally.Tofacilitatethisprocess,uniquefilenameswillbeassignedtoeachaudiotrack. Examplesubmissioncallingformats extractFeatures.sh/path/to/scratch/folder/path/to/featureExtractionListFile.txt TrainAndClassify.sh/path/to/scratch/folder/path/to/trainListFile.txt/path/to/testListFile.txt/path/to/outputListFile.txt extractFeatures.sh/path/to/scratch/folder/path/to/featureExtractionListFile.txt Train.sh/path/to/scratch/folder/path/to/trainListFile.txt Classify.sh/path/to/scratch/folder/path/to/testListFile.txt/path/to/outputListFile.txt myAlgo.sh-extract/path/to/scratch/folder/path/to/featureExtractionListFile.txt myAlgo.sh-train/path/to/scratch/folder/path/to/trainListFile.txt myAlgo.sh-classify/path/to/scratch/folder/path/to/testListFile.txt/path/to/outputListFile.txt Multi-processorcomputenodeswillbeusedtorunthistask,however,weaskthatsubmissionsusenomorethan4cores(aswewillberunningalotofsubmissionsandwillneedtorunsomeinparallel).Ideally,thenumberofthreadstouseshouldbespecifiedasacommandlineparameter.Alternatively,implementationsmaybeprovidedinhard-coded1,2or4thread/coreconfigurations. extractFeatures.sh-numThreads4/path/to/scratch/folder/path/to/featureExtractionListFile.txt TrainAndClassify.sh-numThreads4/path/to/scratch/folder/path/to/trainListFile.txt/path/to/testListFile.txt/path/to/outputListFile.txt myAlgo.sh-extract-numThreads4/path/to/scratch/folder/path/to/featureExtractionListFile.txt myAlgo.sh-TrainAndClassify-numThreads4/path/to/scratch/folder/path/to/trainListFile.txt/path/to/testListFile.txt/path/to/outputListFile.txt Packagingsubmissions Allsubmissionsshouldbestaticallylinkedtoalllibraries(thepresenceofdynamicallylinkedlibrariescannotbeguaranteed).IMIRSELshouldbenotifiedofanydependenciesthatyoucannotincludewithyoursubmissionattheearliestopportunity(inordertogivethemtimetosatisfythedependency). BesuretofollowtheBestCodingPracticesforMIREX BesuretofollowtheMIREX2019SubmissionInstructions AllsubmissionsshouldincludeaREADMEfileincludingthefollowingtheinformation: Commandlinecallingformatforallexecutablesincludingexamples Numberofthreads/coresusedorwhetherthisshouldbespecifiedonthecommandline Expectedmemoryfootprint Expectedruntime Approximatelyhowmuchscratchdiskspacewillthesubmissionneedtostoreanyfeature/cachefiles? Anyrequiredenvironments/architectures(andversions)suchasMatlab,Java,Python,Bash,Rubyetc. Anyspecialnoticeregardingtorunningyouralgorithm NotethattheinformationthatyouplaceintheREADMEfileisextremelyimportantinensuringthatyoursubmissionisevaluatedproperly. Timeandhardwarelimits Duetothepotentiallyhighnumberofparticipantsinthisandotheraudiotasks,hardlimitsontheruntimeofsubmissionswillbeimposed. Ahardlimitof24hourswillbeimposedonfeatureextractiontimes. Ahardlimitof48hourswillbeimposedonthe3training/classificationcycles,leadingtoatotalruntimelimitof72hoursforeachsubmission. PotentialParticipants name/email Retrievedfrom"https://www.music-ir.org/mirex/w/index.php?title=2020:Audio_Classification_(Train/Test)_Tasks&oldid=13147" NavigationmenuViewsPageDiscussionViewsourceHistoryPersonaltoolsLoginRequestaccountMIREXbyYearMIREX2021MIREX2020MIREX2019MIREX2018MIREX2017MIREX2016MIREX2015MIREX2014MIREX2013MIREX2012MIREX2011MIREX2010MIREX2009MIREX2008MIREX2007MIREX2006MIREX2005ResultsbyYearMIREX2021ResultsMIREX2020ResultsMIREX2019ResultsMIREX2018ResultsMIREX2017ResultsMIREX2016ResultsMIREX2015ResultsMIREX2014ResultsMIREX2013ResultsMIREX2012ResultsMIREX2011ResultsMIREX2010ResultsMIREX2009ResultsMIREX2008ResultsMIREX2007ResultsMIREX2006ResultsMIREX2005ResultsAccountRequestRequestFormSearch NavigationMIREXCENTRALHOMECommunityportalCurrenteventsRecentchangesRandompageHelpToolsWhatlinkshereRelatedchangesSpecialpagesPrintableversionPermanentlinkPageinformationThispagewaslasteditedon1June2020,at22:43.ContentisavailableunderGNUFreeDocumentationLicense1.2unlessotherwisenoted.PrivacypolicyAboutMIREXWikiDisclaimers