Wikipedia:Database download

WikiTaxi is an offline-reader for wikis in MediaWiki format. It enables users to search and browse popular wikis like Wikipedia, Wikiquote, or WikiNews, without ... Wikipedia:Databasedownload FromWikipedia,thefreeencyclopedia Jumptonavigation Jumptosearch Informationondownloadingdumpsofthewikidatabase Forscheduling,relatedtoolsetc.,seeDatadumpsonMeta-Wiki. "WP:DD"redirectshere.Fordeletiondiscussions,seeWikipedia:Deletiondiscussions. Thishelppageisahow-toguide.Itdetailsprocessesorproceduresofsomeaspect(s)ofWikipedia'snormsandpractices.ItisnotoneofWikipedia'spoliciesorguidelines,andmayreflectvaryinglevelsofconsensusandvetting.ShortcutsWP:DUMPWP:DUMPS Readers'FAQ AboutWikipedia Administration FAQs Assessingarticlequality Authoritycontrol Books Categories Censorship Copyright Disambiguation Imagesandmultimedia ISBN Microformats Mobileaccess Offlineaccess Navigation Otherlanguages Pagenames Portals Searching Studenthelp ResearchingwithWikipedia CitingWikipedia Readers'glossary Readers'index Reader'sguidetoWikipedia vte Wikipediaoffersfreecopiesofallavailablecontenttointerestedusers.Thesedatabasescanbeusedformirroring,personaluse,informalbackups,offlineuseordatabasequeries(suchasforWikipedia:Maintenance).Alltextcontentismulti-licensedundertheCreativeCommonsAttribution-ShareAlike3.0License(CC-BY-SA)andtheGNUFreeDocumentationLicense(GFDL).Imagesandotherfilesareavailableunderdifferentterms,asdetailedontheirdescriptionpages.Forouradviceaboutcomplyingwiththeselicenses,seeWikipedia:Copyrights. Contents 1OfflineWikipediareaders 2WheredoIgetit? 2.1English-languageWikipedia 3ShouldIgetmultistream? 3.1Howtousemultistream? 3.2Otherlanguages 4Wherearetheuploadedfiles(image,audio,video,etc.)? 5Dealingwithcompressedfiles 6Dealingwithlargefiles 6.1Filesystemlimits 6.2Operatingsystemlimits 6.3Tips 6.3.1Detectcorruptedfiles 6.3.2ReformattingexternalUSBdrives 6.3.3LinuxandUnix 7Whynotjustretrievedatafromwikipedia.orgatruntime? 7.1Pleasedonotuseawebcrawler 7.1.1Sampleblockedcrawleremail 7.2DoingSQLqueriesonthecurrentdatabasedump 8Databaseschema 8.1SQLschema 8.2XMLschema 9Helptoparsedumpsforuseinscripts 9.1DoingHadoopMapReduceontheWikipediacurrentdatabasedump 10HelptoimportdumpsintoMySQL 11WikimediaEnterpriseHTMLDumps 12StaticHTMLtreedumpsformirroringorCDdistribution 12.1Kiwix 12.2AardDictionary/Aard2 12.3E-book 12.4WikiviewerforRockbox 12.5Olddumps 13DynamicHTMLgenerationfromalocalXMLdatabasedump 13.1XOWA 13.1.1Features 13.1.2Mainfeatures 13.2WikiFilter 13.2.1WikiFiltersystemrequirements 13.2.2HowtosetupWikiFilter 13.3WikiTaxi(forWindows) 13.3.1WikiTaxisystemrequirements 13.3.2WikiTaxiusage 13.4BzReaderandMzReader(forWindows) 13.5EPWING 14Mirrorbuilding 14.1WP-MIRROR 15Seealso 16References 17Externallinks OfflineWikipediareaders SomeofthemanywaystoreadWikipediawhileoffline: XOWA:(§ XOWA) Kiwix:(§ Kiwix) WikiTaxi:§ WikiTaxi(forWindows) aarddict:§ AardDictionary BzReader:§ BzReaderandMzReader(forWindows) SelectedWikipediaarticlesasaprinteddocument:Help:Printing WikiasE-Book:§ E-book WikiFilter:§ WikiFilter Wikipediaonrockbox:§ WikiviewerforRockbox Someofthemaremobileapplications–see"listofWikipediamobileapplications". WheredoIgetit? English-languageWikipedia DumpsfromanyWikimediaFoundationproject:dumps.wikimedia.organdtheInternetArchive EnglishWikipediadumpsinSQLandXML:dumps.wikimedia.org/enwiki/andtheInternetArchive DownloadthedatadumpusingaBitTorrentclient(torrentinghasmanybenefitsandreducesserverload,savingbandwidthcosts). pages-articles-multistream.xml.bz2–Currentrevisionsonly,notalkoruserpages;thisisprobablywhatyouwant,andisover19GBcompressed(expandstoover86GBwhendecompressed). pages-meta-current.xml.bz2–Currentrevisionsonly,allpages(includingtalk) abstract.xml.gz–pageabstracts all-titles-in-ns0.gz–Articletitlesonly(withredirects) SQLfilesforthepagesandlinksarealsoavailable Allrevisions,allpages:Thesefilesexpandtomultipleterabytesoftext.Pleaseonlydownloadtheseifyouknowyoucancopewiththisquantityofdata.GotoLatestDumpsandlookoutforallthefilesthathave'pages-meta-history'intheirname. TodownloadasubsetofthedatabaseinXMLformat,suchasaspecificcategoryoralistofarticlessee:Special:Export,usageofwhichisdescribedatHelp:Export. Wikifront-endsoftware:MediaWiki[1]. Databasebackendsoftware:MySQL. Imagedumps:Seebelow. ShouldIgetmultistream? TL;DR: GETTHEMULTISTREAMVERSION!(andthecorrespondingindexfile,pages-articles-multistream-index.txt.bz2) pages-articles.xml.bz2andpages-articles-multistream.xml.bz2bothcontainthesamexmlcontents.Soifyouunpackeither,yougetthesamedata.Butwithmultistream,itispossibletogetanarticlefromthearchivewithoutunpackingthewholething.Yourreadershouldhandlethisforyou,ifyourreaderdoesn'tsupportititwillworkanywaysincemultistreamandnon-multistreamcontainthesamexml.Theonlydownsidetomultistreamisthatitismarginallylarger.Youmightbetemptedtogetthesmallernon-multistreamarchive,butthiswillbeuselessifyoudon'tunpackit.Anditwillunpackto~5-10timesitsoriginalsize.Pennywise,poundfoolish.Getmultistream. NOTETHATthemultistreamdumpfilecontainsmultiplebz2'streams'(bz2header,body,footer)concatenatedtogetherintoonefile,incontrasttothevanillafilewhichcontainsonestream.Eachseparate'stream'(orreally,file)inthemultistreamdumpcontains100pages,exceptpossiblythelastone. Howtousemultistream? Formultistream,youcangetanindexfile,pages-articles-multistream-index.txt.bz2.Thefirstfieldofthisindexisthenumberofbytestoseekintothecompressedarchivepages-articles-multistream.xml.bz2,thesecondisthearticleID,thethirdthearticletitle. Cutasmallpartoutofthearchivewithddusingthebyteoffsetasfoundintheindex.Youcouldtheneitherbzip2decompressitorusebzip2recover,andsearchthefirstfileforthearticleID. Seehttps://docs.python.org/3/library/bz2.html#bz2.BZ2Decompressorforinfoaboutsuchmultistreamfilesandabouthowtodecompressthemwithpython;seealsohttps://gerrit.wikimedia.org/r/plugins/gitiles/operations/dumps/+/ariel/toys/bz2multistream/README.txtandrelatedfilesforanoldworkingtoy. Otherlanguages Inthedumps.wikimedia.orgdirectoryyouwillfindthelatestSQLandXMLdumpsfortheprojects,notjustEnglish.Thesub-directoriesarenamedforthelanguagecodeandtheappropriateproject.Someotherdirectories(e.g.simple,nostalgia)exist,withthesamestructure.ThesedumpsarealsoavailablefromtheInternetArchive. Wherearetheuploadedfiles(image,audio,video,etc.)? ImagesandotheruploadedmediaareavailablefrommirrorsinadditiontobeingserveddirectlyfromWikimediaservers.Bulkdownloadis(asofSeptember2013)availablefrommirrorsbutnotoffereddirectlyfromWikimediaservers.Seethelistofcurrentmirrors.Youshouldrsyncfromthemirror,thenfillinthemissingimagesfromupload.wikimedia.org;whendownloadingfromupload.wikimedia.orgyoushouldthrottleyourselfto1cachemisspersecond(youcancheckheadersonaresponsetoseeifwasahitormissandthenbackoffwhenyougetamiss)andyoushouldn'tusemorethanoneortwosimultaneousHTTPconnections.Inanycase,makesureyouhaveanaccurateuseragentstringwithcontactinfo(emailaddress)soopscancontactyouifthere'sanissue.YoushouldbegettingchecksumsfromthemediawikiAPIandverifyingthem.TheAPIEtiquettepagecontainssomeguidelines,althoughnotallofthemapply(forexample,becauseupload.wikimedia.orgisn'tMediaWiki,thereisnomaxlagparameter). Unlikemostarticletext,imagesarenotnecessarilylicensedundertheGFDL&CC-BY-SA-3.0.Theymaybeunderoneofmanyfreelicenses,inthepublicdomain,believedtobefairuse,orevencopyrightinfringements(whichshouldbedeleted).Inparticular,useoffairuseimagesoutsidethecontextofWikipediaorsimilarworksmaybeillegal.Imagesundermostlicensesrequireacredit,andpossiblyotherattachedcopyrightinformation.Thisinformationisincludedinimagedescriptionpages,whicharepartofthetextdumpsavailablefromdumps.wikimedia.org.Inconclusion,downloadtheseimagesatyourownrisk(Legal) Dealingwithcompressedfiles Compresseddumpfilesaresignificantlycompressed,thusafterbeingdecompressedwilltakeuplargeamountsofdrivespace.AlargelistofdecompressionprogramsaredescribedinComparisonoffilearchivers.Thefollowingprogramsinparticularcanbeusedtodecompressbzip2.bz2.zipand.7zfiles. Windows BeginningwithWindowsXP,abasicdecompressionprogramenablesdecompressionofzipfiles.[1][2]Amongothers,thefollowingcanbeusedtodecompressbzip2files. bzip2(command-line)(fromhere)isavailableforfreeunderaBSDlicense. 7-ZipisavailableforfreeunderanLGPLlicense. WinRAR WinZip Macintosh(Mac) OSXshipswiththecommand-linebzip2tool. GNU/Linux MostGNU/Linuxdistributionsshipwiththecommand-linebzip2tool. BerkeleySoftwareDistribution(BSD) SomeBSDsystemsshipwiththecommand-linebzip2toolaspartoftheoperatingsystem.Others,suchasOpenBSD,provideitasapackagewhichmustfirstbeinstalled. Notes Someolderversionsofbzip2maynotbeabletohandlefileslargerthan2GB,somakesureyouhavethelatestversionifyouexperienceanyproblems. Someolderarchivesarecompressedwithgzip,whichiscompatiblewithPKZIP(themostcommonWindowsformat). Dealingwithlargefiles Asfilesgrowinsize,sodoesthelikelihoodtheywillexceedsomelimitofacomputingdevice.Eachoperatingsystem,filesystem,hardstoragedevice,andsoftware(application)hasamaximumfilesizelimit.Eachoneofthesewilllikelyhaveadifferentmaximum,andthelowestlimitofallofthemwillbecomethefilesizelimitforastoragedevice. Theolderthesoftwareinacomputingdevice,themorelikelyitwillhavea2GBfilelimitsomewhereinthesystem.Thisisduetooldersoftwareusing32-bitintegersforfileindexing,whichlimitsfilesizesto2^31bytes(2GB)(forsignedintegers),or2^32(4GB)(forunsignedintegers).OlderCprogramminglibrarieshavethis2or4GBlimit,butthenewerfilelibrarieshavebeenconvertedto64-bitintegersthussupportingfilesizesupto2^63or2^64bytes(8or16EB). Beforestartingadownloadofalargefile,checkthestoragedevicetoensureitsfilesystemcansupportfilesofsuchalargesize,andchecktheamountoffreespacetoensurethatitcanholdthedownloadedfile. Filesystemlimits Therearetwolimitsforafilesystem:thefilesystemsizelimit,andthefilesystemlimit.Ingeneral,sincethefilesizelimitislessthanthefilesystemlimit,thelargerfilesystemlimitsareamootpoint.Alargepercentageofusersassumetheycancreatefilesuptothesizeoftheirstoragedevice,butarewrongintheirassumption.Forexample,a16GBstoragedeviceformattedasFAT32filesystemhasafilelimitof4GBforanysinglefile.Thefollowingisalistofthemostcommonfilesystems,andseeComparisonoffilesystemsforadditionaldetailedinformation. Windows FAT16supportsfilesupto4GB.FAT16isthefactoryformatofsmallerUSBdrivesandallSDcardsthatare2GBorsmaller. FAT32supportsfilesupto4GB.FAT32isthefactoryformatoflargerUSBdrivesandallSDHCcardsthatare4GBorlarger. exFATsupportsfilesupto127PB.exFATisthefactoryformatofallSDXCcards,butisincompatiblewithmostflavorsofUNIXduetolicensingproblems. NTFSsupportsfilesupto16TB.NTFSisthedefaultfilesystemformodernWindowscomputers,includingWindows2000,WindowsXP,andalltheirsuccessorstodate.VersionsafterWindows8cansupportlargerfilesifthefilesystemisformattedwithalargerclustersize. ReFSsupportsfilesupto16EB. Macintosh(Mac) HFSPlus(HFS+)supportsfilesupto8EBonMacOSX10.2+andiOS.HFS+wasthedefaultfilesystemforOSXcomputerspriortomacOSHighSierrain2017whenitwasreplacedasdefaultwithAppleFileSystem,APFS. Linux ext2andext3supportsfilesupto16GB,butupto2TBwithlargerblocksizes.Seehttps://users.suse.com/~aj/linux_lfs.htmlformoreinformation. ext4supportsfilesupto16TB,using4KBblocksize.(limitremovedine2fsprogs-1.42(2012)) XFSsupportsfilesupto8EB. ReiserFSsupportsfilesupto1EB,8TBon32-bitsystems. JFSsupportsfilesupto4PB. Btrfssupportsfilesupto16EB. NILFSsupportsfilesupto8EB. YAFFS2supportsfilesupto2GB FreeBSD ZFSsupportsfilesupto16EB. FreeBSDandotherBSDs UnixFileSystem(UFS)supportsfilesupto8ZiB. Operatingsystemlimits Eachoperatingsystemhasinternalfilesystemlimitsforfilesizeanddrivesize,whichisindependentofthefilesystemorphysicalmedia.Iftheoperatingsystemhasanylimitslowerthanthefilesystemorphysicalmedia,thentheOSlimitswillbethereallimit. Windows Windows95,98,MEhavea4GBlimitforallfilesizes. WindowsXPhasa16TBlimitforallfilesizes. Windows7hasa16TBlimitforallfilesizes. Windows8,10,andServer2012havea256TBlimitforallfilesizes. Linux 32-bitkernel2.4.xsystemshavea2TBlimitforallfilesystems. 64-bitkernel2.4.xsystemshavean8EBlimitforallfilesystems. 32-bitkernel2.6.xsystemswithoutoptionCONFIG_LBDhavea2TBlimitforallfilesystems. 32-bitkernel2.6.xsystemswithoptionCONFIG_LBDandall64-bitkernel2.6.xsystemshavean8ZBlimitforallfilesystems.[3] GoogleAndroid GoogleAndroidisbasedonLinux,whichdeterminesitsbaselimits. Internalstorage: Android2.3andlaterusestheext4filesystem.[4] Android2.2andearlierusestheYAFFS2filesystem. Externalstorageslots: AllAndroiddevicesshouldsupportFAT16,FAT32,ext2filesystems. Android2.3andlatersupportsext4filesystem. AppleiOS(seeListofiOSdevices) AlldevicessupportHFSPlus(HFS+)forinternalstorage.Nodeviceshaveexternalstorageslots.Deviceson10.3orlaterrunAppleFileSystemsupportingamaxfilesizeof8EB. Tips Detectcorruptedfiles ItisusefultochecktheMD5sums(providedinafileinthedownloaddirectory)tomakesurethedownloadwascompleteandaccurate.Thiscanbecheckedbyrunningthe"md5sum"commandonthefilesdownloaded.Giventheirsizes,thismaytakesometimetocalculate.Duetothetechnicaldetailsofhowfilesarestored,filesizesmaybereporteddifferentlyondifferentfilesystems,andsoarenotnecessarilyreliable.Also,corruptionmayhaveoccurredduringthedownload,thoughthisisunlikely. ReformattingexternalUSBdrives IfyouplantodownloadWikipediaDumpfilestoonecomputeranduseanexternalUSBflashdriveorharddrivetocopythemtoothercomputers,thenyouwillrunintothe4GBFAT32filesizelimit.Toworkaroundthislimit,reformatthe>4GBUSBdrivetoafilesystemthatsupportslargerfilesizes.IfworkingexclusivelywithWindowsXP/Vista/7computers,thenreformattheUSBdrivetoNTFSfilesystem. LinuxandUnix Ifyouseemtobehittingthe2GBlimit,tryusingwgetversion1.10orgreater,cURLversion7.11.1-1orgreater,orarecentversionoflynx(using-dump).Also,youcanresumedownloads(forexamplewget-c). Whynotjustretrievedatafromwikipedia.orgatruntime? SupposeyouarebuildingapieceofsoftwarethatatcertainpointsdisplaysinformationthatcamefromWikipedia.Ifyouwantyourprogramtodisplaytheinformationinadifferentwaythancanbeseenintheliveversion,you'llprobablyneedthewikicodethatisusedtoenterit,insteadofthefinishedHTML. Also,ifyouwanttogetallthedata,you'llprobablywanttotransferitinthemostefficientwaythat'spossible.Thewikipedia.orgserversneedtodoquiteabitofworktoconvertthewikicodeintoHTML.That'stimeconsumingbothforyouandforthewikipedia.orgservers,sosimplyspideringallpagesisnotthewaytogo. ToaccessanyarticleinXML,oneatatime,accessSpecial:Export/Titleofthearticle. ReadmoreaboutthisatSpecial:Export. PleasebeawarethatlivemirrorsofWikipediathataredynamicallyloadedfromtheWikimediaserversareprohibited.PleaseseeWikipedia:Mirrorsandforks. Pleasedonotuseawebcrawler Pleasedonotuseawebcrawlertodownloadlargenumbersofarticles.Aggressivecrawlingoftheservercancauseadramaticslow-downofWikipedia. Sampleblockedcrawleremail IPaddressnnn.nnn.nnn.nnnwasretrievingupto50pagespersecondfromwikipedia.orgaddresses.Somethinglikeatleastaseconddelaybetweenrequestsisreasonable.Pleaserespectthatsetting.Ifyoumustexceeditalittle,dosoonlyduringtheleastbusytimesshowninoursiteloadgraphsatstats.wikimedia.org/EN/ChartsWikipediaZZ.htm.It'sworthnotingthattocrawlthewholesiteatonehitpersecondwilltakeseveralweeks.TheoriginatingIPisnowblockedorwillbeshortly.Pleasecontactusifyouwantitunblocked.Pleasedon'ttrytocircumventit –we'lljustblockyourwholeIPrange. Ifyouwantinformationonhowtogetourcontentmoreefficiently,weofferavarietyofmethods,includingweeklydatabasedumpswhichyoucanloadintoMySQLandcrawllocallyatanyrateyoufindconvenient.Toolsarealsoavailablewhichwilldothatforyouasoftenasyoulikeonceyouhavetheinfrastructureinplace. Insteadofanemailreplyyoumayprefertovisit#mediawikiconnectatirc.libera.chattodiscussyouroptionswithourteam. DoingSQLqueriesonthecurrentdatabasedump YoucandoSQLqueriesonthecurrentdatabasedumpusingQuarry(asareplacementforthedisabledSpecial:Asksqlpage). Databaseschema SQLschema Seealso:mw:Manual:Databaselayout ThesqlfileusedtoinitializeaMediaWikidatabasecanbefoundhere. XMLschema TheXMLschemaforeachdumpisdefinedatthetopofthefile.AndalsodescribedintheMediaWikiexporthelppage. Helptoparsedumpsforuseinscripts Wikipedia:Computerhelpdesk/ParseMediaWikiDumpdescribesthePerlParse::MediaWikiDumplibrary,whichcanparseXMLdumps. Wikipediapreprocessor(wikiprep.pl)isaPerlscriptthatpreprocessesrawXMLdumpsandbuildslinktables,categoryhierarchies,collectsanchortextforeacharticleetc. WikipediaSQLdumpparserisa.NETlibrarytoreadMySQLdumpswithouttheneedtouseMySQLdatabase WikiDumpParser–a.NETCorelibarytoparsethedatabasedumps. DictionaryBuilderisaRustprogramthatcanparseXMLdumpsandextractentriesinfiles ScriptsforparsingWikipediadumps­–Pythonbasedscriptsforparsingsql.gzfilesfromwikipediadumps. parse-mediawiki-sql–aRustlibraryforquicklyparsingtheSQLdumpfileswithminimalmemoryallocation gitlab.com/tozd/go/mediawiki–aGopackageprovidingutilitiesforprocessingWikipediaandWikidatadumps. DoingHadoopMapReduceontheWikipediacurrentdatabasedump YoucandoHadoopMapReducequeriesonthecurrentdatabasedump,butyouwillneedanextensiontotheInputRecordFormatto haveeachbeasinglemapperinput.Aworkingsetofjavamethods(jobControl,mapper,reducer,andXmlInputRecordFormat)isavailableatHadoopontheWikipedia HelptoimportdumpsintoMySQL See: mw:Manual:ImportingXMLdumps m:Datadumps WikimediaEnterpriseHTMLDumps AspartofWikimediaEnterpriseapartialmirrorofHTMLdumpsismadepublic.Dumpsareproducedforaspecificsetofnamespacesandwikis,andthenmadeavailableforpublicdownload.Eachdumpoutputfileconsistsofatar.gzarchivewhich,whenuncompressedanduntarred,containsonefile,withasinglelineperarticle,injsonformat.Thisiscurrentlyanexperimentalservice. StaticHTMLtreedumpsformirroringorCDdistribution MediaWiki1.5includesroutinestodumpawikitoHTML,renderingtheHTMLwiththesameparserusedonalivewiki.Asthefollowingpagestates,puttingoneofthesedumpsonthewebunmodifiedwillconstituteatrademarkviolation.Theyareintendedforprivateviewinginanintranetordesktopinstallation. IfyouwanttodraftatraditionalwebsiteinMediawikianddumpittoHTMLformat,youmightwanttotrymw2htmlbyUser:Connelly. Ifyou'dliketohelpdevelopdump-to-staticHTMLtools,pleasedropusanoteonthedevelopers'mailinglist. StaticHTMLdumpsarenowavailablehere,butarenotcurrent. Seealso: mw:AlternativeparserslistssomeothernotworkingoptionsforgettingstaticHTMLdumps Wikipedia:Snapshots Wikipedia:TomeRaiderdatabase Kiwix KiwixonanAndroidtablet KiwixisbyfarthelargestofflinedistributionofWikipediatodate.Asanofflinereader,Kiwixworkswithalibraryofcontentsthatarezimfiles:youcanpick&choosewhicheverWikimediaproject(Wikipediainanylanguage,Wiktionary,Wikisource,etc.),aswellasTEDTalks,PhETInteractiveMaths&Physicssimulations,GutenbergProject,etc. Itisfreeandopensource,andcurrentlyavailablefordownloadon: Android iOS macOS Windows&Windows10(UWP) GNU/Linux ...aswellasextensionsforChrome&Firefoxbrowsers,serversolutions,etc.SeeofficialWebsiteforthecompleteKiwixportfolio. AardDictionary/Aard2 AardDictionaryisanofflineWikipediareader.Noimages.Cross-platformforWindows,Mac,Linux,Android,Maemo.RunsonrootedNookandSonyPRS-T1eBooksreaders. ItalsohasasuccessorAard2. E-book Thewiki-as-ebookstoreprovidesebookscreatedfromalargesetofWikipediaarticleswithgrayscaleimagesfore-book-readers(2013). WikiviewerforRockbox ThewikiviewerpluginforrockboxpermitsviewingconvertedWikipediadumpsonmanyRockboxdevices. Itneedsacustombuildandconversionofthewikidumpsusingtheinstructionsavailableathttp://www.rockbox.org/tracker/4755.Theconversionrecompressesthefileandsplitsitinto1GBfilesandanindexfilewhichallneedtobeinthesamefolderonthedeviceormicrosdcard. Olddumps ThestaticversionofWikipediacreatedbyWikimedia:http://static.wikipedia.org/Feb.11,2013-Thisisapparentlyofflinenow.Therewasnocontent. Wiki2static(sitedownasofOctober 2005[update])wasanexperimentalprogramsetupbyUser:Alfiotogeneratehtmldumps,inclusiveofimages,searchfunctionandalphabeticalindex.Atthelinkedsiteexperimentaldumpsandthescriptitselfcanbedownloaded.AsanexampleitwasusedtogeneratethesecopiesofEnglishWikiPedia24April04,SimpleWikiPedia1May04(olddatabase)formatandEnglishWikiPedia24July04SimpleWikiPedia24July04,WikiPediaFrancais27Juillet2004(newformat).BozMousesaversiontogenerateperiodicstaticcopiesatfixedreference(sitedownasofOctober2017). DynamicHTMLgenerationfromalocalXMLdatabasedump InsteadofconvertingadatabasedumpfiletomanypiecesofstaticHTML,onecanalsouseadynamicHTMLgenerator.BrowsingawikipageisjustlikebrowsingaWikisite,butthecontentisfetchedandconvertedfromalocaldumpfileonrequestfromthebrowser. XOWA XOWAisafree,open-sourceapplicationthathelpsdownloadWikipediatoacomputer.AccessallofWikipediaoffline,withoutaninternetconnection! Itiscurrentlyinthebetastageofdevelopment,butisfunctional.Itisavailablefordownloadhere. Features DisplaysallarticlesfromWikipediawithoutaninternetconnection. Downloadacomplete,recentcopyofEnglishWikipedia. Display5.2+millionarticlesinfullHTMLformatting. Showimageswithinanarticle.Access3.7+millionimagesusingtheofflineimagedatabases. WorkswithanyWikimediawiki,includingWikipedia,Wiktionary,Wikisource,Wikiquote,Wikivoyage(alsosomenon-wmfdumps) Workswithanynon-EnglishlanguagewikisuchasFrenchWikipedia,GermanWikisource,DutchWikivoyage,etc. WorkswithotherspecializedwikissuchasWikidata,WikimediaCommons,Wikispecies,oranyotherMediaWikigenerateddump Setupover660+otherwikisincluding: EnglishWiktionary EnglishWikisource EnglishWikiquote EnglishWikivoyage Non-Englishwikis,suchasFrenchWiktionary,GermanWikisource,DutchWikivoyage Wikidata WikimediaCommons Wikispecies ...andmanymore! Updateyourwikiwheneveryouwant,usingWikimedia'sdatabasebackups. Navigatebetweenofflinewikis.Clickon"LookupthiswordinWiktionary"andinstantlyviewthepageinWiktionary. Editarticlestoremovevandalismorerrors. Installtoaflashmemorycardforportabilitytoothermachines. RunonWindows,LinuxandMacOSX. ViewtheHTMLforanywikipage. SearchforanypagebytitleusingaWikipedia-likeSearchbox. BrowsepagesbyalphabeticalorderusingSpecial:AllPages. Findawordonapage. Accessahistoryofviewedpages. Bookmarkyourfavoritepages. Downloadsimagesandotherfilesondemand(whenconnectedtotheinternet) SetsupSimpleWikipediainlessthan5minutes Canbecustomizedatmanylevels:fromkeyboardshortcutstoHTMLlayoutstointernaloptions Mainfeatures Veryfastsearching Keyword(actually,titlewords)basedsearching Searchproducesmultiplepossiblearticles:youcanchooseamongstthem LaTeXbasedrenderingformathematicalformulae Minimalspacerequirements:theoriginal.bz2fileplustheindex Veryfastinstallation(amatterofhours)comparedtoloadingthedumpintoMySQL WikiFilter WikiFilterisaprogramwhichallowsyoutobrowseover100dumpfileswithoutvisitingaWikisite. WikiFiltersystemrequirements ArecentWindowsversion(WindowsXPisfine;Windows98andMEwon'tworkbecausetheydon'thaveNTFSsupport) Afairbitofharddrivespace(toinstallyouwillneedabout12–15Gigabytes;afterwardsyouwillonlyneedabout10Gigabytes) HowtosetupWikiFilter StartdownloadingaWikipediadatabasedumpfilesuchasanEnglishWikipediadump.ItisbesttouseadownloadmanagersuchasGetRightsoyoucanresumedownloadingthefileevenifyourcomputercrashesorisshutdownduringthedownload. DownloadXAMPPLITEfrom[2](youmustgetthe1.5.0versionforittowork).Makesuretopickthefilewhosefilenameendswith.exe Install/extractittoC:\XAMPPLITE. DownloadWikiFilter2.3fromthissite:http://sourceforge.net/projects/wikifilter.Youwillhaveachoiceoffilestodownload,somakesurethatyoupickthe2.3version.ExtractittoC:\WIKIFILTER. CopytheWikiFilter.sointoyourC:\XAMPPLITE\apache\modulesfolder. EdityourC:\xampplite\apache\conf\httpd.conffile,andaddthefollowingline: LoadModuleWikiFilter_module"C:/XAMPPLITE/apache/modules/WikiFilter.so" WhenyourWikipediafilehasfinisheddownloading,uncompressitintoyourC:\WIKIFILTERfolder.(IusedWinRARhttp://www.rarlab.com/demoversion –BitZipperhttp://www.bitzipper.com/winrar.htmlworkswelltoo.) RunWikiFilter(WikiIndex.exe),andgotoyourC:\WIKIFILTERfolder,anddraganddroptheXMLfileintothewindow,clickLoad,thenStart. Afteritfinishes,exitthewindow,andgotoyourC:\XAMPPLITEfolder.Runthesetup_xampp.batfiletoconfigurexampp. Whenyoufinishwiththat,runtheXampp-Control.exefile,andstartApache. Browsetohttp://localhost/wikiandseeifitworks Ifitdoesn'twork,seetheforums. WikiTaxi(forWindows) WikiTaxiisanoffline-readerforwikisinMediaWikiformat.ItenablesuserstosearchandbrowsepopularwikislikeWikipedia,Wikiquote,orWikiNews,withoutbeingconnectedtotheInternet.WikiTaxiworkswellwithdifferentlanguageslikeEnglish,German,Turkish,andothersbuthasaproblemwithright-to-leftlanguagescripts.WikiTaxidoesnotdisplayimages. WikiTaxisystemrequirements AnyWindowsversionstartingfromWindows95orlater.LargeFilesupport(greaterthan4GBwhichrequiresanexFATfilesystem)forthehugewikis(Englishonlyatthetimeofthiswriting). ItalsoworksonLinuxwithWine. 16MBRAMminimumfortheWikiTaxireader,128MBrecommendedfortheimporter(moreforspeed). StoragespacefortheWikiTaxidatabase.Thisrequiresabout11.7GiBfortheEnglishWikipedia(asof5April2011),2GBforGerman,lessforotherWikis.Thesefiguresarelikelytogrowinthefuture. WikiTaxiusage DownloadWikiTaxiandextracttoanemptyfolder.Noinstallationisotherwiserequired. DownloadtheXMLdatabasedump(*.xml.bz2)ofyourfavoritewiki. RunWikiTaxi_Importer.exetoimportthedatabasedumpintoaWikiTaxidatabase.Theimportertakescaretouncompressthedumpasitimports,somakesuretosaveyourdrivespaceanddonotuncompressbeforehand. Whentheimportisfinished,startupWikiTaxi.exeandopenthegenerateddatabasefile.Youcanstartsearching,browsing,andreadingimmediately. Afterasuccessfulimport,theXMLdumpfileisnolongerneededandcanbedeletedtoreclaimdiskspace. ToupdateanofflineWikiforWikiTaxi,downloadandimportamorerecentdatabasedump. ForWikiTaxireading,onlytwofilesarerequired:WikiTaxi.exeandthe.taxidatabase.Copythemtoanystoragedevice(memorystickormemorycard)orburnthemtoaCDorDVDandtakeyourWikipediawithyouwhereveryougo! BzReaderandMzReader(forWindows) BzReaderisanofflineWikipediareaderwithfastsearchcapabilities.ItrenderstheWikitextintoHTMLanddoesn'tneedtodecompressthedatabase.RequiresMicrosoft.NETframework2.0. MzReaderbyMun206workswith(thoughisnotaffiliatedwith)BzReader,andallowsfurtherrenderingofwikicodeintobetterHTML,includinganinterpretationofthemonobookskin.Itaimstomakepagesmorereadable.RequiresMicrosoftVisualBasic6.0Runtime,whichisnotsuppliedwiththedownload.AlsorequiresInetControlandInternetControls(InternetExplorer6ActiveX),whicharepackagedwiththedownload. EPWING OfflineWikipediadatabaseinEPWINGdictionaryformat,whichiscommonandanout-datedJapaneseIndustrialStandards(JIS)inJapan,canbereadincludingthumbnailimagesandtableswithsomerenderinglimits,onanysystemswhereareaderisavailable(Boookends).TherearemanyfreeandcommercialreadersforWindows(includingMobile),MacOSX,iOS(iPhone,iPad),Android,Unix-Linux-BSD,DOS,andJava-basedbrowserapplications(EPWINGViewers). Mirrorbuilding WP-MIRROR Important:WP-mirrorhasn'tbeensupportedsince2014,andcommunityverificationisneededthatitactuallyworks.Seetalkpage. WP-MIRRORisafreeutilityformirroringanydesiredsetofWMFwikis.Thatis,itbuildsawikifarmthattheusercanbrowselocally.WP-MIRRORbuildsacompletemirrorwithoriginalsizemediafiles.WP-MIRRORisavailablefordownload. Seealso DBpedia WikiReader m:Export m:Help:Downloadingpages m:Import Meta:Datadumps/Othertools,forrelatedtools,e.g.extractorsand"dumpreaders" Wikipedia:WikipediaCDSelection Wikipedia:SizeofWikipedia meta:MirroringWikimediaprojectXMLdumps meta:Staticversiontools Wikimediaofflineprojects References ^"Benchmarked:What'stheBestFileCompressionFormat?".HowToGeek.How-ToGeek,LLC.Retrieved18January2017. ^"Zipandunzipfiles".Microsoft.Microsoft.Retrieved18January2017. ^LargeFileSupportinLinux ^Android2.2andbeforeusedYAFFSfilesystem;December14,2010. Externallinks Wikimediadownloads. Domasvisitslogs(readthis!).Also,olddataintheInternetArchive. Wikimediamailinglistsarchives. User:Emijrp/WikipediaArchive.AnefforttofindalltheWiki[mp]ediaavailabledata,andtoencouragepeopletodownloaditandsaveitaroundtheglobe. ScripttodownloadallWikipedia7zdumps. Retrievedfrom"https://en.wikipedia.org/w/index.php?title=Wikipedia:Database_download&oldid=1075442894" Categories:Wikipediahow-toWikipediadownloadsWikipediadatabasereportsHiddencategories:Projectpageswithshortdescription Navigationmenu Personaltools NotloggedinTalkContributionsCreateaccountLogin Namespaces ProjectpageTalk English expanded collapsed Views ReadViewsourceViewhistory More expanded collapsed Search Navigation MainpageContentsCurrenteventsRandomarticleAboutWikipediaContactusDonate Contribute HelpLearntoeditCommunityportalRecentchangesUploadfile Tools WhatlinkshereRelatedchangesUploadfileSpecialpagesPermanentlinkPageinformationWikidataitem Print/export DownloadasPDFPrintableversion Inotherprojects WikimediaCommonsMediaWikiMeta-WikiWikidataWikivoyage Languages বাংলাБългарскиČeštinaDanskDeutschΕλληνικάEspañolفارسیFrançaisGalego한국어BahasaIndonesiaItalianoעבריתLatinaLietuvių日本語Oʻzbekcha/ўзбекчаPolskiPortuguêsРусскийShqipSuomiSvenskaதமிழ்ไทยTürkçeУкраїнська吴语中文 Editlinks