How to become a Cloud cost-control guru on GCP

Learn how to get in full control of your costs with Google Cloud Platform ... This will export every single cost transaction to a table in BigQuery at the ... Skiptocontent Cloudcomputingcostsaredifficulttograsp.Mostofushavejustgivenupontryingtounderstandclearlywhereallthecostfiguresonyourbillcomefrom.Youtakethemforgrantedandraiseanalarmonlywhenitgoestotallyoutofcontrol.BeingreactiveinsteadofproactiveiswhatmostofusdowhenitcomestoCloudcosts.Soundsfamiliar?Everreceivedanunexpectedhugeinvoice?Thenthisarticleisforyou,becauseGoogleCloudPlatform(GCP)allowsyoutobeinfullcontrolofyourcostthroughdifferenttools.I’llexplainyouhowyoucanbecomeaCloudcost-controlguruonGCPandstartyourcloudcostoptimisation. Controlthethingsthatyoucan;costonGCPisoneofthem  Thispostisco-authoredbyouramazinginternYannisPagès. AtFourcastweareconsistentlyusingthefollowingtoolsandtechniquestohelpcustomerscontroltheirCloudcostsinanefficientandaccessiblemanner.  CostControlDashboards Financecontrollerslovedashboards,somakethemhappy.WearenotgoingtoreinventthewheelandGCPprovidesafulltutorialtogetabasicversionofcostcontroldashboards.ButthisishowIrecommendtoproceedtogetyourcostcontroldashboardsupandrunning. FirstofallstartbyexportingyourbillingdatatoBigQuery.ThiswillexporteverysinglecosttransactiontoatableinBigQueryatthemostgranularlevelyoucanimagine.FromthereyoucanrunSQLqueriesandgetquickanswerstoanyquestionsyouhaveregardingyourbilling(ie“howmuchdomyVMsinEuropeconsume?”,“HowmuchdoIpayforallthereservedIPsIhave?”,etc.).IfyouarenotfamiliarwithSQL,thislinkwillgetyoustartedwithpre-madebillingqueries.  OnceyourbillingdataisinBigQuery,nextstepistocreatedashboardsinDataStudio.Youcanfindconvenienttemplatedashboardshere.  Youcaneasilygetmoreadvanced/customiseddashboardsbyplayingaroundwithDataStudio.HavingallthedatainBigQuery,it’sjustamatterofplottingitthewayyouwant. Forinstance,ifyouhavesomeapplicationsrunningonKubernetes,youcanactivatethe“usagemetering”onyourcluster(followtheinstructionsonthislink).ThiswillexportdetailedusageprofiledatatoatableinBigQuery.Byjoiningdatafromthistabletotheoriginalbillingdatatable,youwillgetthedatayouneedtograsphowyourmoneyisspentinyourdifferentclusters.Andagain,DataStudioattherescuetoplotit. AtFourcast,wehavedevelopedsomeadvanceddashboardsthatenabletofullyunderstandhowthecostsaresplitwithinyourcluster.Reachouttousifyouwanttoknowmoreaboutit. BillingalertsthroughSlacknotifications Slackhasgainedmajorpopularityinthedevcommunityandiscertainlyatabthatisconstantlyopenonyourcomputer.Ifyouwanttoconveyinformationtosomeone,youcanbesurethatSlackisthewaytogo.Therefore,whynotkeepingyourteampostedontheirGCPcostwithautomatednotificationsinaSlackchannelwhentheyreachsomethresholdsaboutamaximumbudget? It’seasytodoexactlythat,andIhavemadethingsconvenientforyoubywrappingallwhatyouneedinadeploymentmanagerconfigfile.Firstaquickarchitectureofthewholesetup: Lookingforthegithub?Igotyoucovered.It’shere. StartbycreatingaSlackappforyourworkspace, activateincomingwebhook andcopytheBotUserOAuthAccessToken(youwillneedtoreplaceitinthevalues.yaml;BOTTOKENvalue) Replacethevaluesinthe“values.yaml”filewithyourownSlackandprojectvalues.Onlythefirstfourvaluesaremandatorytochange;therestcanbeleftasdefault. InstalltheSlackappyoujustcreatedinyourSlackworkspaceandinviteittothechannelwhereitwillsendthealerts.  Installytttemplatinglibrary(youwon’tregretit,it’sanamazingtemplatingtool) Runthefollowingcommandattheroot.Itwillinjectthevaluesfrom“values.yaml”intothedifferentconfigurationfilesandcreateanewfolder(“output-dir”)withthefilesneededforthedeployment. ActivatethedeploymentmanagerAPI Runthefollowingcommandattheroot(don’tforgettoreplaceyourprojectID) ThiswilldeployrespectivelyaPub/Subtopic,aCloudFunctionandatableinBigQuery(theresourcesinthebluerectangleabove).Asyoudon’twanttoreceivenotificationsevery30minutes(frequencyatwhichPub/Subreceivesdatafrombilling),theCloudFunctionwillcheckwithBigQueryifithasalreadysentanotificationforthecurrentbillingalert;ifyes,nonotificationissenttoSlack.  Lastbutnotleast,youneedtocreateyourbillingalertsthroughthebillingUIandticktheboxtosendthisdatatothePub/Subtopiccreatedwithdeploymentmanager(topicbilling-alerts).ThisstepwillfeedyourPub/Subtopicwiththerelevantbillingdataandisthefirststepinthearchitectureabove. Slackslashcommands Nowthatwehavedashboardsandautomaticalertingmessages,let’sgiveanyonethepossibilitytointeractwithyourbillingviaSlack;thisisdonethroughslashcommands. SlashCommandsareinitiatedfromthemessageboxinSlack,buttheyaren’tmessages.AsubmittedSlashCommandwillcauseapayloadofdatatobesentfromSlacktoanapp,allowingtheapptorespondinwhateverwayitwants.–OfficialSlackdocumentation SlashCommandsareeasytosetupfromannew/existingapplicationintoSlackAPI:youjusthavetoprecisethecommandname.Then,togetSlacklinkedwithyourGCPproject,createaCloudFunctionwithanHTTPTriggersetupwiththe“RequestURL”valuefromyourslashcommand.DoingsowilltriggeryourCloudFunctioneverytimeyouwillruntheslashcommandfromanychanneloftheSlackworkspace(s)wheretheSlackappisinstalled. What’slefttodoafterwards?Well,Iguessit’stimeforcoding!Indeed,togetSlackcommandstowork,thesearetheonlytwotaskstodo.Ifyou’reworkingwithBigQuery,thecodingprocesscanbeevenfasterasthefunction“getBigQuery”youcanfindinourrepoisallyouneedtoqueryyourBigQueryinstancefromyourCloudFunction.  Slackcommandsfor+3secondscomputingtime However,beawarethatSlackcommandsaretimingoutafterrunningfor3s.Ifyourqueryneedslesstimetocompute,thensticktothestructureofour“current-billing”function. Nevertheless,forbiggercomputingtime,youmaywanttolookatour“delta-billing”function.Inthatcase,youwillneedtosendapackagebeforeprocessingyourquerytonothaveyourcommandcancelledandthensendanotherHTTPmessagemanuallyattheendofyourfunction.Toknoweverythingyouneedaboutdelayedmessages,pleasecheckthislinkattheSlackofficialdocumentation. Deployingfunctions OK,timetodeploytwofunctionswehavebuiltforyouandthatyoucanuseasSlackslashcommand.Todeploythefirstone,runthefollowinggcloudcommand: Oncedeployed,youcanretrievetheURLofthisfunctionbyclickingonthefunctionitself.KeepthisURLcloseandgointheSlackappyoucreated.Clickonthe“slashcommand”sectionandcreateanewone.ThemostimportantparameteristheRequestURL.Pastetheonefromyourfunction.Specifyaswellthecommandas“/currentbilling”andsavethecommand.YouruserscannowaskforthetotalamountspentonGCPforthismonthbywriting“/currentbilling”. Wehavebuiltanotherfunctionthatgetsyouthedeltabillingbetweentwodates.It’susefulforinstancetocheckeachdayhowyourbillinghasincreasedfromthedaybefore,andtopinpointquicklyananomalythatyoucanthendeeperinvestigate.Todeploythisfunction,runthefollowinggcloudcommand: Thenrepeatthestepsyoudidforthepreviousslashcommand.CopytheURLofthenewfunction,createanewslashcommand,etc.Thisslashcommandexpectstwodateargumentsafterthecommand.Forinstance,ifyounamedyourcommand“/deltabilling”thentogetthedifferencebetweenthe05/07/2019and06/07/2019,youwillhavetowrite: “/deltabilling2019-07-052019-07-06” YouhavenowtransformedyourSlackintoacostmonitoringcenterforyourGCP.Feelfreetocreatemorefunctionsandmakeitevenmorepowerful.  Ifyouhavereaduntilhereandimplementedthesethreesolutions,youarenowmasteringyourCloudcostsinaproactiveway.YouhavedynamicdashboardstomonitorandgetinsightsaboutyourGoogleCloudPlatformcosts,coupledwithdynamicnotificationsandslashcommandsinSlacktowarnyouaboutbudgetalerts.Bymasteringyourcosts,youwillalsofindnewwaystocuttheminefficientways.  IfyouneedanyassistanceineitherbuildingcustomiseddashboardsorCloudFunctionsdeployment,feelfreetoreachmeatcharles.verleyen@fourcast.io.Iwillbegladtohelpyou.  WanttolearnmoreaboutGoogleCloudPlatform?  Readmore HighcontrastDefaultcontrast Searchfor close search close StrategicPartners Countries Global English Belgium English Français Nederlands Denmark Dansk France Français Germany Deutsch Italy English Luxembourg English Mexico Español Netherlands English Dutch Norway Norsk Serbia English Serbian Spain Español Turkey English UK English Regions Africa (Morocco,Tunisia) English Alps (Austria,CzechRep,Switzerland,Slovakia) English German French MiddleEast (SaudiArabia,UnitedArabEmirates,Jordan) English Arabic