Introducing AWS Application Cost Profiler, offering user ...

Last week we announced the launch of AWS Application Cost Profiler (ACP), a service that allows customers, running multi-tenant applications ... BlogHome Category  Edition  Follow  Architecture AWSCloudFinancialManagement AWSCloudOperations&Migrations AWSPartnerNetwork AWSPodcast AWSMarketplace AWSNews BigData BusinessProductivity Compute ContactCenter Containers Database Desktop&ApplicationStreaming Developer DevOps EnterpriseStrategy Front-EndWeb&Mobile GameTech HPC Infrastructure&Automation Industries InternetofThings MachineLearning Media Messaging&Targeting Networking&ContentDelivery OpenSource PublicSector QuantumComputing Robotics SAP Security,Identity,&Compliance Startups Storage Training&Certification WindowsonAWS 中国版 ÉditionFrançaise DeutscheEdition 日本版 한국에디션 EdiçãoemPortuguês EdiciónenEspañol EnglishEdition Версиянарусском EdisiBahasaIndonesia TürkçeSürüm MailingList AWSCloudFinancialManagement IntroducingAWSApplicationCostProfiler,offeringuser-basedcostdataofsharedAWSresources Voicedby AmazonPolly   October6,2021:Thisblogpostisunderconstruction.Pleasereferbacktothispostinadayortwoforthemostaccurateandhelpfulinformation. Ithasbecomeincreasinglycommonforcustomerstodeployapplicationsthatshareinfrastructureresourcesforthepurposeofcloudcostoptimization.Asmoreorganizationsaremovingtothesharedinfrastructuremodels,theycontinuouslylookforwaystounderstandtheconsumptionoftheirendusersortenants.Thisisespeciallytruewithsoftware-as-a-service(SaaS)providers,whooftenachieveeconomiesofscalebysharingcompute,storage,andothercloudservices.Ideally,SaaSprovidersneedaccesstometricstoapportioninfrastructurecostsatthetenantleveltounderstandmarginsandconsumptionpatterns.AdoptionofSaaStechnologiesbycustomersandindependentsoftwarevendors(ISVs)isgrowing,demandingthecapabilitytomonitorandtrackcostpatternsofsharedresourcesbyvariousdimensions.UnderstandingcostpertenantinanaccurateandstandardizedwayiskeytobusinessesrunningSaaSoranymulti-tenantapplications. Inamulti-tenantapplication,tenantssharesomeormostoftheinfrastructureresources.Asaresult,allocatingeachtenant’scostcanbechallenging.Itbecomesmorecomplicated,assystemsdeploymentandconsumptioncanvarybyservicetype(forexample,computevsstorage,networkvsdatabases).Togetgranularcostbreakdown,applicationownershavetoinvestheavilyonengineeringandinfrastructureresourcestobuildcustomsolutions,querylogsfrommultiplesourcesforad-hocanalysis.Orthey’dsettleforusingstatic,roughestimates,likenumberofusersorgigabytesofdatastoredtore-distributeAWScostsacrosstenantbase.Theexistingcloudmanagementtoolsoftenlackthemulti-tenancyconsumptionviewtodisplaytenant’sproportionofcostandusage,whichaddscomplexitytomanageapplicationswithsharedinfrastructure. LastweekweannouncedthelaunchofAWSApplicationCostProfiler(ACP),aservicethatallowscustomers,runningmulti-tenantapplications,togetgranularcostbreakdownofsharedAWSresourcesacrosstenantbase.AWSApplicationCostProfilerprovidescustomers,especiallySaaSISVs,astandardmechanismtocorrelateandreporttheirinfrastructurecostforeachcustomerortenant.WiththegranularcostinsightsofAWSresourceconsumption,ISVscandevelopgo-to-marketstrategieswithtier-basedsupportorconsumption-basedpricingfortheirproductsandeffectivelymanagecostsofamulti-tenantarchitecturemodel.Or,organizationsrunningmulti-tenantapplicationscanusethedatatodefineaccuratecostallocationmodelforchargebackpurpose.ApplicationCostProfilerrequiresapplicationownerstoinstrumenttheirapplicationfortenantmetadatainacontextualformat.Wewillcoverwhatcustomersneedtodoinordertoproperlyinstrumenttheirtenantmetadatalaterinthisblog.   GettingstartedwithAWSApplicationCostProfiler(ACP) GettingstartedwithAWSApplicationCostProfiler(ACP),isatwo-stepprocess:step1:aone-timesetuptoconfigureACPforgeneratingfinalconsumptioninsightsreport;step2:Instrumenttheapplicationfortenantmetadata. ThetwostepscanbedoneinanyorderbutbothstepsneedtobecompletedbeforeACPcangenerateconsumptioninsights. Tosavetimeforyourinitialsetup,wecreatedthreeAWSCloudFormationtemplatesthatyoumayfindhelpful. Seedetailsinthesectionsbelow. Step1:ConfiguretheApplicationCostProfilerforreportingconsumptioninsights ApplicationCostProfilerreportconfigurationmustbedefinedintheAWSConsole,viatheAWSCommandLineInterface,orusingoneoftheAWSSDKs.WewilldemonstratehowyouconfigurethisintheAWSconsole.ThereportconfigurationistoinstructAWSApplicationCostProfilerwheretodeliverthetenantcostreportsto.ThedestinationisanAmazonSimpleStorageService(AmazonS3)bucketwiththeproperpermissionsforApplicationCostProfilertowritethereportsto. Prerequisites Forthiswalkthrough,youshouldhavethefollowingprerequisites: AnAWSaccount; AWSIdentityandAccessManagement(IAM)userwithconsoleaccessandadminprivileges; EnableAWSCostExplorer. SetupAmazonS3bucketandreportconfiguration 1.LogintotheAWSConsole; 2.VerifythatCostExplorerhasbeenenabled(ImportantasAWSApplicationCostProfilerwillnotprocesstenantusagedatawithoutCostExplorerbeingenabled); 3.ClickonthelaunchstackbuttonbelowtolaunchourfirstCloudFormationstackthatwillinstallandconfigureaS3bucketwithproperApplicationCostProfilerpermissions andserversideencryptionsettings,anAmazonEventBridgerule,andanAmazonSimpleNotificationService(AmazonSNS) TopicforAWSApplicationCostProfilereventsthatyoucanoptionallysubscribeto; 4.OncethelaunchoftheCloudFormationstackhascompleted,intheN.Virgina/us-east-1regionusetheconsolesearchfunctiontonavigatetothe“AWSApplicationCostProfiler”landingpage; 5.IntheAWSApplicationCostProfilerdashboard,clickon“Getstartednow”.     6.Setupanewreportconfiguration: a.ReportName–Thisisuserdefinedandcannotbechangedoncesaved b.ReportDescription–Thisisauserdefineddescriptionofthereportconfiguration(optional) c.AmazonS3BucketName–ThisistheAmazonS3bucketwhereAWSApplicationCostProfilerwilldeliverthereportsto.ThisbucketwascreatedusingthepreviousCloudFormationtemplate.Bucketisnamed“acp-{REGION}-{ACCOUNT_ID}”,substituting{REGION}withtheAWSregionthattheCloudFormationtemplatewasdeployedto,e.g.us-east-1,and{ACCOUNT_ID}withtheactualAWSaccountidthatwasusedtodeploytheCloudFormationtemplate.Soforexample“acp-us-east-1-987654321”.ThisReportBucketnamecanalsobefoundinthe“Resources”sectionoftheCloudFormationstackdeployedabove. d.S3Prefix–ThisistheprefixintheS3bucketusedabovewhereAWSApplicationCostProfilerwilldeliverthereportsto.Thes3BucketdeployedintheCloudFormationtemplateaboveenabledwritepermissionsforAWSApplicationCostProfilertothe“reports”prefix,thereforeenter“reports”forS3prefixhere. e.TimeFrequency–ChoosewhetherthereportisgeneratedonaDailyorMonthlycadence,orBoth. f.ReportOutputFormat–ChoosethetypeoffilethatwillbecreatedwithinyourAmazonS3bucket.IfyouchooseCSV,ApplicationCostProfilercreatesacomma-separatedvaluestextfilewithgzipcompressionforthereports.IfyouchooseParquet,aParquetfileisgeneratedforthereports.     7.Clickthe“Configure”button.ApplicationCostProfilerwillverifytheexistenceoftheAmazonS3bucketdefinedaboveaswellastheserviceswritepermissionstotheprefixdefinedabove.Ifsuccessfulyouwillseeaconfirmation.     8.Click“OK”toreturntotheAWSApplicationCostProfiler Nowthatyou’vesetupaS3bucketdestinationwithpermissionsandsetupareportconfigurationwithintheAWSCostApplicationProfilerconsole,youarereadytoprepare,upload,andimportyourtenantusagedata. Step2:Reportingtenantusagedatafromyourservices–anexample Inordertogeneratereports,AWSApplicationCostProfilerrequiresyoutoprovidetenantusagedata.ThisinformationneedstobeuploadedtoanS3locationthatAWSApplicationCostProfilerhaspermissionstoreadfrom.TheS3bucketcreatedinthefirstCloudFormationtemplateabovehasgrantedAWSApplicationCostProfilerreadaccesstothe“import/”prefix. BygivingApplicationCostProfileraccesstoyourusagedatafromS3bucket,youallowApplicationCostProfilertotemporarilycopysuchusagedataobjectstotheUSEast(NorthernVirginia)Regionwhileprocessingreports.ThesedataobjectswillbekeptintheUSEast(NorthernVirginia)Regionuntilthemonthlyreportgenerationiscomplete.Toavoidincurringdatatransfercharges,youcanconfigurerequesterpaysonbucket. Asanexample,let’sinstrumentasampleserverlessapplicationtotrackcostacrossyourtenantbase.OursecondCloudFormationtemplatewilldeployabasicserverlessapplicationusingAmazonAPIGatewayandAWSLambda.     TheLambdafunctionperformsafewcomplexmathoperationstosimulatebusinesslogic. exports.handler=(event,context)=>{ console.log("TENANT_ID\t%s",event.tenantId); mySlowFunction(13); constresponse={ statusCode:200, body:"SUCCESS" }; returnresponse; }; … Deploythesampleserverlessapplication 1. LogintotheAWSConsole 2.ClickonthelaunchstackbuttonbelowtolaunchthesecondCloudFormationstackthatwillinstallandconfigurethesampleserverlessapplication.It’simportantthatthisstackbelaunchedinthesameregionastheCloudFormationstacklaunchedpreviously. 3.OncethisCloudFormationtemplatehascompleteddeploying,gototheOutputssectionofthestackandnotethetheapiGatewayInvokeURLvalue,itwillbeinthefollowingformat: https://i3pbggxbf2.execute-api.us-east-1.amazonaws.com/call?tenantId=123 4.UsingtheapiGatewayInvokeURLidentifiedabove,openawebbrowserandpastethevalueintothelocationwindow.Thiswillexecutethesampleserverlessapplicationsimulatingusebytenant“123”.NowchangethevalueaftertenantId=inthebrowserlocationwindowtosimulatemultipletenants,suchastenantId=10ortenantId=20,etc.pressingentereachtimetoexecutethesampleserverlessapplication.InvokingthisendpointinabrowserwillcausetheLambdafunctiontoexecuteandloganentryinthecorrespondingAmazonCloudWatchlog.YoucanalsouseutilitieslikeJMetertosimulatehundredsoreventhousandsoftenantsinvokingthissampleserverlessapplication. PleaseNote:Thisserverlessapplicationisoneoftheseveralapproachesyoucanusetotrackyourtenantusage.AWSApplicationCostProfilerrequiresthetenantusagereporttobeinCSVfileformatonly,whichistheendresultofthisexample.Asmentioned,therearemanydifferentwaystogenerateyourtenantusageCSVreport,dependingonyourapplicationarchitectureandyourcurrenttenantmodel. NowthatyouhavethetenantIDavailableintheLambdaCloudWatchlogsofthesampleserverlessapplication,it’stimetogeneratethetenantusagedatafile.Thisfilewillneedtobestructuredinaformatasdescribedbelow.Asareminder,onlyCSVfileswithfilenameas“.csv,.csv.gz,.csv.gzip”aresupportedbyACP. ApplicationTenantUsageDataElements ApplicationId–Identifiestheapplicationorproductinyoursystemthatisbeingused.Definesthescopeofthetenantmetadata. TenantId–Anidentifierinyoursystemforthetenantwhoisconsumingthespecifiedresource.ApplicationCostProfileraggregatestothislevelwithintheApplicationId. TenantDesc–(ValueOptional)Additionaldataaboutthetenantforyourownadditionalreporting. UsageAccountId–Theaccountthattheresourcerunsin(importantforaccountsthatarepartofanorganization). StartTime–Timestamp(inmilliseconds)fromEpoch,inUTC.Indicatesthestarttimeoftheperiodfortheusagebythespecifiedtenant. EndTime–Timestamp(inmilliseconds)fromEpoch,inUTC.Indicatestheendtimeoftheperiodfortheusagebythespecifiedtenant.\ ResourceId–AmazonResourceName(ARN)forresourcebeingused. Name–(Optional)AsanalternativetospecifyingaResourceId,youcanspecifyaNameresourcetagtoattributecoststoasetofresources(thefieldmustincludethevalueyouwanttousefortheNametag).ResourcetagsareenabledaspartofyourAWSCostandUsageReport.Formoreinformationaboutresourcetags,seeResourcetagsdetailsintheAWSCostandUsageReportUserGuide. PleaseNote:ApplicationId,TenantId,TenantDesc,UsageAccountId,StartTime,EndTime,ResourceId*areAWSApplicationCostProfiler-reservedKeywordsandcannotbeusedasNametagnames. InthisexamplewearegoingtogeneratethetenantusagedataaccordingtotheformatabovebyprocessingandreadingtheinformationfromtheCloudWatchlogsoftheLambdafunctionwewanttoreporton.Fortheinstrumentationinfrastructure,wewillbeusingthefollowingAWSresources: CloudWatchLogs:Inthisexample,wewilluseCloudWatchLogstoprovideustheneededinformationtogeneratetenantusagedatafilesfortheexampleLambda.APNsolutionslikeNewRelicandSplunkcouldalsomeetthisneed. CloudWatchEvent:WewilluseAWSCloudWatcheventstotriggertheinstrumentationLambdahourly,andsubmithourlytenantusagedatatoAWSApplicationCostProfiler. LambdaFunction:ThisLambdawillhavethelogictoextractinformationfromCloudWatchLogs,generatethetenantusagedatafilesanduploadthefilestotheS3bucketcreatedintheCloudFormationtemplatedeployedabove IAM:therightIAMpolicyneedstobeinplacefortheLambdafunctiontowritetotheS3bucket. CloudFormationTemplates:wewilluseaCloudFormationtemplatesoyoucaneasilydeploythesamplefunctionandtriggertogeneratethehourlytenantusagedata. Deploythesampletenantusagegenerator 1.LogintotheAWSConsole 2.ClickonthelaunchstackbuttonbelowtolaunchourthirdCloudFormationstackthatwillinstallandconfiguretheCloudWatchtenantusagegeneratorexample.It’simportantthatthisstackbelaunchedinthesameregionastheCloudFormationstacklaunchedpreviously.Thedefaultparameterfor“LambdaName”ispresetto“ACPSampleLambda”whichisthenameoftheLambdafunctionthatwasdeployedinthepreviousCloudFormationtemplate.ThisistheLambdafunctionwhichthecorrespondingCloudWatchlogswillbeanalyzedfortenantusagedata. WiththeabovestackdeployedandassumingtheoriginalAWSLambdafunctiontobereportedonhasbeenruninthelasthour,youshouldnowseeanewfilecreatedintheacp-{REGION}-{ACCOUNT_ID}bucketunderthe“imports”prefixeveryhourthattheInstrumentationLambdafunctionexecutesonthescheduledefinedabove.Thefilewillcontaintenantusagedatainthecsvformatsimilartothis: ApplicationId,TenantId,TenantDesc,UsageAccountId,StartTime,EndTime,ResourceId MyApp,Tenant1,,123456789012,1613681437032.9001,1613681437041.5312,arn:aws:ec2:useast-1:123456789012:instance/1234-abcd-example-1234 MyApp,Tenant2,,123456789012,1613681245531.4426,1613681245551.1323,arn:aws:ec2:useast-1:123456789012:instance/1234-abcd-example-1234 MyApp,Tenant1,,123456789012,1613681904815.3381,1613681904930.0972,arn:aws:ec2:useast-1:123456789012:instance/1234-abcd-example-1234 MyApp,Tenant2,,123456789012,1613681904765.1956,1613681904946.574,arn:aws:ec2:useast-1:123456789012:instance/1234-abcd-example-1234 TheInstrumentationLambdafunctionabovealsoinvokedtheapplicationcostprofiler.importApplicationUsageAPIwhichinformedAWSApplicationCostProfilertoprocessthegeneratedfileduringthenightlyand/ormonthlyreportgenerationcycle. ApplicationCostProfilerData IfthetimefrequencyforyourApplicationCostProfiler(ACP)reportwassetuptobedaily,itcantakeupto24hours,aftertheInstumentationLambdaruncallstheImportApplicationUsageAPIfortheACPdatatobeavailableintheS3ReportBucketandprefixedconfigured.UsingtheaboveCloudFormationsetup,theACPdatawillbeplacedinS3://acp-{REGION}-{ACCOUNT_ID}/reports/YYYY/MM/DD/part-*.csv.gzaccordingly.AnAmazonEventBridgeeventwillalsobegeneratedwhenACPdataisavailableinthefollowingformat: { "version":"0", "id":"01234567-EXAMPLE", "detail-type":"ApplicationCostProfilerReportGenerated", "source":"aws.application-cost-profiler", "account":"123456789012", "time":"2021-03-31T10:23:43Z", "region":"us-east-1", "resources":[], "detail":{ "message":"ApplicationCostProfilerreportdeliveredinbucket:acp-us-east-1-123456789,key:reports/2020/05/01/part-c744af14-57bd-4c51-b1d4-badc155ef7ef-c000.csv.gz" } } ThefollowingdataisavailableinthetenantcostcsvfilegeneratedbyACP AWSApplicationCostProfilerTenantCostBreakdownElementsonoutputfile PayerAccountId–ThemanagementaccountIDinanorganization,ortheaccountIDiftheaccountisnotpartofAWSOrganizations. UsageAccountId–TheaccountIDfortheaccountwithusage. LineItemType–Thetypeofrecord.AlwaysUsage. UsageStartTime–Timestamp(inmilliseconds)fromEpoch,inUTC.Indicatesthestarttimeoftheperiodfortheusagebythespecifiedtenant. UsageEndTime–Timestamp(inmilliseconds)fromEpoch,inUTC.Indicatestheendtimeoftheperiodfortheusagebythespecifiedtenant. ApplicationIdentifier–TheApplicationIdspecifiedintheusagedatasenttoApplicationCostProfiler. TenantIdentifier–TheTenantIdspecifiedintheusagedatasenttoApplicationCostProfiler.Datawithnorecordintheusagedataiscollectedinunattributed. TenantDescription–TheTenantDescspecifiedintheusagedatasenttoApplicationCostProfiler. ProductCode–TheAWSproductbeingbilled(forexample,AmazonEC2). UsageType–Thetypeofusagebeingbilled(forexample,BoxUsage:c5.large). Operation–Theoperationbeingbilled(forexample,RunInstances). ResourceId–TheresourceIDorAmazonResourceName(ARN)fortheresourcebeingbilled. ScaleFactor–Ifaresourceisover-allocatedforanhour,forexample,theusagedatareportedisequalto2hoursinsteadof1hour,ascalefactorisappliedtomakethetotalequaltheactualbilledamount(inthiscase,0.5).Thiscolumnreportsthescalefactorusedforthespecificresourceforthathour.Thescalefactorisalwaysgreaterthanzero(0)andlessthanorequalto1. TenantAttributionPercent–Thepercentageoftheusageattributedtothespecifiedtenant(betweenzero(0)and1). UsageAmount–Theamountofusageattributedtothespecifiedtenant. CurrencyCode–Thecurrencythattherateandcostarein(forexample,USD). Rate–Thebillingratefortheusage,perunit. TenantCost–Thetotalcostforthatresourceforthespecifiedtenant. Region–TheAWSRegionoftheresource. Name–IfyoucreatedresourcetagsforyourresourcesontheCostandUsagereport,orthroughtheresourceusagedata,theNametagisshownhere.Formoreinformationaboutresourcetags,seeResourcetagsdetailsintheCostandUsageReportUserGuide. ThefollowingisanexampleoftheactualcsvoutputintheACPreport. ThiscsvfilecanthenbequerieddirectlyusingAmazonAthenaorintegratedintoyourexistinganalyticsandreportingtoolssuchasAmazonQuickSightsuchastheexamplehere.     Conclusion Inthiswalkthrough,youhavesetupanewAWSApplicationCostProfiler(ACP)reportconfigurationincludingadestinationAmazonS3ReportBucketwithappropriatepermissions.YouthencompletedtheprocessofgeneratingtenantusagedataforasampleLambdaapplication,importedtheusagedataintoACP,andthenreceivedtheresultreports. Ourserviceteamprovidesreferenceimplementationguidancefordifferentarchitecturepatternstofacilitatetheinstrumentation.WealsohaveadedicatedSolutionArchitectsteamtosupportyouthroughtheinstrumentationandfacilitateon-boarding.Ifyouneedsupportforon-boarding,reachouttousataws-acp@amazon.comorcontactyouraccountteamtogetsupport. Tobetterunderstandthevalueandimportanceofcostpertenant,pleaserefertotheCalculatingTenantCostsinaSaaSEnvironmentblogpost,aswellastheAWSre:Inventsessionthatoutlinesthebasicapproachandchallengesassociatedwithcapturingthisinformation. We’dliketoextendourthankstotheserviceteam,specificallyourcolleagues,PreritPanwar(SeniorProductManager,AWSPartner),andMichaelKane(AWSCloudInfrastructureArchitect).