503 Service Unavailable - Backend Server | Apigee Edge

The client application receives an HTTP response status 503 with the message Service Unavailable following an API proxy call. Error messages. You can see one of ... ApigeeEdge PrivateCloud Latest v4.51.00 End-of-life v4.50.00 Deprecated/Unsupported v4.19.06 v4.19.01 v4.18.05 v4.18.01 v4.17.09 v4.17.05 v4.17.01 v4.16.09 v4.16.05 v4.16.01 Hybrid APIMonitoring Sense APIs Troubleshooting Releases Pricing Community Language English 日本語 Signin Troubleshooting ApigeeEdge PrivateCloud More Hybrid APIMonitoring Sense APIs Troubleshooting Releases Pricing Community Allsupport Antipatterns Introductiontoantipatterns Allantipatterns Policyantipatterns UsewaitForComplete()inJavaScriptcodeSetalongexpirationtimeforOAuthtokensUsegreedyquantifiersintheRegularExpressionProtectionpolicyCacheerrorresponsesStore>512kincacheLogto3rd-partywithJavaScriptpolicyInvokeMessageLoggingmultipletimesinanAPIproxyConfigureanon-distributedquotaReuseaQuotapolicyUsetheRaiseFaultpolicyunderinappropriateconditionsAccessmulti-valueHTTPheadersincorrectlyinanAPIProxyUseServiceCallouttoinvokebackendserviceinnotargetproxy Performanceantipatterns LeaveunusedNodeJSproxiesdeployed Genericantipatterns InvokemanagementAPIfromaproxyInvokeaproxywithinaproxyManageEdgeresourceswithoutusingsourcecontrolmanagementDefinemultiplevirtualhostswithsamehostaliasandportnumberLoadBalancewithasingletargetserverwithMaxFailuressettoanon-zerovalueAccesspayloadwithstreamingDefinemultipleProxyEndpointsinanAPIProxy Backendantipatterns AllowaslowbackendDisableHTTPpersistent(reusablekeep-alive)connections EdgeforPrivateCloudantipatterns AddcustominformationtoApigee-ownedschemainPostgresdatabase Errorcatalogs Runtimeerrorcatalog Producterrorplaybooks Introductiontoplaybooks Problemcategories Diagnosticsinformationcollectionguides Diagnosticscollectionintroduction PrivateCloud AnalyticsproblemsCassandraproblemsDeploymenterrorsRuntimeissuesUnabletostartEdgeComponentsZooKeeperproblems Errormessagereference Allplaybooks Analytics AnalyticsdatastuckinQpidddeadletterqueueAnalyticsreportstimeoutCustomvariablenotvisibleNodatainanalyticsdashboardsPostgresserveroutofdiskspaceAddinganddeletinganalyticscomponentsinanalyticsgroupsCustomDimensionsnotappearingwhenmultipleaxgroupshavebeenconfiguredReporttimedout Debugging EnablingNginxdebuglogsontheRoutersHowtomakedirectAPIrequeststoroutersormessageprocessors Deployment ErroraccessingdatastoreErrorfetchingchildrenforpathErrorprocessingupdatesConfigurationfailedTimeouterror Developerportal(Drupal) OverviewCommunicationissuesInternalerror Installation Unabletoopentheapigee-repo.rpm Integratedportal CustomdomainconfigurationfailswithinvalidvirtualhosterrorinintegrateddeveloperportalCustomdomaindoesnotmatchCNAMErecordinintegrateddeveloperportalTLScertificateexpiredinintegrateddeveloperportalcustomdomainconfiguration OpenLDAP TroubleshootingOpenLDAPproblems UI/Management APIrequestsnotcapturedintheEdgeUIErrorcreatingtracesessionHTTP304responseintracesessionPasswordresetissues Monetization TroubleshootingmonetizationproblemsMonetizationnotificationsarenotworking Nginx(Router) Badconfigfiles Permissions ForbiddenerrorsInsufficientpermissionserrors PostgreSQL Upgradefailure Runtime ErrorCode-messaging.adaptors.http.flow.* 400BadRequest-DecompressionFailureAtRequest404Multiplevirtualhostswiththesamehostalias404Unabletoidentifyproxyforhost500InternalServerError-BackendServer502BadGateway-DecompressionFailureAtResponse502Badgatewaytimeout502BadgatewayunexpectedEOF503Serviceunavailable503Serviceunavailable-backendserver503Serviceunavailable-NoActiveTargets503Serviceunavailable-NoActiveTargets-HealthCheckFailures503Serviceunavailable-prematureclosurebybackendserver503ServiceUnavailable-SSLHandshakeFailure504Gatewaytimeout504Gatewaytimeoutfrombackendserver ErrorCode-protocol.http.* 400BadRequest-DuplicateHeader413RequestEntityTooLarge-TooBigBody414Request-URITooLong-TooBigLine415UnsupportedMediaType-UnsupportedEncoding431RequestHeaderFieldsTooLarge-TooBigHeaders500InternalServerError-BadFormData500InternalServerError-BadPath500InternalServerError-EmptyPath502BadGateway-DuplicateHeader502BadGateway-ResponseWithBody502BadGateway-Response405withoutAllowHeader502BadGateway-TooBigBody502BadGateway-TooBigHeaders502BadGateway-TooBigLine503ServiceUnavailable-Proxytunnelcreationfailedwith403SSLhandshakefailures-badclientcertificateTLS/SSLhandshakefailures400Badrequest-plainHTTPrequestsenttoHTTPSport400Badrequest-SSLcertificateerror499Clientclosedconnection500Internalserver500Internalserver-streamingenabled502Badgateway504Gatewaytimeout-routertimingout SSO/SAML SSOZoneadministrationpage:unauthorizedrequesterror Zookeeper UnabletostartZookeeperConnectionlosserrorsDataissues ApigeeAdapterforEnvoyplaybooks IntroductiontoApigeeAdapterforEnvoyplaybooks Allplaybooks EnvoyproxyfailswithHTTP403ForbiddenerrorinApigeeAdapterforEnvoy EdgeMicrogatewayplaybooks IntroductiontoEdgeMicrogatewayplaybooks Allplaybooks 502BadGateway-Self-signedcertificateinchain502BadGateway-Sockethangup Policyerrorplaybooks Introductiontopolicyplaybooks Policyruntimeerrors AccessControlAssignMessageBasicAuthenticationDecodeJWTExtractVariablesFlowCalloutJavaCalloutJavaScriptJSONThreatProtectionJSONtoXMLKeyValueMapOperationsPythonScriptQuotaResetQuotaServiceCalloutSOAPMessageValidationSpikeArrestXMThreatProtectionXMLtoJSONXSLTransform Policydeploymenterrors AssignMessageBasicAuthenticationDecodeJWTExtensionCalloutExtractVariablesGenerateJWTInvalidateCacheJavaCalloutJavaScriptKeyValueMapOperationsLookupCacheMessageLoggingPopulateCachePythonScriptQuotaRegularExpressionProtectionResetQuotaResponseCacheSAMLAssertionServiceCalloutSOAPMessageValidationSpikeArrestStatisticsCollectorVerifyJWTXMLtoJSONXSLTransform Servicerequests Introductontoservicerequests Apigeeservicerequests PortalusermanagementrequestsProvisioningrequestsStress/Load/PenetrationtestrequestsInfrastructurecapacitymanagementrequestsFeaturemanagementrequestsSAMLconfigurationrequestsVirtualhostconfigurationrequestsApigeesubscriptionrequestsServicerequestsFAQ Othertroubleshootingresources Diagnostictoolsandlogs sosreporttool HTTPstatuscodes PrivateCloudtroubleshootingguide(PDFversion) Trace,analytics,andotherresources Latest v4.51.00 End-of-life v4.50.00 Deprecated/Unsupported v4.19.06 v4.19.01 v4.18.05 v4.18.01 v4.17.09 v4.17.05 v4.17.01 v4.16.09 v4.16.05 v4.16.01 GoogleiscommittedtoadvancingracialequityforBlackcommunities. Seehow. ApigeeEdge Troubleshooting Sendfeedback 503ServiceUnavailable-BackendServer You'reviewingApigeeEdgedocumentation.ViewApigeeXdocumentation. Wasthistroubleshootingplaybookhelpful?Pleaseletusknow byclicking SendFeedback . Videos Watchthefollowingvideotolearnmoreaboutsolving503ServiceUnavailableErrors. Video Description 503ServiceUnavailableErrorfromBackendServer Learnaboutthefollowing: Introductionto503ServiceUnavailableErrorinApigeeEdge Troubleshootingandresolvingareal-time503ServiceUnavailablefrom BackendServer Symptom TheclientapplicationreceivesanHTTPresponsestatus 503withthemessageServiceUnavailablefollowing anAPIproxycall. Errormessages Youcanseeoneofthefollowingerrormessages: HTTP/1.1503ServiceUnavailable HTTP/1.1503ServiceUnavailable:Back-endserverisatcapacity Youmightalsoseeanerrormessagelikethefollowing intheHTTPresponse: Theserveristemporarilyunabletoserviceyourrequestdueto maintenancedowntimeorcapacityproblems.Pleasetryagainlater. Note:Theaboveresponsecodeanderrormessageareonlyexamples. Insomecases,youmayreceiveonlytheerrorresponsecodewithoutanyerrormessage. Theformatandthecontentoftheerrorresponsecodeandtheerrormessagemayvarydependingon thebackendserverimplementation. Causes TheHTTPstatuscode503meansthattheserveriscurrentlyunabletohandletheincoming requests.Usually,thiserroroccursbecausetheserveristoobusyoris temporarilydownformaintenance. Possiblecausesforthe503ServiceUnavailableresponseare: Cause Description Whocanperformthetroubleshootingsteps OverloadedServer Thebackendserverisoverloadedorbeyonditscapacityandcannothandleanynew incomingclientrequests. EdgePublicandPrivateCloudusers Serverundermaintenance Thebackendservermaybeundermaintenancetemporarily. EdgePublicandPrivateCloudusers Cause:OverloadedServer/ServerunderMaintenance InApigeeEdge,the503ServiceUnavailableErrorcanbereturnedfromabackendserver undereitherofthefollowingcircumstances: Abackendserverisoverloaded/busyandcannothandleanynewrequests. Thebackendserverisdownforatemporaryperiodduetomaintenance. Diagnosis Todiagnosetheerror,youcanuseanyofthefollowingthreemethods: Tracetool Nginxaccesslogs Directcalltobackendserver Clickthetabsbelowtolearnabouteachmethod. Tracetool Enablethetracesession, andmaketheAPIcalltoreproducetheissue-503ServiceUnavailable. Selectoneofthefailingrequestsandexaminethetrace. Navigatethroughvariousphasesofthetraceandlocatewherethefailureoccurred. Ifyoufindthatthe503errorisreturnedasaresponsefromthetargetserver, thecausefor503erroristhetargetserver. Here’sasampletracescreenshotshowing503ServiceUnavailableresponsereceived fromthetargetserver: ClicktheResponsereceivedfromtargetserverphaseandgothroughthe ResponseHeadersandResponseContentsectionstoseeiftheyhaveanyusefulinformation: TheResponseHeadersmaycontaintheServerheader,whichindicates wheretheerrorresponsewassentfrom. TheResponseContentmaycontainadditionalinformationaboutwhy thetargetserversentthe503responsecode. Note:Theformatandthecontentoftheresponsepayloadmaydifferbasedonthe targetserverimplementation. Confirmthatthe503erroriscomingfromthetargetserverbychecking thevaluesofX-Apigee-fault-sourceandX-Apigee-fault-codeintheAX (AnalyticsDataRecorded)Phaseinthetraceusingthestepsgivenbelow: ClickonAX(AnalyticsDataRecorded)Phaseasshowninthescreenshotbelow: ScrolldownthePhaseDetailstotheResponseHeaderssectionanddeterminethevalues ofX-Apigee-fault-codeandX-Apigee-fault-sourceasshownbelow: IfthevaluesofX-Apigee-fault-sourceandX-Apigee-fault-codematchthevalues showninthebelowtable,youcanconfirmthatthe503erroriscomingfromthe targetserver: ResponseHeaders Value X-Apigee-fault-source target X-Apigee-fault-code messaging.adaptors.http.flow.ErrorResponseCode Checkifyouareusingproxychainingi.e.ifthetargetserver/targetendpointis invokinganotherproxyinApigee.Todeterminethis: NavigatebacktotheRequestsenttotargetserverphaseand clicktheShowCurlbuttonanddeterminethetargetserverhostalias. Ifthetargetserverhostaliasispointingtoavirtualhostalias,thenit's proxychaining.Inthiscase,youneedtorepeatalltheabovestepsforthechained proxytillyoudeterminewhatisactuallycausingthe503ServiceUnavailableerror. Inthesecases503ServiceUnavailablemayhappeninotherchainedproxiesatother stagesaswellwhichcanbediagnosedusing thisplaybook. Ifthetargetserverhostaliaspointstoyourbackendserver,thengoto Resolution. Nginxaccesslogs Note:ThestepsinthissectionareforEdgePrivateCloudusersonly. YoucanalsorefertoNginxlccesslogstodeterminewhetherthe503statuscodewassent bythebackendserver.Thisisparticularlyusefuliftheissuehasoccurredinthepast oriftheissueisintermittentandyouareunabletocapturethetraceinUI. UsethefollowingstepstodeterminethisinformationfromNginxaccesslogs: ChecktheNginxaccesslogs. /opt/apigee/var/log/edge-router/nginx/~._access_log Searchforany503ErrorsforthespecificAPIproxyduringaspecificduration (iftheproblemhappenedinthepast)orforanyrequestsstillfailingwith503. Ifthereareany503Errors,thencheckiftheerroriscomingfromthebackendserver. IfthevaluesofX-Apigee-fault-sourceandX-Apigee-fault-codematchthe valuesshown inthetablebelow,the503erroriscomingfromthebackendserver: ResponseHeaders Value X-Apigee-fault-source target X-Apigee-fault-code messaging.adaptors.http.flow.ErrorResponseCode Here'sasampleentryshowingthe503errorcausedbythetargetserver: ReviewthespecificAPIProxyandensurethatyouareusing proxychainingi.e., ifthe targetserver/targetendpointisnotinvokinganotherproxyinApigee.Ifyouareusing proxychaining,youneedtorepeatalltheabovestepsforthechainedproxyuntil youdeterminewhatisactuallycausingthe503ServiceUnavailableerror.Inthesecases, 503ServiceUnavailablemayhappeninotherchainedproxiesatotherstagesaswell, whichyoucandiagnoseusingthisplaybook. Ifyouconfirmthatyouarenotusingproxychaining,andthe503erroriscomingfromyour backendserver,thengotoResolution. CalltoBackendServer Youcanmakeadirectcalltothebackendserverandverifythatyouaregettingthesame 503ServiceUnavailableresponseasreceivedwhentherequestwasmadethroughApigeeEdge. Ensurethatyouhavealltherequiredheaders,queryparametersandanycredentialsthat needtobepassedtothebackendserveraspartoftherequest. Ifthebackendserviceispubliclyaccessible,youcanusethecurlcommand, PostmanoranyotherRESTClientandinvokethebackendserverAPIdirectly. IfthebackendserverisaccessibleonlyfromtheMessageProcessors,youcanusethe curlcommand,PostmanoranyotherRESTClientandinvokethebackendserverAPIdirectly fromtheMessageProcessor. Note:ThiscanbeperformedonlybyPrivateCloudusers. Verifythatthebackendserviceisindeedreturning503ServiceUnavailableerror. Resolution Ifyouascertainthatthe503erroriscomingfromthebackendserver,youcandothe followingtoresolvetheissue: Iftheissueiscausedbecausethebackendserverisdownformaintenance, youcanbringthebackendserveronlineafterthemaintenanceperiod. Iftheissueiscausedbecausethebackendserverisoverloaded,then fixtheissueifyouhaveaccesstothebackendserver.Otherwise youmayneedtoworkwithyourbackendserverteamtofixtheissue. DiagnoseIssuesusingAPIMonitoring Note:IfyouareaPrivateClouduser,skipthisprocedure. APIMonitoringenablesyoutoisolate problemareasquicklytodiagnoseerror,performance, andlatencyissuesandtheirsource,suchasdeveloperapps,APIproxies,backendtargets, ortheAPIplatform. Stepthroughasample scenariothatdemonstrateshowtotroubleshoot5xxissueswithyourAPIs usingAPIMonitoring.Forexample,youmaywanttosetupanalerttobenotifiedwhenthenumber ofmessaging.adaptors.http.flow.ErrorResponseCodefaultsexceedsaparticularthreshold. Note: APIMonitoringlists503errorsunderthefaultcode messaging.adaptors.http.flow.ErrorResponseCode,iftheseerrorsarecomingfromthebackend serverorfromanotherproxyinvokedaspartofproxychaining. Ifyouwanttobenotifiedwhena503errorresponseissentfromthebackendserver, youwouldneedtosetupthealertfor503statuscodewithfaultcodeas messaging.adaptors.http.flow.ErrorResponseCode. Mustgatherdiagnosticinformation Iftheproblempersistsevenafterfollowingtheaboveinstructions,pleasegatherthe followingdiagnosticinformation,andthencontact ApigeeSupport. Ifyouareapublicclouduser,providethefollowinginformation: OrganizationName EnvironmentName APIProxyName Completecurlcommandtoreproducethe503error Tracefilecontainingtherequestswith503ServiceUnavailableerror Ifthe503errorsarenotoccurringcurrently,providethetimeperiodwiththetimezone informationwhen503errorsoccurredinthepast. Ifyouareaprivateclouduser,providethefollowinginformation: Completeerrormessageobservedforthefailingrequests. Organization,EnvironmentNameandAPIProxyNameforwhichyouareobserving503errors. APIProxyBundle. Tracefilecontainingtherequestswith503ServiceUnavailableerror. Nginxaccesslogs. /opt/apigee/var/log/edge-router/nginx/~._access_log MessageProcessorlogs. /opt/apigee/var/log/edge-message-processor/logs/system.log Thetimeperiodwiththetimezoneinformationwhenthe503errorsoccurred. Sendfeedback Exceptasotherwisenoted,thecontentofthispageislicensedundertheCreativeCommonsAttribution4.0License,andcodesamplesarelicensedundertheApache2.0License.Fordetails,seetheGoogleDevelopersSitePolicies.JavaisaregisteredtrademarkofOracleand/oritsaffiliates. Lastupdated2022-05-13UTC. [{ "type":"thumb-down", "id":"missingTheInformationINeed", "label":"MissingtheinformationIneed" },{ "type":"thumb-down", "id":"tooComplicatedTooManySteps", "label":"Toocomplicated/toomanysteps" },{ "type":"thumb-down", "id":"outOfDate", "label":"Outofdate" },{ "type":"thumb-down", "id":"samplesCodeIssue", "label":"Samples/codeissue" },{ "type":"thumb-down", "id":"otherDown", "label":"Other" }] [{ "type":"thumb-up", "id":"easyToUnderstand", "label":"Easytounderstand" },{ "type":"thumb-up", "id":"solvedMyProblem", "label":"Solvedmyproblem" },{ "type":"thumb-up", "id":"otherUp", "label":"Other" }] Needtotellusmore? AboutApigee We'repartofGoogle Events Partners eBooksandwebcasts Communityandsupport Community Supportoverview Status Supportportal Resources Developeroverview 4-minutevideosfordevelopers Apigeeblog Android Chrome Firebase GoogleCloudPlatform Allproducts Terms Privacy Language English 日本語