Resource Exhausted Error When Users Attempt to Log In

Knowledge Solutions
, , ,
1

Overview

This article will discuss the ‘resource_exhausted’ error that can affect end-user logins. For reference, this error message is returned from a ‘failed login’ log like so (abbreviated for convenience with sensitive information redacted):

{
  "audience": "redacted",
  "client_id": "redacted",
  "client_name": "redacted",
  "connection_id": "",
  "date": "redacted",
  "description": "resource_exhausted",
  "details": {
    "actions": {
      "executions": [
        "redacted"
      ]
    },
    "body": {},
    "error": {
      "message": "resource_exhausted",
      "oauthError": "access_denied",
      "type": "oauth-authorization"
    },
    "qs": {
   redacted
    },
    "riskAssessment": {
      redacted
      },
      "confidence": "redacted",
      "version": "1"
    },
    "session_id": "redacted",
    "stats": {
      redacted
    }
  },
  "hostname": "redacted",
  "ip": "redacted",
  "scope": [
   redacted
  ],
  "type": "f",
  "user_agent": "redacted",
  "user_id": "redacted",
  "user_name": "redacted",
  "log_id": "redacted",
  "tenant_name": "redacted",
  "_id": "redacted",
  "isMobile": redacted
}

Applies To

  • Extensibility
  • Custom Database Action Scripts
  • Actions
  • Webtask

Cause

This error is produced when the concurrency limit for a given customer’s tenant has been exceeded. The concurrency limit is based on Auth0 subscription.

Excerpt from linked documentation:

To ensure system availability and fair use of system resources, Auth0 limits the number of concurrent in-flight requests across all extensibility products. Tenants that exceed their concurrent request limits should expect errors for new requests until in-flight requests are completed.

With this definition in mind, this situation can present itself if there is significant latency with any Actions or Custom Database Action Scripts that make external API calls. This latency causes in-flight requests to accumulate and eventually cause the ‘resource_exhausted’ error until latency improves and/or when end-user traffic decreases to reduce strain on that pipeline.

Solution

The reasons for latency in an Action or Custom Database’s external API calls will vary based on customer implementation. Since this issue occurs due to custom code executing in a given Auth0 tenant, Auth0 Engineers will have limited visibility in understanding which action or script, if there are multiple, is causing the latency.

Due to this limited visibility, it’s recommended that the customer experiencing this issue immediately begin an investigation on their end to review the code and API endpoints they are attempting to reach out to. Additionally, it’s recommended to review the ‘Actions Details’ tab in Auth0 Logs to review each Action in more depth and get more details about its execution. In particular, the ‘stats’ object, included in the ‘Actions Details’ tab, will provide information on how long a given Action took to execute. Below is a generic example of an Action performing under high latency:

    "stats": {
      "total_request_duration_ms": 19900,
      "total_runtime_execution_duration_ms": 19980,
      "runtime_processing_duration_ms": 2,
      "action_duration_ms": 19820,
      "boot_duration_ms": 2,
      "network_duration_ms": 14
    }

Note the following definitions for the above stats:

  • total_request_duration_ms - total time from when actions made a request to the runtime and got a response
  • total_runtime_execution_duration_ms - total time in the runtime service (webtask), including customer code execution
  • action_duration_ms - the amount of time the customer’s action code was executing in the runtime
  • runtime_external_call_duration_ms - The amount of time spent downloading dependencies and customer code. This is cached in the runtime, so it will often be zero / omitted.
  • boot_duration_ms - time for the container process in the runtime to provision
  • network_duration_ms - time the request spent “on the wire” between actions and the runtime