Intermittent read timeout errors when running tests against Auth0

Help
1

Please include the following information in your post:

  • Which SDK this is regarding: auth0-python
  • SDK Version: 3.19.0
  • Platform Version: Python 3.8.12
  • Code Snippets/Error Messages/Supporting Details/Screenshots:

This is similar to Socket timeout errors? , though since that’s over a year old, I figured a new topic would be better.

When we run our test suite, which makes hundreds of calls to Auth0, we get intermittent read timeout errors - perhaps 1 out of every 2 or 3 runs. The full stacktrace is below.

We’ve set our default to 30s, and these still occur (this is done via RestClientOptions - i.e. RestClientOptions(timeout=30.0)). They’re definitely not rate limit errors, as we’ve seen the expected 429 error when those occur. I can force them to happen while doing local testing by setting the timeout to 1s, and it’s of course not that unexpected to get a read timeout with such a low value. But for 30s, we don’t expect any timeouts to occur. All the operations in our test suite feature very low amounts of data - we’re not doing any performance testing, just functional testing with one or two clients/connections/etc at most. At the end of a test, any Auth0 objects that were created are deleted as well. So there’s a lot of chatter, but we don’t get rate limit errors - we just intermittently get read timeouts.

I’ve tossed in the Python tenacity library to provide retry support on ReadTimeout and ReadTimeoutError, but that’s just a bandage. Wondering if anyone else has experienced these and if you found a workaround to prevent them from occurring, or are they just an expected fact of life?

Stacktrace:

timeout: The read operation timed out
File “urllib3/connectionpool.py”, line 445, in _make_request
six.raise_from(e, None)
File “”, line 3, in raise_from
# Permission is hereby granted, free of charge, to any person obtaining a copy
File “urllib3/connectionpool.py”, line 440, in _make_request
httplib_response = conn.getresponse()
File “http/client.py”, line 1348, in getresponse
response.begin()
File “http/client.py”, line 316, in begin
version, status, reason = self._read_status()
File “http/client.py”, line 277, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “socket.py”, line 669, in readinto
return self._sock.recv_into(b)
File “ssl.py”, line 1241, in recv_into
return self.read(nbytes, buffer)
File “ssl.py”, line 1099, in read
return self._sslobj.read(len, buffer)
ReadTimeoutError: HTTPSConnectionPool(host=‘(redacted).auth0.com’, port=443): Read timed out. (read timeout=1.0)
File “requests/adapters.py”, line 439, in send
resp = conn.urlopen(
File “urllib3/connectionpool.py”, line 755, in urlopen
retries = retries.increment(
File “urllib3/util/retry.py”, line 532, in increment
raise six.reraise(type(error), error, _stacktrace)
File “urllib3/packages/six.py”, line 770, in reraise
raise value
File “urllib3/connectionpool.py”, line 699, in urlopen
httplib_response = self._make_request(
File “urllib3/connectionpool.py”, line 447, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File “urllib3/connectionpool.py”, line 336, in _raise_timeout
raise ReadTimeoutError(
ReadTimeout: HTTPSConnectionPool(host=‘(redacted).auth0.com’, port=443): Read timed out. (read timeout=1.0)
File “init.py”, line 407, in call
result = fn(*args, **kwargs)
File “(redacted)_api/auth0_apis.py”, line 121, in retry_on_read_timeout
return fun(*args)
File “auth0/v3/management/clients.py”, line 69, in all
return self.client.get(self._url(), params=params)
File “auth0/v3/management/rest.py”, line 133, in get
response = requests.get(url, params=params, headers=headers, timeout=self.options.timeout);
File “requests/api.py”, line 76, in get
return request(‘get’, url, params=params, **kwargs)
File “requests/api.py”, line 61, in request
return session.request(method=method, url=url, **kwargs)
File “requests/sessions.py”, line 542, in request
resp = self.send(prep, **send_kwargs)
File “requests/sessions.py”, line 655, in send
r = adapter.send(request, **kwargs)
File “requests/adapters.py”, line 529, in send
raise ReadTimeout(e, request=request)

1 Like
3

Hi @rjrudin,

Welcome to the Auth0 Community!

I understand that you’ve been encountering errors when trying to run your Python tests with Auth0.

There could be several reasons why this is happening, and determining why the Network frequently times out is slightly more complex, like whether if it’s the client application (python application), or the server (Auth0) that’s causing the issue, but also anything in-between. For now, I can confirm that the errors are thrown because your code had exceeded the timeout value. It might be worth trying to set the highest reasonable value for your timeout value and see if that helps.

Before we continue, I have some clarifying questions.

First, could you please clarify when you started observing the timeout errors happening?

Next, could you also please clarify the frequency and magnitude of tests you are performing? (i.e 3 run 1000 tests)

And what kind of tests are you performing? (i.e user create, and update)

Lastly, could you please clarify whether you have any custom scripts enabled like Rules/Hooks/Actions?

Thank you.

5

Thanks for the response, and let me know if a support ticket will be more appropriate for debugging this.

Answers:

1 - We added the tests that involve our code and the Auth0 Python SDK to our CI cycle several weeks ago; I am fairly confident we’ve had intermittent timeout issues since then.

2 - I just ran the test suite locally; it took about 11 minutes to complete, and 670 calls were made to the Auth0 API (I am counting this via a function that we have that returns an instance of the Auth0 client; I added a line of logging and just counted the number of times that line shows up in the log output for the test suite run; also, we construct one instance of an Auth0 object and then cache that for future usage)

3 - The tests involve many different Auth0 API calls. A common thing for a test to do is create a “customer”, which consists of an Auth0 connection, client, and organization. At the end of such a test, a test fixture deletes that connection/client/organization so that we don’t leave dozens of test-only resources lying around in our Auth0 instance.

4 - We have 2 custom post-login actions. Both should be very fast as neither involves an Auth0 API call or accessing any external resource. Both are effectively just doing “If this condition is true, then add this to the ID or access token”.

I’m currently using the tenacity library to provide retry support on ReadTimeout and ReadTimeoutError occurrences. That seems to be an effective hack, but I’m still wondering why a timeout would occur on a fairly simple operation when the timeout is 20 or 30s. I know what you mean though about debugging this stuff - had a similar problem at my last job where our client library would mysteriously fail with a dropped connection.

Re: the two custom actions - I am going to try a test run where I can reproduce the error somewhat (can normally do this by dropping the timeout to a much lower value), and then remove the two actions and see if I get roughly the same number of intermittent timeouts. Won’t be conclusive, but seems worth investigating.

1 Like