Problem statement
We’ve encountered an issue where our Auth0 AD Connector appears to be restarting under load. This occurred during a load test. Based on our findings, it seems that this restart happens after the connector is hit with a burst of around 70 requests per second. Following this burst, the connector appears to restart, leading to “Not In This Node” errors.
Troubleshooting
From the AD/LDAP troubleshooting logs, there are two separate errors occurring, and our engineering team has provided insights into each:
- “Callback was already called” error - This is a known issue that is on our backlog to be fixed. It does not seem to affect the actual functioning of the connector.
- “Connection to ldap was closed” error - Our engineering team believes that this error is likely due to the LDAP server failing to respond.
Error output:
Error: read ECONNRESET at TCP.onStreamRead (internal/stream_base_commons.js:208:20) 2023-04-18T05:21:48.340Z - error: [2023-04-18 05:21:48] Connection to LDAP was closed. 2023-04-18T05:21:48.340Z - debug: [2023-04-18 05:21:48] User ****: Authentication attempt failed. Reason: unexpected error
Cause
If you’ve encountered a similar situation, you might discover that a firewall is in place, but it’s configured to respond only to failed responses. This setup can make it extremely challenging to pinpoint the exact nature of the issue. In this scenario, the firewall’s actions lead to the termination of the AD connection, subsequently triggering the restart of the AD Connectors.
Solution
If you’re facing this problem, consider addressing the firewall settings. You can successfully resolve the issue by adjusting the firewall configurations to permit the AD connection to remain open. This adjustment should help prevent the unnecessary restart of the AD Connectors due to the firewall’s response behavior.
Cause
The customer discovered that they had a firewall in place which was responding only to failed responses. This made it very difficult to determine exactly what was occurring. The firewall was terminating the AD connection, causing the AD Connectors to restart.
Solution
The customer successfully resolved the issue by adjusting their firewall settings to allow the AD connection to remain open.