Excessive time delay when retrieving id and access tokens using an authorisation code

I am seeing a call time of 5 seconds when I access the oauth/token API endpoint in order to obtain id and access tokens in exchange for an authorisation code.
When we have large numbers of authorisation codes coming in to our server in a short space of time, this 5 second delay essentially ties up all our apache threads whilst they wait the required time to talk to Auth0.
Is there something that can be done to reduce this 5 second response time?