Time by time continually system gets timeouts while sending requests to Auth0 JP-1 server from .NET C# 8 application (recreated)

Hi, I work on product and we got big issue with requests timeout from 2024-06-15. We have Azure VMs in China region and use tenant on JP-1 server. Time by time we got errors with two logs.

First: The request was canceled due to the configured HttpClient.Timeout of 120 seconds elapsing.
Second:Unexpected Server Error. Type: System.Net.Http.HttpRequestException. Message: System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer.
—> System.Net.Sockets.SocketException (104): Connection reset by peer`

However, auth0 status page showing that JP-1 server is OK all time.

Implementation
Framework: .NET C# 8

Libraries: Auth0.AuthenticationApi 7.24.0, Auth0.ManagementApi 7.24.0

Service registration:
services.AddSingleton<IAuthenticationConnection, CustomAuthenticationConnection>();

IAuthenticationConnection library interface
CustomAuthenticationConnection custom class

Class implementation:

`using Auth0.AuthenticationApi;
using Auth0.Core.Exceptions;
using Newtonsoft.Json;
using System.Reflection;
using System.Text;
using Microsoft.IdentityModel.Tokens;

namespace Vitalerter.AuthService.Infrastructure.Auth0
{
public class CustomAuthenticationConnection : IAuthenticationConnection, IDisposable
{
private static readonly JsonSerializerSettings _jsonSerializerSettings = new JsonSerializerSettings
{
NullValueHandling = NullValueHandling.Ignore,
DateParseHandling = DateParseHandling.DateTime
};

    private readonly TimeSpan _timeout = new TimeSpan(0, 2, 0);
    private readonly HttpClient _httpClient;
    private readonly string _agentString;

    private bool _ownHttpClient;

    public CustomAuthenticationConnection(HttpClient httpClient = null)
    {
        _ownHttpClient = httpClient == null;
        this._httpClient = httpClient ?? new HttpClient();
        this._httpClient.Timeout = _timeout;
        if (_ownHttpClient)
        {
            this._httpClient.DefaultRequestHeaders.Add("Auth0-Client", CreateAgentString());
        }
        else
        {
            _agentString = CreateAgentString();
        }
    }

    public CustomAuthenticationConnection(HttpMessageHandler handler)
     : this(new HttpClient(handler ?? new HttpClientHandler()))
    {
        _ownHttpClient = true;
    }

    public async Task<T> GetAsync<T>(Uri uri, IDictionary<string, string> headers = null, CancellationToken cancellationToken = default(CancellationToken))
    {
        using HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, uri);
        ApplyHeaders(request, headers);
        return await SendRequest<T>(request, cancellationToken).ConfigureAwait(continueOnCapturedContext: false);
    }

    public async Task<T> SendAsync<T>(HttpMethod method, Uri uri, object body, IDictionary<string, string> headers = null, CancellationToken cancellationToken = default(CancellationToken))
    {
        using HttpRequestMessage request = new HttpRequestMessage(method, uri)
        {
            Content = BuildMessageContent(body)
        };
        ApplyHeaders(request, headers);
        return await SendRequest<T>(request, cancellationToken).ConfigureAwait(continueOnCapturedContext: false);
    }

    private async Task<T> SendRequest<T>(HttpRequestMessage request, CancellationToken cancellationToken = default(CancellationToken))
    {
        if (!_ownHttpClient)
        {
            request.Headers.Add("Auth0-Client", _agentString);
        }

        using HttpResponseMessage response = await _httpClient.SendAsync(request, cancellationToken).ConfigureAwait(continueOnCapturedContext: false);
        if (!response.IsSuccessStatusCode)
        {
            throw await ApiException.CreateSpecificExceptionAsync(response).ConfigureAwait(continueOnCapturedContext: false);
        }

        string text = await response.Content.ReadAsStringAsync().ConfigureAwait(continueOnCapturedContext: false);
        return (T)((typeof(T) == typeof(string)) ? ((object)(T)(object)text) : ((object)JsonConvert.DeserializeObject<T>(text, _jsonSerializerSettings)));
    }

    private void ApplyHeaders(HttpRequestMessage request, IDictionary<string, string> headers)
    {
        if (headers == null)
        {
            return;
        }

        foreach (KeyValuePair<string, string> header in headers)
        {
            if (header.Key != null && header.Value != null)
            {
                request.Headers.Add(header.Key, header.Value);
            }
        }
    }

    private HttpContent BuildMessageContent(object body)
    {
        if (body == null)
        {
            return null;
        }

        if (body is IDictionary<string, string> parameters)
        {
            return CreateFormUrlEncodedContent(parameters);
        }

        return CreateJsonStringContent(body);
    }

    private static HttpContent CreateJsonStringContent(object body)
    {
        return new StringContent(JsonConvert.SerializeObject(body, _jsonSerializerSettings), Encoding.UTF8, "application/json");
    }

    private static HttpContent CreateFormUrlEncodedContent(IDictionary<string, string> parameters)
    {
        return new FormUrlEncodedContent(parameters.Select((KeyValuePair<string, string> p) => new KeyValuePair<string, string>(p.Key, p.Value ?? "")));
    }

    private static string CreateAgentString()
    {
        string target = "NETSTANDARD2.0";
        Version version = typeof(CustomAuthenticationConnection).GetTypeInfo().Assembly.GetName().Version;
        string s = JsonConvert.SerializeObject(new
        {
            name = "Auth0.Net",
            version = version.Major + "." + version.Minor + "." + version.Revision,
            env = new { target }
        }, Formatting.None);
        return Base64UrlEncoder.Encode(Encoding.UTF8.GetBytes(s));
    }

    protected virtual void Dispose(bool disposing)
    {
        if (disposing && _ownHttpClient)
        {
            _httpClient.Dispose();
            _ownHttpClient = false;
        }
    }

    public void Dispose()
    {
        Dispose(disposing: true);
    }
}

}`

Any thoughts? Any info about that?

Hi there @a.donchenko ,

Thank you very much for sharing the specifics of your implementation and connectivity issue.

I did the initial research and found this Knowledge Article about timeouts.

I’m aware the KM references AWS while you host your application on Azure, but it’s likely that Azure and AWS share architectural similarities, so you could try to apply the suggested in the article solution.

Please let me know if that sounds good to you and if you have follow-up questions :slight_smile:

So, you suggest to increase timeout time on any level of the system, right? This kind of solution I understand like temporary solution, because looks like this temporary long requests are OK. However, it’s affect our clients: they are signed out, they can’t login to the system. I am looking for some best practice or information about the issue.

Anyway, thank you very much for feedback. It was useful for me anyway.

UPD: Issue is still happening and now it’s consistently. Sometime we got timeout while sending request to Auth0, sometimes it’s OK. However it’s OK to get management token. When application try to get some info, e.g. user, it’s timed out.

Check picture below. Red logs are from class that can’t be converted to custom in order to get details of auth0 response to the application. Green logs are from class that was converted to custom and we could check details of auth0 response to the application. Green logs only about getting management token.

Hi @a.donchenko ,

Could you please share (1) what’s the timeout for idle connections set in your NAT gateway? (After what time your NAT gateway will terminate idle connection?)

And (2) what’s your application TCP keepalive timeout? And (3) what’s the Operating system’s TCP keepalive timeout set (in the VMs where you run your application)?