I/O Timeout when trying to access /.well-known/openid-configuration

Help
,
1

Hello everyone,

So I recently started having a weird issue with our login stack using Auth0. Our web server has consistently been unable to access the provider information at https://recondev.us.auth0.com/.well-known/openid-configuration.

The server is running in EC2 and the egress firewall rule is set to allow all on 0.0.0.0/0.

This is what a curl of the endpoint from the web server get’s me and also on another spun up instance just to test:

$curl -ivk https://recondev.us.auth0.com/.well-known/openid-configuration
*   Trying 104.16.170.253...
* TCP_NODELAY set
*   Trying 2606:4700::6810:abfd...
* TCP_NODELAY set
* Immediate connect fail for 2606:4700::6810:abfd: Network is unreachable

But when I curl it from my local machine I get a connection. One thing that stands out is that the EC2 instances seem to be trying to connect to it over IPv6 for some reason.

Has anyone seen something like this? It’s making it impossible for users to login so it’s a rather pressing issue.

2

The curl will attempt to connect both to IP4 and IP6 so I’m not sure the failure for IP6 is relevant.

Nonetheless, have you already tried forcing curl, when issued from the impacted machines, to use only IP4 in order to try to reduce the scope of analysis?

3

So I set the -4 flag and now it just hangs at Trying…

Another odd artifact is if I curl http endpoints everything is fine, but https I get network unreachable errors. So as an example “curl http://ifconfig.io” returns my public ip but
“curl https://ifconfig.io” hangs.

I double checked AWS and the machines iptables rules and nothing is blocking 443 egress traffic.

4

If its hanging for every HTTPS request the issues seems global and given the stage at which it hangs you may have to try to see if you can get more info from a TCP capture. From the TCP traffic capture you should be able to see where stuff hangs maybe.

5

Here is the tcpdump output when trying to curl https://google.com

[ec2-user@ip-172-31-72-116 ~]$ sudo tcpdump -vv dst 172.217.164.174
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 16:48:08.002585 
IP (tos 0x0, ttl 255, id 44801, offset 0, flags [DF], proto TCP (6), length 60)
ip-172-31-72-116.ec2.internal.58200 > iad23s69-in-f14.1e100.net.https: Flags [S], cksum 0x464a 
(incorrect -> 0x637a), seq 2221673898, win 26883, options [mss 8961,sackOK,TS val 2112037644 
ecr 0,nop,wscale 7], length 0
16:48:09.014790 IP (tos 0x0, ttl 255, id 44802, offset 0, flags [DF], proto TCP (6), length 60)
ip-172-31-72-116.ec2.internal.58200 > iad23s69-in-f14.1e100.net.https: Flags [S], cksum 0x464a 
(incorrect -> 0x5f86), seq 2221673898, win 26883, options [mss 8961,sackOK,TS val 2112038656 
ecr 0,nop,wscale 7], length 0
16:48:11.030791 IP (tos 0x0, ttl 255, id 44803, offset 0, flags [DF], proto TCP (6), length 60)
ip-172-31-72-116.ec2.internal.58200 > iad23s69-in-f14.1e100.net.https: Flags [S], cksum 0x464a 
(incorrect -> 0x57a6), seq 2221673898, win 26883, options [mss 8961,sackOK,TS val 2112040672 
ecr 0,nop,wscale 7], length 0
16:48:15.158792 IP (tos 0x0, ttl 255, id 44804, offset 0, flags [DF], proto TCP (6), length 60)
ip-172-31-72-116.ec2.internal.58200 > iad23s69-in-f14.1e100.net.https: Flags [S], cksum 0x464a 
(incorrect -> 0x4786), seq 2221673898, win 26883, options [mss 8961,sackOK,TS val 2112044800 
ecr 0,nop,wscale 7], length 0
16:48:23.350796 IP (tos 0x0, ttl 255, id 44805, offset 0, flags [DF], proto TCP (6), length 60)
ip-172-31-72-116.ec2.internal.58200 > iad23s69-in-f14.1e100.net.https: Flags [S], cksum 0x464a 
(incorrect -> 0x2786), seq 2221673898, win 26883, options [mss 8961,sackOK,TS val 2112052992 
ecr 0,nop,wscale 7], length 0

My TCP foo is lacking so I’m not sure what can be gleaned from that.