Couldn't retrieve remote JWK set: Read timed out | Spring boot | Java

abhishek.kulkarni · October 28, 2021, 8:55am

Facing below error while booting the spring boot application. Setting value to jwk-set-uri in application.yml also didn’t work. Any solution to fix this?

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'springSecurityFilterChain' defined in class path resource [org/springframework/security/config/annotation/web/configuration/WebSecurityConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.servlet.Filter]: Factory method 'springSecurityFilterChain' threw exception; nested exception is java.lang.IllegalStateException: com.nimbusds.jose.RemoteKeySourceException: Couldn't retrieve remote JWK set: Read timed out
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918) ~[spring-context-5.3.9.jar:5.3.9]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) ~[spring-context-5.3.9.jar:5.3.9]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145) ~[spring-boot-2.5.4.jar:2.5.4]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-2.5.4.jar:2.5.4]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) ~[spring-boot-2.5.4.jar:2.5.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:338) ~[spring-boot-2.5.4.jar:2.5.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-2.5.4.jar:2.5.4]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) ~[spring-boot-2.5.4.jar:2.5.4]
	at com.hyve.oracle.OracleRegistrationServiceApplication.main(OracleRegistrationServiceApplication.java:14) ~[classes/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.boot.devtools.restart.RestartLauncher.run(RestartLauncher.java:49) ~[spring-boot-devtools-2.5.4.jar:2.5.4]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [javax.servlet.Filter]: Factory method 'springSecurityFilterChain' threw exception; nested exception is java.lang.IllegalStateException: com.nimbusds.jose.RemoteKeySourceException: Couldn't retrieve remote JWK set: Read timed out
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.3.9.jar:5.3.9]
	at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653) ~[spring-beans-5.3.9.jar:5.3.9]
	... 26 common frames omitted
Caused by: java.lang.IllegalStateException: com.nimbusds.jose.RemoteKeySourceException: Couldn't retrieve remote JWK set: Read timed out
	at org.springframework.security.oauth2.jwt.JwtDecoderProviderConfigurationUtils.getSignatureAlgorithms(JwtDecoderProviderConfigurationUtils.java:107) ~[spring-security-oauth2-jose-5.5.2.jar:5.5.2]
	at org.springframework.security.oauth2.jwt.JwtDecoders.withProviderConfiguration(JwtDecoders.java:122) ~[spring-security-oauth2-jose-5.5.2.jar:5.5.2]
	at org.springframework.security.oauth2.jwt.JwtDecoders.fromOidcIssuerLocation(JwtDecoders.java:66) ~[spring-security-oauth2-jose-5.5.2.jar:5.5.2]
	at com.hyve.oracle.security.SecurityUtil.jwtDecoder(SecurityUtil.java:19) ~[classes/:na]
	at com.hyve.oracle.security.SecurityConfig.configure(SecurityConfig.java:50) ~[classes/:na]
	at org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter.getHttp(WebSecurityConfigurerAdapter.java:217) ~[spring-security-config-5.5.2.jar:5.5.2]
	at org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter.init(WebSecurityConfigurerAdapter.java:315) ~[spring-security-config-5.5.2.jar:5.5.2]
	at org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter.init(WebSecurityConfigurerAdapter.java:93) ~[spring-security-config-5.5.2.jar:5.5.2]
	at com.hyve.oracle.security.SecurityConfig$$EnhancerBySpringCGLIB$$5f560a16.init(<generated>) ~[classes/:na]
	at org.springframework.security.config.annotation.AbstractConfiguredSecurityBuilder.init(AbstractConfiguredSecurityBuilder.java:338) ~[spring-security-config-5.5.2.jar:5.5.2]
	at org.springframework.security.config.annotation.AbstractConfiguredSecurityBuilder.doBuild(AbstractConfiguredSecurityBuilder.java:300) ~[spring-security-config-5.5.2.jar:5.5.2]
	at org.springframework.security.config.annotation.AbstractSecurityBuilder.build(AbstractSecurityBuilder.java:38) ~[spring-security-config-5.5.2.jar:5.5.2]
	at org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration.springSecurityFilterChain(WebSecurityConfiguration.java:127) ~[spring-security-config-5.5.2.jar:5.5.2]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.9.jar:5.3.9]
	... 27 common frames omitted
Caused by: com.nimbusds.jose.RemoteKeySourceException: Couldn't retrieve remote JWK set: Read timed out
	at com.nimbusds.jose.jwk.source.RemoteJWKSet.updateJWKSetFromURL(RemoteJWKSet.java:167) ~[nimbus-jose-jwt-9.10.1.jar:9.10.1]
	at com.nimbusds.jose.jwk.source.RemoteJWKSet.get(RemoteJWKSet.java:260) ~[nimbus-jose-jwt-9.10.1.jar:9.10.1]
	at org.springframework.security.oauth2.jwt.JwtDecoderProviderConfigurationUtils.getSignatureAlgorithms(JwtDecoderProviderConfigurationUtils.java:90) ~[spring-security-oauth2-jose-5.5.2.jar:5.5.2]
	... 44 common frames omitted
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.base/java.net.SocketInputStream.socketRead0(Native Method) ~[na:na]
	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115) ~[na:na]
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168) ~[na:na]
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:68) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1104) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:823) ~[na:na]
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) ~[na:na]
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292) ~[na:na]
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:746) ~[na:na]
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1604) ~[na:na]
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1509) ~[na:na]
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:245) ~[na:na]
	at com.nimbusds.jose.util.DefaultResourceRetriever.getInputStream(DefaultResourceRetriever.java:305) ~[nimbus-jose-jwt-9.10.1.jar:9.10.1]
	at com.nimbusds.jose.util.DefaultResourceRetriever.retrieveResource(DefaultResourceRetriever.java:257) ~[nimbus-jose-jwt-9.10.1.jar:9.10.1]
	at com.nimbusds.jose.jwk.source.RemoteJWKSet.updateJWKSetFromURL(RemoteJWKSet.java:165) ~[nimbus-jose-jwt-9.10.1.jar:9.10.1]
	... 46 common frames omitted

dan.woda · October 29, 2021, 8:06pm

@abhishek.kulkarni

It’s hard to say without more info. The error is saying that the request to get the remote key set timed out, which could be from a series of things.

Here are a few similar topics that might help:

github.com/microsoft/azure-spring-boot

Couldn't retrieve remote JWK set: connect/read timed out

opened 07:49PM - 27 Dec 19 UTC

closed 08:50AM - 09 Mar 21 UTC

sandboxbohemian

aad investigate azure-spring

## Environment - Spring boot starter: - active directory spring boot starter… - OS Type: Linux - Java version: 1.8 ## Summary We have a Spring Boot app that uses Spring Security with the Azure AD starter. We have observed sporadic connect and read timeouts while the library tries to get a response from the JWKS URL (https://login.microsoftonline.com/common/discovery/keys). ## Reproduce steps We are using the following dependencies in our stack - `org.springframework.boot:spring-boot-starter-security:2.1.0.RELEASE` - `org.springframework.security:spring-security-oauth2-client:5.1.3.RELEASE` - `org.springframework.security:spring-security-oauth2-jose:5.1.3.RELEASE` - `com.microsoft.azure:azure-active-directory-spring-boot-starter:2.1.2` - `com.nimbusds:oauth2-oidc-sdk:5.64.4` We have the same code running in two different environments under two different tenant ids, and we have observed timeouts in only one of them. Following is the stack trace. ``` Servlet.service() for servlet [dispatcherServlet] in context with path [/api/v1/src-annuities-query] threw exception [com.nimbusds.jose.RemoteKeySourceException: Couldn't retrieve remote JWK set: connect timed out] with root cause j.n.SocketTimeoutException: connect timed out at j.n.PlainSocketImpl.socketConnect(PlainSocketImpl.java) at j.n.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at j.n.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at j.n.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at j.n.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at s.s.s.SSLSocketImpl.connect(SSLSocketImpl.java:666) at s.net.NetworkClient.doConnect(NetworkClient.java:175) at s.n.w.h.HttpClient.openServer(HttpClient.java:463) at s.n.w.h.HttpClient.openServer(HttpClient.java:558) at s.n.w.p.h.HttpsClient.<init>(HttpsClient.java:264) at s.n.w.p.h.HttpsClient.New(HttpsClient.java:367) at s.n.w.p.h.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) at s.n.w.p.h.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) at s.n.w.p.h.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) at s.n.w.p.h.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) at s.n.w.p.h.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570) at s.n.w.p.h.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) at s.n.w.p.h.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268) at c.n.j.u.DefaultResourceRetriever.getInputStream(DefaultResourceRetriever.java:249) ... 66 frames truncated ``` Following #417 , we set the timeout parameters as below, but even then we have observed timeouts (but definitely fewer). `azure.activedirectory.jose.connect-timeout=2000` `azure.activedirectory.jose.read-timeout=2000` `azure.activedirectory.jose.size-limit=51200` Stepping through the code, we found that the `JWTResourceRetriever` is conditional, so we set it up to ensure that it gets overwritten with the parameters above. ``` @Bean public ResourceRetriever getJWTResourceRetriever(@Autowired JoseConfigurationProperties properties) { return new DefaultResourceRetriever(properties.getConnectTimeout(), properties.getReadTimeout(), properties.getSizeLimit()); } ``` And in a different @Configuration class to avoid cyclic dependencies ``` @Autowired private AADAuthenticationFilter azureAdFilter; @Override protected void configure(HttpSecurity http) throws Exception { http.authorizeRequests() .antMatchers("api/**/*") .authenticated(); http.authorizeRequests() .anyRequest() .permitAll(); http.addFilterBefore(azureAdFilter, UsernamePasswordAuthenticationFilter.class); } ``` *JoseConfigurationproperties.java* ``` @Configuration @ConfigurationProperties(prefix = "azure.activedirectory.jose") public class JoseConfigurationProperties { private int connectTimeout; private int readTimeout; private int sizeLimit; // getters and setters... } ``` ## Expected Results The connection should not have timed out. Over a period of 7 days, we have a 55-45 split between connect timeout and read timeout. Also, since this is serving static content, is there any way to force caching of the JWKS content and not perform a fetch on every incoming request? ## Actual Results ``` Servlet.service() for servlet [dispatcherServlet] in context with path [/api/v1/src-annuities-query] threw exception [com.nimbusds.jose.RemoteKeySourceException: Couldn't retrieve remote JWK set: connect timed out] with root cause j.n.SocketTimeoutException: connect timed out at j.n.PlainSocketImpl.socketConnect(PlainSocketImpl.java) at j.n.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at j.n.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at j.n.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at j.n.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at s.s.s.SSLSocketImpl.connect(SSLSocketImpl.java:666) at s.net.NetworkClient.doConnect(NetworkClient.java:175) at s.n.w.h.HttpClient.openServer(HttpClient.java:463) at s.n.w.h.HttpClient.openServer(HttpClient.java:558) at s.n.w.p.h.HttpsClient.<init>(HttpsClient.java:264) at s.n.w.p.h.HttpsClient.New(HttpsClient.java:367) at s.n.w.p.h.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) at s.n.w.p.h.HttpURLConnection.plainConnect0(HttpURLConnection.java:1162) at s.n.w.p.h.HttpURLConnection.plainConnect(HttpURLConnection.java:1056) at s.n.w.p.h.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) at s.n.w.p.h.HttpURLConnection.getInputStream0(HttpURLConnection.java:1570) at s.n.w.p.h.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) at s.n.w.p.h.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:268) at c.n.j.u.DefaultResourceRetriever.getInputStream(DefaultResourceRetriever.java:249) ... 66 frames truncated ```