We run microservices on k8s with multiple instances to increase resilience and decrease downtime, and we recently switched from ADB2C to auth0. To give a bit more background, when a user calls an API, they only authenticate against a single microservice and none of their permissions propagate. We implement a zero trust security model, meaning a service needs to authenticate itself when calling other services directly. This is done using M2M Authentication with the client credential grant.
To give an example. When a user wants to do something with service A and service A needs to call service B, the user only required permissions for the action on service A and service A needs permission for its action on service B.
Currently, each Microservice uses the same M2M Client ID across all instances. We want to enable refresh token rotation, but doing so would prevent the individual instances from managing their own token, since a refresh would break all but one instance (the last one to acquire a token).
To get around this issue, we would need a centralized cache and/or management unit. These would need to be highly available themselves, further increasing complexity and introducing another potential point of failure. Now either the services would need to monitor for a token refresh, be notified of such, or have the token be injected by a proxy of sorts. The number of problems keeps increasing the further you iterate, as you need to avoid race conditions when refreshing, get access to the token in a dev environment, etc.
Has anybody tackled this issue before or does the majority avoid these issues altogether, by not enabling that particular switch, for example?
Regardless of that switch, sharing a token between instances would keep the number of token refreshes down and prevent a token to be issued on each pod restart.
Any ideas would be appreciated.