Auth0 Access Control using Azure AD groups

I tried adding an authorization rule wherein it will throw an Access denied authorization error if the user logging in does not belong to specified allowed AD groups. This rule is working fine but in the scenario wherein I remove a particular user from the allowed AD group, the Access Denied auth error is not kicking in and still allows the user to log in. Is this perhaps due to the AD group membership being cached and when the rule is executed, it still sees that the user is still a member of the allowed AD group?
Has anyone else encountered this same issue?
Thanks so much to anyone who can provide some light.