This can sometimes happen if the audience and/or scope is not included in the AuthModule.forRoot() or if you are requesting an access token for a different audience at a later point. Could this be the case for your app?
Also, are you using cacheLocation: "localstorage"?