Hi there @ahmed.khouaja welcome to the community!
I think you’re always going to get an error thrown at that point if there is one.
As to why the error is occurring, I’ve seen this happen in a couple of scenarios. One possibility is if you are rotating refresh tokens, and trying to reuse them. Another being that if a refresh token is revoked, grants are also deleted and thus all subsequent tokens with the same user, client_id, and audience are invalid. Some more on that here:
Hope this helps!