An invalid refresh token can occur if:
One important detail is that when you revoke a token, for security reasons the grants associated with that token are deleted. This means that all other refresh tokens issued to the same combination of application, user and audience effectively become invalid. If you are revoking refresh tokens as part of your flow, this might explain it.