Automatic calls to /oauth/token fail on second one when useRefreshToken

When I have useRefreshTokens set to true and make a call to getTokenSilently(), the first time the token is about to expire and it make that call to /oauth/token with the refresh token, it works fine however when it needs to make the call to /oauth/token the second time, it fails with the error Error: Unknown or invalid refresh token. and I am confused as to why since the refresh token it is sending has never been used and is before its expire time.