You need to use the management API to get the user’s Google refresh token and you shouldn’t be doing this directly from a public client. You’ll want fetch this token from your backend/API.
There’s a good write up about it here (ignore the first posts, they are about getting an Auth0 refresh token. I linked to the correct post directly):
It sounds like you aren’t being returned the refresh token from google. Have you gone through the post I linked and confirmed you are sending the correct params with the request?