Thanks @remus.ivan for the response. Unfortunately due to my site being on a different domain than my tenant, I can’t rely on getTokenSilently (at least not the iframe method that calls /authorize) since many browsers block third party cookies. This is why I was using refresh tokens in the first place. Unfortunate that there’s no way to make this work.
I guess I will look into updating my apps so that I can use something like the id_token instead and not rely on /userinfo.