Again, this all comes down to the audience - This would be passed to getTokenSilently via authorizationParams.
If the only thing you are doing is using the access token to call /userinfo then you are correct in that you do not need an audience, nor do you need to validate the token per se. In most cases, the access token is used to access an API (your API) and in that case you’d want a JWT to inspect which does require an audience (your API identifier in Auth0) as you’ve seen.