You must configure your resource server (aka API) in the APIs section of the dashboard; as part of this configuration you will provide a unique identifier for the API and also currently configure the signing algorithm used in the JWT access tokens that are issued for authorization requests targeting this API.
In addition, your client application must ensure that it performs an authorization request that states that it wants to receive an access token suitable to your API. It can do this by including an audience parameter where the value is the identifier you configured previously for the API.
Having met the above requirements then the access token received by the client application will indeed be a JWT (in future other formats may be supported, but you would be able to choose at API configuration time).
As additional note, the access token you’re receiving is likely only suitable to call the /userinfo endpoint because you either did not specify an audience or specify one associated with that endpoint. In this case since it’s the /userinfo endpoint that validates the access token the actual format is unspecified; at this time is an opaque token, but again that can change without notice because both the issuer and consumer are controlled by the Auth0 service itself.