Access token is not long / JWT after refresh

When I initially ask for a access_token in a native app client, providing an audience for my API, I get a proper JWT access_token.
But when I do a refresh I get a short access_token again, and obviously can’t provide an audience.
What do I have to do on my client / server to make it work?