I don’t want the user to have to log in twice (once for the Management API access token and again for an access token for my API) and it isn’t possible to specify multiple audiences. So as a solution, can I just use the Management API access token to authorize access to my own API (possibly checking scope as well as audience)?
No, you should not use such approach. You’re correct that requesting multiple audiences is currently not possible, however, making multiple requests for an API access token does not necessarily imply that the user has to login again.
In particular, if you’re using universal login a session should have been established when the user completed the first login request so under certain conditions it should be possible to perform a second request for another API without the user having to actively authenticate again. In other words the session that is leveraged for SSO across applications can also be leveraged by the same applications for requesting multiple access tokens to different audiences in certain scenarios.