The resource server associated with the Management API requires explicit consent from the end-user when performing an authorization request; to my knowledge this is not configurable, so the silent authentication will only succeed after the user has provided consent in an interactive authentication flow. This means that the error in question is unavoidable unless you first use an interactive flow that would allow the user to authenticate and provide consent.
Additionally, have in mind that most of the scopes associated with the Management API won’t be available to be provided in access token issued as part of an end-user flow. At most end-users would be able to request scopes that are associated to their current user.
In general, if you want to provide certain operation that can be triggered by end-users and that require Management API requests you would broker this access in your back-end and/or API and do not request Management API access tokens directly from an end-user flow. Instead you would use client credentials; see the reference documentation for additional information on this.