Thanks for getting in touch with us at Auth0 Community.
The Try button within connections actually uses the tenant’s canonical domain as opposed to the custom domain if one is setup. To test you would need to either try the connection from one of your applications that uses the custom domain or you can try this method https://community.auth0.com/t/try-button-for-any-sso-connection-has-trouble-with-redirect-uri/91004
Warm regards.