Hi @jp1
Welcome to the Auth0 Community!
As mentioned in this knowledge article, the Directory.All.Read permission is not strictly necessary and the connection should work only with the User.Read permission.
If you have any other questions, feel free to leave a reply!
Kind Regards,
Nik