Welcome to the Auth0 Community!
As mentioned in this knowledge article, the Directory.Read.All permission is not strictly necessary. You should be able to set up and use the connection with the User.Read permission and any other ones you deem necessary for your implementation.
If you have any other questions, let me know!
Kind Regards,
Nik