Regarding the first two points: the right place to put your logic would usually be a Pre-User-Registration Hook Explore Flows and Triggers, however it should be noted that at the moment, the hooks do only return a generic error message only (“An error occured.”) upon failure (which your logic would trigger), thus it’s not possible to return a customized error message (at least not if using the standard Universal Login Page).
So you might need to use a custom UI with additional frontend validation as well.
Captcha is not supported (and not recommended) by Auth0 out of the box. See:
and this blog article (“Avoid CAPTCHAs”):