Passwordless connections have a Disable Sign Ups setting which should achieve this. You can enable this setting in the passwordless connection settings:
Dashboard > Connections > Passwordless > Your connection > Disable Sign Ups (bottom of the connection settings)