Problem Statement
Some attack protection features are not available for passwordless connections, such as Bot Detection or Suspicious IP Throttling.
Solution
The recommended approach is that you can host your custom SMS gateway and implement logic to filter the requests before sending out the SMS. The custom SMS gateway will act as a layer in front of your messaging service’s API. For this approach, you will be responsible for figuring out which requests to block (based on IP or phone number). Perhaps you could use another third-party service for this.
Other points to consider is: If the attack is on passwordless/start, you can switch to embedded passwordless and filter the requests on your server. Probably a little drastic unless this issue is chronic. If the attack is on oauth/token, you can switch to a regular web app architecture so that the token endpoint requires a secret and filter the requests on their server. If the attack is something else, like just trying to spam users in one country or of one phone provider (rare), the SMS provider might have options to filter our texts to just a set of numbers getting spammed.