Problem statement
There was an attack on our tenant where the attacker directly hit the passwordless/start endpoint.
We enabled the Bot Detection feature on our tenant before the attack. However, Bot Detection let a majority of the attacks through and end up requesting SMS OTP on our SMS account.
Symptoms
Bot detection enabled
Passwordless SMS connection in use
Many requests pass through bot detection and still trigger SMS to be sent
Troubleshooting
verification_required error is expected if bot detection is triggered.
With “when risky” enabled for bot detection on passwordless flow
However, this only caught a small percentage of requests:
The attacker used the same user agent (Mozilla/5.0 (Linux; Android 9; SM-N976N Build/QP1A.190711.020; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/92.0.4515.131 Mobile Safari/537.36) but from unique IP addresses. User agent parsed to “Android WebView 92 on Android (Pie) Samsung SM-N976N”
Solution
Our bot detection model is probabilistic. There are certain scenarios and attack patterns where it performs worse than the documented average. We are working towards improving on them. If Bot Detection is not working sufficiently well during an attack, we recommend changing the captcha setting to “always on”.
The captcha can be skipped for IPs that are on your Bot detection allow listed IPs as those should be trusted applications.