Looks like the same issue I had at this issue here.
This is caused by the “application type” you have selected. I believe passwordless OTP is only supported in SPA & Regular Web App types.
To test this out, try creating a new application, set the application type to one of the above and then enable passwordless OTP.