Answering my own question for anybody who runs into the same issue.
In my specific case I had an API accessing Auth0 Management API (a Machine to Machine connection).
This also had to be enabled for Passwordless Connections and once that was done, the consent form was no longer coming up.
Passwordless Connections → Email → Applications (enable main RWA and whatever other connection your setup needs also)