Custom SMTP 5.7.4 Unrecognized authentication type

We’ve confirmed for our custom SMTP that TLS is enables as well as STARTTLS, yet we keep getting this error when testing email:

“error”: “Error sending email: Invalid login sequence while waiting for "334 VXNlcm5hbWU6": 504 5.7.4 Unrecognized authentication type”
},

We’ve tested these same settings with Free Online SMTP Server Test - Send and Check your Email Server and others and they work fine. Thoughts?