Google Workspace not working with multiple domains

I set up a Google Workspaces enterprise connection that I expect to be used by multiple domains, however I am NOT using home realm discovery.

If I don’t set the Google Workspace Domain to the domain of the logging in email I get.

Email domain mismatch: somedomain.com is not in the list of allowed email domains for this identity provider.

Setting it to * doesn’t work.

I do not want to set up a separate conection per domain or turn on home realm discovery, and this is definately on the auth0 side because when I connect to the same google app from postman with the same params it works fine.

Any ideas on how I can get this to work?