Two things:
(1) On my older tenants under settings → advanced there was a toggle for Enable Seamless SSO. Toggling that on or off would change the behavior that occurred when I found this problem. On newer tenants that option didn’t exist, and the behavior was effectively always in the ON position.
(2) The fix so all the sites worked with the sso option enabled was that in my /logout endpoint on my own site I had to add a header redirect for the users browser to my auth0 api /v2/logout url. I also passed it a returnTo my own site and the client id to log it out of.
The reason things worked fine for me before this change is I was just calling the auth0 logout function from the provided sdk which cleared the session in the browser and so they had to log in again. Since Auth0 made the sso option default (and not optional on new tenants) instead of having to authenticate each time, if that users browser is already authenticated it will log them in automatically. You must now redirect the browser (or probably something like end the session via the api on the backend) to the auth0 logout endpoint itself to completely log them out.