The initial behavior you described is generally referred to as Home Realm Discovery (HRD) and in this case it basically consists of automatically detecting how the user performs authentication by analysis of the provided email address.
This is currently available for enterprise connections and for some types of enterprise connections this is optional. For example, if you configure a generic SAML-based enterprise connection you can choose to not specify the email domains that could trigger HRD.
However, for Google Apps you’ll need to specify the main domain and the only optional part is other domain aliases that could also be in use. This has the side-effect that when a Google Apps connection is enabled and you input an email associated with your main domain then HRD will always be triggered.
As a workaround specific to database connection, if you configured the database connection to also require a username than Alice should be able to authenticate with username/password credentials by providing the associated username instead of an email.
In relation to your second point, the HRD happens purely by looking into the domain part of the email address so when you input xxx@mycompany.com, the @mycompany.com is used as a way to discover the user authenticates with Google and as such should be redirected to it.
Given the current browser session already has an authenticated Google session for Alice, even though you wrote xxx@mycompany.com the IdP (Google in this case) can detect the existing session and automatically redirect based on that authenticated session.
UPDATE: (regarding your comment)
Specifying prompt=none would mean the request fails if it requires user interaction; not specifying does not imply that user interaction will have to happen. There are other options for prompt that could indeed force login or interaction, but support for them is on a case by case basis in terms of identity providers and by default I don’t believe we are sending any specific prompt value.
The behavior you see is also mostly likely Google deciding that reusing the session is the right thing to do based on the request. From our side, we execute HRD and detect that we need to redirect to Google. Additionally, we’ll most likely include a login_hint parameter indicating that this request was initiated due someone with an xxx@mycompany.com email wanting to login.
However, it’s at Google discretion to detect that the specified login hint does not map to a valid email address and as such deciding to reuse the session that already exists (the Alice one). When you input a different email, one that really maps to another account, Google won’t reuse the session has it know it’s for a different user.
In my personal opinion, it seems a bit misleading and not doing it with could probably have been a batter choice, but I might be missing something. If you consider that login_hint is just that; a hint which Google might be completely ignoring if it’s not a valid email which then leads to the reuse of the existing session then the behavior is not so unexpected. You can confirm that this theory is correct by looking at the network requests in your browser and inspecting the request to the Google identity provider; having confirmed this, you could also try to follow-up the reason for this behavior with Google itself.