The opening of a link from the email, to my knowledge, will always open in a new tab/browser window given that at this point this is a completely independent operation; the only actor in play that has context this is part of a specific authentication flow is the end-user. Both the email client, operating system and browser treat this as any other operation that involves opening an URL from an email.
Having said that your own client application has a bit more context given that if a magic link was requested then it’s highly likely that this will be followed by the actual completion of the authentication. With this in mind the client application, as you mentioned, can indeed implement a mechanism to listen or poll to that authentication event and automatically update its UI to reflect the authenticated user (even if the actual authentication was completed in a new tab). The mechanism in question would highly depend on the type of client application and possibly also on the technology stack.
With the above you would now still have two open tabs, but both would reflect the authenticated user. Besides the user copy/pasting the magic link to the already open tab I don’t think there will be a way for you to avoid having the two tabs open.