It’s an interesting approach. Though if access token’s are meant purely for the client and do not dictacte what a given user can or cannot do, I was more leaning towards it being the responsibility of the client to make the call whether or not the auth’d user should be there or not.
In your case, the user would authenticate to your portal (client), and once the portal received and validated the idtoken, it could then make a call to a central auth and see if that user has access to that client.