Hey there @aidan.temple sorry for the delayed response, and welcome to the community!
I don’t believe this is possible 
While I’m not totally clear on your case, it sounds like you might be interested in Role-Based Access Control (RBAC) - Basically, you could create a role, assign it to users, and if a user does leave the organization you remove the role. It’s pretty straightforward to add a user’s roles to a token, see this post.
https://auth0.com/docs/manage-users/organizations/configure-organizations/add-member-roles
Hopefully this at least gives you something to work with!