Hey there @ben.zoon !
I can see options for that 
But rather than describing all of them, let me please ask - are you open to implementing the Organizations feature? One organization = one customer (users of this customer) having assigned their own IdP/IdPs.
Does one of the IdPs you plan to implement communicate via the Web Services Federation Protocol? (Asking because the Organizations feature doesn’t support this protocol).
Awaiting your feedback and, of course, any follow-up questions!