The difference in the flow that works and the one that doesn’t is that one is SP-Initiated and the other is IdP-Initiated.
You mention that you configured IdP-Initiated with the expected client application, but then you mention that you selected SAML as the response type. I’m assuming you meant response protocol and if that’s the case it implies that a SAML assertion will be posted to your client application. However, the libraries you mention as being used in the client application do not support processing SAML assertions. This is consistent with the fact that you mention that when it works you see #access_token=foo&... in the URL; that is not a SAML response, it’s an OpenID Connect one.
You’ll need to change Response Protocol to be OpenID Connect so that the final stretch of the process is done according to the protocol the client application expects. In addition, you may also need to set the Query String to be response_type=id_token token so that the response tokens are immediately delivered in the fragment component of the callback URL.