Thanks for the detailed description of your use case
This is a common question for which there is plenty of “controversy” around. Basically, as long as your client and API are within the logical bounds of the same application it may be OK to use the ID Token in this way. I won’t go into too much detail here but the following response in a blog discussion is really helpful.
Here is the video that is referenced, also very helpful!
Albeit a bit confusing, hopefully this helps to clear things up a bit!
