Turns out the library idtoken-verifier uses the current time as set by the OS on the client to verify the id-token. This was wildly off on my system, and Windows was not automatically updating it. After setting the correct time and time zone, all works as expected.