The problem occurred without page refresh. I was making API calls every 1s and each was getting authorized with a new token.
The issue I had was resolved when ID Token expiry was increased from 60s to 15min. Access token expiry was also set to a much larger value. Looks like there is some bug-ish behaviour in the SDK.
Regards,
Bart