Double check that your callback and logout are the same. I originally just had a typo, but now I believe the issue is that my website is not using https and it fails randomly. The issue has not be resolved according the post below:
If anyone has any idea if there is a workaround or https is a hard requirement for it to work 100% of the time (which is kind of inconvenient).