You’ll likely have to go ask the spec authors or try to go spelunking in the OAuth working group mailing lists to see if the rationale was ever discussed there. My opinion would be that give JSON does not mandate a specific format, a format that would have good interoperability and be commonly supported would be a good choice.
In relation to why seconds and not milliseconds, not sure, likely milliseconds precision was not required and for a format aiming to be compact that would reduce a few characters.