Well, sometimes guidance is just like opinions; everyone has one. Another problem with guidance is context, for example, if you say to me the access token is only used every two hours and the lifetime of that token is configured to be 5 hours, caching it or not does not make a substantial difference.
If however the token is continuously used it would be a different matter. I also agree that caching can take different levels, you can cache it just in-memory, likely the easiest approach and in order for that token to leak the process would need to be running and a more advanced attack employed.
On the other hand if you wanted to cache it across multiple executions of the application then you need to me more careful where you persist that token. However, if you already need to cache refresh tokens then you likely already solved the problem of where to store sensitive information so storing one more piece of information should not make much difference.