Hey @Wik,
Welcome to the Auth0 Community! 
Let me try to answer your questions and clarify a bit.
is there any such mechanism in the Access Token that allows me to use it to when sending requests from my frontend (React Web App) to my backend?
If you are referring to mechanisms that bind an access token to the client-API channel, they do exist. They are mentioned in the article and are known as sender constraint.
Or should Access Tokens only be used between backends?
Absolutely no! Access tokens can be used by a frontend to call a backend.
If so, how will my backend ever know that I successfully sign in with (e.g.) Auth0/Google/Github or other IdP?
Not sure I understand correctly.
If your need is for your backend to know if the user is still logged in, you can do that with traditional cookies.
If you need to know which specific IdP the user has authenticated with, the frontend can send this information to the backend by extracting it from the ID token (issuer claim).
I hope I helped clear up your doubts.