ASP.NET Core 3.1 with Razorpages

Hi. I have just upgraded our codebase from asp core 2.2 to 3.1. That went ok, but now I cannot get it to work with auth0 authentication. It keeps failing with correlation error after the challenge has been issued, I have succesfully entered credentials using the univeral login, it returns to the callback, and then it cannot seem to find the cookie.
I tried the sample code for asp core 3.1 and it works, but it uses MVC not razor pages, so I suspect it might have something to do with the routing? I copied all of the Startup.cs from the working sample into my project only leaving the .UseRazorPages-related calls to keep my razor pages working. That seems to be the only difference (I am using the same from Program.cs as well)
Anyone has any ideas to help me out? I have spent 10 hours on this today alone (and some yesterday as well)