Web crawling a website using Auth0 javascript library to authenticate

Help
, , , ,
1

I’m trying to web crawl using Scrapy a website using Auth0 login. The login form parses the username and password fields using javascript to authenticate with an Auth0 function call to login. Scrapy has built in form authentication but because the Auth0 login page is not a form Scrapy fails and i have tried converting the login page to a form so it can fill in the form but still not authenticate.

Has anyone tried web crawling a website with an Auth0 custom login page and can provide advice or help?

2

I never came across a scenario similar to the one you describe; scraping is highly dependent of the application/website in question and some may make it more difficult or outright impossible. From your description, it seems the current blocking issue is the use of JS for authentication instead of an HTML form.

Given the use of JS to perform login may be relatively common, my only advice is that if you haven’t done so already you may want to try assistance in more generic forums like StackOverflow, in particular, looking for help with the use of Scrapy with JS-based authentication instead of HTML forms.

3

Thanks for the insight, I will try StackOverflow.

4

Thanks for the insight, I will try StackOverflow.