Quick question: are web crawlers allowed on a Auth0 protected site?
I ask this because I am interested on implementing SEO to my Auth0 protected site. As some of you may know, web crawlers (such as for Google and Bing) need to crawl your site in order to index it on the search engine. I was wondering if those types of crawlers are allowed to enter on an Auth0 protected site (for SEO purposes).
Hi @ysaez ,
Generally, crawlers wont be able to access content on pages that are password protected or require an account to access.
I wondered, does this includes the hosted login on Auth0 Tenants?
You could try testing this.
@dan.woda Hello again, I wondered if the Tenant’s Auth0 domain (ex: example-co.au.auth0.com) has a robots.txt file preventing Google from crawling the site. Just curious 
.
A quick google search will show you how to find it 
I’m going to close this thread as the original question was answered. Please open a new topic if you have another question.