Overview
When using a custom domain for an Auth0 tenant, it is possible to experience issues with Google indexing due to a restrictive robots.txt file being hosted. This article explains why the restriction is in place.
Applies To
- Custom Domain
- Search Engine Indexing
- robots.txt
Cause
The custom domain should be used only for Auth0 API endpoints (Authentication API, Management API), which should not be crawled or indexed by search engines.
Solution
The presence of the robots.txt resource in an Auth0 domain discourages automated web crawling mechanisms (such as search engines like Google) from crawling the domain because no content in an Auth0 domain can benefit from web indexing.