Salesforce Community Website / Auth0 / Searchengine Whitelisting

Right no we do a paywall on a community website, that comes up for not logged in users. What we like to perform is to whitelist searchengine-bots by an IP List or so . is there any known standard solution or config or any code help to do so?