Issue with robots.txt file serving JavaScript instead of plain text

Hi, Access to your website outside a browser is restricted with a JavaScript security system, which will affect tools like CURL:

However, most search engine crawlers, like GoogleBot, can still properly crawl and index your website. They won’t be affected by this restriction.

7 Likes