Bingbot reading HTTPS robots.txt incorrectly

No, just the Live URL (which allows Bing to show you what it sees).

It still indexes your website, but you just can’t use certain tools (like Live URL) that attempt to pull website information with a system that doesn’t use cookies and javascript. Bing web crawlers, however, are permitted as they use JS and cookies, so the security system lets them crawl.

5 Likes