Ahrefs website checker fails due to robots.txt file not accessible.
HTTP status code: Timeout
Robots.txt error: Internal error
We either got an HTTP code that denies access to the file or the sever failed to satisfy the request. Most likely reason is that your website is blocking our crawler. Check that you do not have a firewall or plugin blocking bots.
Bing webmaster tools also unable to access this file.
Hi and welcome to the forum! Maybe this is caused by our security system:
You might try to use Cloudflare since you have a custom domain to bypass it, but if the tool you’re using uses cURL to check your website this might not work as well.
Thank you for the quick response. I’m a novice and this is simply a personal website, small and nothing fancy. I’m just wanting to get indexed by search engines. Google seems to be fine with it so far for indexing my pages. Bing and Ahrefs are problematic it appears.
I just read the link you sent and it is right on the mark. Bing and the website checker Ahrefs balk at the code sent to check for javascript. I also noticed that if I pay for the not-free, premium version this security check is no longer an issue. Thanks again.
I thought I understood the fix for my issue but upon further analysis, I wanted to try again to state my case and seek help. In June 2023 there is a great deal of discussion on IF vs Bing. My exact situation was presented regarding Bing whereby the javascript response belies ever actually reaching my website. The result of that posting seems to say that many IF sites do indeed play nice with Bing. I wanted to try to better understand if my site will indeed play nice or if it is hopeless. I guess my focus right now is the fact that my robots.txt file passes muster as a url; but when the command to Fetch Current is issued, that is when the aes.js / javascript response fails. Thanks.