Ahrefs Crawl Keeps Failing


So I tried to use the Site Audit feature on Ahrefs but it keeps showing this error every single day.

That probably happens because of this:


Probably due to this

1 Like

Oh man :frowning: Is there no way to bypass this or an alternative? I mean Ahrefs crawlers are good bots.

No, you cannot bypass this security measure, you can, however, use an alternate service.

1 Like

Use cloud flare.

I am using cloudflare (and not with the broken cpanel integration, I’ve changed to cloudflare’s nameservers). I still get the crawl failed message.

Huh, idk then…

Check your firewall logs in cloudflare

That was the first thing I did. But the log was empty. I even paused Cloudflare just to be sure but it was still failing. That’s how I came to the conclusion that this was a hosting problem.

How would Cloudflare allow the bots? Cloudflare is just a DNS and protection service, all the traffic still goes to Infinity where it is than blocked by the security protocol.

@Citrus_Titan, I believe that premium hosting allows this, but you would have to ask them before upgrading if you would like to do so. There is no way to bypass this measure on the free service. And since most other free hosts are based on this service, it is unlikely you will find a free host that allows this.

Not quite. When using Cloudflare, our browser validation system is disabled under the assumption that Cloudflare’s security settings block all bad traffic. Those same security settings could be blocking that crawler.

1 Like

Oh, did not know that, thanks for the info and apologies to @Deveroonie (Although maybe you could give more info in your answer so I don’t make that mistake next time)

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.