Error 520 at ahrefs This site can't be crawled

Hosting Support
1

When trying to add my site to aherfs I run into 520 issues according to their tools. I have Cloudflare configured as flexible and have really simple SSL installed and visually dont see an issue. Any ideas? I have a few other sites setup on other host the same way without issues.

When trying to add my site to aherfs I see error 520. This site can’t be crawled as seen below.

This website can’t be crawled.
https://tutorialhelpdesk.com/robots.txt
robots.txt page returns an HTTP error with code 520
4 Oct, 12:19 AM

http://tutorialhelpdesk.com/robots.txt
The HTTP server encountered an internal error (520). The server may currently have issues.
This website is protected by the Cloudflare firewall
4 Oct, 12:19 AM

https://www.tutorialhelpdesk.com/robots.txt
robots.txt page returns an HTTP error with code 520
4 Oct, 12:19 AM

http://www.tutorialhelpdesk.com/robots.txt
The HTTP server encountered an internal error (520). The server may currently have issues.
This website is protected by the Cloudflare firewall
4 Oct, 12:19 AM

2

If you enabled Bot Fight Mode in Cloudflare, this is expected.

3 Likes
3

Definitely no bot blocking or anything else in the cloudflare security section.

1 Like
4

image
image1040×606 32.6 KB

I think that’s normal

first http redirects to https (that’s why http doesn’t work)

in addition, the www version redirects to naked (not www)
so again it is normal to throw an error

while the right URL, as far as I can see, properly recognizes and reads.



Test here

4 Likes
5

So aside from ahrefs showing this issue do you think I can ignore the errors without any major issues with functionality?

6

Now it shows me again that it doesn’t work even on HTTPS (all 4 variants 520)

You can always check on Cloudflare
Firewall Events Activity Log for an entry that corresponds with Ahrefs

But I don’t think it’s blocked - if something is blocked then it would show you 403 and not 520

It can also be that it depends on what configuration you use on CF

Maybe that test tool of ahrefs is silly too
520 comes from Cloudflare, but only that ahrefs test reports that - while your site works properly through the browser

or is some rate limit active (do you have any plugin for protection on your WP?)

I don’t really know what to tell you, because sometimes the test works, sometimes it doesn’t.

3 Likes
7

I definitely have no additional protections. I just want to make sure as content is created the site can index correctly. Right now I am not sure if it can.

8

Ahrefs is a tool that gives you some statistics
with which you can “achieve” better SEO with your additional actions on site.

Is not something in the sense of a google bot and does not belong under a search engine.

You don’t really need any tools, just quality content
that’s what it’s all about - quality original content

The core goal for SEO should be to attract customers to your website, not obsess over how much search engines or other bots love you. The latter will come inevitably if the former is achieved.

I have that bot blocked

image
image967×538 29.1 KB

why?

4 Likes
9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.