(please specify the URL of the site on which you are experiencing the problem)
403 Forbidden
Various bots of Yandex search system cannot index any page of my site due to error 403 (Forbidden). I cannot change any settings related to that, because I don’t have them. So the problem is in the hosting settings, obviously. How can it be solved?
so, there won’t be any answer, I suppose. That’s just great.
The honest answer is that I don’t know. There are more people reporting issues with their sites not getting indexed by Yandex, but other sites do seem to be getting indexed without any problems. I don’t know that much about Yandex, so I have no idea why it works for some sites and not for others, I can’t imagine how our hosting would discriminate somehow.
Those others who had problems were also using Cloudflare,
and on the CF support forum, it was stated that CF has problems with Yandex because their bot has a habit of frequently changing the IP range and regularly introducing some new IPs.
Because of this, it is difficult for CF to determine whether it is a valid bot or a malicious one that will then be blocked.
Probably something like that is the cause here as well
Thank you for the answer. Then, I’ll try to find the answer myself.
That’s probably not my case, 'cause I don’t use Cloudflare.
It may not be, but it can help offer an explanation.
InfinityFree uses this security system:
Normally, web crawling bots from search engines such as Yandex are whitelisted, however if Yandex keeps changing the IP ranges they are using, the system, unaware of those changes, would enforce the security checks on Yandex’s requests and cause this issue, much like how Cloudflare can block legitimate Yandex requests thinking they are malicious when this happens.
This is just speculation though; I also do not know how Yandex operates, or how the whitelist is implemented, so I cannot be sure this is the case.
Hi ChrisPAR and oldfagtv,
You may reference this link:
Yandex is a Russian search engine platform and its user agent is often spoofed, however, they do not offer any other validation methods other than reverse DNS toward their domain.
Unless a solution to check for the reverse DNS per hit, it’s likely Yandex indexing is not compatible with InfinityFree.
Cheers!
this likely has something to do with the way infinityfree redirects sites using the ?=1 at the end of the url. not sure what this is for but redirects can cause sites like yandex or archive.org to not index.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.