HTTP/1.1 403 Forbidden
pls add support for YandexBot user-agent
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
as examplle for google bot user-agent 200 OK
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Yandex crawlers are not allowed to access our hosting. They are very aggressive and generate high server load, while only contributing very few legitimate visitors.
Also, we don’t whitelist based on user agent alone. Because as you know: it’s trivial to change your user agent. You’ll get a 200 response with a Google user agent, but that response will not be your actual web page.
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.