I have submitted my sitemap to both bing webmaster and google analytics. Google can fetch and index my site normally, but bing keeps reportting error. I used the URL check tool on bing webmaster to check the url of my site, it replied 403 access denied. I asked AI for help and configured the htaccess to specifically allow bing agent, but it still can not fetch my website. AI instructed me to look into the server log, I don’t know where to find the server log.
Can anyone please tell me how to resolve this issue? Thanks!
Bing should be able to index your website (although Bing’s url checker and other similar tools in webmaster don’t work due to that system… see this for clarification). What happens when you submit the sitemap to Bing webmaster? Does it give you an error?
When I submit the sitemap (whethere in xml or txt format), the Bing webmaster also reports 403 (Forbidden). Bing says I can Fetch as Bingbot to test the sitemap url, I did it, and also got 403 (curl -I -A “Bingbot” https://psych-scholars.com/), but event I can fetch the url even using any virtual agent (curl -I -A “My-agent” https://psyche-scholars.com/)
The Bing bot can access your website, but the “Bing bot” must actually try to connect from the Bing servers. You cannot just bypass our security systems with a custom user agent.
The request with the custom user agent doesn’t work though. You’ll get a “successful” response, but the response will be the browser validation challenge, not your actual web page.
I was not trying to bypass the security systems. I was just trying to figure out the problem. The situation is that the bing webmaster reports 403 error to all of the following:
Currently, I am not able to find my site in the search results of Bing, but able to do this on Google. I am not sure whether it is the “security system” that caused this. Is this normal for freeplan hosted sites, or is this a issue of my site that needs to do something to fix it?
