Hello,
I am experiencing issues with the indexing and accessibility of my sitemap.xml. Despite having a correctly generated Sitemap, Google Search Console is showing errors such as “Unable to fetch” or “Blocked by robots.txt.”
I have confirmed that the Sitemap is accessible through the browser, but it seems that Googlebot is unable to crawl it correctly.
Could you please assist me in resolving this issue and ensuring that my Sitemap is correctly indexed?
I have the following plugins installed: SEO Optimized Images, Spice Starter Sites, Yoast SEO, and Akismet Anti-spam. Could the issue be related to one of these plugins?
Websites on our hosting can just get indexed by search engines. When I search Google for site:newshub.pp.ua, I see many pages on your website are being indexed successfully.
Note that we have a security system that blocks access from things that are not browsers. We’ve made sure that actual search engine crawlers can access your site, but sitemap checkers might not be able to do.
Part of this security system is also that visitors may be redirected to a URL with ?i=1 at the end of it. It’s possible that these are blocked by your robots.txt through this line:
Disallow: /*?*
You might want to try and remove this line and see if it helps.
You might be right that I don’t fully understand everything, but regarding the indexing – the pages are being indexed because I submitted them manually through Google Search Console for indexing. I also tried to add the sitemap path in the Sitemaps section of Google Search Console, but I immediately get the error “Couldn’t fetch”. I removed the line Disallow: /? from the robots.txt file, but nothing changed. What could be the problem then?
Hello, could you please tell me how much time it usually takes, because when I go to the path https://newshub.pp.ua/robots.txt . I can already see the updated content.
I’m not Google. Google decides when it’s time to check your site again. It can be the next second, days, weeks, or even months. But I cannot really tell anything.
Submitting every URL by hand is not necessary. The fact the pages are indexed means that Google can access your website. And if they can access your website, it means they can navigate it and find all the pages on your website on their own.
A website doesn’t need to have a sitemap to be indexed.
Compare it to visiting a new city.
It helps if you have map of the city beforehand, so you know where all the interesting places are, and you know how to navigate between them effectively.
But not having a map beforehand doesn’t mean you’re not allowed to go to the city. You can still go there and just follow the road signs, or just wander around, and still have a good time. You might not find some things that you could have seen on the map, but you might also discover things that the map didn’t show.
Search engines largely work the same. A sitemap is really just a map to your site. Search engines only need to know that your website exists, and as long as they are not denied entry, they will visit it, navigate it, and index it.