The sitemap generator (https://www.xml-sitemaps.com) reports that the crawler bot is being blocked when I submit my homepage URL.
Other Information
My site has 250 pages. Generator support reports that the bot is getting blocked by Cloudflare. I disabled Cloudflare and re-ran and the bot reports that another block is in place, with the following crawler bot response:
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 08 Feb 2025 19:06:41 GMT
Content-Type: text/html
Content-Length: 841
Connection: keep-alive
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Cache-Control: no-cache
My site does not directly use javascript. The error report shows one of my webpages “news.html.” I don’t know why it’s showing that. I removed the link to it from my homepage and re-ran the generator to no avail. I have no clue where the block is coming from nor the javascript / aes.js references. Thank you.
Bing Webmaster Tools also reports that the site scan will not work because I am using dynamic html. I am not to my knowledge. I honestly don’t know how to tell. I’m a novice, sorry folks.
Thank you Dan. I remember dealing with security limitations by incorporating Cloudflare. Does it not allow for things such as sitemap.xml generation? That was a few months ago. It seems like things are different now.
If I may add my two cents: sitemap generator sites are useless anyway.
The point of a sitemap is that it helps search engine crawlers understand the structure of your site. But you have to understand that search engine crawlers are quite sophisticated and will probably understand you site pretty well without it.
When you use an online sitemap generator, you’re basically using a rudimentary crawler to generate a structure that should give extra information to a much more sophisticated crawler.
The added value of sitemaps is for things a crawler cannot easily guess. So sitemaps only make sense if they are either crafted by hand or generated by your website building software (which knows more about the structure of your site than any crawler).
Your website will get crawled and indexed just fine without a sitemap. If you decide you’re going to add a sitemap anyway, make sure that it actually adds value and isn’t just junk spat out by one crawler being fed into another crawler.
