What I’m seeing is: While I’m trying to figure out why online sitemap generators only able to fetch single webpage but not any other, I went into the SEO tools provided by infinityfree cpanel where one can find Robots.txt Checker tool which helps to identify the robots.txt file in any website. So when I tried my website it says robots.txt not found for “website”.
Also I tried to see whether some of crawlers like httpstatus.io can able to identify robots.txt file by providing the url path to check status. Even it says “error” - Unable to validate first certificate.
Can anyone has any kind of solution to this? Kindly let me know. Thank you.
Online tools are blocked by the server so they are unable to view all the content of your site and create a corresponding sitemaps !
(Google is allowed - major popular crawlers are allowed)
The solution is to use some offline Sitemap generator and instruct it to review the local content of the folder in which is your website. (or simply write it yourself)
It’s best to use Cloudflare
you get free SSL (plus other benefits) and online tools like that sitemap generator you used are not blocked so you can easily generate it
P.S. Google supports also txt format for sitemap
If your sitemap includes only web page URLs, you can provide Google with a simple text file that contains one URL per line. For example:
Encode your file using UTF-8 encoding.
Your text file should contain nothing but the list of URLs.
You can name the text file anything you wish, provided it has a .txt extension (for instance, sitemap.txt).
