Hello, Ive been trying to fix an issue I had for a while about robots.txt, everytime when I try to analyze my website in page speed insights I get this message : (robots.txt is not validLighthouse was unable to download a robots.txt file) (If your robots.txt file is malformed, crawlers may not be able to understand how you want your website to be crawled or indexed).
I really don’t know what to do with that because it appears totally fine for me and when I try to run an ahrefs site audit it tells me that my robot.txt is not accessible.
Is there an issue with the robot.txt for my website?
And another question for info:
Does my issue affect why my website is totally not appearing in Google search results?
Because it is not appearing at all even after typing “site:merchandisemuseum.kesug.com”
It might be blocked by this though I hadn’t experienced similar issues before:
I don’t see any issues.
Usually not, Google would be able to access your site normally. The reason why you’re not seeing it is because that’s just not how it works. Google isn’t some magic and knows every site on this world. You have to give it time to index your site and eventually shows up in search results.
The ultimate solution is to get a custom domain and use cloudflare.
If you insist on using a free subdomain then you probably would keep on facing the same issue. There might also be a problem with Lighthouse, so I don’t think you should worry as so many sites here got indexed successfully.