# Robots.txt issue **URL:** \ **Category:** Hosting Support\ **Created:** [March 9, 2024, 5:26am UTC](https://forum.infinityfree.com/t/robots-txt-issue/90886 "2024-03-09T05:26:24Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![MerchandiseMuseum](https://forum.infinityfree.com/user_avatar/forum.infinityfree.com/merchandisemuseum/32/46653_2.png) [@MerchandiseMuseum](https://forum.infinityfree.com/u/MerchandiseMuseum)\ **Post date:** [March 9, 2024, 5:26am UTC](https://forum.infinityfree.com/t/robots-txt-issue/90886/1 "2024-03-09T05:26:24Z")
### Website URL ([https://merchandisemuseum.kesug.com/robots.txt](https://merchandisemuseum.kesug.com/robots.txt)) ### Other Information Hello, Ive been trying to fix an issue I had for a while about robots.txt, everytime when I try to analyze my website in page speed insights I get this message : (robots.txt is not validLighthouse was unable to download a robots.txt file) (If your robots.txt file is malformed, crawlers may not be able to understand how you want your website to be crawled or indexed). I really don’t know what to do with that because it appears totally fine for me and when I try to run an ahrefs site audit it tells me that my robot.txt is not accessible. Is there an issue with the robot.txt for my website? And another question for info: Does my issue affect why my website is totally not appearing in Google search results? Because it is not appearing at all even after typing “site:merchandisemuseum.kesug.com” ---
**Author:** ![Meishin](https://forum.infinityfree.com/user_avatar/forum.infinityfree.com/meishin/32/59902_2.png) [@Meishin](https://forum.infinityfree.com/u/Meishin)\ **Post date:** [March 9, 2024, 6:09am UTC](https://forum.infinityfree.com/t/robots-txt-issue/90886/2 "2024-03-09T06:09:20Z")
> [@MerchandiseMuseum](#): > > Lighthouse was unable to download a robots.txt file It might be blocked by this though I hadn’t experienced similar issues before: > [@Ensuring only web browsers can access your website](https://forum.infinityfree.com/t/ensuring-only-web-browsers-can-access-your-website/49353): > > InfinityFree is a website hosting service. That means that the hosting accounts we provided are intended for hosting websites. Websites contain pages that are accessed through web browsers. InfinityFree is not intended to be used for file sharing, API hosting, database hosting or background tasks/tools. To help enforce this, free hosting enforces a security system that makes sure that anyone trying to access your website is using a normal web browser. This is done by checking whether the web br… > [@MerchandiseMuseum](#): > > Is there an issue with the robot.txt for my website? I don’t see any issues. > [@MerchandiseMuseum](#): > > Does my issue affect why my website is totally not appearing in Google search results Usually not, Google would be able to access your site normally. The reason why you’re not seeing it is because that’s just not how it works. Google isn’t some magic and knows every site on this world. You have to give it time to index your site and eventually shows up in search results. ---
**Author:** ![MerchandiseMuseum](https://forum.infinityfree.com/user_avatar/forum.infinityfree.com/merchandisemuseum/32/46653_2.png) [@MerchandiseMuseum](https://forum.infinityfree.com/u/MerchandiseMuseum)\ **Post date:** [March 9, 2024, 6:17am UTC](https://forum.infinityfree.com/t/robots-txt-issue/90886/3 "2024-03-09T06:17:27Z")
even if you hadnt experienced a similar issue like this before do you know a solution for it? Because if there is no issue in my robots.txt file, Lighthouse should be easily able to download a robot.txt file. Any ideas or tips? ---
**Author:** ![Meishin](https://forum.infinityfree.com/user_avatar/forum.infinityfree.com/meishin/32/59902_2.png) [@Meishin](https://forum.infinityfree.com/u/Meishin)\ **Post date:** [March 9, 2024, 6:22am UTC](https://forum.infinityfree.com/t/robots-txt-issue/90886/4 "2024-03-09T06:22:42Z")
> [@MerchandiseMuseum](#): > > do you know a solution for it? The ultimate solution is to get a custom domain and use cloudflare. If you insist on using a free subdomain then you probably would keep on facing the same issue. There might also be a problem with Lighthouse, so I don’t think you should worry as so many sites here got indexed successfully. ---
**Author:** ![MerchandiseMuseum](https://forum.infinityfree.com/user_avatar/forum.infinityfree.com/merchandisemuseum/32/46653_2.png) [@MerchandiseMuseum](https://forum.infinityfree.com/u/MerchandiseMuseum)\ **Post date:** [March 9, 2024, 7:16am UTC](https://forum.infinityfree.com/t/robots-txt-issue/90886/5 "2024-03-09T07:16:18Z")
cant i just use the same subdomain domain to use with cloudflare? or will that not work? but according to your answer, if i keep it as it it i should fine for my website to indexed, I hope so. thank you. ---
**Author:** ![Meishin](https://forum.infinityfree.com/user_avatar/forum.infinityfree.com/meishin/32/59902_2.png) [@Meishin](https://forum.infinityfree.com/u/Meishin)\ **Post date:** [March 9, 2024, 7:18am UTC](https://forum.infinityfree.com/t/robots-txt-issue/90886/6 "2024-03-09T07:18:58Z")
> [@MerchandiseMuseum](#): > > cant i just use the same subdomain domain to use with cloudflare No, Cloudflare don’t accept these. > [@MerchandiseMuseum](#): > > if i keep it as it it i should fine for my website to indexed Yes. ---
**Author:** ![system](https://infinityfree-forum-uploads.s3.dualstack.eu-central-1.amazonaws.com/original/3X/b/0/b024fd3fa88bae093c9576e55c2c94e33f890206.svg) [@system](https://forum.infinityfree.com/u/system)\ **Post date:** [March 16, 2024, 7:19am UTC](https://forum.infinityfree.com/t/robots-txt-issue/90886/7 "2024-03-16T07:19:58Z")
Closed on March 16, 2024, 7:19am UTC This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.