How to use Robots.txt Generator
- Choose whether the rules apply to all crawlers or a named user-agent.
- Add one path per line to Allow or Disallow.
- Enter the full sitemap URL if you have one.
- Generate the file and review the directives carefully.
- Save it as robots.txt at the root of the correct hostname, then test the live file.
What to know
A robots.txt file gives cooperating crawlers instructions about which URL paths they may request. It is not access control, a privacy barrier or a way to remove a URL from search results by itself. Never use it to protect private files or sensitive pages.
Frequently asked questions
Does Disallow remove a page from Google?
Not necessarily. A blocked URL can still be indexed from links or other signals. Use appropriate noindex mechanisms when a page should not appear in search, and ensure crawlers can access the noindex instruction.
Does robots.txt secure private content?
No. It is publicly readable and voluntary for compliant crawlers. Use authentication and server-side access controls for private content.
Where should robots.txt be placed?
At the root of the hostname, for example https://example.com/robots.txt. A file in a subfolder does not govern the whole site.