🤖 SEO Robots.txt & Sitemap Directives Generator
A robots.txt generator with correct Allow/Disallow precedence — and why Disallow blocks crawling but does not stop a URL from being indexed.
Generated robots.txt File
What SEO Robots.txt & Sitemap Directives Generator Does
robots.txt tells crawlers which paths they may request. It lives at the root of a domain, is fetched before crawling begins, and applies only to that exact host and protocol — a file on https://example.com does not govern the www subdomain.
The rule almost everyone gets wrong is what Disallow actually does. It stops a compliant crawler from *fetching* a URL. It does not stop that URL from being *indexed*: if Google finds enough links pointing at a blocked page, it can list the URL in results with no description, because it was never allowed to read the page to find the noindex tag you added.
That produces a genuine catch: to keep a page out of the index you must let Google crawl it so it can see the noindex directive. Blocking it in robots.txt guarantees the opposite of what you wanted.
How to Use SEO Robots.txt & Sitemap Directives Generator
- Configure default crawler access and specify any private directories to disallow
- Input your XML sitemap URL
- Click Copy Robots.txt and save to the root of your web server
What Each Directive Does
| Directive | Effect | Supported by Google |
|---|---|---|
| User-agent: * | Applies the following rules to all crawlers | Yes |
| Disallow: /path | Do not fetch anything under /path | Yes |
| Allow: /path/file | Exception carved out of a broader Disallow | Yes |
| Sitemap: https://…/sitemap.xml | Points to your sitemap; absolute URL required | Yes |
| Crawl-delay: 10 | Seconds between requests | No — Google ignores it |
| Noindex: /path | Never officially supported; removed in 2019 | No |
Source: Google Search Central — Create and submit a robots.txt file
Blocking vs De-indexing
These solve different problems and are frequently confused.
| Goal | Correct method | Why not robots.txt |
|---|---|---|
| Keep a page out of search results | <meta name="robots" content="noindex"> or an X-Robots-Tag header | A blocked page cannot be read, so the noindex is never seen |
| Save crawl budget on low-value URLs | robots.txt Disallow | This is what it is for |
| Keep content private | Authentication | robots.txt is public and advisory only |
| Remove a URL urgently | Search Console removal tool, then noindex | robots.txt does not remove anything already indexed |
Pattern Matching
| Pattern | Matches |
|---|---|
| Disallow: / | The entire site |
| Disallow: /admin/ | Everything under /admin/ |
| Disallow: /*.pdf$ | Any URL ending in .pdf |
| Disallow: /*? | Any URL containing a query string |
| Allow: /admin/public/ | Re-permits one subtree inside a blocked one |
| Disallow: | Nothing — an empty value blocks nothing |
How to Read Your Result
Never block CSS and JavaScript
Google renders pages to evaluate them. Blocking /assets/ or /static/ means the renderer sees an unstyled, non-functional page and judges it on that. This was once common advice and is now actively harmful. If a crawler cannot load your stylesheet, it cannot confirm your page is mobile-friendly.
robots.txt is public
Anyone can read yours at /robots.txt, and listing a path there advertises it. Blocking /admin-secret-panel/ tells every visitor exactly where to look. Anything that must not be reached needs authentication; robots.txt is a request, not a lock, and hostile crawlers simply ignore it.
The longest matching rule wins
When Allow and Disallow both match a URL, Google applies the more specific rule — the one with the longer path — not the one listed first. So Disallow: /folder/ with Allow: /folder/page.html permits that one page. If a rule is not behaving as expected, count the characters.
One file per host and protocol
https://example.com/robots.txt governs only that origin. The www subdomain, a staging subdomain and the http version each need their own file. Sites that redirect www to non-www still serve robots.txt from both until the redirect resolves, which is a common source of confusion.
Limitations & Accuracy Notes
- robots.txt is voluntary. Well-behaved crawlers honor it; scrapers and malicious bots ignore it entirely.
- It controls crawling only. It cannot remove existing indexed pages, and it cannot keep content private.
- Crawl-delay is ignored by Google. Use the crawl rate setting in Search Console if you need to slow it down.
- Rules are case-sensitive in the path, so /Admin/ and /admin/ are different. The filename itself must be lowercase robots.txt.
- A syntax error can be interpreted more permissively or more strictly than you intended. Test the finished file with Google Search Console's robots.txt report before relying on it.
- The file must be under 500 KiB for Google; content beyond that is ignored.
Frequently Asked Questions
What is a robots.txt file?
Where should the robots.txt file be uploaded?
Does Disallow stop a page being indexed?
Is robots.txt a security control?
Is Crawl-delay honored?
Where must the file live?
Should I block CSS and JavaScript?
What happens if the file is missing?
References & Further Reading
- Google Search Central — Create and submit a robots.txt file — Official directive support, size limits and matching rules
- RFC 9309 — Robots Exclusion Protocol — The 2022 IETF standardization of robots.txt