Robots.txt Generator
Prepare a robots.txt draft from explicit crawler, allow, disallow, sitemap, and optional crawl-delay values. The page distinguishes crawl control from deindexing.
Browser-localFreeNo upload required
What this tool is for
Prepare a robots.txt draft from explicit crawler, allow, disallow, sitemap, and optional crawl-delay values. The page distinguishes crawl control from deindexing.
How to use
- Choose the user agent and add only paths you genuinely want crawlers to allow or disallow.
- Add the canonical sitemap
URLand review every generated line. - Test the draft against representative public and private paths before publishing `/robots.txt`.
Example
Allow the site, disallow `/internal-preview/`, and add the canonical sitemap URL; then verify that no rule blocks CSS, JavaScript, or the whole site.
FAQ
Can robots.txt remove an indexed page?
Not reliably. Use an indexable-page `noindex` response or removal workflow as appropriate.
Is crawl-delay supported by Googlebot?
Googlebot does not use the crawl-delay directive; support differs by crawler.
Reproducible examples
Known inputs and expected results
Run these examples in the workspace above and compare the result with the documented output.
One crawler group
- Input
- {"userAgent":"*","allow":["/"],"disallow":["/private/"],"sitemaps":["https://example.com/sitemap.xml"]}
- Options
- Use one group and one sitemap.
- Action
Generaterobots.txt.- Expected output
- User-agent: *, Allow: /, Disallow: /private/, and the absolute Sitemap line.
- Explanation
- The rules are text directives and do not remove an already indexed
URL.
Multiple rule lines
- Input
- {"userAgent":"ExampleBot","allow":["/public/"],"disallow":["/","/tmp/"]}
- Options
- Keep the rule order visible.
- Action
Generateand inspect every line.- Expected output
- A group for ExampleBot with one Allow and two Disallow directives.
- Explanation
- Crawler-specific longest-match behavior must still be tested against the target crawler.
Error or unsupported case
Malformed configuration
- Input
- {"userAgent":"*","sitemaps":["not a url"]
- Expected error or limit
- Invalid
JSONor an invalid sitemapURLproduces a visible error rather than a silent permissive fallback. - Safer alternative
- Correct the configuration and verify the published root robots.txt and page-level indexing controls.
When not to use this tool
- Removing a page from search or protecting confidential content
- Guaranteeing support for crawl-delay or every crawler's interpretation
Algorithm and assumptions
Generation supports the documented Robots Exclusion Protocol subset; JSON input must be an object, sitemap values are absolute URLs, and noindex/canonical remain separate controls.
Input and output
Input is a manual rule set. Output is plain robots.txt text for review and copying.
Supported formats and behavior
- User-agent groups
- Allow and Disallow paths
- Sitemap and optional crawl-delay lines
Limits
- No live-site fetch or official crawler simulation
- Robots rules are public and should never contain secrets
Edge cases and common errors
- `Disallow: /` blocks the entire selected user agent
- A blank Disallow permits crawling
- Different crawlers resolve conflicting rules differently
Release testing
Release checks cover allow/disallow output, full-site block warnings, sitemap lines, blank rules, invalid sitemap values, copy, and reset.
Privacy boundary
The input and result stay in this browser session. ZZP Box does not upload or store the values entered in this tool.
Related reviewed tools
Choose the next tool by the operation you need, then review that page's stated input and limits.