Robots.txt Generator: Build Rules Without Breaking

A robots.txt generator assembles a valid robots.txt file from crawler groups, allow and disallow paths, an optional crawl delay and a sitemap line. Choose the crawlers, list the paths, and copy the generated text into public/robots.txt. The tool never writes to your server, so it cannot break a live site while you experiment.

The most common robots.txt disaster is a single character: Disallow: / blocks everything, and a staging rule copied to production can remove a whole site from search. This tool flags the obvious versions of that mistake before you copy the file, but it cannot know your intent. Read the output before publishing it.

#What goes into a robots.txt

A robots.txt file is a set of groups. Each group starts with one or more User-agent lines and is followed by rules for that crawler.

• User-agent — the crawler token the group applies to, or * for all crawlers.
• Disallow — paths the crawler should not request. Matching starts at the beginning of the path, so /admin also covers /administrator.
• Allow — paths inside a blocked folder that should still be crawled. A more specific Allow overrides a matching Disallow for most crawlers.
• Crawl-delay — seconds between requests. Bing and Yandex honour it; Google ignores it.
• Sitemap — an absolute URL to your sitemap. This line is outside the groups and can appear anywhere.

The generator offers groups for all crawlers, Googlebot, Googlebot-Image, Bingbot, YandexBot, DuckDuckBot and Baiduspider. Those tokens are written exactly as shown; a typo in a user-agent name means the group silently does nothing.

#How to generate robots.txt

1. Open the robots.txt generator.
2. Tick the crawlers the rules should apply to. * is selected by default.
3. Choose the access mode: block listed paths, allow everything, or emit no group for the selected crawlers.
4. List paths to disallow, one per line. A missing leading slash is added for you.
5. Add any Allow exceptions. These are useful for a single public file inside a blocked directory.
6. Set a crawl delay only if you have a reason. Zero omits the line.
7. Paste a full sitemap URL if you have one.
8. Press Run, then copy the text to public/robots.txt.

Paths are trimmed and de-duplicated. Blank lines are ignored. Wildcards such as * and the end anchor $ are passed through as text, because support varies between crawlers.

#Which access mode to use

The third mode is easy to misread. "Ignore" here means "do not write a group for this crawler", not "block this crawler". To block a crawler, select it and disallow /. To leave it governed by the wildcard group, select nothing that overrides it.

#Common problems and fixes

The whole site disappeared from search. The file contains Disallow: /. That blocks every path. Remove it, request a recrawl, and check that the file is not still cached. The generator warns when it sees this line, but it will still build it if you asked for it.

A private page is still in the index. Disallow prevents crawling, not indexing. If other sites link to the URL, or it was indexed before the rule, it can remain in results without a snippet. To keep a page out of the index, use a noindex robots meta tag or an X-Robots-Tag header, and allow crawling so the crawler can see the instruction.

The sitemap line was dropped. The sitemap must be an absolute http or https URL with a host. A bare path such as /sitemap.xml is not valid and is left out with a note. Enter https://example.com/sitemap.xml.

A crawler ignores the rule. User-agent names are case-insensitive in the spec but must match the crawler's token. Googlebot-News is not Googlebot. Check the exact token and remember that some crawlers ignore groups they do not recognise.

The rule blocks more than intended. Path matching is prefix-based. /admin blocks /admin, /administrator and /admin-tools. Use a trailing slash (/admin/) or a more specific path if you meant only one directory. Wildcards behave differently across crawlers, so test with the crawler's own checker.

Crawl-delay did nothing. Google ignores Crawl-delay. Use Search Console's crawl rate settings for Google, and treat the line as guidance for Bing and Yandex.

The file returns a 404 or HTML. robots.txt must be served as plain text at the site root. A soft 404 or an HTML error page is treated as no robots.txt at all. Check the response code and content type after uploading.

A staging file reached production. Keep the production file separate from staging. A staging file that blocks everything is correct on staging and catastrophic live.

#Rules worth remembering

• robots.txt is a hint, not access control. Anything disallowed can still be fetched by a human or a bot that chooses to ignore the file. Protect private data with authentication, not a text file.
• Do not block CSS or JavaScript. Google renders pages and needs those files to understand the layout. Blocking them can make a page look broken to the crawler.
• Do not block a page you want indexed. Crawling has to happen before indexing. Disallow and noindex together are contradictory: the crawler cannot see the noindex if it cannot fetch the page.
• Keep the file small and simple. A short list of real paths is easier to audit than a hundred rules accumulated over years.
• Use one group per crawler. Do not repeat the same user-agent in several groups and expect the rules to merge; behaviour differs.
• Test after upload. Fetch https://yourdomain.com/robots.txt in a browser and confirm it is the file you expected.

#Robots.txt, sitemaps and canonical tags

These three files solve different problems, and they are often confused:

• robots.txt says which paths crawlers may request.
• A sitemap lists the URLs you want indexed, with optional last-modified dates. Build one with the XML sitemap generator, then add its URL to the Sitemap: line here.
• A canonical tag chooses one URL when several serve the same content. The canonical tag builder lists the variants you need to cover.

Before publishing a page, the meta tag analyzer checks the title, description, robots meta, viewport and social tags from pasted source. It complements robots.txt: one controls crawling, the other tells the crawler how to treat the page once it arrives.

#Frequently asked questions

#Bottom line

A robots.txt file is short, easy to get wrong and slow to notice. Generate it deliberately: choose the crawlers, list only the paths you mean to block, add the sitemap line, and read the output before copying it. Start with the robots.txt generator, use the sitemap generator for the file it points at, and check each page's tags with the meta tag analyzer.