Robots.txt Generator

robots.txt tells crawlers which URLs they may fetch. That is the whole of it. It does not hide anything from a person: a disallowed page still loads in a browser when you type its address, and a disallowed asset is still downloadable if you know the URL.

It is also not an access control. Anything that must be genuinely private — an admin area, a paid download, personal data — needs authentication at the server, never a robots.txt rule. Treat the file as a polite instruction about what is not worth crawling.

The file is a series of groups. Each starts with one or more User-agent lines followed by rules that apply to exactly those crawlers. A crawler reads the group whose user-agent token matches its own name exactly — Googlebot matches User-agent: Googlebot but not User-agent: Google.

The catch that surprises people: when a specific group matches, the catch-all * group is ignored for that crawler. Listing rules for Googlebot and nothing else means every other crawler, including Bing, is governed solely by whatever the * group says — or by no group at all.

Disallow: with nothing after the colon means allow everything; that is the only correct way to express it. Disallow: / means block everything. The distinction is the source of a well-known publishing mistake, and it is worth reading the generated file once before you ship it.

Within a group, the most specific matching rule wins, and a matching Allow beats a matching Disallow regardless of length. That is how a single file is opened inside a blocked folder — block /admin, then allow /admin/public.

Two wildcards are supported. * matches any run of characters, so /admin/* covers everything under that folder. $ anchors the end of the URL, so /page.php$ matches that one file but not /page.php?id=4.

Paths are matched from the beginning of the path, not as complete strings. A rule of /admin therefore also covers /administrator and /admin-panel. A leading slash is added for you if you leave it out.

Crawl-delay asks a crawler to wait between requests. Google has ignored it for years, while Bing and Yandex still honour it, so it is worth setting for a large site that is not bandwidth-sensitive rather than for a small one.

The Sitemap: line is the most reliably useful part of the file for most sites. It points crawlers at your XML sitemap and needs no group, so it sits at the end, outside every group. A full absolute URL is required; a relative path is ignored.

The file must live at the root of your domain — https://example.com/robots.txt — and nowhere else. A file in a subfolder, or one blocked by a server rule, does nothing at all while looking like it is working.

Before you publish, check two things: that the first group does not contain Disallow: /, and that a path you definitely want indexed is not caught by a prefix rule. Then confirm the live file by opening its URL directly — if the site is on a framework that rewrites unknown paths to a 200 page, you may be looking at your homepage instead of a 404, which hides the problem rather than fixing it.