Robots.txt Generator
Create a correct robots.txt by adding user-agent rules, allow and disallow paths, crawl delay and a sitemap reference — with inline validation.
Examples
Disallow /admin
User-agent: * Disallow: /admin Sitemap: https://site.com/sitemap.xml
How it works
The tool writes one rule group: a User-agent line, then your Allow paths, then your Disallow paths, an optional Crawl-delay and a Sitemap line. If you list no Disallow paths it writes an empty “Disallow:”, which means nothing is blocked.
User-agent: * Allow: /blog/drafts/public Disallow: /blog/drafts Disallow: /cart Sitemap: https://example.com/sitemap.xml
The “Block everything” preset writes “Disallow: /” and ignores the path boxes. Start each path with a slash; the tool does not check them. It builds a single group, so for different rules per crawler, generate each group separately and paste them into one file. Googlebot ignores Crawl-delay.
How to use Robots.txt Generator
Why use this tool
Frequently asked questions
No. It controls crawling, not indexing. To keep a page out of results, use a noindex meta tag or HTTP header.
Disallow blocks a path and Allow re-opens part of it. When rules conflict, Google follows the most specific (longest) matching rule. For example, "Disallow: /private/" plus "Allow: /private/public-page.html" blocks the folder but still lets crawlers reach that one page.
Use "User-agent: *" followed by an empty "Disallow:" line (or "Allow: /"). Both tell every crawler it may access the whole site.
Write "User-agent: *" then "Disallow: /folder-name/". Everything not listed under Disallow stays allowed by default.
It must be at the root of the host, e.g. https://example.com/robots.txt. A subdomain such as blog.example.com needs its own robots.txt file.
Found a bug or have an idea?
ToolOrbit is actively developed — feedback directly shapes what gets built next.
Send feedback