A robots.txt file is the first file search engine bots look for when visiting your website. It acts as a traffic control guide, instructing crawlers like Googlebot, Bingbot, and AI scrapers which parts of your website they are permitted to crawl and which directories they should avoid.
A single misplaced slash or wildcard in your robots.txt can accidentally de-index an entire website or waste your server's crawl budget. Here is how robots.txt rules work, along with 10 common real-world examples.
The core robots.txt directives
- User-agent: Identifies the bot the rule applies to (* means all bots, Googlebot specifies Google).
- Disallow: Specifies a path crawlers should not fetch (e.g., /admin/).
- Allow: Explicitly allows access to a sub-path within an otherwise disallowed parent directory.
- Sitemap: Provides the full absolute URL of your XML sitemap.
- Crawl-delay: Requests a delay (in seconds) between requests (honored by Bing/Yandex, ignored by Google).
10 Common robots.txt rule examples
1. Allow all crawlers complete access
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
2. Block all bots from entire site (Staging / Dev)
User-agent: * Disallow: /
3. Block specific private directories (Admin / Cart / Checkout)
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /checkout/ Disallow: /api/
4. Block a specific file
User-agent: * Disallow: /config/database.json
5. Allow one file inside a blocked folder
User-agent: * Disallow: /assets/ Allow: /assets/logo.png
6. Block specific search query strings
User-agent: * Disallow: /*?q= Disallow: /*?search=
7. Block a specific file extension (e.g., PDFs)
User-agent: * Disallow: /*.pdf$
8. Block specific AI scrapers (GPTBot, CCBot)
User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: /
9. Set rules specific to Googlebot only
User-agent: Googlebot Disallow: /temporary-promo/
10. Complete standard production setup
User-agent: * Disallow: /admin/ Disallow: /tmp/ Disallow: /private/ Sitemap: https://example.com/sitemap.xml
Frequently asked questions
Robots.txt must be placed in the top-level root directory of your domain (e.g., https://example.com/robots.txt). Search engine crawlers will not check subdirectories.
No. Robots.txt manages crawling, not indexing. If external links point to a disallowed URL, Google may still index the URL. To prevent indexing, use a <meta name="robots" content="noindex"> tag instead.
Use: User-agent: *\nDisallow: /admin/
Use ToolOrbit's free Robots.txt Generator to select crawlers, specify paths, attach your sitemap, and download a valid robots.txt file in seconds.
