robots.txt is the site-level file that tells crawlers what to access and what to ignore. When written correctly, it helps search engines focus on your important pages and avoid wasting crawl budget on private or irrelevant sections.
User-agent rules explained
Each section starts with one or more User-agent directives. Use "*" to target all crawlers, or a specific bot name to limit rules to a single crawler.
Allow and Disallow
- Disallow blocks a path from being crawled.
- Allow explicitly permits a path under a broader disallow rule.
- Paths should start with / and are case-sensitive on most servers.
- Do not use robots.txt to hide sensitive data; use authentication or noindex instead.
User-agent: * Disallow: /admin Allow: /admin/public Sitemap: https://example.com/sitemap.xml
Add your sitemap
Including a Sitemap directive helps crawlers discover your important URLs. It is especially useful for large sites, new sites, or pages that are not well-linked internally.
Validation matters
A malformed robots.txt can confuse crawlers and cause them to ignore the whole file. Use a generator to ensure the syntax is correct.
ToolOrbit’s Robots.txt Generator lets you build and validate the file with no guesswork.
