A robots.txt file is a plain text file at the root of a website that tells crawlers which URLs they may and may not request. It controls crawling, not indexing, and that one distinction is behind most of the robots.txt mistakes that cause SEO problems.
This guide covers the syntax, how Google decides which rule wins, how to handle AI crawlers, and how Google behaves when your robots.txt is missing or broken.
If you want to check a file as you read, use my free robots.txt tester and validator, or build one from scratch with the robots.txt generator.
What robots.txt does (and what it doesn't)
When a well-behaved crawler such as Googlebot visits your site, it fetches /robots.txt first and checks whether it is allowed to request a URL before requesting it. A Disallow rule means "don't fetch this." It does not mean "don't show this in search results."
If a blocked URL has links pointing to it, Google can still index it without ever seeing its content. It then appears in Search Console as "Indexed, though blocked by robots.txt", usually with no description in the search result. To keep a page out of the index, allow crawling and use a noindex meta tag or an X-Robots-Tag HTTP header instead.
robots.txt is also not a security measure. The file is public, so listing private paths in it actually advertises them, and bad bots simply ignore it. Protect anything sensitive with authentication.
Where the robots.txt file lives
robots.txt must sit at the root of the host it applies to, for example https://www.example.com/robots.txt. A file in a subfolder is ignored. Each combination of protocol, host and port is treated separately, so blog.example.com and shop.example.com each need their own file, and the rules on www.example.com do not cover them.
The file should be UTF-8 plain text. Google reads only the first 500 KiB, so on very large sites anything past that limit is ignored. Keep the file lean by consolidating rules with wildcards instead of listing hundreds of paths.
robots.txt syntax, with an example
A robots.txt file is made of groups. Each group starts with one or more User-agent lines, followed by the Allow and Disallow rules for those crawlers. Here is a typical file for a WordPress site:
# Rules for all crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/
Sitemap: https://www.example.com/sitemap.xmlUser-agent
Names the crawler the group applies to, such as Googlebot, Bingbot or GPTBot. An asterisk (*) matches any crawler that doesn't have a more specific group of its own.
Disallow
A path the crawler should not request. Rules match from the start of the URL path, so Disallow: /wp-admin/ blocks everything under that folder. An empty Disallow: line blocks nothing, while Disallow: / blocks the entire site.
Allow
Makes an exception to a broader Disallow. In the example, admin-ajax.php stays crawlable because many WordPress themes and plugins load front-end content through it.
Sitemap
Points crawlers to your XML sitemap using an absolute URL. It isn't tied to any user-agent group, so it can go anywhere in the file, and you can list several. See my sitemap best practices for what to put in the sitemap itself.
Comments
Anything after a # is ignored by crawlers. Use comments to note why a rule exists and who added it, because nobody remembers why /tmp-2019/ was blocked three years later.
Crawl-delay (Google ignores it)
Crawl-delay asks a crawler to wait a number of seconds between requests. Bing and some other crawlers respect it, but Googlebot does not. If Google is overloading your server, it slows down automatically when it sees 500, 503 or 429 responses.
How Google decides which rule applies
Most robots.txt bugs come from misunderstanding precedence. Three rules explain almost everything.
1. A crawler follows only its most specific group
If a group names Googlebot, Googlebot follows that group and ignores the * group completely. Rules are not merged. In the file below, Googlebot is free to crawl /search/:
User-agent: *
Disallow: /search/
User-agent: Googlebot
Disallow: /private/If you create a crawler-specific group, repeat every rule you still want that crawler to follow.
2. The longest matching rule wins
Within a group, order doesn't matter. Google picks the rule with the longest matching path, and if an Allow and a Disallow match with equal length, Allow wins. Here, /blog/robots-txt-guide/ can be crawled even though /blog/ is blocked:
User-agent: *
Disallow: /blog/
Allow: /blog/robots-txt-guide/3. Paths are case-sensitive; directive names are not
User-agent, user-agent and USER-AGENT all work. But Disallow: /Admin/ does not block /admin/, because URL paths are case-sensitive.
4. Wildcards: * and $
Google and Bing support two special characters. An asterisk matches any sequence of characters, and a dollar sign marks the end of the URL:
User-agent: *
# Block any URL containing a sort parameter
Disallow: /*?sort=
# Block URLs ending in .pdf
Disallow: /*.pdf$When to use robots.txt, and when not to
robots.txt is the right tool when you want to stop crawlers spending time on URLs that have no search value:
- Internal search results. Search pages can generate endless URL combinations and are low-quality landing pages.
- Faceted navigation and parameters. Filter and sort combinations on e-commerce sites are a common crawl budget drain.
- Infinite spaces. Calendars, session IDs and other URL patterns that never end.
- Crawlers you don't want. For example, AI training crawlers (covered below).
It is the wrong tool for these jobs:
- Duplicate content. If Google can't crawl a duplicate, it can't see the canonical tag on it. Use rel="canonical" instead.
- Removing pages from Google. Use noindex, and leave the page crawlable until it drops out.
- Keeping PDFs or images out of search. Send an X-Robots-Tag: noindex HTTP header on those files.
- Staging sites and private areas. Put them behind a password. A robots.txt block doesn't stop them being indexed through links, and it doesn't stop people.
How to block AI crawlers with robots.txt
AI companies publish user-agent tokens you can target in robots.txt. The important nuance is that several companies now separate training crawlers from search or retrieval crawlers, so blocking one does not block the other. The most common tokens are:
- GPTBot: OpenAI's crawler for model training. OAI-SearchBot is separate and powers ChatGPT search results.
- ClaudeBot: Anthropic's crawler for model training. Anthropic uses separate agents for search and user-requested fetches.
- Google-Extended: Controls whether your content is used for Gemini models. It is a control token rather than a separate crawler, and it does not affect Google Search rankings or AI Overviews, which rely on Googlebot.
- CCBot: Common Crawl, whose open dataset is widely used to train AI models.
- PerplexityBot: Indexes pages for Perplexity's answers.
You can list several user-agents in one group. This example blocks the training crawlers while leaving AI search crawlers alone, so you can still be cited in AI answers:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /Before you block, decide what you actually want. Blocking training crawlers is a content licensing choice. Blocking search and retrieval crawlers means you won't be cited in those AI tools at all. Crawler names also change, so check each company's documentation and revisit your rules every few months.
What happens if robots.txt is missing or broken
Google's behaviour depends on the HTTP status code the file returns:
- 2xx: Google reads and follows the rules.
- 4xx (such as 404): Google treats this as having no robots.txt and crawls everything. You don't need a robots.txt file if you have nothing to block.
- 5xx or 429: Google treats the whole site as disallowed and pauses crawling. If the error persists for about 30 days, it falls back to the last cached version, or crawls without restrictions if it has none. A robots.txt that keeps returning server errors can stall crawling across your entire site.
Google generally caches robots.txt for up to 24 hours, so changes aren't picked up instantly.
Common robots.txt mistakes
- Launching with the staging file. A Disallow: / carried over from staging blocks the whole site. Check robots.txt as part of every launch or migration checklist.
- Blocking pages that carry noindex. Google never sees a noindex tag on a page it isn't allowed to crawl, so the page can stay indexed indefinitely.
- Blocking CSS and JavaScript. Google needs these files to render pages. Blocking them can make pages look broken or empty to Googlebot.
- Forgetting group precedence. A new Googlebot-specific group quietly overrides every rule in the * group for Google.
- Assuming one file covers subdomains. Each host needs its own robots.txt.
- Missing trailing slashes or over-broad paths. Disallow: /blog blocks /blog/, but also /blog-news/ and /blogger.html. Use Disallow: /blog/ if you mean only the folder.
How to test your robots.txt
Google retired its standalone robots.txt Tester at the end of 2023. It was replaced by the robots.txt report in Search Console, which shows the version Google last fetched, when it fetched it, and any parsing errors.
To test whether a specific URL is blocked, paste your rules and the URL into my robots.txt tester, then use URL Inspection in Search Console to confirm how Google sees the live page. After you change the file, you can ask Google to recrawl your robots.txt in Search Console instead of waiting for the cache to expire.
For the formal specification, see RFC 9309, which standardised the Robots Exclusion Protocol in 2022 after nearly 30 years as an informal convention.
FAQs about robots.txt
Does blocking a page in robots.txt remove it from Google?
No. robots.txt stops crawling, not indexing. A blocked URL can still appear in search results if other pages link to it. To remove a page, allow crawling and add a noindex meta tag or X-Robots-Tag header, then wait for Google to recrawl it.
Can a website have more than one robots.txt file?
Each host has exactly one robots.txt file, at its root. But subdomains and protocols count as separate hosts, so www.example.com, blog.example.com and shop.example.com each have their own file.
Is robots.txt case-sensitive?
Directive names such as User-agent and Disallow are not case-sensitive. URL paths are, so Disallow: /Admin/ does not block /admin/.
How long does Google take to see robots.txt changes?
Google generally caches robots.txt for up to 24 hours. You can request a recrawl from the robots.txt report in Google Search Console to speed this up.
Do I need a robots.txt file?
Not necessarily. If the file returns a 404, Google crawls everything, which is fine for many small sites. You need one when there are URLs you want crawlers to skip, or when you want to point crawlers to your sitemap.
