Last Updated on October 20, 2026
Most robots.txt problems come from a short list of mistakes: a leftover Disallow: / that blocks the whole site, using Disallow to try to remove pages from Google, unsupported noindex or crawl-delay lines, blocked CSS and JavaScript, a file in the wrong place, and server errors on /robots.txt. Each issue below has its cause and a fix you can copy.
Everything here is checked against Google’s robots.txt introduction, Google’s robots.txt specification and RFC 9309, the IETF standard for the Robots Exclusion Protocol published in September 2022. Where Google and other crawlers behave differently, we say so.
How does robots.txt Disallow work?
A robots.txt file is a plain text file at the root of a host. It is made of groups: one or more User-agent lines naming the crawler, followed by Allow and Disallow rules that match the start of URL paths. Google supports four fields: user-agent, allow, disallow and sitemap (Google).
# Keep all crawlers out of the cart and internal search results
User-agent: *
Disallow: /cart/
Disallow: /search
Sitemap: https://www.example.com/sitemap.xmlThree things to remember before you troubleshoot anything:
- Disallow controls crawling, not indexing. Google says robots.txt “is not a mechanism for keeping a web page out of Google” (Google).
- Rules are prefix matches on the path.
Disallow: /searchalso blocks/search-tips/and/searchable.html. - Compliance is voluntary. Reputable crawlers obey it; others may not. RFC 9309 states the rules “are not a form of access authorization” (RFC 9309).
1. Disallow: / is blocking your entire site
Symptom: traffic falls off a cliff, Search Console’s page indexing report fills with “Blocked by robots.txt”, and new pages stop appearing. The usual cause is a staging or pre-launch robots.txt that went live with the site.
# This blocks every compliant crawler from every URL
User-agent: *
Disallow: /Fix: replace it with a file that allows crawling. Either of these allows everything (an empty Disallow matches nothing):
User-agent: *
Allow: /User-agent: *
Disallow:Then open the robots.txt report in Search Console and use Request a recrawl so Google picks up the change sooner than its normal cache refresh (Google). To keep staging sites private, use password protection on the staging host instead of relying on robots.txt.
2. Using Disallow to remove pages from Google
Symptom: you disallowed a section months ago and the URLs still show in search, usually with no description. Search Console lists them as “Indexed, though blocked by robots.txt”.
Cause: a blocked page can still be indexed if other pages link to it; Google just can’t read its content (Google, page indexing report). Worse, Disallow hides your noindex tag from Google: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file” (Google).
Fix: remove the Disallow rule for those URLs, then add noindex in one of these two forms and let Google recrawl:
<meta name="robots" content="noindex">X-Robots-Tag: noindexOnce the pages have dropped out, you can disallow them again if crawl volume matters. For pages that no longer exist, a 404 or 410 status also removes them once crawled. For content that must stay private, use a login, not robots.txt: the file is public, so listing /admin-backup/ in it tells everyone where to look.
3. noindex, nofollow or crawl-delay lines in robots.txt
Symptom: pages you “noindexed” in robots.txt are still indexed, or Googlebot ignores the crawl delay you set.
Cause: Google retired all code handling unsupported robots.txt rules, including noindex, on September 1, 2019 (Google Search Central blog). Its current spec names crawl-delay as unsupported (Google).
# Google ignores both of these lines
User-agent: *
Noindex: /old-landing-pages/
Crawl-delay: 10Fix: move indexing control to the meta robots tag or X-Robots-Tag header (issue 2). If Googlebot is overloading your server, Google’s advice is to return 500, 503 or 429 for a short period (hours, at most 1 to 2 days) or file a request through its Googlebot report form (Google). Some other crawlers do read Crawl-delay; Anthropic, for example, says ClaudeBot supports it (Anthropic). Keep it only in groups for crawlers that document support.
4. Blocking CSS, JavaScript or images Google needs to render pages
Symptom: the URL Inspection tool’s rendered screenshot looks broken, content loaded by JavaScript is missing from Google’s view, or mobile issues appear that real users don’t see.
Cause: rules like Disallow: /wp-includes/, Disallow: /assets/ or Disallow: /*.js$. Google says you can block unimportant resources, but not ones that Google needs to understand the page (Google).
Fix: delete those rules. If you must keep a directory blocked, add a longer Allow for the files inside it (longer rules win, see issue 8):
User-agent: *
Disallow: /app/
Allow: /app/*.css$
Allow: /app/*.js$5. The file is in the wrong place, or covers the wrong host
Symptom: your rules have no effect at all, or work on one version of the site but not another.
Cause: the file must be named robots.txt (lowercase), and there is one per host, at the root. Crawlers do not look for it in subdirectories, so /blog/robots.txt is ignored. The rules apply only to the exact protocol, host and port that serve the file: https://www.example.com/robots.txt does not cover https://shop.example.com/ or http://www.example.com/ (Google, Google).
Fix: publish one file at the root of every host you want to control, including subdomains, and merge any rules you had scattered elsewhere. If your blog lives in a subfolder (as ours does at /blog/), its rules go in the root file:
User-agent: *
Disallow: /blog/wp-admin/
Allow: /blog/wp-admin/admin-ajax.php6. robots.txt returns a server error, a 404 or a redirect chain
How Google reacts depends on the HTTP status of /robots.txt itself (Google):
| Status of /robots.txt | What Google does |
|---|---|
| 2xx | Uses the file as served. |
| 3xx | Follows at least five redirect hops, then treats it as a 404. JavaScript and meta refresh redirects are not followed. |
| 4xx (except 429) | Acts as if there is no robots.txt: everything may be crawled. |
| 5xx, timeouts, DNS errors | Stops crawling the site for the first 12 hours while retrying, then uses the last cached copy for up to 30 days. After 30 days it treats the site as having no robots.txt if the site is otherwise available. |
The trap: a firewall, CDN rule or broken PHP handler that makes /robots.txt return 500 or 503 can halt crawling of the whole site. A 404 is harmless only if you didn’t need any rules; a 401 or 403 also counts as “no restrictions”, so it is not a way to slow Google down.
Fix: make sure /robots.txt returns 200 with Content-Type: text/plain, directly on the canonical host. Check it from the command line:
curl -sI https://www.example.com/robots.txtThen check “Fetch status” in the robots.txt report, which shows the files Google found for your top 20 hosts, when each was last checked and any parsing issues.
7. Encoding, formatting or file size problems
Cause: files saved from a word processor, served as HTML, or so large that rules get cut off. Google requires a UTF-8 plain text file, ignores invalid lines, and enforces a 500 KiB limit: anything after that is ignored (Google). RFC 9309 sets 500 KiB as the minimum crawlers should parse, so other bots may also stop there (RFC 9309).
Fix: edit in a plain text editor, save as UTF-8, put each rule on its own line, and start each group with its User-agent line. If the file is approaching 500 KiB, it almost always lists individual URLs that a single wildcard pattern could cover:
# Instead of hundreds of lines like Disallow: /products?sort=price
User-agent: *
Disallow: /*?sort=
Disallow: /*&sort=8. Misreading which rule wins when Allow and Disallow conflict
Cause: assuming the first matching line wins, as in some firewall configs. Order in the file does not matter. For Google and under RFC 9309, the most specific rule wins, meaning the longest matching path. If an Allow and a Disallow match equally, the least restrictive one (Allow) wins (Google, RFC 9309).
User-agent: *
Disallow: /blog/
Allow: /blog/guides/With this file, /blog/guides/robots-txt/ is allowed (the Allow path is longer) and /blog/news/ is blocked. Put the line order however reads best; the result is the same.
Fix: when a URL is blocked unexpectedly, list every rule in the applicable group that matches its path, then pick the longest. Google’s open-source parser, linked from the robots.txt report help page, lets developers test this locally.
9. Adding a Googlebot group and losing your general rules
Symptom: you add a group for one crawler and it starts crawling sections that the * group blocks.
Cause: a crawler obeys only one group: the one with the most specific matching user-agent. A specific group and the * group are not combined (Google).
User-agent: *
Disallow: /cart/
Disallow: /search
# Googlebot now ignores everything above and only blocks /staging/
User-agent: Googlebot
Disallow: /staging/Fix: repeat the shared rules in every named group:
User-agent: *
Disallow: /cart/
Disallow: /search
User-agent: Googlebot
Disallow: /cart/
Disallow: /search
Disallow: /staging/Also note that Google’s AdsBot crawlers ignore the * group and must be named explicitly (Google).
10. Wildcards and prefix matches that block too much (or too little)
Google supports two special characters: * matches zero or more of any character, and $ marks the end of the URL. A trailing * does nothing, because every rule is already a prefix match (Google).
| Rule | Blocks | Does not block |
|---|---|---|
Disallow: /private |
/private/, /private-offers.html, /privatelabel/ | /Private/, /about/private/ |
Disallow: /private/ |
/private/ and everything under it | /private, /private-offers.html |
Disallow: /*.pdf$ |
/files/guide.pdf | /files/guide.pdf?v=2 |
Disallow: /*? |
any URL with a query string | /shoes/ |
Disallow: /private* |
same as /private (trailing * is ignored) |
/Private/ |
Fix: end directory rules with a slash, add $ only when you mean “ends exactly here”, and test a sample of real URLs from your logs or crawl before deploying. Disallow: /*? is a common culprit: it also blocks paginated or filtered pages you may want crawled.
11. Case mismatches in paths
Field names are not case-sensitive (disallow and Disallow both work), but path values are. Google’s example: /fish does not match /Fish.asp (Google). User-agent names are matched case-insensitively under RFC 9309 (RFC 9309).
Fix: copy paths exactly as they appear in your URLs. If your CMS serves both cases, either add a rule for each or, better, redirect to one lowercase version:
User-agent: *
Disallow: /Downloads/
Disallow: /downloads/12. A missing or broken Sitemap line
Cause: relative sitemap paths, the wrong protocol or host, or a sitemap URL that redirects or returns 404. Google requires a fully qualified URL. You can list several sitemaps, they may sit on a different host, and the line applies to all crawlers regardless of where it appears in the file (Google).
# Wrong: relative path
Sitemap: /sitemap.xml
# Right
Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/blog/sitemap_index.xmlFix: use the canonical protocol and host, confirm each sitemap URL returns 200, and also submit it in Search Console’s Sitemaps report so you can see errors. For a wider check of crawl and index settings, see our technical SEO guide.
How do you block AI crawlers in robots.txt?
Each AI company publishes its own user-agent tokens, and they control different things. Blocking a training crawler does not necessarily remove you from that company’s AI search answers, and vice versa. Here is what each vendor documents as of October 2026:
| Token | Operator | What blocking it does |
|---|---|---|
GPTBot |
OpenAI | Signals your content should not be used to train OpenAI’s foundation models (OpenAI). |
OAI-SearchBot |
OpenAI | Opted-out sites are not shown in ChatGPT search answers, though they can still appear as navigational links (OpenAI). |
ChatGPT-User |
OpenAI | User-initiated visits; OpenAI says robots.txt rules may not apply (OpenAI). |
ClaudeBot |
Anthropic | Excludes your future content from Anthropic’s model training data (Anthropic). |
Claude-SearchBot, Claude-User |
Anthropic | Search indexing and user-requested fetches; blocking may reduce your visibility in Claude’s answers (Anthropic). |
PerplexityBot |
Perplexity | Removes you from Perplexity search results; Perplexity says it is not used to crawl for training foundation models (Perplexity). |
Perplexity-User |
Perplexity | User-requested fetches that Perplexity says generally ignore robots.txt (Perplexity). |
Google-Extended |
Controls use of your content for training future Gemini models and for grounding in Gemini Apps and Vertex AI. It does not affect inclusion or ranking in Google Search, and has no separate user-agent string in your logs (Google). | |
CCBot |
Common Crawl | Stops Common Crawl from adding your pages to its open web archive (Common Crawl). |
A common setup for a business that wants to appear in AI search answers but opt out of model training:
# Opt out of AI model training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
# Keep AI search crawlers and search engines allowed
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: *
Disallow: /cart/
Sitemap: https://www.example.com/sitemap.xmlTwo mistakes we see often: blocking Googlebot in an attempt to stay out of Gemini (that removes you from Google Search; Google-Extended is the right token), and blocking OAI-SearchBot or PerplexityBot while also trying to win citations in those tools. If AI visibility is a goal, our generative engine optimization services start with exactly this file. Also note that the user-triggered fetchers above may not obey robots.txt at all, so blocking them there is not reliable.
How do you test and refresh robots.txt changes?
The old robots.txt Tester in Search Console has been retired. Google now points to the robots.txt report for file status and to the URL Inspection tool for checking whether a specific URL is blocked (Google). Use this checklist after every change:
- Load
https://yourdomain.com/robots.txtin a browser and withcurl -sI: confirm a200status and plain text content on every host and subdomain. - Open the robots.txt report (Search Console, Settings) and check Fetch status and Issues.
- Click Request a recrawl. Otherwise, Google generally caches robots.txt for up to 24 hours, longer if it can’t refresh the file (Google).
- Run important URLs through the URL Inspection tool to confirm they are not blocked and render with their CSS and JavaScript.
- Watch the page indexing report over the following weeks for “Blocked by robots.txt” and “Indexed, though blocked by robots.txt” (Google).
Robots.txt mistakes are among the first things we check in an SEO audit, because one line can undo everything else on the site.
FAQ
What does “Disallow: /” mean in robots.txt?
Under the user-agent it follows, Disallow: / blocks crawling of every URL on that host, because every path starts with a slash. With User-agent: *, it tells all compliant crawlers to stay out of the whole site. It does not remove pages already in Google’s index; for that, allow crawling and use noindex.
Does robots.txt Disallow stop a page from being indexed?
No. Disallow stops crawling, but Google can still index a blocked URL if other pages link to it, usually showing it without a description. To keep a page out of search results, allow crawling and add a noindex meta tag or X-Robots-Tag header, or put the page behind a login.
Which wins in robots.txt, Allow or Disallow?
For Google and under RFC 9309, the rule with the longest matching path wins, regardless of the order of lines in the file. If an Allow and a Disallow rule match with equal length, Allow wins because it is the least restrictive. Allow: /blog/guides/ therefore overrides Disallow: /blog/ for URLs inside the guides folder.
How long does Google take to see robots.txt changes?
Google generally caches robots.txt for up to 24 hours, and longer if it cannot fetch a fresh copy because of timeouts or server errors. To speed things up, use Request a recrawl in Search Console’s robots.txt report. Newly unblocked pages still need to be recrawled before they show up normally in search.
Does blocking Google-Extended remove my site from Google Search?
No. Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. It controls whether content Google crawls can be used to train future Gemini models and to ground answers in Gemini Apps and Vertex AI. Blocking Googlebot, by contrast, removes you from Search.
