Tool

robots.txt validator and URL tester

Check a robots.txt against RFC 9309 and documented crawler behaviour, then test a URL and see which rule matches it — for Googlebot and other crawler tokens, side by side.

Syntax findings and configuration warnings are listed separately, with evidence and suggested fixes. A matched rule describes robots.txt permission, not proof of a crawler’s actual visit or removal from search. Guidance beside each blocked token explains distinctions such as search, training and user-requested visits. How this tool decides.

Preparing editor…

Load a robots.txt

Fetch a live file from any host, upload one, or type straight into the editor below. Everything updates as you type.

No site context set. File syntax and URL-path tests use the current editor text; sitemap host comparisons are skipped. Changing this address does not fetch a file until you choose Fetch /robots.txt.

robots.txt

6 lines · 0.1 KB · 1 group · 2 rules

Test a URL

Enter a path, or a full URL on the exact site origin above. Each crawler below is matched against the text currently in the editor, and the exact rule responsible is named — not just the verdict.

URL or path

Matched against: /admin/secret

BLOCKED16 crawlers share this result
Disallow: /admin/ — in group User-agent: *
- GooglebotGoogle
  Blocking this prevents Googlebot from crawling the page. The URL can still appear in Google Search.
- Googlebot-ImageGoogle
  Controls image crawling for Google Images, Discover, Google Video and the Search features that show images, logos and favicons.
- BingbotMicrosoft
  Blocking this prevents Bingbot from crawling the page. For removal from Bing, use noindex on a crawlable page.
- DuckDuckBotDuckDuckGo
  Blocking this affects DuckDuckGo’s own crawl.
- ApplebotApple
  Blocking this restricts Apple’s search crawl for Spotlight, Siri and Safari. Without an Applebot group, Applebot follows the rules for Googlebot.
- GPTBotOpenAI
  Disallowing this signals that content should not be used to train OpenAI’s foundation models. Search access is separate.
- OAI-SearchBotOpenAI
  Disallowing this opts the content out of ChatGPT search answers; navigational links can still appear.
- ChatGPT-UserOpenAIMay ignore robots.txt
  Handles visits requested by ChatGPT users. OpenAI says robots.txt rules may not apply to these requests.
- ClaudeBotAnthropic
  Disallowing this opts future content out of Anthropic’s training crawl. Search and user requests use separate agents.
- Claude-SearchBotAnthropic
  Disallowing this stops Anthropic indexing the content to improve Claude’s search results. Training uses ClaudeBot.
- Claude-UserAnthropic
  Fetches pages when Claude users ask about them. Disallowing this stops Claude retrieving the content for those requests.
- Google-ExtendedGoogle
  Controls use for Gemini training and grounding in Gemini Apps and Vertex AI. It does not affect Google Search inclusion or ranking.
- PerplexityBotPerplexity
  Controls Perplexity’s search crawl. Blocking it does not guarantee exclusion from answers fetched at a user’s request.
- Perplexity-UserPerplexityMay ignore robots.txt
  Fetches pages when Perplexity users ask about them. Perplexity says this fetcher generally ignores robots.txt, so a block may not apply.
- CCBotCommon Crawl
  Blocking this tells Common Crawl’s crawler not to fetch the page. Common Crawl notes that other crawlers sometimes pretend to be CCBot.
- Applebot-ExtendedApple
  Controls use of content collected by Applebot for Apple’s model training. It does not control crawling or search inclusion.

Findings

Findings update from the current text and site context. Errors, configuration warnings and informational notes are listed separately.

0 critical
0 warnings
0 informational

Counts describe the complete text currently in the editor.

Nothing to report.

No issues were detected in the current text. This does not verify live crawling, sitemap contents or indexing.

Automatic corrections

No automatic correction is needed. No errors or warnings with a safe automatic fix were found. Informational notes may still need review. Your file is unchanged.