docsntools

Robots.txt Tester

Paste a robots.txt and a list of URLs to see which are allowed or blocked for Googlebot, Bingbot, GPTBot, or any custom user-agent.

100% in your browser · no uploads · no ad trackers

About Robots.txt Tester

Robots.txt Tester parses a robots.txt file the same way Google's crawler does - longest-match wins, with Allow beating Disallow on ties - and shows you exactly which URLs are blocked or allowed for any user-agent. Use it to debug indexing problems, validate edits before deploying, and confirm AI crawler rules (GPTBot, ClaudeBot, PerplexityBot).

robots.txt

Test

Common: Googlebot, Bingbot, GPTBot, Applebot, * (any)

Results

Add a robots.txt and URLs to see allow / block decisions.

How to use

  1. 1.Paste your robots.txt content.
  2. 2.Enter a user-agent (e.g. Googlebot).
  3. 3.List URLs to test - see allow/block per rule.

FAQ

Which matching algorithm is used?
Google's spec: the most specific (longest) matching pattern wins, and Allow beats Disallow on ties. Wildcards `*` and end-of-path `$` are supported.
Can I test AI crawlers?
Yes - type any user-agent string (GPTBot, ClaudeBot, PerplexityBot, Bytespider, etc.) and rules with that token will apply.
Does it fetch live URLs?
No. Everything runs locally in your browser; nothing is requested or uploaded. Paste robots.txt content directly.
Is Robots.txt Tester free?
Yes. Every tool on docsntools is 100% free with no signup, no watermarks, and no upload limits.
Are my files uploaded to a server?
No. All processing happens entirely in your browser. Your files never leave your device.
Does it work on mobile?
Yes - every tool is responsive and works in modern mobile browsers (Chrome, Safari, Firefox).

Related tools