#542 · Developer Tool

Robots.txt Rule Tester

Test one URL against the Allow and Disallow records in a robots.txt file. Enter a crawler token such as Googlebot and an absolute URL or path. The tester selects the most specific matching user-agent group, compares applicable path patterns, and applies the longest-match rule. It supports the common * wildcard and trailing $ anchor, then shows the winning rule and the candidate count. No request is sent to the target site.

Developer Input

Web metadata input
Ad space

How to use this developer tool

  1. Paste the requested source text or load a local file.
  2. Set any comparison values or processing options shown below the input.
  3. Select the primary action or press Ctrl/Cmd + Enter.
  4. Review the diagnostics, then copy or download the output.

What this developer tool does

The decision explains which crawler group and path rule won for the supplied URL.

A specific matching user-agent group takes precedence over the wildcard group. Among its rules, the longest matching pattern wins; Allow wins a tie.

Crawler implementations may differ on non-standard directives, percent encoding, and Unicode normalization.

Example

Input

User-agent: *
Disallow: /private/
Allow: /private/public/

User-agent: Googlebot
Disallow: /tmp/

Run the sample to see the exact report and metrics produced by the current implementation.

Use cases

  • Pre-deployment review of generated site files
  • Debugging crawler or link-preview configuration
  • Auditing metadata during a migration
  • Creating repeatable QA evidence for a release

Tips for reliable output

  • Paste the complete source when grouping or document context matters.
  • Use absolute public URLs unless the field explicitly accepts a path.
  • Retest after redirects, route rules, or locale mappings change.
  • Keep a deliberately invalid sample for regression checks.
  • Confirm the deployed result with the relevant external platform.

Processing details

A specific matching user-agent group takes precedence over the wildcard group. Among its rules, the longest matching pattern wins; Allow wins a tie. The parser runs entirely in the current browser tab and returns copy-ready text plus structured exports when useful.

Crawler implementations may differ on non-standard directives, percent encoding, and Unicode normalization.

Frequently asked questions

Can I use this tool without uploading data?

Yes. Processing happens in your browser, and the page makes no request with the supplied input.

What happens when the input is malformed?

The Robots.txt Rule Tester stops, displays an actionable error, and does not produce a misleading successful result.

Does the result guarantee search engine behavior?

No. The result checks the supplied markup or rules; crawlers and social platforms may apply additional policies and cached data.

Can I download the result?

Yes. Use Download for the primary output, or the CSV and JSON buttons when structured exports are available.

How should I test the result before deployment?

Run a representative valid sample and a deliberately invalid case, then verify the deployed public URL with the relevant crawler or platform tool.

Checks and output

AreaBehavior
InputProcessed locally
ErrorsReported without silent correction
ExportCopy, download, and structured data when available

Web Meta & Server Files

Inspect crawler directives, sitemap files, canonical URLs, localized alternates, and social metadata.

View category hub