Code · Made a little simpler

Robots TXT Rule Conflict Explainer

Explain which supplied robots.txt groups apply to a crawler product token and why a specific Allow or Disallow wins for a path. Review the other matched rules and line numbers.

Free to useNo account neededProcessed on your device
Your workspaceProcessed on your device

Uses the supplied rules for this path. URLs are parsed locally.

Add this tool to your website

Paste this code into your page to add the tool and a visible ToolMellow credit link.

HTML embed code
Preview embed

Robots TXT Rule Conflict Explainer: how to use it

  1. Paste robots.txt or open a file. Enter the crawler product token and a target path or HTTP URL.
  2. Select Explain rules. Review exact groups, wildcard fallback and the normalized comparison path, then compare matched rule lengths.
  3. Change the path or agent to see which group or rule wins. Download a text trace for review.

Limits and notes

Checks 500 KiB of UTF-8 text. Uses exact case-insensitive product-token matching and the shared path matcher. Compares paths and queries up to 2,047 characters, strips fragments and applies Allow on equal normalized lengths. Does not fetch a deployed file, check server responses or implement provider-specific aliases.

How your privacy works →

Robots TXT Rule Conflict Explainer: common questions

Why is the wildcard group ignored?

A product token with an exact match uses the matching groups. The wildcard group applies only when no exact product group applies.

What happens when Allow and Disallow have the same length?

For matched rules of equal normalized length, this model chooses Allow. The trace shows the other tied row and the deciding line.

Can the query string change the result?

Yes. A terminal dollar sign requires the entire compared path and query to match. Adding a query can make a rule ending in that sign stop matching.

Guides