robots.txt Tester
Fetch and parse any site's robots.txt — see the rules, sitemaps, and test whether a path is crawlable. Free, no signup.
One stray Disallow can hide a whole section of your site from search. This tester reads the live robots.txt and splits it into the groups and rules crawlers actually follow. Type in a path and it tells you whether that page is reachable, so the pages you want indexed stay reachable and the ones you don’t stay blocked.
How it works
- 01
Enter a site URL
Paste any URL on the site — Sitewell reads the robots.txt from that origin's root for you.
- 02
Read the rules
See each user-agent group with its Allow/Disallow rules and every sitemap the file declares.
- 03
Test a path
Type a path and a crawler name to check whether that page is crawlable under the current rules.
Frequently asked questions
- What does the robots.txt tester check?
- It fetches the robots.txt file at the root of the domain, then breaks it down into its user-agent groups, the Allow and Disallow rules under each, and any Sitemap lines. You can also type a path and a crawler name to see whether that page would be crawled.
- How does the path checker decide allowed or blocked?
- It follows the same longest-match rule Google uses: the rule with the most specific (longest) matching path wins, and when an Allow and a Disallow tie, the Allow takes priority. A path with no matching rule is treated as crawlable.
- Does blocking a page in robots.txt remove it from Google?
- No. robots.txt stops compliant crawlers from fetching a page, but a blocked URL can still appear in results if other pages link to it. To keep a page out of the index, use a noindex meta tag or header and leave it crawlable so Google can read that directive.
- Where should robots.txt live?
- It has to sit at the root of the host — example.com/robots.txt. A robots.txt in a subfolder is ignored, and each subdomain needs its own file. This tester always reads the file from the origin root for you.
Related tools
DNS Lookup
Look up A, AAAA, MX, TXT, NS, CNAME and SOA records.
Meta Tag Checker
Audit titles, descriptions and Open Graph tags.
HTTP Header Checker
Inspect response and security headers for any URL.
HTTP Status Checker
Check status codes for many URLs at once.
Redirect Checker
Trace 301/302 redirect chains and catch loops.
Broken Link Checker
Find dead links and 404s on any page.
SSL & Domain Expiry Checker
Check SSL certificate and domain expiry dates.
Website Audit
One-URL health score across SSL, headers, meta and more.
Core Web Vitals Checker
Measure LCP, CLS and INP via Google PageSpeed.