Robots.txt Checker
Find out which AI crawlers your robots.txt is blocking. Inspect every User-agent group, see matched rules with line numbers, and test if a specific URL is allowed or blocked.
/robots.txt from the domain.
View raw robots.txt
How to Use This Tool
Enter a URL or paste robots.txt
Paste any URL from a site (even the homepage) and we will fetch its robots.txt automatically. Or paste your robots.txt content directly.
Review AI crawler access
See which AI crawlers, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more, are blocked on your site with the matched rule and line number.
Test specific URLs
Drop in any URL on the site to see if it is allowed or blocked, which User-agent group was matched, and exactly which Allow/Disallow line applied.
Related tools
Robots.txt Generator
Create a robots.txt with rules for Googlebot, Bingbot and AI crawlers.
XML Sitemap Generator
Turn a list of URLs into a valid, upload-ready sitemap.xml.
Schema Markup Generator
Generate JSON-LD for Organization, LocalBusiness, FAQ, Article and Product.
Rather have experts fix it? We do this daily.
Our team handles technical SEO, content and AI-search visibility for US businesses. Start with a free audit of your site.