Enter your domain to fetch its robots.txt, see all crawl rules, and test whether Googlebot is allowed or blocked from any URL path. The single file that controls what Google can see.
Robots.txt existence
Confirms the file is reachable — a 404 means all crawlers have unrestricted access
User-agent groups
Shows all bot-specific rules (Googlebot, GPTBot, *) and their Allow/Disallow directives
URL path testing
Enter any path (e.g. /blog/) to instantly check if it's blocked for Googlebot
Sitemap directives
Extracts all Sitemap: lines — this is how Google discovers your sitemap
Crawl-delay
Flags crawl-delay directives (Google ignores them, but other bots don't)
Full block detection
Warns if Disallow: / is set for all bots — a common staging mistake that goes live
Indexa monitors your robots.txt and alerts you the moment a rule change could affect crawlability — before it costs you traffic.
Try Indexa free