Media
2 items
Description
Audit which paths a site disallows in robots.txt and how many of them Google has indexed anyway. This extension fetches the site's robots.txt, extracts the Disallow rules that apply to Googlebot, converts each one into a Google site: query, and shows how many URLs Google keeps in their index under each disallowed pattern. Because: A site can ask Googlebot not to crawl certain paths, but those URLs can still end up in the index. This tool makes it easy to spot where that's happening at scale. Click the icon on any site and the extension: 1. Fetches the site's robots.txt 2. Extracts every Disallow rule that applies to Googlebot 3. Builds a Google site: query for each rule 4. Reports how many URLs Google still has indexed under each disallowed pattern A site can ask Googlebot not to crawl certain paths, but those URLs can still end up in the index. This tool makes it easy to spot where that's happening at scale. FEATURES • Handles every robots.txt pattern: plain prefixes, wildcards, end-of-URL anchors, filetype suffixes, query-string patterns, URL-encoded UTF-8 paths, and Allow exceptions • Tokenization and grouping match Google's open-source robots.txt parser • Allow rules carve out exceptions from Disallows via combined Google queries • 7-day local cache — re-running an audit is instant for cached rules • Sortable table, copy-as-TSV / Markdown (e.g., for client deliverables) • Google CAPTCHA recovery flow for large audits PRIVACY No data collection. No telemetry. No remote logging. The extension only fetches the public robots.txt and runs Google search queries through your normal browser session. All run state and cached counts live on your device only.