Media
2 items
Description
Audits robots.txt to verify if AI, LLM, and Search Engine bots are whitelisted or blocked. # AI Bot Inspector 🤖 A premium Chrome Extension designed for web developers, SEO specialists, and site operators to audit `robots.txt` compliance against 40+ top AI bots, LLM training scrapers, search engines, and RAG agents. Ensure your content-crawling policies match your intent with a single click. --- ## 🚀 Key Features * **⚡ Automatic Single-Click Audits**: Grabs the active tab's domain, retrieves `/robots.txt`, and parses it instantly without console copy-pasting. * **🔬 Multi-Agent Compliance Engine**: Vets rules for 40+ crawlers, including OpenAI , Google , , Meta, Apple, and Common Crawl. * **📊 Traffic Light Classification**: Categorizes compliance levels into **Errors** (explicitly blocked 🔴), **Warnings** (implicit fallback allowed 🟡), and **Passes** (explicitly whitelisted 🟢). * **🏗️ Interactive Sandbox Mode**: Paste, test, and live-debug custom `robots.txt` text blocks before deploying them to production. * **🔍 Search & Group Filters**: Instantly find specific bots or filter listings by severity category and purpose. * **📥 CSV Audit Exporter**: Download standard audit spreadsheets for reporting and compliance tracking. --- ## 🛠️ Installation & Setup 1. Clone this directory or extract the zip archive. 2. Open Google Chrome and navigate to `chrome://extensions/`. 3. Toggle **Developer mode** (top-right corner) to **ON**. 4. Click **Load unpacked** (top-left corner). 5. Select this project directory (`ai-bot-inspector`). --- ## 📜 Supported Crawlers We audit classifications across: * **Search Engines** * **AI Model Training** * **AI User Fetches / RAG**