AI Crawler Check: robots.txt and llms.txt

Description

See which AI crawlers a site allows or blocks. Checks robots.txt and llms.txt for ChatGPT, Claude, Perplexity and Gemini. Most sites block AI crawlers by accident. A robots.txt written in 2019 does not know what GPTBot is. A default CMS rule can shut out the engines your buyers now ask for recommendations. It will not show up in Search Console, and it will not show up in your analytics. AI Crawler Check reads any site's robots.txt and llms.txt and tells you in one click which AI crawlers get in. It checks 16 crawler user agents from the major AI providers, covering their search crawlers, training crawlers and user-triggered fetchers. The full list of what it checks is on the tool page linked below. It also checks whether the site publishes an llms.txt file, and shows you what is in it. How it works: click the icon on any page. The extension reads robots.txt and llms.txt from that domain, parses the rules the way a crawler would, and shows allowed, blocked, or partly blocked for each bot. Everything runs in your browser. No account, no tracking, nothing sent anywhere. Built by GOJI, a digital agency in Melbourne. We publish the Australian AI Visibility Index, which measures how often businesses appear in AI generated answers. This is one of the tools we built for that work.

Details & SEO