Media
5 items
Description
See which AI crawlers this site allows or refuses, read from its own robots.txt and meta tags. Which AI crawlers does this site allow? Click the icon on any website and MurmCrawl reads that site's own robots.txt, the robots directives on the page, and its response header, then tells you which named AI crawlers it refuses, which it names and allows, and which it never mentions at all. Silence is permission The third answer is the one most sites give and the one nobody reports. A crawler that is not named in a robots.txt is not blocked by it — the default is to allow, so an unmentioned crawler may read the whole site. MurmCrawl counts those separately from the ones a site deliberately let in, because "we allowed them" and "we never thought about them" are different decisions and only one of them was made on purpose. It will not tell you whether anybody obeyed This is the number every tool in this space is tempted to imply, and it cannot be known from here or from anywhere. A robots.txt is a request, not a fence. It stops nobody who ignores it, nobody who renames their user agent, and it removes nothing from a model that has already read the site. Where a figure cannot be known, the panel prints an em-dash and the reason. Twenty-two crawlers, matched exactly The tokens of twenty-two named AI crawlers, from the large model operators down to the open research corpora, each with who runs it and what they say it is for. Matching is on the token as written and never on a substring, so a rule for one product is never counted as a rule for a similarly named one. The list is a snapshot: operators add, rename and retire these tokens with no notice and no central registry, and the panel says so rather than implying it is complete. It reads the file properly Groups that name the same crawler twice are merged, as the specification requires, so a file that splits its rules across two blocks is not read backwards. A block on one folder is not reported as a block on the site. A rule file that answers with a web page — which a great many hosts do for any unknown path — is reported as absent rather than parsed as a page with no rules in it, because "this site refuses nobody" and "there is nothing here to read" are different answers. Two permissions, and nothing runs until you click activeTab and scripting. No host permissions, no background worker, no content script. When you click, it asks the site you are on for its own rules file and nothing else, without cookies, and asks no other host for anything. Free. No account. Privacy Everything happens in your browser. No account, no tracking, no analytics, and no data leaves your machine. https://murmtools.com/murmcrawl-privacy Part of MurmTools — small research extensions that show their working. https://murmtools.com