Which AI crawlers do sites block? 219 homepages, six categories.
We read the robots.txt of 219 homepages in six categories. Few block an AI crawler outright; many apply a Disallow rule that is usually an ordinary path.
Two Sites, Launched With Good SEO, Invisible to AI Crawlers
AgaOne Commodities scanned their new corporate sites after launch, fixed what the report ranked first, and scanned again. Average score moved from 56.5 to 94.5.
Who Controls Your Brand's AI Representation?
76% of consumer brands have an llms.txt file, but the files we checked by hand were platform-generated and said nothing about the brand. A scan of 64 sites across B2B SaaS and consumer brands.
See where your own site lands.
Scan your site free