From Dev.to
I checked 30 top websites for AI-crawler readiness. The ones ChatGPT cites most scored worst.
Everyone "knows" that the web is locking out AI crawlers. Cloudflare flips the default to block, robots.txt blocklists go viral, every SEO newsletter screams about GPTBot. So I ran the actual test. I...