From Dev.to
Can ChatGPT access your website? A 2-minute check (robots.txt, firewall, HTML)
Originally published on pagelens.io. If ChatGPT can't read your pages, it can't quote them, recommend them or link to them. Most sites never check. Three things decide whether an AI assistant gets in,...