From Dev.to
Which AI Crawlers Does Your robots.txt Block? Check GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot Per URL With Python
Most teams now have an opinion about AI crawlers. Block training, allow search. Or allow everything so the product shows up in ChatGPT and Perplexity answers. Or block it all. The opinion is the easy...