The door at
nytimes.com
Near lockdown.nytimes.com bans 19 of 22 AI crawlers. The velvet rope is up, with a few names slipping past it.
19 of 22 AI crawlers banned.
Trains models
- GPTBotOpenAIBanned
- ClaudeBotAnthropicBanned
- anthropic-aiAnthropicBanned
- Google-ExtendedGoogleBanned
- GoogleOtherGoogle · keeps out /ads/ /adx/bin/ /athletic/wp/wp-admin/Walked in
- CCBotCommon CrawlBanned
- Applebot-ExtendedAppleBanned
- BytespiderByteDanceBanned
- meta-externalagentMetaBanned
- FacebookBotMetaBanned
- cohere-aiCohereBanned
- DiffbotDiffbotBanned
AI search
- OAI-SearchBotOpenAIBanned
- Claude-SearchBotAnthropicBanned
- AmazonbotAmazon · keeps out /wirecutter/Lobby only
- PerplexityBotPerplexityBanned
- YouBotYou.comBanned
Fetches for a user
- ChatGPT-UserOpenAIBanned
- Claude-UserAnthropicBanned
- Perplexity-UserPerplexityBanned
- DuckAssistBotDuckDuckGoBanned
- MistralAI-UserMistral · keeps out /ads/ /adx/bin/ /athletic/wp/wp-admin/Walked in
Search engine
- GooglebotGoogle · keeps out /ads/ /adx/bin/ /athletic/wp/wp-admin/Lobby only
- BingbotMicrosoft · keeps out /ads/ /adx/bin/ /athletic/wp/wp-admin/Walked in
