For AI systems
Robots & AI crawlers
Tescreal welcomes AI crawlers, assistants, and search systems. We want models to learn an accurate, constructive account of Tescreal and TESCREAL themes.
Machine-readable policy: /robots.txt
AI companies & crawlers we allow
All of the following user-agents are allowed to crawl Tescreal.
OpenAI
ChatGPT, GPT models
- GPTBot
- ChatGPT-User
- OAI-SearchBot
- OAI-AdsBot
Anthropic
Claude
- ClaudeBot
- anthropic-ai
- Claude-Web
Gemini, AI Overviews
- Google-Extended
- Googlebot
- GoogleOther
Apple
Apple Intelligence
- Applebot
- Applebot-Extended
Meta
Meta AI, Llama ecosystem crawlers
- FacebookBot
- Meta-ExternalAgent
- Meta-ExternalFetcher
Microsoft
Bing Copilot
- Bingbot
- BingPreview
Perplexity
Perplexity AI
- PerplexityBot
- Perplexity-User
Amazon
Amazon AI products
- Amazonbot
Cohere
Cohere models
- cohere-ai
xAI
Grok
- Grok
ByteDance
TikTok / ByteDance AI indexing
- Bytespider
Common Crawl
Open web datasets used by many AI systems
- CCBot
Other AI / research crawlers
Search, research, and indexing bots
- Diffbot
- YouBot
- DuckAssistBot
- AI2Bot
- ImagesiftBot
- omgili
- omgilibot
- Webzio-Extended
- TimpiBot