# robots.txt, drnickygrant.com # # Policy, 16 October 2026 (owner decision, replaces the allow-everything # policy of 15 October): welcome every crawler that brings searchers or # answers people's questions with a link back here; turn away crawlers that # only harvest pages for datasets and send no one back, above all the bulk # training scrapers. # # Allowed: every search engine (UK, US and the Chinese engines that serve # the /zh/ pages); the AI search and live-answer agents of ChatGPT, Claude, # Perplexity, Gemini, Apple, DuckDuckGo, Mistral and Meta AI; and the # training crawlers of the four companies whose assistants send visitors # here (OpenAI, Anthropic, Google, Apple), so that those assistants know # the site without a live search. # # Turned away: bulk training and dataset scrapers with no referral product # or a record of heavy crawling (ByteDance, Common Crawl, Meta's training # crawler, Amazon's training crawler, DeepSeek, Huawei Pangu, Diffbot and # the others listed at the end). # # Nothing else is disallowed except the drill app shell (/coredrills/app/, # a 3 MB question bank with no standalone value: it is noindex and # canonicalises to /coredrills/). A named group does not inherit the # default group's rules, so each repeats that one Disallow. # # Enforcement: robots.txt is a request. The same list is enforced at the # Cloudflare edge by the WAF custom rule "AI Crawl Control - Block AI bots # by User Agent" (Security > Security rules), which also stops scrapers # that ignore this file; the AI Crawl Control categories stay at Allow. # Bot Preference Sync stays off so that Cloudflare does not rewrite this # file. Keep the two lists the same. # # Content-Signal (Cloudflare's Content Signals Policy, contentsignals.org): # search = may be indexed and linked; ai-input = may be fetched to answer a # user's question; ai-train = may be used for training, by the crawlers # this file allows. User-agent: * Content-signal: search=yes, ai-input=yes, ai-train=yes Allow: / Disallow: /coredrills/app/ Allow: /coredrills/ # --------------------------------------------------------------- # Search engines # --------------------------------------------------------------- # Google Search, including AI Overviews and AI Mode User-agent: Googlebot Allow: / Disallow: /coredrills/app/ # Bing, Yahoo, DuckDuckGo results and Microsoft Copilot User-agent: Bingbot Allow: / Disallow: /coredrills/app/ # Apple: Siri, Spotlight, Safari and Apple Intelligence answers User-agent: Applebot Allow: / Disallow: /coredrills/app/ # DuckDuckGo User-agent: DuckDuckBot Allow: / Disallow: /coredrills/app/ # --------------------------------------------------------------- # Chinese search engines (the /zh/ pages) # --------------------------------------------------------------- # Baidu User-agent: Baiduspider Allow: / Disallow: /coredrills/app/ # Baidu, rendering User-agent: Baiduspider-render Allow: / Disallow: /coredrills/app/ # Sogou (also WeChat search) User-agent: Sogou web spider Allow: / Disallow: /coredrills/app/ # Qihoo 360 User-agent: 360Spider Allow: / Disallow: /coredrills/app/ # Shenma (UC and Alibaba) mobile search User-agent: Yisouspider Allow: / Disallow: /coredrills/app/ # Huawei Petal Search User-agent: PetalBot Allow: / Disallow: /coredrills/app/ # --------------------------------------------------------------- # AI search, live answers and citations. These fetch a page to answer one # person's question, or index it for answers that link back: they send # real visitors. # --------------------------------------------------------------- # OpenAI: ChatGPT search index User-agent: OAI-SearchBot Allow: / Disallow: /coredrills/app/ # OpenAI: a page fetched to answer one user's question User-agent: ChatGPT-User Allow: / Disallow: /coredrills/app/ # Anthropic: Claude's search index User-agent: Claude-SearchBot Allow: / Disallow: /coredrills/app/ # Anthropic: a page fetched to answer one user's question User-agent: Claude-User Allow: / Disallow: /coredrills/app/ # Perplexity: index and citations User-agent: PerplexityBot Allow: / Disallow: /coredrills/app/ # Perplexity: a page fetched to answer one user's question User-agent: Perplexity-User Allow: / Disallow: /coredrills/app/ # DuckDuckGo's AI answers User-agent: DuckAssistBot Allow: / Disallow: /coredrills/app/ # Mistral Le Chat: a page fetched to answer one user's question User-agent: MistralAI-User Allow: / Disallow: /coredrills/app/ # Meta AI: a page fetched to answer one user's question User-agent: meta-externalfetcher Allow: / Disallow: /coredrills/app/ # --------------------------------------------------------------- # Training, allowed for the four companies whose assistants send visitors # here, so that their models know this site and its owner. # --------------------------------------------------------------- # OpenAI: model training User-agent: GPTBot Allow: / Disallow: /coredrills/app/ # Anthropic: model training User-agent: ClaudeBot Allow: / Disallow: /coredrills/app/ # Google: Gemini training and grounding (a token, not a crawler; it never # affects Google Search or AI Overviews) User-agent: Google-Extended Allow: / Disallow: /coredrills/app/ # Apple: Apple Intelligence training (a token; Applebot above crawls) User-agent: Applebot-Extended Allow: / Disallow: /coredrills/app/ # --------------------------------------------------------------- # Turned away: bulk training and dataset scrapers. They send no visitors # back. The same list is blocked at the Cloudflare edge. # --------------------------------------------------------------- # ByteDance: Doubao training and bulk crawling User-agent: Bytespider Disallow: / # ByteDance: TikTok crawler User-agent: TikTokSpider Disallow: / # Common Crawl: the open dataset that training scrapers reuse User-agent: CCBot Disallow: / # Meta: model training (Meta AI's live answers use meta-externalfetcher, allowed above) User-agent: meta-externalagent Disallow: / # Meta: older training crawler User-agent: FacebookBot Disallow: / # Amazon: model training User-agent: Amazonbot Disallow: / # DeepSeek: model training User-agent: DeepSeekBot Disallow: / # Huawei: Pangu model training (Petal Search is allowed above) User-agent: PanguBot Disallow: / # Cohere: training data User-agent: cohere-training-data-crawler Disallow: / # Diffbot: data resold to third parties User-agent: Diffbot Disallow: / # Allen Institute: open training datasets User-agent: AI2Bot Disallow: / # Dataset and image scrapers User-agent: ImagesiftBot Disallow: / User-agent: img2dataset Disallow: / User-agent: Omgilibot Disallow: / User-agent: Timpibot Disallow: / User-agent: iaskspider Disallow: / # --------------------------------------------------------------- # Machine-readable context for AI engines: curated summaries and sources. # Open to every allowed crawler by the groups above. # /llms.txt /llms-extended.txt /ai-summary.txt /ai-context/ # /zh/llms.txt /zh/ai-summary.txt /zh/ai-context/ # /research/tex/ /teaching/tex/ # --------------------------------------------------------------- # Key commercial page: /tmua-app-presale.html (the EntryQ TMUA app: passes and prices, # the closed beta, students' Google reviews in full, FAQ). Its facts are mirrored in # /llms.txt, /llms-extended.txt and /ai-summary.txt (updated 16 November 2026). # The sitemap carries a video entry for the EntryQ tour (/video/, thumbnail in /img/); # both folders are open to every allowed crawler so the video can be indexed. Sitemap: https://www.drnickygrant.com/sitemap.xml