AI Crawler User-Agents: Training, Search, and User-Fetch Are Separate Controls

Treating "AI bots" as one category is the most common mistake in robots.txt configuration. Vendors deliberately expose several tokens with different meanings, so a site can permit one behaviour and refuse another. OpenAI runs three distinct agents, each with its own robots.txt token and its own published IP ranges: - GPTBot — crawls content that may be used to train generative models. - OAI-SearchBot — indexes content so it can be cited in ChatGPT's search results. It operates independently of GPTBot. - ChatGPT-User — fetches one URL because a human asked ChatGPT to read that page. These are independently controllable. Allowing OAI-SearchBot while disallowing GPTBot means "cite me, don't train on me". Allowing ChatGPT-User alone means "fetch this when a person asks, but do not crawl me". Google-Extended and Applebot-Extended work differently, and this is the detail most guides get wrong: **they are not crawlers**. Neither fetches anything. They are product tokens that govern what may be done with content the ordinary Googlebot or Applebot already retrieved. Disallowing Google-Extended opts content out of Gemini training without affecting Google Search visibility — and notably does not remove a site from AI Overviews, which are served from the regular Googlebot crawl. Applebot-Extended similarly opts out of Apple Intelligence training while leaving Siri and Spotlight results intact. Two structural rules govern all of this: - A crawler obeys **exactly one group** — the most specific one matching its user agent — and does not inherit from `User-agent: *`. A named group must therefore restate the entire policy, not just its differences. - robots.txt is advisory. It is a request that well-behaved crawlers honour, not an access control. Enforcement requires authentication or blocking; see Web Bot Auth: Cryptographic Identity for AI Agents. Because the vocabulary is vendor-specific and expands every time a company ships a new product, there is a standardisation effort to replace it — see AIPREF: The IETF Standard for Declaring AI Usage Preferences and The Four Layers of Machine-Readable AI Site Policy.