The Four Layers of Machine-Readable AI Site Policy
Site-level AI policy separates into four independent layers — access, preference, licensing and identity — that answer different questions, are at different maturity levels, and are routinely conflated in guidance that treats them as one decision.
Advice about "controlling AI access to your site" collapses four separate questions into one. They have different mechanisms, different maturity, and different failure modes, and a site can hold a different position on each. **1. Access — may you fetch this?** robots.txt, governed by RFC 9309 and extended in practice by vendor-specific tokens. Mature, universally implemented, and entirely voluntary. See AI Crawler User-Agents: Training, Search, and User-Fetch Are Separate Controls. **2. Preference — what may you do with it once fetched?** The AIPREF work at the IETF, expressed as `Content-Usage` in a header or a robots.txt rule, with `train-ai` and `search` as separate categories. Standards-track but not yet deployed. See AIPREF: The IETF Standard for Declaring AI Usage Preferences. **3. Licensing — on what terms, and for what compensation?** RSL, spanning free and attribution through pay-per-crawl and pay-per-inference. An industry standard with an associated collective, unresolved as to enforcement. See Really Simple Licensing (RSL): Machine-Readable AI Licensing Terms. **4. Identity — who is actually asking?** Web Bot Auth, signing each request under RFC 9421. The newest layer and, structurally, the one the others depend on. See Web Bot Auth: Cryptographic Identity for AI Agents. The dependency runs bottom-up, and this is what most guidance misses. Layers 1 through 3 are declarations addressed to whoever is asking — and until layer 4 exists, "whoever is asking" is a self-reported string. A preference expressed at anyone claiming to be GPTBot is a preference expressed at anyone at all. This is the same structural weakness that metadata systems have always had when the party supplying the metadata benefits from misstating it — the argument in Metacrap (Doctorow Essay), arriving here with the roles reversed: the publisher is honest and the consumer's self-description is the unreliable part. A practical consequence: the layers are worth adopting in a different order than they are usually presented. Identity verification is available now at the infrastructure layer and changes what the others can mean, while the preference and licensing layers are cheap forward-looking declarations whose value is contingent on ratification. Publishing all four costs little; expecting the top three to bind anyone without the fourth is the error. A site that actively *wants* AI consumption inverts every default in the standard advice — it allows the crawlers, declares permissive preferences, licenses freely, and treats verified identity as a reason to serve agents better rather than to throttle them. See llms.txt: The Proposed AI-Readable Site Summary, and Its Adoption Gap and Generative Engine Optimization: Optimizing to Be Cited, Not Ranked.