Web Bot Auth: Cryptographic Identity for AI Agents
Web Bot Auth lets a bot prove its identity per request using HTTP Message Signatures (RFC 9421) with an Ed25519 key and a published key directory, replacing user-agent strings and IP allowlists — which are trivially forged and hard to maintain.
Every mechanism that names a crawler — robots.txt groups, analytics, allowlists — rests on the user agent string, which any client can set to any value. Web Bot Auth replaces that with cryptographic proof. The scheme, led by Cloudflare and standardised through an IETF working group chartered in 2026 after a Birds-of-a-Feather session at IETF 123, has a bot sign each request using HTTP Message Signatures (RFC 9421). Concretely: an Ed25519 key, a `Signature-Agent` header identifying where the operator's keys live, and a published JWKS directory a verifier can fetch to confirm the signature. Two drafts carry the design — one for the key directory, one for the signing protocol. What this fixes is specific. Identifying bots by user agent is forgeable by anyone, and identifying them by IP range means every operator publishes a list and every site owner re-fetches it forever. A per-request signature collapses both problems: the request either verifies against the operator's advertised key or it does not, and no allowlist has to be maintained. Adoption is unusually concrete for a draft standard. Verification is implemented at the infrastructure layer by Cloudflare and by AWS WAF, and agents signing their requests include those operated by Anthropic, OpenAI, Perplexity and Common Crawl. The strategic consequence is that it enables graduated policy rather than a binary. A site that cannot tell a declared crawler from a scraper wearing its name has only "block" and "allow". A site that can verify identity can serve verified agents a richer or faster path — a documented API, higher rate limits — while treating unverified traffic conservatively. It is also the precondition for anything transactional, since pay-per-crawl terms of the sort in Really Simple Licensing (RSL): Machine-Readable AI Licensing Terms are unbillable against an identity nobody can verify. Related: AI Crawler User-Agents: Training, Search, and User-Fetch Are Separate Controls and The Four Layers of Machine-Readable AI Site Policy.