Content-Usage (aipref): robots.txt and HTTP header snippets

Declare AI usage preferences with a Content-Usage rule in robots.txt or a Content-Usage HTTP header, for example train-ai=n, search=y. The syntax comes from the IETF's AI Preferences (aipref) working group. It states intent and blocks nothing. Both specifications are still Internet-Drafts, and nothing obliges a crawler to read them (vocabulary draft, attachment draft).

In the readiness stack this sits one step after access control. robots.txt decides who may fetch a page (how to control AI crawlers); usage preferences say what a crawler that did fetch it may do next: train a model, feed an answer or only index. Unlike llms.txt, which only points at content, it states permission. It is one layer of AI agent readiness; the 30-minute audit shows how it fits with the rest. Status is as of October 2026.

Key takeaways

• Two drafts, no RFC: vocabulary -08 (14 September 2026) and attachment rules -05 (19 August 2026). The vocabulary text says the draft as a whole, and Sections 3 and 4 in particular, does not reflect working-group consensus (IETF datatracker).
• Three categories, two values: train-ai, ai-use, search, each y or n. Omitted means "unknown", and the draft names no default (vocabulary -08).
• One string, two attach points: a Content-Usage line in a robots.txt group, or a response header (attachment -05).
• It enforces nothing. The recipient chooses whether to follow it, and Google's robots.txt documentation lists four supported fields, none of them this one (Google).
• Cloudflare's Content-Signal is a different vocabulary (ai-train=no, not train-ai=n). Our own observation from the code: aiSiteReady's content_usage_policy check reads Content-Signal, RSL (Really Simple Licensing), TDMRep (the TDM Reservation Protocol) and noai, but not Content-Usage yet.

What is Content-Usage, and what do the aipref drafts define?

Content-Usage is the name of a robots.txt rule and an HTTP header that carry a short statement of AI usage preferences. Two Internet-Drafts from the IETF's AI Preferences working group define it.

The vocabulary has three categories, each with a lowercase label (vocabulary -08).

A statement assigns y (allow) or n (disallow) to any of these, serialized as a Structured Fields dictionary: comma-separated members, lowercase keys, bare y or n tokens (IETF RFC 9651). So train-ai=n, search=y means "do not train on this, search may use it" and says nothing about ai-use. A missing key, a quoted "n" or any token other than y and n yields "unknown", not an error.

search is deliberately narrow: it applies only when the result links back to the original and excerpts help users judge relevance. It does not cover summaries.

The drafts are also moving. After the working group's last call, an October 2025 update described categories called Foundation Model Production, AI Output and Search (IETF blog); today's draft says AI Training, AI Use and Search. The charter's August 2026 milestone for sending both documents to the IESG, the IETF's approval body, has passed. The datatracker still lists both as working-group documents (IETF datatracker), so treat everything here as a snapshot from October 2026.

How do you declare Content-Usage in robots.txt?

Add one or more Content-Usage lines inside a User-Agent group, with an optional path before the preference. The attachment draft calls this a two-stage arrangement: Allow and Disallow decide whether a crawler may fetch a URL, and Content-Usage states what it may do with the content afterwards (attachment -05).

User-Agent: *
Allow: /
Disallow: /admin/
Content-Usage: train-ai=n, search=y
Content-Usage: /blog/ train-ai=y, ai-use=y, search=y

A line without a path covers everything the group may crawl. A line starting with / is a path prefix matched like Allow and Disallow, and the longest match wins, so here /blog/ is open for training and the rest is closed. /admin/ is disallowed, and the draft says no preference is implied for a resource that cannot be crawled.

Three traps before you copy anything:

1. Named groups do not inherit. A crawler obeys the group matching its product token and falls back to * only when none matches (IETF RFC 9309). In the draft's own example, ExampleBot uses its own group and none of the * group's rules. If you write separate groups for AI crawlers, repeat the line in each group, as the robots.txt guide advises for Disallow.
2. Crawlers cache robots.txt. The draft allows up to 24 hours, and changes do not apply retroactively to content already fetched.
3. Unknown lines are ignored, and may be flagged. Google skips fields it does not list (Google), and Cloudflare notes Search Console may report Syntax not understood for newer directives, with no observed crawl or SEO effect (Cloudflare docs).

User-Agent: *
Allow: /
Content-Usage: train-ai=n, search=y

User-Agent: GPTBot
Allow: /
Content-Usage: train-ai=n, search=y

How do you send Content-Usage as an HTTP header?

Send it on the response in exactly the same syntax as the robots.txt value. The draft defines it as representation metadata: it applies to the content of the message, not the resource, works for any content type, and has no special effect on caching. It defines no HTML <meta> form and no other way to embed the preference in a page (attachment -05).

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Content-Usage: train-ai=n, search=y

On nginx, one line in the server block covers every response, including images and PDFs.

# "always" also adds it to error responses (4xx/5xx).
add_header Content-Usage "train-ai=n, search=y" always;

nginx has an inheritance rule that trips people up: it passes add_header down only if the current level defines none, so a location with its own add_header silently drops this one (nginx docs). Repeat the line there. If your application sets headers itself, a Nitro plugin does the same job, using the beforeResponse pattern aiSiteReady already uses for its RateLimit-Policy header.

// server/plugins/content-usage.ts
export default defineNitroPlugin((nitroApp) => {
  nitroApp.hooks.hook("beforeResponse", (event) => {
    // A streamed response has already sent its headers.
    if (event.node.res.headersSent) return;
    setResponseHeader(event, "Content-Usage", "train-ai=n, search=y");
  });
});

Use the header when preferences differ per response or file type, or when you cannot edit the robots.txt that covers your content. Otherwise the robots.txt rule is simpler: one file to audit.

Which policy should you publish? Three examples

Pick one of three values: train-ai=n, search=y (search, no training), train-ai=y, ai-use=y, search=y (allow all) or train-ai=n, ai-use=n, search=n (decline all). Copy the matching line into both attach points. Anything you omit stays "unknown", so state the categories you care about.

# robots.txt - pick one line inside your User-Agent group
Content-Usage: train-ai=n, search=y
# Content-Usage: train-ai=y, ai-use=y, search=y
# Content-Usage: train-ai=n, ai-use=n, search=n

Content-Usage: train-ai=n, search=y

"Block all three" does not stop a request; it only declares your wish. To keep a crawler out, use Disallow for crawlers that honor it, or a firewall rule for those that do not.

search=y is broader than it looks: the draft says it covers a search application's internal processing, even model training, if the results link back and show relevance-helping excerpts. It also says search overrides the other categories where they overlap. That rule sits in Section 4, and the draft says the text as a whole, and that section in particular, does not yet reflect working-group consensus.

How is Content-Usage different from Content-Signal, TDMRep, RSL and noai?

Each one uses its own vocabulary, so none replaces another. Content-Usage overlaps most with Cloudflare's Content Signals Policy, published as a Content-Signal line in robots.txt with per-purpose yes/no values.

A ? in the matrix means a source mentions the placement without defining it, such as Cloudflare's changelog referring to a content-signal response header (Cloudflare changelog), or aiSiteReady's own check reading noai in X-Robots-Tag. Content-Signal and Content-Usage are close cousins, and the labels map like this.

train-ai=no is not a valid IETF preference, because only y and n count, so it silently parses to "unknown". Cloudflare is also testing an optional fourth field, use=immediate, reference or full, so Content Signals is still evolving too (Cloudflare docs). If you publish both, make them say the same thing.

What happens when signals conflict?

Within Content-Usage the drafts answer clearly: the more specific path wins, and where values still clash, the most restrictive applies. Between Content-Usage and everything else, no document names a winner, so never publish contradictions.

1. Longer path wins. In the draft's example, a pathless train-ai=n covers the site and /ai-ok/ train-ai=y overrides it for that subtree.
2. Same path, different values: most restrictive wins. The draft notes this differs from Allow and Disallow, where a tie resolves to the more permissive option (attachment -05, RFC 9309).
3. robots.txt rule against HTTP header. The attachment draft sets no precedence. The vocabulary's default applies "absent some other means of resolving conflicts": any disallow wins, then any allow, otherwise unknown (vocabulary -08).
4. Disallow on the path. Preferences apply only to resources the crawler may fetch.
5. Contracts beat signals. The vocabulary notes that a license explicitly permitting a use likely applies despite a machine-readable "no".

Other mechanisms use other rules. TDMRep says rights holders should use one technique only, and if several appear, a header supersedes the well-known file and HTML metadata supersedes both (W3C). I found no conflict rule for Content-Signal in the Cloudflare material I read. So if TDMRep reserves rights while Content-Usage says train-ai=y, no specification decides which one wins.

Who actually honors Content-Usage today?

I could not find a primary source from a major crawler operator saying it reads Content-Usage, so treat support as unknown and uneven. What I could check:

• The specification. It does not ensure preferences are followed, leaves that choice to the recipient, and says preferences are not a security mechanism (vocabulary -08).
• Google. Its robots.txt documentation lists user-agent, allow, disallow and sitemap and says other fields are unsupported (Google). Its documented AI control is the Google-Extended token (Google).
• OpenAI. Its crawler page documents GPTBot and OAI-SearchBot as independent robots.txt tokens; I found no mention of Content-Signal or Content-Usage there on 2 October 2026 (OpenAI).
• Cloudflare. Content Signals express preferences, not technical countermeasures, and some companies might ignore them (Cloudflare).

The one adoption figure I can cite: Cloudflare's scan of Radar's top 200,000 domains, published in April 2026, found that 4% declared AI usage preferences in robots.txt via Content Signals (Cloudflare). I have none for Content-Usage.

So build in layers: real crawler tokens for access, a usage declaration so your intent is on the record, and server or firewall rules for enforcement. Legal weight depends on your jurisdiction; Cloudflare's Content Signals text frames its restrictions as rights reservations under the EU's 2019/790 copyright directive (Cloudflare), so ask a lawyer about your case.

How do you verify your Content-Usage setup?

Fetch the file and the header from the public URL, not from localhost: a proxy, a CDN or an nginx location can change what a crawler sees.

# 1. The robots.txt rule (also shows any Content-Signal line)
curl -sS https://example.com/robots.txt | grep -iE '^content-(usage|signal):'

# 2. The response header on a page
curl -sSI https://example.com/ | grep -i '^content-usage:'

# 3. The same header on a non-HTML file from the same stack
curl -sSI https://example.com/logo.png | grep -i '^content-usage:'

Expect Content-Usage: train-ai=n, search=y from the first command and content-usage: train-ai=n, search=y from the others (HTTP/2 lowercases names). No output means the signal is missing; grep exits with status 1. I ran the first command against aiSiteReady's production site on 2 October 2026 and got one line, Content-Signal: search=yes, ai-input=yes, ai-train=yes. There is no Content-Usage rule or header there yet.

To see what a consumer would conclude, parse the value as the draft describes and combine statements. This is a deliberately small reading of the format; use a full RFC 9651 parser in production.

type Pref = "allow" | "disallow" | "unknown";
type Prefs = Record<"train-ai" | "ai-use" | "search", Pref>;
const unknown = (): Prefs => ({ "train-ai": "unknown", "ai-use": "unknown", search: "unknown" });

export function parseUsage(value: string): Prefs {
  const prefs = unknown();
  const seen = new Map<string, string>();
  for (const member of value.split(",")) {
    const [rawKey, rawVal] = member.trim().split("=");
    const key = rawKey.split(";")[0];
    if (/[A-Z]/.test(key)) return unknown(); // keys are case-sensitive: parse fails
    seen.set(key, (rawVal ?? "").split(";")[0].trim()); // repeated key: last one wins
  }
  for (const key of Object.keys(prefs) as (keyof Prefs)[]) {
    const v = seen.get(key);
    if (v === "y") prefs[key] = "allow";
    else if (v === "n") prefs[key] = "disallow"; // "no", "yes" and "\"n\"" stay unknown
  }
  return prefs;
}

// Default conflict rule: any disallow wins, then any allow.
export function combine(...all: Prefs[]): Prefs {
  const out = unknown();
  for (const key of Object.keys(out) as (keyof Prefs)[]) {
    const v = all.map((p) => p[key]);
    out[key] = v.includes("disallow") ? "disallow" : v.includes("allow") ? "allow" : "unknown";
  }
  return out;
}

// robots.txt says "train-ai=y, search=y", the header says "train-ai=n":
console.log(combine(parseUsage("train-ai=y, search=y"), parseUsage("train-ai=n")));
// { 'train-ai': 'disallow', 'ai-use': 'unknown', search: 'allow' }
console.log(parseUsage("train-ai=no, search=y"));
// { 'train-ai': 'unknown', 'ai-use': 'unknown', search: 'allow' }

I ran this exact code and it prints the outputs shown. These are the mistakes I would check first:

Can aiSiteReady check your usage signals?

Partly: its content_usage_policy check finds Content-Signal, TDMRep, RSL and noai, but not Content-Usage yet. The methodology page places the check in the Bot Governance category. The details below come from its source code, which I read while writing this article.

The check fetches the page you scan (usually the homepage), plus /robots.txt and /.well-known/tdmrep.json. In robots.txt it recognizes Content-Signal pairs and RSL License: lines. Elsewhere it reads valid TDMRep entries, TDMRep meta tags and the tdm-reservation and tdm-policy headers; noai and noimageai in a robots meta tag or X-Robots-Tag; RSL Link relations; and vendor-style policy headers and meta tags such as ai-policy and content-usage-policy.

One strong signal is a PASS, only weak or malformed signals are PARTIAL, and no signals at all is a FAIL. It scores whether an explicit declaration exists, not which way you chose, so reserving and permitting both pass.

Its header list contains content-usage-policy, a vendor-style name, but not the IETF's content-usage. The check never reads a Content-Usage line in robots.txt either. Other robots.txt checks only list it as an unknown directive in their evidence, which never changes their status.

By my reading of the code, a site publishing aipref alone would currently be reported as having no usage signals. So publish a recognized signal such as Content-Signal alongside it, and keep the two consistent. aiSiteReady's own robots.txt publishes a permissive Content-Signal line, which is what passes this check on our site.

Run a free scan to see which of Content-Signal, TDMRep, RSL and noai your site publishes, which are weak, and what to fix first, in English, Ukrainian or Russian. The scanner is read-only, fetches through an SSRF-safe fetcher and runs no JavaScript.

Content-Usage lets you say train-ai=n, search=y once and attach it to robots.txt and your HTTP responses. It is a draft nobody has to obey, so publish it carefully, keep it consistent with any Content-Signal you already have, and back it with real access rules where it matters.

IMozz has 20 years in software development, with the past year spent building with LLMs. He builds aiSiteReady, a read-only scanner that checks whether AI agents can read a site. It server-renders its own content as a working example.