RateLimit-Policy header: let AI agents throttle themselves

The RateLimit-Policy header tells an AI agent how many requests your service accepts per time window, and its companion RateLimit header tells it how many are left right now. An agent that reads both can pace itself, pause before it runs dry, and wait the right number of seconds, instead of discovering your limit by running into a 429 Too Many Requests or an IP block.

Rate limits belong to the Bot Governance layer of the readiness stack. robots.txt says whether a crawler may come in; these headers say how fast it may go. They matter most for an API you expose to agents (see make your site agent-callable and the API catalog), but the same two lines on a home page already tell a crawler what to expect.

Both fields come from an IETF Internet-Draft, not an RFC, so I mark what could still move. For the bigger picture, start with what AI agent readiness means.

Key takeaways

• Two response fields, both lists of Structured Field items: RateLimit-Policy (q quota, w window, optional qu unit and pk partition key) and RateLimit (r remaining, t effective window in seconds) (IETF draft).
• Status as of October 2026: Internet-Draft revision 11, dated 23 May 2026, no RFC number; the syntax last changed in revision 08, but open pull requests would rename RateLimit's r to a and t to w, allow unquoted policy names, and add a RateLimit-Partition field. Treat the examples as the current shape, not a promise.
• 429 with Retry-After is what makes a client stop (RFC 6585, RFC 9110); the draft says Retry-After wins when both are present.
• Advertise the standing policy on pages and the live remainder on uncached API responses, only for limits you really enforce. As of October 2026, our own site hard-codes q=20;w=60 in the frontend while the backend reads its limit from config, a sync risk described below.
• aiSiteReady's rate_limit_hints check passes on a parseable RateLimit or RateLimit-Policy, or a valid Retry-After; legacy X-RateLimit-* and a bare 429 earn only a PARTIAL (details at the end).

What do RateLimit-Policy and RateLimit actually say?

They are two response header fields that split one job in half: RateLimit-Policy states the standing rule, and RateLimit reports the current state under that rule. Both are Structured Field lists (RFC 9651) whose items are quoted strings naming a policy, followed by ;-separated parameters. The policy name is what ties the two headers together.

Here is a worked example with numbers that agree with each other. The policy "api";q=100;w=60 allows 100 requests per 60-second window. Twenty-five seconds into the window the agent has spent 60 of them, so the server answers "api";r=40;t=35: at most 40 more requests in the next 35 seconds, the time left in this window.

The parameters are short, and each is defined in the draft (IETF draft):

Three rules from the draft shape the implementation. RateLimit-Policy should stay constant across responses, while RateLimit may change on every one. Clients may ignore the policy details, must ignore malformed fields, and must not assume later responses carry the headers at all. And a positive r is a hint, not a guarantee: clients must not treat it as a service-level agreement. A list can carry several policies, such as "burst";q=100;w=60,"daily";q=1000;w=86400.

Is it a standard yet, and which headers should you send?

Not yet: it is an IETF Internet-Draft. Send the revision 08 pair, RateLimit-Policy and RateLimit, plus Retry-After on every 429. The syntax has changed several times, most recently in revision 08. As of October 2026 the latest revision is draft-ietf-httpapi-ratelimit-headers-11, dated 23 May 2026. It belongs to the IETF HTTPAPI working group, its header says Standards Track, and the datatracker shows no RFC number and an expiry of 24 November 2026 (datatracker).

Support is uneven, and that deserves one plain statement. GitHub's REST API sends x-ratelimit-* headers whose reset is a UTC epoch timestamp (GitHub docs). Envoy's rate limit filter can emit X-RateLimit-* headers following draft-polli-03, not the current names (Envoy docs). Anthropic's crawler FAQ says its bots respect a robots.txt Crawl-delay "where appropriate" (Anthropic).

The signal every client already understands is 429 plus Retry-After; Google, for one, treats a 429 as a server-overload signal and slows its crawl (Google). The new headers are still worth two lines: they cost almost nothing, and anything that does read them, scanners such as aiSiteReady included, sees your limits.

The draft's history explains why old tutorials disagree. I compared the archived text of the revisions to build this table:

Which should you send? The revision 08-and-later pair, because it is the current shape and the only one with named policies. Keep legacy X-RateLimit-* only if a client you know relies on it, with identical numbers. Don't add the revision 00 to 06 RateLimit-Limit/-Remaining/-Reset trio or the revision 07 dictionary syntax.

aiSiteReady's scan endpoint (POST /api/v1/scans) sends both families on a 429: Retry-After, the X-RateLimit-* trio, and the modern pair. That is a hedge while the draft settles, not a model to copy forever.

How do 429 and Retry-After fit in?

They are the enforcement half; the new headers are the forecast. 429 Too Many Requests means the client sent too many requests in a given time. Caches must not store a 429, and the response may include Retry-After (RFC 6585, section 4). Retry-After carries either a number of seconds or an HTTP-date (RFC 9110, section 10.2.3).

The draft is independent of the status code: RateLimit may appear on any response, and its own FAQ says the spec is not tied to 429. When a response carries both RateLimit and Retry-After, the client must follow Retry-After, and the server should not set it earlier than the end of the effective window. A complete throttled response therefore looks like this:

HTTP/1.1 429 Too Many Requests
Content-Type: application/problem+json
Retry-After: 35
RateLimit-Policy: "api";q=100;w=60
RateLimit: "api";r=0;t=35

{
  "type": "https://iana.org/assignments/http-problem-types#quota-exceeded",
  "title": "Too Many Requests",
  "status": 429,
  "violated-policies": ["api"]
}

The type URI is a problem type the draft proposes for registration, built on RFC 9457; a plain JSON error body works too. The headers do the real work, and Retry-After equals t here on purpose, so the two signals never contradict each other.

How should an AI agent budget requests and back off?

An agent should pace itself to the policy, pause when r nears zero, obey Retry-After on a 429, and cap every wait. First, read RateLimit-Policy once and pace yourself: for "api";q=100;w=60 that is one request every 0.6 seconds (60 divided by 100), which can never exceed the quota.

Second, read RateLimit after every response, and when r is close to zero, hold the next request for t seconds. The draft allows both styles: slowing down as you approach the limit, or spending the whole quota and then waiting.

Third, on a 429 or 503, wait for Retry-After; if it is missing, wait the RateLimit t seconds, then fall back to exponential backoff with jitter. Fourth, cap what you will wait and ignore garbage: the draft's security section warns about values such as Retry-After: 1000000 and tells clients to set thresholds of their own.

Here is that loop (steps two to four) in Node 18 or later, with global fetch. In October 2026 I ran it against the FastAPI middleware shown below with a small quota of 4 requests per 8 seconds: it paused between windows and never saw a 429, and when I exhausted the quota with curl first, it waited out Retry-After instead.

const sleep = (s: number) => new Promise((r) => setTimeout(r, s * 1000));

// Minimal reader for the common form "api";r=40;t=35. Use an RFC 9651
// Structured Fields library for anything more exotic (partition keys, etc.).
function readRateLimit(value: string | null) {
  if (!value) return null;
  const items = value.split(/,(?=\s*")/).flatMap((member) => {
    const [name, ...params] = member.trim().split(";");
    const p = Object.fromEntries(params.map((s) => s.trim().split("=")));
    const r = Number(p.r);
    const t = p.t === undefined ? undefined : Number(p.t); // effective window, seconds
    if (!/^".+"$/.test(name) || !Number.isInteger(r) || r < 0) return []; // malformed: ignore
    if (t !== undefined && (!Number.isInteger(t) || t < 0)) return []; // same for t
    return [{ policy: name.slice(1, -1), remaining: r, reset: t }];
  });
  // With several policies, obey the one closest to exhaustion.
  return items.sort((a, b) => a.remaining - b.remaining)[0] ?? null;
}

// Retry-After is delay-seconds or an HTTP-date (RFC 9110, section 10.2.3).
function retryAfterSeconds(value: string | null): number | null {
  if (!value) return null;
  if (/^\d+$/.test(value.trim())) return Number(value);
  const at = /[a-z]/i.test(value) ? Date.parse(value) : NaN; // HTTP-dates contain letters
  return Number.isNaN(at) ? null : Math.max(0, Math.ceil((at - Date.now()) / 1000));
}

const MAX_WAIT = 600; // seconds; refuse absurd waits and report them instead
let notBefore = 0; // epoch ms: do not send the next request before this

export async function politeFetch(url: string, attempts = 4): Promise<Response> {
  for (let i = 0; ; i++) {
    const pause = notBefore - Date.now();
    if (pause > 0) await sleep(pause / 1000);

    const res = await fetch(url, {
      headers: { "User-Agent": "acme-agent/1.0 (+https://acme.example/bot)" },
    });
    const limit = readRateLimit(res.headers.get("ratelimit"));

    if (res.status === 429 || res.status === 503) {
      // Retry-After wins over RateLimit when both are present.
      const wait = retryAfterSeconds(res.headers.get("retry-after")) ?? limit?.reset ?? 2 ** i;
      if (i + 1 >= attempts || wait > MAX_WAIT) return res; // give up, let the caller see it
      notBefore = Date.now() + (wait + Math.random()) * 1000; // jitter
      await res.body?.cancel(); // discard the body before retrying
      continue;
    }

    // Nearly out of quota: hold the next request for t seconds (the effective window).
    if (limit && limit.remaining <= 1 && limit.reset !== undefined && limit.reset <= MAX_WAIT) {
      notBefore = Date.now() + limit.reset * 1000;
    }
    return res;
  }
}

The loop is minimal: one origin, one process. A real agent keeps a budget per origin, shares it across workers, and sends a descriptive User-Agent.

How do you publish the headers from nginx, Nitro, and FastAPI?

Publish the standing policy wherever an agent lands first, and the live remainder only where a limiter counts requests. A static RateLimit-Policy suits cacheable pages because it never changes, and a live RateLimit does not: the draft tells clients to ignore both fields on responses served from a cache (IETF draft, section 7.3). Losing a static policy on a cache hit costs little, while a cached live r would be wrong.

aiSiteReady splits it exactly that way: its HTML pages carry "scan";q=20;w=60, and the scan endpoint adds the live pair. scan is our label for the scan API; the draft gives policy names no meaning beyond linking the two headers, so a crawler may apply q=20;w=60 to the pages themselves, a pace we are happy with.

nginx can enforce a limit and advertise the policy, but it cannot report the remaining quota. The limit_req module is a leaky bucket and exposes only a $limit_req_status variable, so you send a static RateLimit-Policy and a Retry-After on rejections (nginx limit_req).

# http {} context
limit_req_zone $binary_remote_addr zone=api:10m rate=60r/m;
limit_req_status 429;                 # the default rejection status is 503

map $status $retry_after {
    429     1;                        # one request per second: a slot frees within 1 s
    default "";                       # an empty value means no header is sent
}

server {
    listen 443 ssl;
    server_name api.example.com;

    location /api/ {
        limit_req zone=api burst=59 nodelay;   # 60 at once, then one per second

        add_header RateLimit-Policy '"api";q=60;w=60' always;
        add_header Retry-After $retry_after always;

        proxy_pass http://app_backend;
    }
}

Three details matter. Without always, add_header applies only to a fixed list of success and redirect codes, so your 429 gets nothing. By default, directives are inherited only if the current level defines no add_header of its own (nginx 1.29.3+ can merge them with add_header_inherit merge;), so one stray line in a location drops the ones above it (nginx headers module). The map trick works because nginx sends no header when the value is empty, as the source shows.

Try limit_req_dry_run on; first and log $limit_req_status to see what would be refused. I ran this configuration in nginx 1.31.1 (Docker): 200s and 404s carried RateLimit-Policy, rejected requests got 429 with Retry-After: 1, and with burst=59 a client could send 60 requests at once before the first 429. With a smaller burst, advertise a smaller quota, for example q=10;w=10 for burst=10. The listen 443 ssl line also needs your ssl_certificate directives, and proxy_pass needs an upstream app_backend block.

Nitro (Nuxt 4) is how aiSiteReady advertises the policy on its own pages. This is an abridged copy of the plugin this site runs; the real one builds the value in a pure function that Vitest can test.

// server/plugins/rate-limit-policy.ts (Nitro 2)
export default defineNitroPlugin((nitroApp) => {
  const policyHeader = `"scan";q=20;w=60`;

  nitroApp.hooks.hook("beforeResponse", (event) => {
    if (event.node.res.headersSent) return; // body already streamed
    const contentType = getResponseHeader(event, "content-type");
    if (typeof contentType !== "string" || !contentType.includes("text/html")) return;
    setResponseHeader(event, "RateLimit-Policy", policyHeader);
  });
});

The text/html gate is deliberate: a font or a JavaScript chunk has no rate limit to advertise. These names are for Nitro 2, which this site runs; the current Nitro documentation shows definePlugin and a response hook instead, so check your version (Nitro 2, Nitro 3).

FastAPI is where the live pair belongs. This middleware keeps a fixed window per client and emits both headers on every response, including the 429. In October 2026 I ran it with uvicorn and tested it with the curl commands below.

import math
import time

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

POLICY, QUOTA, WINDOW = "api", 100, 60  # must match what the limiter below enforces
_windows: dict[str, tuple[float, int]] = {}  # client -> (window start, count)


def hit(client: str) -> tuple[bool, int, int]:
    """Count one request; return (allowed, remaining, seconds until reset)."""
    now = time.monotonic()
    start, count = _windows.get(client, (now, 0))
    if now - start >= WINDOW:  # window over: start a new one
        start, count = now, 0
    count += 1
    _windows[client] = (start, count)
    reset = max(1, math.ceil(start + WINDOW - now))
    return count <= QUOTA, max(0, QUOTA - count), reset


@app.middleware("http")
async def rate_limit(request: Request, call_next):
    client = request.client.host if request.client else "unknown"
    allowed, remaining, reset = hit(client)
    headers = {
        "RateLimit-Policy": f'"{POLICY}";q={QUOTA};w={WINDOW}',
        "RateLimit": f'"{POLICY}";r={remaining};t={reset}',
    }
    if not allowed:
        headers["Retry-After"] = str(reset)
        return JSONResponse({"detail": "Too many requests"}, status_code=429, headers=headers)
    response = await call_next(request)
    response.headers.update(headers)
    return response

The @app.middleware("http") form comes from the FastAPI docs (FastAPI). In production, swap the dictionary for a shared store; aiSiteReady's limiter uses an atomic Redis INCR plus EXPIRE so every API replica counts the same window. Behind a reverse proxy, key on the real client address, not the proxy's. The part worth copying from aiSiteReady's backend is that both headers come from the limiter's decision, never from constants:

headers = {
    "RateLimit-Policy": f'"scan";q={rule.limit};w={rule.window_seconds}',
    "RateLimit": f'"scan";r={decision.remaining};t={decision.retry_after_seconds}',
}

What mistakes break rate-limit headers?

Most failures are a number that stopped matching reality, or a header missing from the response an agent read.

As of 2 October 2026, the last row is a gap in aiSiteReady's own API. Browsers expose only seven response headers to cross-origin scripts by default, and neither RateLimit nor Retry-After is among them (MDN). The backend's CORS configuration exposes only Content-Disposition, so a script on another origin would not see them. A server-side agent is unaffected; a browser-based one would miss them.

The mistake I'm living with is the second copy of the numbers. When I added the header in June 2026, the policy for our HTML pages became two constants in a shared frontend module:

/** Quota (`q`): max scan requests per window. Mirrors `rate_limit_scan_ip_requests`. */
export const SCAN_RATE_LIMIT_QUOTA = 20;
/** Window (`w`) in seconds. Mirrors `rate_limit_scan_window_seconds`. */
export const SCAN_RATE_LIMIT_WINDOW_SECONDS = 60;

The backend reads rate_limit_scan_ip_requests and rate_limit_scan_window_seconds from its settings, with defaults of 20 and 60 that an operator can override through the environment. The comment above the constants says to keep them in sync. The frontend test checks the header against its own constants, and the backend tests check the "scan";q=20;w=60 output for limits configured as 20 and 60, but I found no test that compares one side to the other.

Raise the backend limit to 60 and the scan endpoint advertises q=60 while the home page still says q=20. Lower it to 10 and the home page promises more than the API grants. The fix is one source of truth or a test that checks both; as of 2 October 2026 it is a comment and two matching defaults.

How do you verify the headers with curl?

Fetch the page or endpoint an agent would fetch first, read the headers from a real GET, then push your own API past its quota. Test the public edge URL, because gateways and CDNs in front of your origin can add or strip headers. Use -D - -o /dev/null rather than curl -I: a HEAD request can draw a 405 or skip your middleware.

# 1. What does an agent see on the first page it fetches?
curl -sS -D - -o /dev/null https://example.com/ | grep -i '^ratelimit'

# 2. Does the live remainder count down on the API?
for i in 1 2 3; do
  curl -sS -D - -o /dev/null https://api.example.com/items/1 | grep -i '^ratelimit:'
done

# 3. Push past the quota (your own staging API only) and read the 429
for i in $(seq 1 120); do
  curl -sS -o /dev/null -w '%{http_code}\n' https://staging.example.com/items/1
done | sort | uniq -c
curl -sS -D - -o /dev/null https://staging.example.com/items/1 | grep -iE '^(HTTP|retry-after|ratelimit)'

In October 2026, against the FastAPI middleware above, the second command printed a counter that dropped by one each time, and the third (run after the window had reset) printed 100 200 and 20 429, followed by the 429's headers. Your numbers will differ: t is rounded up and shrinks as the window ages.

ratelimit: "api";r=97;t=60
ratelimit: "api";r=96;t=60
ratelimit: "api";r=95;t=60

HTTP/1.1 429 Too Many Requests
ratelimit-policy: "api";q=100;w=60
ratelimit: "api";r=0;t=60
retry-after: 60

Header names are case-insensitive, so ratelimit: and RateLimit: are the same header. Put the cheap part into CI. This script checks the RateLimit-Policy half of what aiSiteReady accepts, in its most common form: a quoted policy name followed directly by an integer q. The scanner is more lenient: any parameter order, a parseable RateLimit, or a valid Retry-After also passes.

#!/usr/bin/env bash
set -euo pipefail
URL="${1:?usage: check-ratelimit.sh https://example.com/}"
HEADERS="$(curl -sS -D - -o /dev/null "$URL" | tr -d '\r')"
grep -qiE '^ratelimit-policy: "[^"]+";q=[0-9]+' <<<"$HEADERS" || { echo 'FAIL: no parseable RateLimit-Policy'; exit 1; }
grep -qiE '^x-ratelimit-' <<<"$HEADERS" && echo 'WARN: legacy X-RateLimit-* present, keep the numbers identical'
echo OK

How does aiSiteReady check rate-limit headers?

aiSiteReady's rate_limit_hints check reads the headers on the one URL you scan, usually your home page, and scores what it finds. It sits in the Bot Governance category next to the AI bot rules check (which reads your robots.txt) and the content-usage check. It is light on your server: it reuses the response the scan already fetched, so it sends no extra request, and falls back to a single HEAD only when that response isn't available.

It is a low-severity check and runs in every scan profile preset, from All checks to Content site and API/Application. If your API lives on another URL, scan that URL too: a home-page scan sees only the policy on that page.

How it scores what it finds, per the check's documentation and code:

The 429 exception is intentional. Most checks treat both 403 and 429 as "the target blocked us", but here a 429 is itself the rate-limit signal, so a 429 with a valid Retry-After still passes.

The methodology page lists the category weights and severity multipliers. This check is Low (×0.5) and shares Bot Governance with the AI bot rules and content-usage checks, so a FAIL costs about 3 points of 100 (3.6 under Content site) and a PARTIAL about half that. It pairs with controlling AI crawlers in robots.txt, which decides who may crawl; for the wider picture, run the 30-minute agent readiness audit.

Run a free scan to see whether the page you scan advertises a parseable RateLimit-Policy, which legacy headers it still sends, and which fix to make first, in English, Ukrainian, or Russian.

The short version: send RateLimit-Policy for the standing rule, RateLimit for the live remainder, and Retry-After on every 429, generate all three from the limiter that really enforces them, and test the 429 path with curl. The draft may still change, but those lines let an agent slow itself down before you have to slow it down for it.

IMozz has 20 years in software development, with the past year spent building with LLMs. He builds aiSiteReady, a read-only scanner that checks whether AI agents can read and call a site. It server-renders its own content as a working example.