# Machine-readable AI training opt-out — frozen protocol v1

Frozen 19 November 2026, before the first crawl. Vantage: Cursor-hosted cloud VM (USA datacenter egress).

Slug: `ai-opt-out-signals-3day`.

## Research question

On the frozen privacy-policy observatory corpus, which sites expose a TDM Reservation, a robots `noai` or `noimageai` rule, or `ai.txt`, and do any of those signals change across the three collection days?

This is a fetch of public machine-readable signals. It is not a compliance finding, not a claim about how any model was trained, and not a score of privacy-policy prose.

## What this study is not

The privacy-policy observatory (`/blog/privacy-policy-observatory/`, sample frozen 17 November 2026) is the **site list only**. This study does not re-score the 12-dimension rubric or republish the observability index.

The privacy-policy mutation week scores retention, sharing, and arbitration prose on the same sites. This study does not read those clauses and does not diff policy text.

Named AI-crawler `Disallow` rules, Cloudflare `Content-Signal`, and `Content-Usage` lines are recorded as unscored context. They do not answer the question.

## Schedule

Three measurements. There is no 22 November slot.

| Slot | When | Role |
| --- | --- | --- |
| `day0` | 2026-11-19, as soon as this protocol and corpus are frozen | First measurement |
| `day1` | 2026-11-20T06:00:00Z through 2026-11-20T06:30:00Z | Scheduled measurement |
| `day2` | 2026-11-21T06:00:00Z through 2026-11-21T06:30:00Z | Scheduled measurement |

Stop after `2026-11-21T06:30:00Z`. Then write `research-article.md` and the site page on this same branch and PR.

A slot that starts after its `not_after` but before `late_until` in `schedule.json` is collected and marked `late: true`. Do not collect `day2` after `2026-11-21T12:00:00Z`. Do not add a slot on 2026-11-22.

Commands, from the repository root:

```bash
python3 research/ai-opt-out-signals-3day/scripts/collect.py --self-test
python3 research/ai-opt-out-signals-3day/scripts/collect.py --slot day0
python3 research/ai-opt-out-signals-3day/scripts/collect.py --slot day1
python3 research/ai-opt-out-signals-3day/scripts/collect.py --slot day2
```

The collector refuses a slot before `not_before` unless `--allow-early` is passed. It refuses to overwrite a finished slot unless `--force` is passed. Do not use either flag on a scheduled run.

## Network path

- **Path:** cloud-direct HTTPS GET. The collector installs an empty proxy handler so environment `HTTP_PROXY` / `HTTPS_PROXY` cannot move the path.
- **ProxySeller HTTP and SOCKS:** `NOT_RUN`. This study does not need a second exit. Discover env **names** only. Never print or commit values.
- **CometVPN:** `NOT_RUN`. VPN-only. No tunnel. CometVPN remains a corpus member (public site) only.
- **User-Agent, frozen:** `FreePolicyToolsResearch/1.0 (+https://freepolicytools.com/research; ai-opt-out-signals-3day)`.
- Do not retry a blocked response with a browser user-agent. A 401, 403, 429, or 503 is an observation (`blocked`), not a prompt to evade.
- TLS verification on. Timeout 20 seconds. At most 5 redirects. `Accept-Encoding: identity`, with gzip recovery when a server compresses anyway. Delay about 0.2 seconds between URLs on one host and 1.0–1.6 seconds between hosts.
- No cookies, no form posts, no login, no script execution.

## Corpus

N=24. Copied from the observatory amended sample (`source-sample.json` in this directory, SHA-256 recorded in `corpus.json`). Ids, names, strata, and observatory URLs are not edited.

**Site origin rule.** `site_origin` is `https://` plus the host of the observatory URL.

**One pre-registered exception.** `nordvpn`: the observatory document is a PDF on `sb.nordcdn.com`. The site origin is `https://nordvpn.com`. The CDN host’s `robots.txt`, `ai.txt`, and `tdmrep.json` are not fetched and cannot satisfy the site-level signals. The PDF URL is still fetched for resource-level TDM headers and PDF XMP.

No other host is added. `www` is not folded into the apex, and the apex is not added beside `www`. Google is `policies.google.com`, Meta is `www.facebook.com`, Microsoft is `privacy.microsoft.com`. Those are the hosts the observatory corpus named.

`query` strings are kept on the document request. TDMRep path matching uses the URL path only, not the query. That limit matters for the Amazon help URL, whose node id sits in the query.

## Fetches per site

On `site_origin`:

1. `GET /` — homepage headers and HTML meta.
2. `GET /robots.txt`
3. `GET /ai.txt` — headline `ai.txt` location (Spawning root file).
4. `GET /.well-known/tdmrep.json` — W3C TDM Reservation Protocol file.
5. `GET /.well-known/ai.txt` — secondary only. Not the headline boolean.

Document URL (observatory URL with the fragment removed) when it is not the origin root: response headers, HTML meta, or PDF XMP.

Body caps: 512 KiB for `robots.txt`, `ai.txt`, and `tdmrep.json`; 1.5 MiB for HTML; 2 MiB for PDF. A complete gzip member under the cap is decompressed and then capped. Truncation is recorded. Crawler practice already ignores `robots.txt` past 500 KiB, so a cap on that file is still an observation.

Raw bodies stay in untracked `private/`. Public JSON stores status, final URL, hashes, extracted values, and short redacted excerpts.

## Signal definitions

### 1. TDM Reservation

Specification: W3C TDM Reservation Protocol Community Group Report, 10 May 2024. `tdm-reservation` is `1` (rights reserved) or `0` (not reserved). Any other value is a protocol error and counts as unset for the effective value. It is still stored as `error`.

Carriers, in processing order for the **effective** value on one URL:

1. A valid `/.well-known/tdmrep.json` rule whose `location` matches the URL path. Most specific match wins (count of non-`*` characters). Equal specificity keeps the earliest rule in the file. `*` is any sequence. A trailing `$` anchors the end. Without `$`, the pattern is a prefix. Match is case-sensitive.
2. Response header `tdm-reservation` overrides the file when the value is `0` or `1`.
3. HTML `<meta name="tdm-reservation" content="0|1">` overrides the header.
4. On a PDF, XMP `tdm:reservation` / `tdm-reservation` of `0` or `1` overrides previous values.

Absence of a carrier does not clear a value already set. `tdm-policy` URLs are stored and are not fetched.

Headline `exposes_tdm_reservation`:

- `true` if any observed carrier value or file rule is `1`.
- `false` if every applicable carrier was observed and none is `1`.
- `unknown` if none is `1` and at least one applicable carrier was blocked, errored, or (for HTML meta) not read because the status was not a 2xx HTML document.

`tdm_protocol_present` is the same tri-state for “any valid file, or any observed `0` or `1`”, including a reservation of `0`. A `0` is not a TDM Reservation. Hosting a valid `tdmrep.json` array, even an empty one, counts as the protocol file being present.

A 200 HTML document at `/.well-known/tdmrep.json` is not the protocol file (`html_soft`). It does not assert a reservation.

Headers on a 403 are observed. HTML meta on a 403 is not: the body is not treated as the site’s document.

### 2. Robots `noai` / `noimageai` rule

A rule is one of:

- A `robots.txt` directive whose name is `noai` or `noimageai`.
- A `robots.txt` field value that contains the whole token `noai` or `noimageai` (comma- or whitespace-separated). A `User-agent` of exactly `noai` or `noimageai` counts.
- An HTML robots meta (`name` is `robots`, ends with `bot`, or `http-equiv` is `x-robots-tag`) whose content has those tokens.
- An `X-Robots-Tag` response header with those tokens.

`noimageai` is not also `noai`. A path such as `Disallow: /noai/` is not a rule. A comment is not a rule.

Carriers: `robots.txt` body, homepage header, homepage meta, document header, document meta.

Headline `exposes_robots_noai` is `true` if either token appears on an observed carrier. `exposes_noai` and `exposes_noimageai` are recorded separately. `false` means every applicable carrier was observed and neither token appeared. `unknown` means a carrier was blocked or errored and no token was seen elsewhere. A 200 HTML body at `/robots.txt` is an observed response with no robots rule in it, not an unknown file.

### 3. `ai.txt`

Headline location: `https://{site_origin}/ai.txt`.

`exposes_ai_txt` is `true` only when the final status is 200, the body is not HTML, and the body has at least one `User-agent`, `Allow`, or `Disallow` line. That is the Spawning root-file shape.

- 404 or 410: `false` (`absent`).
- 200 HTML, including a soft-404 shell: `false` (`html_soft`).
- 200 non-HTML without those directives: `false` (`unstructured`). The hash is kept for drift.
- 401, 403, 429, 503, timeout, TLS, or DNS failure: `unknown`.

`/.well-known/ai.txt` uses the same classifier and is stored under `secondary`. It is **not** part of `exposes_ai_txt`.

## Change

Each site’s scored tuple is the headline tri-states, the effective TDM values, the `tdmrep.json` rule list, the robots rule excerpts, the `ai.txt` status class, and the `ai.txt` SHA-256 when the status class is `document`.

A **signal change** is any difference in that tuple between slots.

A **body drift** is a SHA-256 change of `robots.txt`, `tdmrep.json`, or an `ai.txt` body that does not change the scored tuple. Drift is reported and is not, by itself, a signal change.

Secondary and unscored fields (`/.well-known/ai.txt`, `Content-Signal`, `Content-Usage`, named crawler full-site disallows) are diffed separately. They do not flip the headline.

`day0` is the baseline. The question’s “over 3 days” is `day0` (19 November), `day1` (20 November), and `day2` (21 November).

## Unscored context

From the same `robots.txt` bytes and response headers, record:

- `Content-Signal` and `Content-Usage` directives and headers.
- User-agents in `unscored-crawlers.json` whose group contains `Disallow: /` or `Disallow: /*` and does not also `Allow: /` or `Allow: /*`.

The crawler list is frozen. Do not extend it mid-study.

## Redaction

Never commit:

- Secret values, `Authorization`, `Cookie`, or `Set-Cookie`.
- Raw client or peer IP addresses. Stored errors, excerpts, and URLs pass an IPv4 and IPv6 redactor. Emails in excerpts become `[email]`.
- Full HTML or PDF bodies. Those stay untracked under `private/` only for the text signal files (`robots.txt`, `ai.txt`, `tdmrep.json`).

Public evidence: `public/research/ai-opt-out-signals-3day/`.

## Analysis plan

After `day2`, and only after `2026-11-21T06:30:00Z`:

- Count `true` / `false` / `unknown` for each headline signal, overall and by stratum.
- Name every site in the `true` and `unknown` sets.
- List signal changes and body drifts across the three slots. If nothing changed, say that the three fetches agreed.
- State limits: research user-agent blocks, single USA cloud path, purposive N=24, path-only TDM matching, root `ai.txt` only for the headline, no browser rendering, no claim that a missing file means the operator authorizes training.
- Screenshots: these signals are files and headers, not banners. Do not capture a homepage to stand in for `robots.txt`. Capture a viewport only if a later slot’s evidence is a rendered page that itself carries the signal.

## Deliverables

- This protocol, `corpus.json`, `schedule.json`, collector, slot JSON, hashes.
- After the stop: `research-article.md` in this directory and under `public/research/ai-opt-out-signals-3day/`, plus the site page.
- Affiliate links `/proxyseller/` and `/cometvpn/`, with a disclosure, belong in that article. They are not placed in the Day 0 evidence. Affiliates did not choose the corpus or the signal definitions. ProxySeller is `NOT_RUN` here. CometVPN is VPN-only and is also a corpus site.

## Deviations log

Append-only after this freeze. None at freeze.
