None of the Pages We Could Read Exposed a Machine-Readable AI Training Opt-Out
The signals in this study are files and headers. The picture names the three places we looked. The counts are in the charts and the public JSON.
The short result
Three cloud-direct crawls, one per day, asked a file question: which of these 24 sites expose a TDM Reservation, a robots noai ou noimageai rule, or a root ai.txt?
The true count is zero on every headline, every day. “Unknown” is a separate bucket. It means a carrier returned 403, or HTML meta could not be read, and no positive signal appeared on a carrier we did read. On 19 November the unknowns were NordVPN (all three headlines) and ExpressVPN (TDM only). On 20 and 21 November Amazon joined them for TDM and robots, because its policy URL changed from HTTP 200 to HTTP 403. Amazon’s /ai.txt stayed a 404.
Day 20 to day 21 changed no scored field. Other files changed bytes. Those changes are body drift: the hash moved, and the extracted signal did not.
What we fetched
Each site, each day: the homepage, /robots.txt, /ai.txt, /.well-known/tdmrep.json, /.well-known/ai.txt, and the observatory document URL with the fragment removed. That is 6 URLs × 24 sites × 3 days = 432 GET requests. Every request returned a status: 211 × 200, 195 × 404, 20 × 403, and 6 × 410. There was no transport error.
The client was Python urllib with TLS on, an empty proxy handler, a 20-second timeout, and at most five redirects. The user-agent was FreePolicyToolsResearch/1.0 (+https://freepolicytools.com/research; ai-opt-out-signals-3day). A 403 was recorded and left alone. We did not retry it with a browser user-agent. Collection ran from a USA cloud datacenter. ProxySeller HTTP, ProxySeller SOCKS, and a CometVPN tunnel were NOT_RUN.
| Slot | UTC window | Rôle |
|---|---|---|
| 19 November | 14:56:02–14:57:36 | First measurement, the same day the corpus froze |
| 20 November | 06:01:27–06:03:11 | Scheduled 06:00–06:30 slot |
| 21 November | 06:01:19–06:03:00 | Last slot. Stop after 06:30:00. |
The origin is https:// plus the host of the observatory URL. www is not folded into the apex. Google is policies.google.com, Meta is www.facebook.com, and Microsoft is privacy.microsoft.com. NordVPN is the one pre-registered exception: the site origin is https://nordvpn.com, and the document stays the privacy PDF on sb.nordcdn.com. The CDN host’s robots and well-known files were not fetched.
Related reading on the prose notice, which this study leaves untouched: ce qu'une politique de confidentialité devrait inclure, where to display it, mistakes that show up in the writing, and the générateur de politique de confidentialité.
How a signal counts
TDM Reservation
Under the W3C community report of 10 May 2024, tdm-reservation: 1 means rights reserved and 0 means not reserved. The headline is true only for an observed 1. A 0 would count as the protocol being present and would still leave the reservation headline false. We observed neither value. Carriers, in order, are a matching rule in /.well-known/tdmrep.json, a response header, an HTML meta tag, and PDF XMP. A later observed 0 or 1 overrides an earlier one. Headers on a 403 count. HTML and PDF bodies on a 403 do not. The legal text the protocol is written against is Directive (EU) 2019/790, Article 4. This page records the files. It does not apply the directive to a model or a company.
Robots noai and noimageai
A hit is a directive of that name, a whole token in a field value, a user-agent of exactly that name, an HTML robots meta, or an X-Robots-Tag header. noimageai is its own token. Disallow: /noai/ is a path. The robots exclusion protocol underneath the parser is RFC 9309. Publishers such as DeviantArt have used the noai token as an opt-out. None of the 24 robots files we parsed contained it.
Root ai.txt
The headline file is https://{origin}/ai.txt. It counts only on HTTP 200, a non-HTML body, and at least one User-agent, Allowou Disallow line. A 404 or 410 is absent. A 200 HTML page, including a soft-404 shell, is false. /.well-known/ai.txt is stored and is not part of the headline.
Counts across three days
Final day, 21 November. Teal is observed-absent (false). Amber is unknown. The true segment is empty because the true count is 0 of 24 on each headline.
21 November, true / false / unknown
Unknown counts by day
The columns are the size of the unknown set. TDM rose from 2 to 3 when Amazon’s policy URL returned 403. Robots rose from 1 to 2 for the same reason. ai.txt stayed at 1 (NordVPN) because Amazon’s root file remained a 404.
Root ai.txt response classes
The same split on all three days. “HTML page” is HTTP 200 with an HTML body, which the protocol scores as false.
sites
- 20 absent (404 or 410)
- 3 HTML pages: Surfshark, DuckDuckGo, Meta
- 1 blocked: NordVPN (403)
| Jour | TDM reservation | Protocol present | robots noai | root ai.txt |
|---|---|---|---|---|
| 19 November | 0 / 22 / 2 | 0 / 22 / 2 | 0 / 23 / 1 | 0 / 23 / 1 |
| 20 November | 0 / 21 / 3 | 0 / 21 / 3 | 0 / 22 / 2 | 0 / 23 / 1 |
| 21 November | 0 / 21 / 3 | 0 / 21 / 3 | 0 / 22 / 2 | 0 / 23 / 1 |
Counts are true / false / unknown. exposes_noai et exposes_noimageai matched the robots column on every day. No extracted robots rule list contained either token.
Every site on 21 November
“Not observed” means the applicable carriers were read and the signal was absent. “Unknown” means a carrier was blocked or unread. Amazon was “not observed” for TDM and robots on 19 November. Every other row was stable for three days.
| Site | Stratum | TDM reservation | robots noai | root ai.txt |
|---|---|---|---|---|
| Proton VPN | VPN | Not observed | Not observed | Sans objet |
| Mullvad VPN | VPN | Not observed | Not observed | Sans objet |
| NordVPN | VPN | Inconnu | Inconnu | Bloqué |
| Surfshark | VPN | Not observed | Not observed | HTML page |
| ExpressVPN | VPN | Inconnu | Not observed | Sans objet |
| IVPN | VPN | Not observed | Not observed | Sans objet |
| CometVPN | VPN | Not observed | Not observed | Sans objet |
| Proxy-Seller | Proxy | Not observed | Not observed | Sans objet |
| Bright Data | Proxy | Not observed | Not observed | Sans objet |
| Oxylabs | Proxy | Not observed | Not observed | Sans objet |
| IPRoyal | Proxy | Not observed | Not observed | Sans objet |
| Webshare | Proxy | Not observed | Not observed | Sans objet |
| Petits | Privacy tool | Not observed | Not observed | Sans objet |
| 1Password | Privacy tool | Not observed | Not observed | Sans objet |
| DuckDuckGo | Privacy tool | Not observed | Not observed | HTML page |
| Signal | Privacy tool | Not observed | Not observed | Sans objet |
| Mainstream | Not observed | Not observed | Sans objet | |
| Meta | Mainstream | Not observed | Not observed | HTML page |
| Amazon | Mainstream | Inconnu | Inconnu | Sans objet |
| Microsoft | Mainstream | Not observed | Not observed | Sans objet |
| Termly | Legal tech | Not observed | Not observed | Sans objet |
| iubenda | Legal tech | Not observed | Not observed | Sans objet |
| CookieYes | Legal tech | Not observed | Not observed | Sans objet |
| Didomi | Legal tech | Not observed | Not observed | Sans objet |
The three unknown cases
NordVPN
Every day, the homepage, /ai.txt, /.well-known/ai.txtet tdmrep.json returned 403. robots.txt returned 200, text/plain, 15,951 bytes, with no noai token, and the hash never moved. The privacy PDF on sb.nordcdn.com returned 200, 107,311 bytes, with XMP reservation unset. The site-level headlines stay unknown because the blocked carriers were never observed, even though the robots file and the PDF that we did read carried no reservation and no noai règle.
ExpressVPN
Homepage, policy HTML, and robots.txt (200, 1,136 bytes) were read every day, with no reservation and no noai rule. Root ai.txt was 404, so that headline is false./.well-known/tdmrep.json et /.well-known/ai.txt were 403, application/xml, 111 bytes. The blocked TDM file is why the TDM headline is unknown. The blocked well-known ai.txt is secondary and does not move the headline.
Microsoft, scored false after a cross-host redirect
Every Microsoft request left privacy.microsoft.com and finished on www.microsoft.com. The privacy homepage and privacy statement were HTTP 200. The redirected /en-us/privacy/robots.txt, ai.txtet tdmrep.json were HTTP 404. Headlines are false on those final responses. The collector did not also request https://www.microsoft.com/robots.txt. That would be a different URL.
The one scored change
Amazon is the only id in diffs/day0-day1.json. Six fields moved from an observed negative to unknown: effective_document, exposes_noai, exposes_noimageai, exposes_robots_noai, exposes_tdm_reservationet tdm_protocol_present. exposes_ai_txt stayed false. diffs/day1-day2.json has an empty signal_changes list.
| Transporteur | 19 November | 20 and 21 November |
|---|---|---|
| Page d'accueil | 200, 882,108 bytes, meta unset | 200, 3,781 bytes, meta unset |
| robots.txt | 200, 7,947 bytes, no noai token | Same status, size, and hash |
| /ai.txt | 404 | 404 |
| tdmrep.json | 404 | 404 |
| Policy document | 200, 379,320 bytes, meta unset | 403, 2,319 bytes, meta unread |
The smaller homepage was still HTTP 200 HTML, so its meta was scored as unset. The unknown comes from the policy URL. This is a loss of observation. The robots file did not gain a noai line, and the 27 named crawler disallows in that file did not change.
Bytes that moved
Surfshark, DuckDuckGo, and Meta returned HTTP 200 HTML at /ai.txt every day, and the hash changed between every pair of days. The status class stayed html_soft. DuckDuckGo’s body is an HTML search page. It includes a script parameter named safe_noai. That name is not a robots directive, a robots meta token, or an ai.txt rule. The same host returned HTTP 410 HTML for /.well-known/ai.txt et tdmrep.json every day, which is known absence.
21 of 24 robots.txt files kept the same SHA-256 for three days. Signal (11,109 bytes) and Google (1,652 bytes) changed hash every day at a constant length. Bright Data stayed 609 bytes on the first two days and grew to 945 bytes on 21 November. Extracted noai rules stayed empty, and the unscored diff of content signals and crawler lists stayed empty. Homepage or document HTML sizes also moved, with status still 200, for Surfshark, ExpressVPN, CometVPN, Proxy-Seller, Bright Data, Oxylabs, Bitwarden, 1Password, Google, Meta, Termly, iubenda, and CookieYes. Page size is not a scored field.
What the files contained instead
Five sites sent Cloudflare Content-Signal values ai-input=yes, ai-train=yeset search=yes: IVPN, Bright Data, Oxylabs, IPRoyal, and CookieYes. CookieYes repeats that same triple 13 times across the robots file and response headers. ai-train=yes is a permission. It was stored as unscored context. Content-Usage was empty on every site. Both diffs report no change in these fields.
Named full-site Disallow: / rules, from the frozen 32-name crawler list, also held still:
- Meta: Amazonbot, Applebot-Extended, ClaudeBot, Google-Extended, GPTBot, PerplexityBot, PetalBot (7).
- Amazon: GPTBot, CCBot, PerplexityBot, Google-Extended, ClaudeBot, meta-externalagent, Bytespider, PetalBot, omgili, AI2Bot, Diffbot, DuckAssistBot, Ai2Bot-Dolma, Google-CloudVertexBot, meta-externalfetcher, cohere-ai, YouBot, Claude-User, Claude-SearchBot, Perplexity-User, ChatGPT-User, OAI-SearchBot, Timpibot, Claude-Web, GoogleOther, ICC-Crawler, ImagesiftBot (27).
- IVPN: Bytespider. CookieYes: PetalBot.
Meta and Amazon published those bot-specific disallows and still had no noai token, no TDM Reservation, and no root ai.txt on the responses we could read. CookieYes’s 404 bodies for the three signal files were truncated at 524,288 bytes. The status was already 404, so those URLs stay absent.
Limites
- The research user-agent can itself draw a 403 or a thin 200. Those responses stay unknown. We did not browse them again as a person.
- One USA cloud path. A residential exit or a browser can see a different status. ProxySeller and CometVPN were not exits.
- N=24 is the observatory list. It is a purposive set, not a prevalence estimate for the web.
- Google, Meta, and Microsoft are scored on the policy host the observatory named.
wwwand the apex are different origins. - TDM path matching ignores the query string. Amazon’s help-page node id never entered a match, and no
tdmrep.jsonwas valid to match against. - Only root
ai.txtcan flip the headline. JavaScript-inserted meta tags were not executed. - A missing file is this fetch’s observation. It is neither a grant of training permission nor a reservation.
ai-train=yeson five sites is an unscored permission in the other direction. - Three days can catch a change inside the window, as Amazon’s document status did. They say nothing about October or December.
Preuves
Public package at /research/ai-opt-out-signals-3day/. Raw robots, ai.txtet tdmrep.json bodies stay in untracked private/. Screenshots are omitted because these signals are files and headers.
protocol.md24 origins and document URLs
corpus.json19 November crawl
slots/day0.json20 November crawl
slots/day1.json21 November crawl
slots/day2.jsonAmazon’s false-to-unknown change
diffs/day0-day1.jsonEmpty signal-change list
diffs/day1-day2.jsonCounts and both diffs
summary.jsonThis article as Markdown
research-article.mdPackage checksums
manifest.sha256
Suggested citation: FreePolicyTools. “None of the Pages We Could Read Exposed a Machine-Readable AI Training Opt-Out.” Original research collected 19–21 November 2026.
Divulgation
This article links to ProxySeller et CometVPN. FreePolicyTools may earn a commission if you buy through those links. Affiliates did not design the protocol, choose the corpus, or see the results before publication. The corpus is the observatory list. Both companies were measured as public websites with the same client as the other 22 sites.
Foire aux questions
Did any of the 24 sites expose a TDM Reservation, a noai rule, or ai.txt?
Across 19, 20, and 21 November 2026 the observed count was zero for each headline. NordVPN, ExpressVPN, and Amazon had at least one blocked carrier, so those headlines stay unknown rather than observed-absent.
Why are NordVPN, ExpressVPN, and Amazon unknown?
NordVPN’s homepage, ai.txt, and tdmrep.json returned 403 on every crawl, while robots.txt and the privacy PDF were read. ExpressVPN’s tdmrep.json returned 403 while its HTML and robots.txt were read. Amazon’s policy URL returned 403 on 20 and 21 November after a 200 on 19 November. A blocked carrier keeps the headline unknown when no positive signal was seen elsewhere.
Did Amazon publish an opt-out on the second day?
The scored change is from observed-absent to unknown. robots.txt stayed a 200 text file of 7,947 bytes with no noai token, and /ai.txt stayed 404. The policy URL stopped returning the document.
What is Content-Signal doing in these results?
Five sites sent ai-input=yes, ai-train=yes, and search=yes. That is a machine-readable permission. It is recorded as unscored context and does not count as a TDM Reservation, a noai rule, or ai.txt.
Does a missing file mean the site allows AI training?
A missing file records that this fetch did not find the signal. It does not record permission to train, and it does not record a reservation.
Did ProxySeller or CometVPN carry this crawl?
Both exits were NOT_RUN. Collection was cloud-direct from a USA datacenter. CometVPN and Proxy-Seller were fetched only as two of the 24 public sites. The affiliate links are separate from the measurements.