None of the Pages We Could Read Exposed a Machine-Readable AI Training Opt-Out

Three document cards labeled robots.txt, ai.txt, and tdmrep.json, with FreePolicyTools at the lower left.

The signals in this study are files and headers. The picture names the three places we looked. The counts are in the charts and the public JSON.

0TDM Reservations observed
0noai or noimageai rules
0root ai.txt documents
1scored change: Amazon, false to unknown

The short result

Three cloud-direct crawls, one per day, asked a file question: which of these 24 sites expose a TDM Reservation, a robots noai oder noimageai rule, or a root ai.txt?

The true count is zero on every headline, every day. “Unknown” is a separate bucket. It means a carrier returned 403, or HTML meta could not be read, and no positive signal appeared on a carrier we did read. On 19 November the unknowns were NordVPN (all three headlines) and ExpressVPN (TDM only). On 20 and 21 November Amazon joined them for TDM and robots, because its policy URL changed from HTTP 200 to HTTP 403. Amazon’s /ai.txt stayed a 404.

Day 20 to day 21 changed no scored field. Other files changed bytes. Those changes are body drift: the hash moved, and the extracted signal did not.

What we fetched

Each site, each day: the homepage, /robots.txt, /ai.txt, /.well-known/tdmrep.json, /.well-known/ai.txt, and the observatory document URL with the fragment removed. That is 6 URLs × 24 sites × 3 days = 432 GET requests. Every request returned a status: 211 × 200, 195 × 404, 20 × 403, and 6 × 410. There was no transport error.

The client was Python urllib with TLS on, an empty proxy handler, a 20-second timeout, and at most five redirects. The user-agent was FreePolicyToolsResearch/1.0 (+https://freepolicytools.com/research; ai-opt-out-signals-3day). A 403 was recorded and left alone. We did not retry it with a browser user-agent. Collection ran from a USA cloud datacenter. ProxySeller HTTP, ProxySeller SOCKS, and a CometVPN tunnel were NOT_RUN.

Collection windows. All three slots started on time.
SlotUTC windowRolle
19 November14:56:02–14:57:36First measurement, the same day the corpus froze
20 November06:01:27–06:03:11Scheduled 06:00–06:30 slot
21 November06:01:19–06:03:00Last slot. Stop after 06:30:00.

The origin is https:// plus the host of the observatory URL. www is not folded into the apex. Google is policies.google.com, Meta is www.facebook.com, and Microsoft is privacy.microsoft.com. NordVPN is the one pre-registered exception: the site origin is https://nordvpn.com, and the document stays the privacy PDF on sb.nordcdn.com. The CDN host’s robots and well-known files were not fetched.

Related reading on the prose notice, which this study leaves untouched: Was eine Datenschutzerklärung beinhalten sollte, where to display it, mistakes that show up in the writing, and the Datenschutzrichtliniengenerator.

How a signal counts

TDM Reservation

Under the W3C community report of 10 May 2024, tdm-reservation: 1 means rights reserved and 0 means not reserved. The headline is true only for an observed 1. A 0 would count as the protocol being present and would still leave the reservation headline false. We observed neither value. Carriers, in order, are a matching rule in /.well-known/tdmrep.json, a response header, an HTML meta tag, and PDF XMP. A later observed 0 or 1 overrides an earlier one. Headers on a 403 count. HTML and PDF bodies on a 403 do not. The legal text the protocol is written against is Directive (EU) 2019/790, Article 4. This page records the files. It does not apply the directive to a model or a company.

Robots noai and noimageai

A hit is a directive of that name, a whole token in a field value, a user-agent of exactly that name, an HTML robots meta, or an X-Robots-Tag header. noimageai is its own token. Disallow: /noai/ is a path. The robots exclusion protocol underneath the parser is RFC 9309. Publishers such as DeviantArt have used the noai token as an opt-out. None of the 24 robots files we parsed contained it.

Root ai.txt

The headline file is https://{origin}/ai.txt. It counts only on HTTP 200, a non-HTML body, and at least one User-agent, Allow, oder Disallow line. A 404 or 410 is absent. A 200 HTML page, including a soft-404 shell, is false. /.well-known/ai.txt is stored and is not part of the headline.

Counts across three days

Final day, 21 November. Teal is observed-absent (false). Amber is unknown. The true segment is empty because the true count is 0 of 24 on each headline.

Unknown counts by day

The columns are the size of the unknown set. TDM rose from 2 to 3 when Amazon’s policy URL returned 403. Robots rose from 1 to 2 for the same reason. ai.txt stayed at 1 (NordVPN) because Amazon’s root file remained a 404.

TDM unknown robots unknown ai.txt unknown

Root ai.txt response classes

The same split on all three days. “HTML page” is HTTP 200 with an HTML body, which the protocol scores as false.

  • 20 absent (404 or 410)
  • 3 HTML pages: Surfshark, DuckDuckGo, Meta
  • 1 blocked: NordVPN (403)
Headline counts. True was 0 in every cell.
TagTDM reservationProtocol presentrobots noairoot ai.txt
19 November0 / 22 / 20 / 22 / 20 / 23 / 10 / 23 / 1
20 November0 / 21 / 30 / 21 / 30 / 22 / 20 / 23 / 1
21 November0 / 21 / 30 / 21 / 30 / 22 / 20 / 23 / 1

Counts are true / false / unknown. exposes_noai und exposes_noimageai matched the robots column on every day. No extracted robots rule list contained either token.

Every site on 21 November

“Not observed” means the applicable carriers were read and the signal was absent. “Unknown” means a carrier was blocked or unread. Amazon was “not observed” for TDM and robots on 19 November. Every other row was stable for three days.

Final-day headline by site. No row is an observed opt-out.
StandortStratumTDM reservationrobots noairoot ai.txt
Proton VPNVPNNot observedNot observedNicht vorhanden
Mullvad VPNVPNNot observedNot observedNicht vorhanden
NordVPNVPNUnbekanntUnbekanntBlockiert
SurfsharkVPNNot observedNot observedHTML page
ExpressVPNVPNUnbekanntNot observedNicht vorhanden
IVPNVPNNot observedNot observedNicht vorhanden
CometVPNVPNNot observedNot observedNicht vorhanden
Proxy-SellerProxyNot observedNot observedNicht vorhanden
Bright DataProxyNot observedNot observedNicht vorhanden
OxylabsProxyNot observedNot observedNicht vorhanden
IPRoyalProxyNot observedNot observedNicht vorhanden
WebshareProxyNot observedNot observedNicht vorhanden
BitwardenPrivacy toolNot observedNot observedNicht vorhanden
1PasswordPrivacy toolNot observedNot observedNicht vorhanden
DuckDuckGoPrivacy toolNot observedNot observedHTML page
SignalPrivacy toolNot observedNot observedNicht vorhanden
GoogleMainstreamNot observedNot observedNicht vorhanden
MetaMainstreamNot observedNot observedHTML page
AmazonMainstreamUnbekanntUnbekanntNicht vorhanden
MicrosoftMainstreamNot observedNot observedNicht vorhanden
TermlyLegal techNot observedNot observedNicht vorhanden
iubendaLegal techNot observedNot observedNicht vorhanden
CookieYesLegal techNot observedNot observedNicht vorhanden
DidomiLegal techNot observedNot observedNicht vorhanden

The three unknown cases

NordVPN

Every day, the homepage, /ai.txt, /.well-known/ai.txt, und tdmrep.json returned 403. robots.txt returned 200, text/plain, 15,951 bytes, with no noai token, and the hash never moved. The privacy PDF on sb.nordcdn.com returned 200, 107,311 bytes, with XMP reservation unset. The site-level headlines stay unknown because the blocked carriers were never observed, even though the robots file and the PDF that we did read carried no reservation and no noai Regel.

ExpressVPN

Homepage, policy HTML, and robots.txt (200, 1,136 bytes) were read every day, with no reservation and no noai rule. Root ai.txt was 404, so that headline is false./.well-known/tdmrep.json und /.well-known/ai.txt were 403, application/xml, 111 bytes. The blocked TDM file is why the TDM headline is unknown. The blocked well-known ai.txt is secondary and does not move the headline.

Microsoft, scored false after a cross-host redirect

Every Microsoft request left privacy.microsoft.com and finished on www.microsoft.com. The privacy homepage and privacy statement were HTTP 200. The redirected /en-us/privacy/robots.txt, ai.txt, und tdmrep.json were HTTP 404. Headlines are false on those final responses. The collector did not also request https://www.microsoft.com/robots.txt. That would be a different URL.

The one scored change

Amazon is the only id in diffs/day0-day1.json. Six fields moved from an observed negative to unknown: effective_document, exposes_noai, exposes_noimageai, exposes_robots_noai, exposes_tdm_reservation, und tdm_protocol_present. exposes_ai_txt stayed false. diffs/day1-day2.json has an empty signal_changes list.

Amazon carriers. The policy URL is the field that changed status.
Beförderer19 November20 and 21 November
Homepage200, 882,108 bytes, meta unset200, 3,781 bytes, meta unset
robots.txt200, 7,947 bytes, no noai tokenSame status, size, and hash
/ai.txt404404
tdmrep.json404404
Policy document200, 379,320 bytes, meta unset403, 2,319 bytes, meta unread

The smaller homepage was still HTTP 200 HTML, so its meta was scored as unset. The unknown comes from the policy URL. This is a loss of observation. The robots file did not gain a noai line, and the 27 named crawler disallows in that file did not change.

Bytes that moved

Surfshark, DuckDuckGo, and Meta returned HTTP 200 HTML at /ai.txt every day, and the hash changed between every pair of days. The status class stayed html_soft. DuckDuckGo’s body is an HTML search page. It includes a script parameter named safe_noai. That name is not a robots directive, a robots meta token, or an ai.txt rule. The same host returned HTTP 410 HTML for /.well-known/ai.txt und tdmrep.json every day, which is known absence.

21 of 24 robots.txt files kept the same SHA-256 for three days. Signal (11,109 bytes) and Google (1,652 bytes) changed hash every day at a constant length. Bright Data stayed 609 bytes on the first two days and grew to 945 bytes on 21 November. Extracted noai rules stayed empty, and the unscored diff of content signals and crawler lists stayed empty. Homepage or document HTML sizes also moved, with status still 200, for Surfshark, ExpressVPN, CometVPN, Proxy-Seller, Bright Data, Oxylabs, Bitwarden, 1Password, Google, Meta, Termly, iubenda, and CookieYes. Page size is not a scored field.

What the files contained instead

Five sites sent Cloudflare Content-Signal values ai-input=yes, ai-train=yes, und search=yes: IVPN, Bright Data, Oxylabs, IPRoyal, and CookieYes. CookieYes repeats that same triple 13 times across the robots file and response headers. ai-train=yes is a permission. It was stored as unscored context. Content-Usage was empty on every site. Both diffs report no change in these fields.

Named full-site Disallow: / rules, from the frozen 32-name crawler list, also held still:

  • Meta: Amazonbot, Applebot-Extended, ClaudeBot, Google-Extended, GPTBot, PerplexityBot, PetalBot (7).
  • Amazon: GPTBot, CCBot, PerplexityBot, Google-Extended, ClaudeBot, meta-externalagent, Bytespider, PetalBot, omgili, AI2Bot, Diffbot, DuckAssistBot, Ai2Bot-Dolma, Google-CloudVertexBot, meta-externalfetcher, cohere-ai, YouBot, Claude-User, Claude-SearchBot, Perplexity-User, ChatGPT-User, OAI-SearchBot, Timpibot, Claude-Web, GoogleOther, ICC-Crawler, ImagesiftBot (27).
  • IVPN: Bytespider. CookieYes: PetalBot.

Meta and Amazon published those bot-specific disallows and still had no noai token, no TDM Reservation, and no root ai.txt on the responses we could read. CookieYes’s 404 bodies for the three signal files were truncated at 524,288 bytes. The status was already 404, so those URLs stay absent.

Grenzwerte

  • The research user-agent can itself draw a 403 or a thin 200. Those responses stay unknown. We did not browse them again as a person.
  • One USA cloud path. A residential exit or a browser can see a different status. ProxySeller and CometVPN were not exits.
  • N=24 is the observatory list. It is a purposive set, not a prevalence estimate for the web.
  • Google, Meta, and Microsoft are scored on the policy host the observatory named. www and the apex are different origins.
  • TDM path matching ignores the query string. Amazon’s help-page node id never entered a match, and no tdmrep.json was valid to match against.
  • Only root ai.txt can flip the headline. JavaScript-inserted meta tags were not executed.
  • A missing file is this fetch’s observation. It is neither a grant of training permission nor a reservation. ai-train=yes on five sites is an unscored permission in the other direction.
  • Three days can catch a change inside the window, as Amazon’s document status did. They say nothing about October or December.

Nachweise

Public package at /research/ai-opt-out-signals-3day/. Raw robots, ai.txt, und tdmrep.json bodies stay in untracked private/. Screenshots are omitted because these signals are files and headers.

Suggested citation: FreePolicyTools. “None of the Pages We Could Read Exposed a Machine-Readable AI Training Opt-Out.” Original research collected 19–21 November 2026.

Unterrichtung

This article links to ProxySeller und CometVPN. FreePolicyTools may earn a commission if you buy through those links. Affiliates did not design the protocol, choose the corpus, or see the results before publication. The corpus is the observatory list. Both companies were measured as public websites with the same client as the other 22 sites.

ProxySeller CometVPN

Häufig gestellte Fragen

Did any of the 24 sites expose a TDM Reservation, a noai rule, or ai.txt?

Across 19, 20, and 21 November 2026 the observed count was zero for each headline. NordVPN, ExpressVPN, and Amazon had at least one blocked carrier, so those headlines stay unknown rather than observed-absent.

Why are NordVPN, ExpressVPN, and Amazon unknown?

NordVPN’s homepage, ai.txt, and tdmrep.json returned 403 on every crawl, while robots.txt and the privacy PDF were read. ExpressVPN’s tdmrep.json returned 403 while its HTML and robots.txt were read. Amazon’s policy URL returned 403 on 20 and 21 November after a 200 on 19 November. A blocked carrier keeps the headline unknown when no positive signal was seen elsewhere.

Did Amazon publish an opt-out on the second day?

The scored change is from observed-absent to unknown. robots.txt stayed a 200 text file of 7,947 bytes with no noai token, and /ai.txt stayed 404. The policy URL stopped returning the document.

What is Content-Signal doing in these results?

Five sites sent ai-input=yes, ai-train=yes, and search=yes. That is a machine-readable permission. It is recorded as unscored context and does not count as a TDM Reservation, a noai rule, or ai.txt.

Does a missing file mean the site allows AI training?

A missing file records that this fetch did not find the signal. It does not record permission to train, and it does not record a reservation.

Did ProxySeller or CometVPN carry this crawl?

Both exits were NOT_RUN. Collection was cloud-direct from a USA datacenter. CometVPN and Proxy-Seller were fetched only as two of the 24 public sites. The affiliate links are separate from the measurements.

Maschinell übersetzt