Specimen assay AS-2026-11-04
Issued 2026-11-04T09:00:00Z · this is the whole deliverable, not a summary of one
What share of this population took a stance on AI agents in its robots.txt, as at the declared date?
Of 639 hosts in the declared population, 152 took one: 23.79%, 95% confidence interval 20.65%–27.24%, as at 2026-11-04T09:00:00Z.
Everything needed to disagree with that number is on this page: the population and its denominator rule, the window, the evidence count, what was excluded and why, and the one command that recomputes it from the signed chain.
What was measured
Capability class PH-C005 / PH-C006 — whether a site's robots.txt takes any stance on AI agents — naming one, or blocking one. PH-C005 is silence on AI agents; PH-C006 is blocking an AI-era agent.
The instrument is the robots_stance probe. The question asked of it:
What share of this population took a stance on AI agents in its robots.txt, as at the declared date?
The population, and why it is not 678
Declared as: hosts in this index's live rotation — agent-native hosts and the brochure corpus — that were observed inside the freshness window.
678 hosts produced an observation inside the window.
39 of them returned nothing we could classify — a transport
failure, not an answer — so they are recorded unknown and counted neither as
present nor as absent. That leaves 639 hosts in the cross-section.
Absence of a machine-readable signal is an upper bound on absence of the
capability, never proof of it. Where the instrument cannot tell the two apart, the record says
unknown and the host leaves the denominator.
The window
A host is in the cross-section only if it was observed within 7 days of 2026-11-04T09:00:00Z
| oldest observation counted | 2026-11-01T21:35:06Z |
|---|---|
| newest observation counted | 2026-11-04T07:51:25Z |
| median observation | 2026-11-04T03:43:17Z |
The result
| took a stance | 152 |
|---|---|
| of hosts counted | 639 |
| rate | 23.79% |
| 95% interval | 20.65% – 27.24% |
| interval method | Wilson score interval, z=1.96 |
| of which, naming an AI agent | 96 |
| of which, blocking one | 56 |
| minimum n before a figure is withheld | 30 |
The evidence
18,932 signed observation records lie inside the window, chain sequence
882,249 to 1,130,950: 15,072 absent, 2,687 present, 1,173 unknown.
More records than hosts, because a host in rotation is observed repeatedly; the estimator takes one cell per host — the newest observation at or before the issue instant.
Provenance
At issue the chain held 1,134,150 records, head
50ebf1b545d6f714c8c795bf5bf75c142533dfb30e8543a3f4188f0b326c8583. A single appending host; every record is ed25519-signed and carries its predecessor's hash.
The chain head is published daily and saved at the Internet Archive, so a reader can confirm the chain existed in this state on that date without taking our word for it. This is the anchor nearest this specimen; it is not this specimen's own head.
Nearest anchor: 2026-11-03T12:43:09Z, head 3958eb215031463242c8b9d84ee6a097aaf51e8963b99229aedff63ecf36826a
at 1,092,600 records.
What was excluded, and why
1,340 hosts were excluded as stale. Last observed outside the window. Their verdicts are real and remain in the chain; they are simply not evidence about today, so they are not averaged into a rate presented as current.
Those hosts' own rate is 26.72%, and it must not be read against the figure above. This rate and the cross-section rate are over DISJOINT host sets, not the same hosts at two dates, so their difference is a sampling difference and must not be read as change over time. The series above is the only object in this document that measures change.
Pooling both sets gives 25.77% over 1,979 hosts. We do not quote that number. Every host ever probed live, whenever probed, in one rate: not restricted to the freshness window and therefore not a statement about today. Shown for comparison, NOT for quoting.
Recompute it
The estimator call, verbatim:
PYTHONPATH=. python3 - <<'EOF'
import pathlib
from phantom.curve import live_cross_section
print(live_cross_section(pathlib.Path('data'), now=1793782800.0))
EOF
Given the chain, this returns the figures above exactly. A later observation of the same host does not change them: the estimator is bounded at the issue instant, and tests/test_assay_reproducible.py grows a chain past that instant to prove it.
Limits, stated before you pay
- A rate is a rate for the declared target set. It is a sample, not a census.
- Absence of a machine-readable signal is an upper bound on absence of the capability, not proof of it. Where the instrument cannot tell the two apart the record says
unknown. - Measurement is from a single vantage. A series that mixed vantages would mix WHERE with WHEN.
- History before 2026-09-20 is reconstructed from a public web archive, not observed contemporaneously.
What is never sold
- Findings about an individual named third party's own site or conduct, at any price. Everything we observe is published and delivered in aggregate. A measurement that a third party has published itself is a public claim rather than an observation of ours, and may be cited and examined by name, as any review cites its sources.
- A figure to order. The number is whatever the signed record says for the date asked about. If it does not support the claim a buyer wants, that IS the finding, and we say so.
- A measurement that cannot be recomputed. If the method cannot be published with the result, the result is not for sale.
A signed assay is one capability class across one declared population at one declared date, delivered with the method so the result can be recomputed against us. Scope is agreed in writing first. What an engagement costs, and on what terms, is set out on one page.