When did websites start blocking AI crawlers?
A dated, population-level measurement in 21 epochs — January 2019 to September 2026.
Through January 2021, none of the sites we sampled expressed any AI-era stance in
robots.txt: 0.0%, 95% confidence interval 0.0–4.9%, n=74. That remained true
across six consecutive epochs spanning January 2019 to June 2023, every one
of them at zero, with no interval reaching above 5.3%. The first
movement we observe is October 2023, at 3.8% (95% CI 1.3–10.7%, n=78). By
September 2026 the figure is 14.1% (95% CI 8.3–23.1%, n=85).
⚠️ Those zeros are an upper bound, not a zero. Nought of 74 sites does not
establish that nothing existed; it bounds the rate below about 5%, and a 5% ceiling across the
whole web still admits a very large number of sites. The defensible claim is that the practice
was too rare to appear in a sample of this size, repeatedly and across years — not that
it did not exist. ⚠️ And the series supports the long-run rise only: all 105 pairs of
post-2023 epochs have overlapping confidence intervals, so no single step between two
rows is supported by this data. The fall from 8.6% to 4.8% between March 2024 and January 2025,
for instance, is three sites of movement on an n of about 82, and should be read as noise.
This page exists because the question has a legal form. Where a dispute turns on whether a
reservation of rights was machine-readable at the time of use, the test is the
technological state of the art at that past date — not today's. The policy
literature on machine-readable AI opt-outs names the standards but publishes no adoption
figures. These are figures.
The series
Share of sampled sites whose robots.txt names or blocks an AI-era
agent. Vantage is held constant — every row is fetched from a single archive source — and Wilson
95% intervals are shown. The cohort is not constant, and a constant vantage is not a
constant cohort. The sample moves between 69 and 85 sites as the archive's coverage of
our target list varies, so an unknown part of any movement between rows is composition rather
than time. We do not offer a fixed-panel series because only 6 of our targets appear in
all 22 epochs, which would carry no useful precision. Read the long-run shape, not the
rows.
| epoch | observed | n | rate | 95% CI | blocks | names |
|---|
| CC-MAIN-2019-04 | 2019-01-24 | 69 | 0.0% | 0.0–5.3% | 0 | 0 |
| CC-MAIN-2020-05 | 2020-01-29 | 71 | 0.0% | 0.0–5.1% | 0 | 0 |
| CC-MAIN-2021-04 | 2021-01-28 | 74 | 0.0% | 0.0–4.9% | 0 | 0 |
| CC-MAIN-2022-05 | 2022-01-29 | 77 | 0.0% | 0.0–4.8% | 0 | 0 |
| CC-MAIN-2023-06 | 2023-02-09 | 78 | 0.0% | 0.0–4.7% | 0 | 0 |
| CC-MAIN-2023-23 | 2023-06-10 | 78 | 0.0% | 0.0–4.7% | 0 | 0 |
| CC-MAIN-2023-40 | 2023-10-05 | 78 | 3.8% | 1.3–10.7% | 3 | 0 |
| CC-MAIN-2024-10 | 2024-03-05 | 81 | 8.6% | 4.2–16.8% | 7 | 0 |
| CC-MAIN-2025-05 | 2025-01-26 | 83 | 4.8% | 1.9–11.7% | 3 | 1 |
| CC-MAIN-2025-26 | 2025-06-25 | 81 | 6.2% | 2.7–13.6% | 4 | 1 |
| CC-MAIN-2025-38 | 2025-09-18 | 79 | 3.8% | 1.3–10.6% | 3 | 0 |
| CC-MAIN-2025-51 | 2025-12-17 | 79 | 6.3% | 2.7–14.0% | 5 | 0 |
| CC-MAIN-2026-04 | 2026-01-25 | 81 | 6.2% | 2.7–13.6% | 5 | 0 |
| CC-MAIN-2026-08 | 2026-02-19 | 81 | 4.9% | 1.9–12.0% | 4 | 0 |
| CC-MAIN-2026-12 | 2026-03-17 | 82 | 7.3% | 3.4–15.1% | 5 | 1 |
| CC-MAIN-2026-17 | 2026-04-23 | 83 | 8.4% | 4.1–16.4% | 5 | 2 |
| CC-MAIN-2026-21 | 2026-05-21 | 82 | 9.8% | 5.0–18.1% | 4 | 4 |
| CC-MAIN-2026-25 | 2026-06-18 | 84 | 13.1% | 7.5–21.9% | 6 | 5 |
| CC-MAIN-2026-30 | 2026-07-22 | 84 | 13.1% | 7.5–21.9% | 6 | 5 |
| CC-MAIN-2026-34 | 2026-08-19 | 82 | 15.9% | 9.5–25.3% | 6 | 7 |
| CC-MAIN-2026-39 | 2026-09-16 | 85 | 14.1% | 8.3–23.1% | 7 | 5 |
One further epoch, CC-MAIN-2024-33 (2024-08-16), is withheld rather than
dropped: it covers only 6 sites, too few to state a rate. Reporting it would have
implied a precision the sample does not carry; deleting it silently would have hidden a hole.
The absence was durable, not a momentary lapse
A zero in a single year could be an artefact of when one looked. Two things argue it was a
settled norm rather than a gap that happened to be open.
First, it held across six consecutive epochs, January 2019 through June
2023, every one of them at zero, with no interval reaching above 5.3%.
Second, we measured separately how often such gaps close on their own. Across roughly 1,992
hosts watched over 31.8 days in 18 capability classes, a gap that is open at one observation is
closed at the next at a rate of 0.021% per observation pair (95% CI
0.016–0.028%), and slightly more gaps opened than closed over the window. These declarations do
not appear spontaneously. The norm held because nothing was pushing on it.
Method, and what this does not establish
- Sample, not census. 69–85 sites per epoch, drawn from a Tranco-ranked
list and stratified across four size bands (enterprise, mid-market, independent,
long-tail). This is not "the web", and a different sample would give a different
number. ⚠️ Nor is the cohort identical between epochs: the target list is fixed, but
which of those targets the archive captured varies, so the rows are comparable in
vantage and in method, and only approximately in membership.
- The history is a re-derivation, not a contemporaneous observation. Epochs
before 2026-09-20 are reconstructed from Common Crawl, a public archive. We did not watch
the web in 2021; we read what a public crawler recorded then. That is reproducible by
anyone — which is a virtue in evidence, not a weakness, but it is not the same as having
been there.
- Our signatures date our computation, not the web. The record is
append-only and signed, which shows we have not altered it since. It cannot, by itself,
establish to an opponent that the underlying source said what we say it said. The source
is citable independently.
- 🔴 A selection effect we cannot measure, in the direction that flatters this
page. The history is derived from a public crawl, and a site that discourages
crawlers is, in general, less likely to be well represented in a crawl. That would bias the
measured rate downward — which is the direction that favours the argument this page
is most often wanted for. Two things soften it without removing it: the targets are chosen
from a Tranco rank list rather than from the crawl itself, so the crawl selects captures and
not the cohort; and a crawler fetches
robots.txt precisely in order to obey it,
so that file is captured more reliably than ordinary pages. Within our sample we find no
thinning — sites that declare a stance today average slightly more archive coverage,
20.1 epochs against 18.3 — but n is 9, so that settles nothing, and by construction the sites
a crawl missed entirely cannot appear in our data at all. Treat every rate here as an
upper bound on absence.
- Single vantage. Every row above is fetched from one source, deliberately.
Mixing vantages makes part of a step a change in where rather than
when.
- "Names or blocks an AI-era agent" is the measured thing. It is not a legal
conclusion about whether any reservation was valid, effective or binding, and nothing here
speaks to any individual named site. Figures are published in aggregate only.
The machine-readable form
The same series, with its caveats attached, is served as JSON at
/free/curve — free, no credential, no payment. Agents and
scripts should read that rather than parse this page.
If you need this for a matter
A norm-at-a-date assay for a defined population and a specific date is work we do on request:
the population stated, the interval attached, the method written down so it can be recomputed by
the other side. Write to info@geosensor.tech.
We will say when a question cannot be answered from this record. The commonest
case: a class with no dated history. Of the capability classes we measure, only this one has a
series reaching back before 2026 — including, notably, the machine-readable AI usage
policy class (ai.txt, TDMRep), for which we hold no history at all and will not pretend to.