Posts tagged "dataset"

25 posts found.

September 05, 2026 28 min read

How Long the Internet Remembers: 43.6 Billion DNS Answers, 40 Months of TTLs, and the Two Companies That Set the Cache Clock for a Third of the Web

We kept the exact TTL of every A record in 28 full crawls of the DomainsProject corpus, April 2023 to August 2026 — 43.6 billion DNS answers, 6.9 billion of them the address record of a registrable domain. The share of domains whose address expires in five minutes or less rose from 21.27% to 27.28%, and among live (non-parked) sites from 23.00% to 34.03%, almost all of it since March 2025. But 60.8% of surviving domains publish the same value 40 months later: GoDaddy's domaincontrol.com (mode 600) and Cloudflare (mode 300, 89% of its zones) set the number for a third of the web, and Cloudflare alone accounts for 47% of every domain at five minutes or less. For DNS and platform operators, SRE and incident-response teams, and Internet measurement researchers.

August 20, 2026 39 min read

MX Flux: Sixteen Months of Email Records for 422 Million Domains — 71% Never Touched Their MX, Google's Lead Grows Without Switchers, and 13 Million Domains Now Refuse Mail

We followed the MX record of every apex domain across eleven full crawls, April 2025 - August 2026 (421.9 million apexes). 70.8% of mail-capable domains sit on the bit-identical MX record set sixteen months later; provider switching runs under 1% a month. Google Workspace's lead over Microsoft 365 widened at every single pass (1.28 to 1.43 by domain count) - yet direct switching between the two is a wash; the gap is made by new domains, which arrive 2.3-to-1 for Google, while Microsoft wins the migrations out of self-hosting. Null MX quadrupled from 3.4M to 13.4M domains, and 77% of it is one aftermarket platform's zone template. For email and security teams, hosting providers, and Internet measurement researchers.

July 23, 2026 22 min read

Who Moves the World's DNS: 15 Months of Delegation Flux, a One-Month Million-Domain Migration, and the Nameservers That Outlive Their Owners

We followed the NS delegation of every apex domain across nine full crawls, April 2025 - July 2026 (404 million apexes). 61% of the web kept the exact same nameservers for fifteen months; operator switching runs at 2.3% a month; and the biggest moves were corporate re-plumbs - GoDaddy consolidating 2.1 million NameFind domains onto Afternic, Namecheap shifting 1.1 million parked names to an undocumented platform in one month. Meanwhile 339,000 domains still delegate to the retired Dan.com. For DNS operators, security teams, and Internet measurement researchers.

July 18, 2026 21 min read

The SaaS Ledger: 15 Months of TXT Records, a 625x AI Adoption Curve, and Five Million Tokens Moved by One Corporate Decision

We re-classified 18 billion DNS TXT queries across ten full crawls (April 2025 - July 2026) and turned the SaaS verification-token census into a time series. Anthropic tokens grew 625x to 131,812 apexes. The Dan.com shutdown moved five million marketplace tokens. Microsoft 365's domain share is flat while Google's climbs. And across every major vendor, only 19-29% of token disappearances are deliberate cleanup - the rest is domain death. For SaaS analysts, security teams, and Internet measurement researchers.

July 11, 2026 17 min read

The Restless Aliases: the CNAME Layer re-crawled after 32 Days — 92.5% Held Still, 12.6 Million Rewired or Vanished, and the Biggest Moves Were Made by Platforms, Not People

Two CNAME-typed crawls of 1.9 billion hostnames, 32 days apart: 169 million cross-domain aliases followed by name. 7.5% of the alias layer rewires or disappears in one month, parking lost 1.54M aliases, one Chinese registrar generated 13% of all new aliases — and the three biggest 'migrations' were platforms re-pointing their own chains, dragging half a million customer domains along. For DNS operators, measurement researchers, and anyone who builds on someone else's CNAME.

July 10, 2026 18 min read

The Parking Lot: We Measured the Parked Web Every Month for 38 Months — It Grew to One Domain in Four, Peaked Five Days After Its Business Model Died, and Has Been Shrinking Since

26 monthly DNS snapshots, April 2023 to June 2026: domains pointing at parking, for-sale, expiry and registrar-default landers grew from 16.0% to a peak of 25.5% of all website records — 73.6 million domains — then began the first sustained decline on record, in the exact quarter Google switched off the parking ad economy and Bodis shut down. GoDaddy front doors hold three-quarters of the lot. For measurement researchers, registrars, security teams, and anyone who counts 'websites'.

July 06, 2026 22 min read

Where the Web Moves: We Followed 110 Million Domains for 38 Months — Two-Thirds Never Left Their Host, and One in Four Domains Now Points at a Parking Lot

We compared the website address of every domain that resolved in both April 2023 and June 2026 — 110.6 million survivors — and classified both endpoints by BGP network. 46.8% answer from the exact same IP three years later; 67.7% never left their provider. The biggest migration wasn't chosen by any webmaster: GoDaddy moved its 10-million-domain lander estate from Google Cloud to AWS anycast, and the parked web doubled to 24.8% of all site records. For hosting strategists, registrars, measurement researchers, and anyone who thinks the cloud wars are moving the web.

July 03, 2026 20 min read

The AI Crawler Census: 14 Bots, 97,454 Requests, and More Knocks Than Every Search Engine Combined

We logged 1,348,706 requests across two production web properties for 174 days and classified every user-agent. AI crawlers now generate 7.2% of all observed traffic — more than Googlebot, Bingbot, DuckDuckBot, Baiduspider, and PetalBot combined. OpenAI alone accounts for 38% of the AI crawl, 69% of AI requests still feed training pipelines, and a robots.txt honeypot shows 99% of AI-crawler violations trace to a single bot. For publishers, infrastructure teams, and anyone deciding what to do about the new tenants of the crawl economy.

July 02, 2026 18 min read

The Half-Life of a Domain: We Tracked 181 Million Resolving Websites for 38 Months — 63.5% Are Still Alive

We followed every one of the 181.1 million domains that resolved in April 2023 across 26 DNS snapshots to June 2026, and did the same for a 246.8-million-domain 2025 cohort. The resolving web loses 13-15% of itself every year — a half-life of roughly five years — but the deaths are wildly unequal: Freenom's five TLDs went 99% extinct in the largest die-off ever recorded in the DNS, the cheap-promo gTLDs lose up to 95% of their web in three years, and Germany's .de keeps 84% alive. For registrars, investors, security teams, and anyone who links to anything.

Page 1 of 3 Older →