DomainsProject.org news feed
Added new 2.8T (133 billion) records for Freya bringing total generated data to 10.2T / 517 billion records
Freya DNS worker surpassed 180 billion processed records
Autovacuum tool has removed over 42 million of invalid records (DNS catchers) reducing dataset size to 342 million records.
- After adding some generic subdomains (.com.xx, .net.xx, etc) resulting dataset grew significantly. Machine ran out of disk space at 3.4T new (7.4T total) / 384 billion records.
- At least several other registrars (
.fmis a known culprit) are doing a dirty trick for non-existent domains. Special thanks to community for catching this.
- 54,081,701 new words for dataset. At least 82,312,348,922 new domains (1.6T) to check, which brings total generated dataset to pretty serious 5.6T.
- crawler code is now closed source and used internally. Most of the job is now done by Freya
- 4.0T dataset of generated DNS names is now being processed. Return is small, about 8-10k domains per 1 million records.
- Some of those are already in the database, so 212 billion records are expected to yield about 20 million new domains.
- There’s a separate process, called
autovacuum, running on a regular basis. It cleans up dataset from unreachable (expired, servfail, etc.) domains.
- Added this news file :-)