DomainsProject.org news feed
Freya DNS worker has surpassed 500 billion (9.8T) processed records
Dataset is back to 380+ million records. ipscan utility is periodically scanning 3.7b IPv4 addresses for domain names.
Added new 1.2T (61 billion) records for Freya bringing total generated data to 11.4T / 578 billion records
Added new 2.8T (133 billion) records for Freya bringing total generated data to 10.2T / 517 billion records
Freya DNS worker has surpassed 180 (3.6T) billion processed records
Autovacuum tool has removed over 42 million of invalid records (DNS catchers) reducing dataset size to 342 million records.
- After adding some generic subdomains (.com.xx, .net.xx, etc) resulting dataset grew significantly. Machine ran out of disk space at 3.4T new (7.4T total) / 384 billion records.
- At least several other registrars (
.fmis a known culprit) are doing a dirty trick for non-existent domains. Special thanks to community for catching this.
- 54,081,701 new words for dataset. At least 82,312,348,922 new domains (1.6T) to check, which brings total generated dataset to pretty serious 5.6T.
- crawler code is now closed source and used internally. Most of the job is now done by Freya
- 4.0T dataset of generated DNS names is now being processed. Return is small, about 8-10k domains per 1 million records.
- Some of those are already in the database, so 212 billion records are expected to yield about 20 million new domains.
- There’s a separate process, called
autovacuum, running on a regular basis. It cleans up dataset from unreachable (expired, servfail, etc.) domains.
- Added this news file :-)