Home / Use cases / Bulk dataset & mirror fetching
Data engineeringBulk dataset & mirror fetching
Pull many files across many hosts where a subset fingerprint TLS, without probing every host from scratch.
The problem
A dataset or mirror job spans dozens of hosts. Some serve everyone; a handful fingerprint TLS and block plain clients partway through. Retrying the whole job, or hand-maintaining a list of which hosts need a browser, is fragile and slow.
How rewget solves it
- ▸Run the job through rewget with the wget flags you already use for recursion and resume.
- ▸Each host is probed once: hosts that never block stay on fast stage-1 wget; hosts that fingerprint escalate automatically.
- ▸The per-domain cache records the winning stage for seven days, so later passes over the same hosts skip straight to what works.
- ▸Resumable and recursive downloads run through stage 1, so large multi-file fetches keep wget's native behaviour.
$ rewget --continue --recursive https://mirror.example/datasets/ [host mirror.example → stage 1, cached] [host cdn.blocked.example → stage 2, cached]
- ·Recursion and resume are wget features handled in stage 1 — rewget does not reinvent them.
- ·No benchmark claims here: speed depends entirely on your network and how many hosts block.
Questions
+ Does rewget re-probe every host on every run?
No. The first time it learns which stage a host needs, it caches that per domain for seven days and reuses it.
+ Do --continue and --recursive still work?
Yes. They are plain wget flags forwarded to stage 1, so resumable and recursive downloads behave exactly as they do in wget.