Skip to content
rewget GitHub

Home / How it works

How rewget gets through

rewget is a thin CLI shim over a daemon. The CLI is what your scripts call; rewgetd owns the expensive resources — the browser pool and the impersonation sessions — and only does work when a site actually blocks you.

The pipeline

  your script / CI / Makefile
             │  rewget <url>   (wget CLI surface, unchanged)
             ▼
  ┌───────────────────────┐
  │  rewget CLI (shim)     │  parses --rewget-* flags, forwards the rest
  └──────────┬────────────┘
             │ IPC
             ▼
  ┌───────────────────────┐
  │  rewgetd (daemon)      │  browser pool · TLS sessions · profile store
  └──────────┬────────────┘
             ▼
   ┌──────────┐ 403/CAPTCHA ┌──────────────┐ still blocked ┌──────────────┐
   │ stage 1  │────────────▶│ stage 2       │──────────────▶│ stage 3       │
   │ wget     │             │ rquest / TLS  │               │ Chromium (JS) │
   └────┬─────┘             └──────┬────────┘               └──────┬───────┘
        │ 200                      │ 200                            │ 200
        ▼                          ▼                                ▼
                  first stage that works wins → cached per-domain (7d)

The three stages, in order

rewget always starts cheap and only escalates when it has to. Every escalation is triggered by a response code you can configure with --rewget-fallback-codes.

  1. 1

    Stage 1 — plain wget

    rewget runs wget with your exact flags. If the response is not a configured fallback code, that result is returned unchanged.

  2. 2

    Stage 2 — browser-impersonating HTTP

    On a fallback code, rewgetd retries through rquest using a real browser TLS and HTTP/2 fingerprint. Most fingerprint blocks resolve here.

  3. 3

    Stage 3 — headless Chromium

    If stage 2 still fails, chromiumoxide drives real Chromium to solve the JavaScript challenge, then exports the session so the download can complete.

  4. 4

    Cache the winner

    rewget records which stage worked for that domain and reuses it for seven days, so later downloads skip straight to the answer.

Per-domain caching

Probing from stage 1 every time would be wasteful for hosts you already know block plain HTTP. rewget caches the winning stage per domain with a seven-day TTL, so a mirror job hitting the same host repeatedly pays the escalation cost once, then runs at full speed for a week.

Signed profile updates

The browser fingerprints in stage 2 come from a profile store. Profiles are Ed25519-signed and verified before use, and new browser releases are pulled in explicitly with --rewget-update-profiles — never fetched silently. Inspect what is installed with --rewget-list-profiles.

Ready to try it? Head to the quickstart, browse the full feature set, or see real use cases.