Home / Use cases / CI & release mirroring
Continuous integrationCI & release mirroring
Stop CI from flaking when a shared runner IP gets a 403 fetching a dependency or release tarball.
The problem
CI runners share IP ranges that upstream sites increasingly rate-limit or fingerprint. A wget step that works on your laptop starts returning 403 or a CAPTCHA on the runner, and your pipeline goes red for reasons that have nothing to do with your code.
How rewget solves it
- ▸Alias wget=rewget in the CI image so the fetch step is unchanged — no pipeline rewrite.
- ▸When the runner hits a block, rewget escalates through the browser-impersonating HTTP stage and, if needed, headless Chromium.
- ▸The winning stage is cached per domain for seven days, so repeated builds against the same host stay fast.
- ▸Use --rewget-no-fallback in the jobs where you want a genuine upstream outage to fail loudly instead of being masked.
$ rewget https://downloads.example/toolchain.tar.gz [stage 1: wget] 429 Too Many Requests → escalating [stage 2: rquest/firefox136] 200 OK
- ·rewget does not defeat authentication — if the artifact needs a token, pass it exactly as you would with wget.
- ·Both the CLI and rewgetd ship in every install package, so the CI image needs no extra wiring.
Questions
+ Will this hide real upstream failures in CI?
Only if you let it. Use --rewget-no-fallback on critical fetches so a genuine outage fails the job instead of escalating silently.
+ Does the cache persist between CI runs?
The per-domain cache has a seven-day TTL. Whether it survives between runs depends on whether your CI persists the rewget state directory; within a run it always applies.