Resumable downloads: the corner cases nobody tests

Resumable downloads look like a five-line feature. The HTTP spec gives you Range, the server gives you 206 Partial Content, the client glues bytes together, and the file on disk is whole. Easy.

Resumable downloads are not a five-line feature. The spec is honest about what the protocol promises, but the protocol promises less than people assume, and CDNs, proxies, and middleboxes have spent the last decade adding edge cases that the spec did not anticipate. Most clients handle the happy path correctly and silently produce wrong output on at least three of the cases below. We have audited the rewget code path against a checklist, and even there we have caveats. This post is the checklist.

The happy path, for reference

You requested GET /file.tar.gz. The connection broke at 17.3MB of 42.1MB. You restart with Range: bytes=17300000-. The server responds 206 Partial Content with Content-Range: bytes 17300000-42100000/42100000. You append the new bytes to the existing file. You verify the size matches. You are done.

That works for static origin servers with stable content. It works for most of S3 and Backblaze B2. It works for plain nginx and Apache serving immutable files. It does not work for any of the cases below.

Corner case 1: the Range header was ignored

Some servers and CDNs will accept your Range request, ignore it, and respond 200 OK with the entire file. They are technically allowed to do this — the spec allows servers to ignore Range — and you will not notice unless you check the response status code before you append.

We have seen this in two flavours. The clean version is a 200 OK with the full body; you can detect it and truncate-then-replace. The dangerous version is a 200 OK with a Content-Range header that lies, or a 200 OK where the body is the entire file but the client assumed the bytes were the suffix and appended them anyway. The file on disk is now garbage. wget catches this. Naive client code does not.

Corner case 2: the file changed underneath you

You downloaded 17.3MB of the original file. You came back six hours later. The origin has rebuilt the artifact. The byte at offset 17,300,000 is different. The server says 206 Partial Content and gives you the new bytes from offset 17,300,000. You append. The file has a clean seam in the middle and is silently corrupted.

The correct mitigation is If-Range, which lets you say “give me the partial content only if the validator (ETag or Last-Modified) still matches.” If the validator does not match, the server is required to respond 200 OK with the entire file, and the client is required to restart from zero.

Validators are not consistent across origins. Some give you a strong ETag. Some give you a weak ETag that changes on every cache miss. Some give you no ETag and a Last-Modified that updates more often than the content. Trusting the validator blindly is wrong; ignoring it is also wrong. rewget records both the validator and a tail hash of the partial file and refuses to resume if either disagrees.

Corner case 3: the Content-Length is missing or wrong

CDNs in front of dynamically generated content sometimes omit Content-Length, or send it as a chunked stream, or send a Content-Length that disagrees with the actual body. The cleanest case is “missing”; the client cannot pre-allocate or know when the download is “complete.” The worst case is a Content-Length of 0 on a 50MB body, which we have seen from at least one well-known CDN under specific cache-miss conditions.

The mitigation here is to never trust Content-Length as a completion signal. The completion signal is the connection closing cleanly after the documented end of the chunked stream, or the byte count matching the Range you asked for. wget handles this. rewget inherits the behaviour.

Corner case 4: byte zero looks different on retry

Bot detection sometimes serves you a CAPTCHA page on the first request and the real content on the second. If your resume logic sees a partial file on disk and sends Range: bytes=17300000- against an origin that just put a CAPTCHA at byte zero, you will append “real file bytes from offset 17,300,000 onward” to “the first 17.3MB of a CAPTCHA page.” The file is corrupted in a way that is impossible to detect from size alone.

This is the case rewget’s stage 2 fallback was built for. If stage 1 wget gets blocked on the initial request, stage 2 retries with browser TLS impersonation. If stage 1 succeeded and stage 2 is asked to resume, rewget verifies that the prefix on disk matches what the new stage would have served — same content type, same first-chunk hash — before letting the resume proceed. If the prefix does not match, rewget restarts from zero rather than producing a corrupted file.

Corner case 5: redirect chains on resume

You started a download from https://example.com/file.tar.gz. The origin redirected you to https://cdn1.example.com/file.tar.gz. You downloaded 17.3MB and the connection dropped. You restart. The origin’s load balancer now sends you to https://cdn2.example.com/file.tar.gz. Are the bytes guaranteed to be identical?

Usually yes. Sometimes no. If the CDN nodes are out of sync, or if the origin is in the middle of a rolling deploy, the answer depends on cache state. The clean mitigation is to pin to the resolved URL after the first redirect (which is what wget does on resume with the same command line) and re-validate on the resumed request. The risky shortcut is to follow the redirect again and trust the new node.

rewget pins to the resolved URL on the first successful stage-1 attempt and replays the resume against the same node when possible. The per-domain cache makes this cheap.

Corner case 6: the byte range cap

Some CDNs cap how many bytes a single Range response can return. You ask for bytes=17300000- against a 42.1MB file; the CDN sends you bytes=17300000-25000000. You have to detect the partial response, append, and request the next slice. This is a perfectly legal CDN behaviour. Clients that assume bytes=N- returns everything from N to end will pause halfway and never resume.

wget handles this correctly. Most “I wrote my own download manager” projects do not, because the test fixtures do not include CDNs that do this.

Corner case 7: clock-shaped resume bugs

You started at 23:59:59. The download spans midnight. The server’s Last-Modified rolled forward at midnight because of a scheduled rebuild. Your If-Modified-Since and If-Range headers now both miss; the server gives you the new file from byte zero; your client appends the new bytes to the old prefix. Silent corruption again.

The mitigation is to anchor the validator at the start of the download, not to recompute it at resume time. If the validator changes, restart.

Corner case 8: the daemon outlived the partial file

In rewget’s case, the daemon survives across CLI invocations. The cache says “this domain works on stage 2.” A second CLI invocation, in a different shell, on the same domain, on the same partial file — does it pick up where the first left off?

The answer is “yes, if the partial file is intact, and no otherwise.” We hash the partial file’s tail on every resume attempt and refuse to continue if the hash does not match what we saw at the last successful stage. This is paranoid; we made it paranoid on purpose because the alternative is a silently corrupted artifact in a CI cache.

What we test for

The rewget test suite covers the cases above with deterministic fixtures: a fake origin that ignores Range, a fake origin that returns a different file mid-stream, a fake origin that lies about Content-Length, a fake CDN that caps byte ranges, and a stage-2 path that gets a different first byte than stage 1 saw. There are 42 tests covering all stages and edge cases; resume is roughly a third of them.

The cases the test suite does not cover are the ones that need a real CDN at a specific load condition. We pin those to a manual regression list and re-run before each release against the WAFs and CDNs we have permission to hit.

What you should do

If you are building a download client: build the resume test fixtures first. The happy-path code is the easy part. The cases above are what separate a download client from a corruption generator.

If you are using rewget: trust the resume path on a stable origin. If the origin is doing something weird and the resume produces a file you do not trust, use --rewget-no-fallback and a fresh download. Debugging a corrupted artifact is more expensive than re-fetching one.