Debugging Object Storage Read Failures Without the Egress Bill

TL;DR

A query over a 260 GB table started failing in production with a DatasetReadError. The obvious way to debug it was to run the failing scan against the real table, repeatedly, and pay egress on every attempt.

We didn't need the real data. We needed its shape and a slow network. We generated a table with the same shape and served it from hadro, a small read-only S3 server that can add latency, cap bandwidth and report what it sees. The query failed locally the same way it did in production. hadro's own stats showed the cause: far more bytes in flight than the link could deliver before the client's deadlines. The fix is now a regression test that runs in CI on every build.

The data never left the laptop, so none of this cost any egress.

The failure

This query started failing:

SELECT * FROM github.events WHERE actor_login = 'joocer';

The table is about 260 GB of Parquet in object storage, and some of its columns are wide strings holding JSON payloads. Queries that only touched the narrow columns were fine. SELECT * failed.

It didn't fail locally either, because on localhost every read finishes instantly. Whatever was going wrong only happened over a real network.

Why not debug it where it happens

Debugging against production means scanning the table from somewhere it isn't. Every scan from a laptop is billed as egress, and one experiment is never one scan: you need a baseline, a run with the change, runs at other settings, and then the same again when the first idea turns out to be wrong. Each run is also slow, and the network is whatever it happens to be that day, so two runs aren't comparable.

What we wanted was the opposite of all three: free to run, fast to iterate, and a network we could set ourselves.

hadro

hadro is a read-only, S3-compatible server. Point it at a directory and every sub-directory becomes a bucket:

pip install hadro
hadro ./data

Any S3 client can then list objects, fetch byte ranges and run S3 Select: boto3, the AWS CLI, MinIO, or Opteryx. With keys set, it checks SigV4 signatures, so a mis-signed request fails against hadro the same way it would against AWS.

It didn't start out as a test tool. It grew from S1 and cache.opteryx.app, an experiment in putting an S3-shaped caching layer in front of Google Cloud Storage. The name was an attempt at humour: S1 covered about a third of the S3 API surface. That third was the part a reader needs, which is listing, ranged GETs, and S3 Select, which was there so filtering could happen in the cache rather than shipping whole objects.

The cache didn't last, but the server did. We repackaged it as hadro, made the local filesystem the default backend, and swapped pyarrow for rugo, the reader Opteryx uses.

The feature that mattered here is that hadro can stop answering instantly:

It also serves HTTPS when given a certificate and key, which mattered because production reads go over TLS.

Reproducing it

We didn't copy the 260 GB. A repro script generated a wide, many-file table with synthetic data in the same shape: a login column to filter on, and wide payload columns full of random hex that zstd can't compress, so each column chunk is a large range request just like the real one.

hadro served it over TLS with production-like shaping:

python scratch/repro_s3_shaped_scan.py --workdir /tmp/repro -- \
    --latency-ms 20 --total-bandwidth-mbps 100

The query failed with the same DatasetReadError as in production. After that, each experiment took seconds and cost nothing.

What the server saw

Opteryx's HTTP client gives every range request a deadline, sized from the request's bytes at an assumed minimum bandwidth. For a single request that's reasonable. A small request that stalls times out quickly, and a big one gets longer.

The problem is that requests don't run alone. The scan has many in flight at once, and they all share one link. On a narrow int64 column, 24 concurrent requests put 17 MB on the wire. On a wide string column, the same 24 requests put 181 MB on the wire. Each request's deadline assumed it had the link to itself, so the last bytes landed after the deadlines had passed, even though the link had been busy the whole time. Nothing was stuck. There was just too much in flight to finish in time.

--stats confirmed it. With the client uncapped, hadro reported 242 MB in flight at its peak. This is a number from the server's side, so it isn't based on anything the client believes it sent.

The fix

The client now has a process-wide cap on bytes in flight. A request is admitted only when its bytes fit under the cap, and the wait happens before its timer starts, so queueing never uses up its deadline. A request bigger than the cap is admitted on its own, so nothing deadlocks.

On the same 100 Mbps link, a 23 MB cap let the wide column complete at 94% of link speed.

hadro also showed us that a fixed cap was the wrong answer. With shaping set to 1200 Mbps, a 25 MB cap made the same column 4× slower (129 → 31 MB/s). So the default is adaptive. The cap follows the link's measured throughput, starting from the assumed minimum bandwidth and growing as the link proves it's faster.

Finding that meant running the same scan at different link speeds and cap settings, again and again. Against the real table, every one of those runs would have been a full scan's worth of egress.

Keeping it fixed

The repro script was a one-off. The behaviour it found is now a test:

with hadro.Server(data=str(tmp_path), bandwidth_mbps=16, stats=True) as server:
    ...  # eight threads fetch overlapping ranges, with the cap set
    with urllib.request.urlopen(f"{server.endpoint}/_shaping/stats") as response:
        peak = json.load(response)["peak_bytes_in_flight"]

assert peak <= cap

hadro.Server runs the server in a background thread on a free port, so a test can start a slow, throttled object store in one line and shut it down when it's done.

The same CI job runs Opteryx's S3 integration tests against hadro with signatures checked. Each s3:// result is compared against the same query over the same bytes on local disk, so the only thing that can differ is the transport.

Installing it in GitHub Actions takes one line, plus a step that checks the install worked:

- name: Install Requirements
  run: |
    python -m uv pip install --upgrade -r tests/requirements.txt
    # --no-deps: hadro depends on the published rugo wheel, which would
    # shadow the rugo we've just built from source.
    python -m uv pip install --upgrade --no-deps hadro

# Skipping is for dev machines without hadro, never how CI passes.
- name: Verify hadro is installed
  run: |
    python -c "import hadro, fastapi, uvicorn, importlib.metadata as m; print('hadro', m.version('hadro'))"

The tests use pytest.importorskip("hadro") so they still run on machines without it. Without the check, a broken install in CI would skip every S3 test and the build would still go green.

Using hadro yourself

pip install hadro

hadro needs Python 3.11+, on Linux (x86-64 or aarch64) or macOS on Apple silicon. It depends on FastAPI, uvicorn and rugo, and doesn't need pyarrow, pandas or numpy.

In a test:

import hadro
import pytest

@pytest.fixture(scope="session")
def s3_endpoint():
    with hadro.Server(data="tests/data", latency_ms=50, bandwidth_mbps=2.5) as server:
        yield server.endpoint

S3 Select from the S1 days still works, now running on rugo. It accepts Parquet, JSON Lines and CSV input, and supports projection, WHERE and LIMIT. Filters are pushed into the reader, so Parquet row groups can be skipped based on footer statistics before anything is decoded. As an extension, it can also return results as Parquet.

hadro has limits. It's read-only, so anything that writes gets 405 MethodNotAllowed. It only supports path-style addressing. It serves whatever certificate you give it but doesn't create one. It isn't an object store. It's a stand-in for one that you can run on your laptop and slow down on purpose.

The source is at mabel-dev/hadro, under Apache 2.0.

— Justin