Avro Deserves Better Than a Dict per Record

TL;DR

• Avro is the format data moves in: every Iceberg table's manifests are Avro, and Kafka Connect sinks land events as Avro files. Yet every Python reader we tried hands it back one dictionary at a time, and PyArrow doesn't read it at all.
• Using Iceberg with Opteryx? Your queries get to work sooner. Working out which files a snapshot holds - the step before planning even starts - is now 4× to 7× faster.
• Got Avro files in a bucket? Query them with SQL: SELECT ... FROM READ_AVRO('events/*.avro'). Over ClickBench's 100 million rows stored as Avro, Opteryx answers the 43 queries in 150 s, against ClickHouse's 590 s on the same files.
• Reading Avro in Python? Rugo gives you columns instead of dictionaries: ~8× faster than fastavro and ~50× faster than the Apache reader, and still ~4× faster than fastavro once every value is a Python object again.

Avro is everywhere you aren't looking

Avro is not glamorous. It's a row-oriented binary format with its schema in the header: compact, self-describing, and built for writing records quickly as they arrive. That's exactly why so much infrastructure is built on it:

• Iceberg. Before an engine reads a single row of an Iceberg table, it reads the snapshot's manifest list and every manifest that points at. Those are Avro. Every Iceberg query starts with Avro.
• Event pipelines. Kafka Connect sinks write Avro files to object storage, as do plenty of logging and change-capture tools.

Now look at what Python gives you to read it with:

• avro, the Apache reference implementation: pure Python, correct, complete - and slow.
• fastavro: much faster, and the default choice, but it still returns a dict per record.
• PyIceberg brings its own Avro reader for manifests, building a Python object per manifest entry.
• PyArrow doesn't read Avro.

Every one of them gives you Python objects, one record at a time. That's fine for a script that copies records from A to B. It's the wrong shape for analysis, which wants a column of numbers you can sum, not a list of dictionaries with a number inside each.

So we built a reader that gives you columns, and put it to work in three places.

If you use Iceberg: your queries get started sooner

This is why the work happened when it did.

opteryx-iceberg lets Opteryx query external Iceberg tables. Before it can plan a query against a snapshot, it needs the snapshot's file list: every live data file, with each column's min/max bounds so the planner can skip files that can't match. Building that list means reading the snapshot's manifests - and until now that went through PyIceberg, one Python object per entry.

It now reads them with Rugo:

To be clear about what's being timed: this is the toll paid before the query is planned and before a single byte of the table's data is read. It sits in front of the first query against every snapshot, and on a table with many manifests it's the difference between noticing it and not.

It shows up end to end, too. Our local Iceberg benchmarks exist mainly to test compatibility rather than speed, but with manifests now read by Rugo the suite runs 25% faster.

The other thing you get is the same answer. The new code makes exactly the decisions PyIceberg's planner makes - which entries are live, which sequence numbers apply, which delete files touch which data files - and it's tested file by file against PyIceberg's own planner across v1 tables, v2 tables, partitioned and unpartitioned tables, and a v1 table upgraded to v2. Tables with merge-on-read deletes are still refused, exactly when PyIceberg would attach a delete, rather than read without them. PyIceberg is still the catalog client - it reads metadata.json, snapshots and schemas, which it's genuinely good at.

If you have Avro files: query them with SQL

Avro is like CSV and JSONL in Opteryx: something you read, not a format you store tables in. So it arrives as a table function, and a glob reads a whole folder:

SELECT status, COUNT(*)
FROM READ_AVRO('gs://my-bucket/events/2026-10-*.avro')
GROUP BY status;

Files written months apart with slightly different schemas still line up - each is read as the first file's schema, with missing fields filled with their defaults. Nested records, maps and arrays come back as JSON, so ->> pulls fields out of them the same way it does for nested Parquet. And COUNT(*) over a folder of Avro files is close to free: every Avro block records how many rows it holds.

To see how it holds up, we wrote ClickBench's 100 million rows as 103 Avro files (14 GB, deflate) and ran its 43 queries against Opteryx, ClickHouse and DuckDB, all reading the same files:

ClickHouse takes about 13 seconds on nearly every query, COUNT(*) included - we answer that one in 0.11 s. Opteryx is ahead on all 43 queries, 4× at the median. DuckDB's Avro extension took 47-52 seconds per query; at that pace the full run is over half an hour, and after three queries we decided we had better things to do with the afternoon.

Filters on wide files are where Avro usually hurts most, so we made them cheap too. When a query filters an Avro file, Opteryx decodes just the columns the filter needs, throws away the rows that fail, and only then goes back for the rest of the row. ClickBench's SELECT * ... WHERE URL LIKE '%google%' - every one of 105 columns, for the roughly 16,000 rows that match - went from 12.2 s to 3.8 s. Queries with filters that keep most rows don't pay for the trick: Opteryx notices, and reads them the plain way.

If you read Avro in Python: take columns, not dictionaries

You don't need a query engine for any of this. pip install rugo and:

from rugo import avro

with avro.read_avro("events.avro", columns=["user_id", "status"]) as reader:
    for batch in reader:
        statuses = batch.column("status").to_pylist()

Ask for the columns you need and only those are decoded. Here's the same file read by each library:

That's 200k records of 20 fields with deflate compression, a mix of what event streams actually carry: numbers, strings, nullable fields, enums, timestamps, decimals, and nested records, arrays and maps. Rugo 0.5.0, fastavro 1.13.1, avro 1.12.2, one thread, best of five. The "Rugo → Python values" column converts every column into Python objects, so it's the fair comparison if Python objects are what you want at the end.

On a bigger, plainer file - a million rows of 20 columns - the gap is wider still:

The fair caveats: fastavro and avro are smaller installs (3.1 MB and 0.5 MB, against Rugo's 8.1 MB, which also covers Parquet, CSV and JSONL), and they write Avro. Rugo only reads it, for now - a writer is on the way.

Why it's quick

Briefly, because the what matters more than the how:

• The schema is worked out once per file, not once per value. Opening a file turns its schema and your column list into a short list of steps, and every record just follows the steps.
• What you didn't ask for is skipped, not read. Unwanted strings are jumped over in one step, and runs of unwanted numbers in one more.
• No Python object is created per value. Values land straight in columns.
• Every core helps with one big file, and compressed files decompress with libdeflate, one of the fastest deflate decoders there is.

We also didn't take the shortcut of wrapping the Apache project's C++ library, because it drags in Boost - and Opteryx and Rugo install with no dependencies at all. The reader is fuzzed with sanitizers on, and anything it can't read correctly (a handful of rarely-used Avro features) it refuses by name rather than guessing.

It's new, and it will have rough edges

As at October 2026, Avro support in Rugo and Opteryx is new. It's tested against fastavro and the Apache reader, checked against PyIceberg's planner, and fuzzed - but Avro files in the wild come from a lot of different writers, and some of them will find corners we haven't. Expect rough edges.

We're going to keep battering the reader, and we're adding a writer, and we'll work through what turns up as we go. If you point Rugo or READ_AVRO at your own files - especially the odd ones - we'd love to hear how it went, good or bad. Open an issue and tell us what you were reading and what happened.

The honest part

Avro is not an analytics format, and no reader changes that.

The same ClickBench data stored as Parquet runs about 26× faster in Opteryx. Our Avro floor is about 3 seconds a query, and profiling says why: 61% of it is decompressing the file, and 35% is walking through rows. Avro stores whole rows, so reading one column still means stepping past every other field of every record - and there are no statistics in the format to skip anything with. A filter on CounterID = 62 reads every row of every file.

We tried to dig under that floor. Skipping runs of numbers with vector instructions came out 35% slower than the plain loop - Avro's numbers are mostly one to three bytes, too short for the trick to pay - so we dropped it.

And that's fine. Avro's job is to carry data reliably from the thing that made it to the thing that stores it. Ours is to make sure that when you need to read it - to start an Iceberg query, to dig through last night's events in a bucket, or to load a file in a notebook - it costs as little as the format allows, and doesn't arrive one dictionary at a time.

— Justin