Try Opteryx

Exploratory Analysis in Notebooks

Opteryx Core fits the notebook workflow well: it installs with pip, runs in the same process as your notebook kernel, and streams results in a form you can hand straight to pandas or a plotting library. It is a good fit when a Parquet dataset is larger than you want to load whole, but you still want to explore it interactively.

Try it yourself: the companion notebook runs every example on this page in order, against a public dataset of 4,630 orbital launches in place of transactions: aggregate in SQL, take a first look, chart a yearly summary with pandas, and iterate rows. It needs no account. Open in Colab or download the notebook.

What You'll Be Able to Do

By the end of this page you'll be able to:

  • set up Opteryx Core inside a notebook kernel and point it at a folder of Parquet
  • explore a dataset larger than memory by aggregating in SQL and pulling back only the summary
  • get a fast first look at an unfamiliar dataset — a sample, a row count, value frequencies
  • turn a small result into a pandas DataFrame and plot it
  • iterate a result row by row when you need Python-level logic
  • confirm the engine itself works using the built-in $planets relation

Setup

bash
pip install opteryx-core pyarrow pandas matplotlib

opteryx-core installs with no dependencies; pyarrow is needed for to_arrow(), and pandas and matplotlib for the chart below.

python
import opteryx
from opteryx.connectors import DiskConnector

opteryx.register_workspace("data", DiskConnector)
session = opteryx.session()

register_workspace maps the data prefix to the ./data folder under the current working directory, so data.transactions reads the Parquet files in ./data/transactions/. See Querying Local Data for how dataset names resolve to folders.

Querying Without Loading Everything

The usual notebook problem is a dataset that does not fit comfortably in memory. Aggregate it in SQL and only pull back the summary:

python
for morsel in session.execute_to_morsels("""
    SELECT
        category,
        COUNT(*)     AS n,
        AVG(amount)  AS avg_amount
      FROM data.transactions
     WHERE ordered_at >= CAST('2024-01-01' AS TIMESTAMP)
     GROUP BY category
     ORDER BY n DESC
"""):
    print(morsel.to_arrow())

The scan, filter, and aggregation run inside Opteryx over the full dataset; only the grouped result — a handful of rows — ever reaches Python.

A Fast First Look

Get the shape of an unfamiliar dataset without reading all of it:

python
# a sample of rows
for morsel in session.execute_to_morsels("SELECT * FROM data.transactions LIMIT 20"):
    print(morsel.to_arrow())
sql
-- row count
SELECT COUNT(*) FROM data.transactions;
sql
-- distinct values and their frequencies
SELECT status, COUNT(*) AS n
  FROM data.transactions
 GROUP BY status
 ORDER BY n DESC;

Straight to a Chart

For a result you know is small, concatenate the morsels into a single Arrow table, which converts to pandas directly:

python
import pyarrow

morsels = session.execute_to_morsels("""
    SELECT
        FORMAT_TIMESTAMP('%Y-%m', ordered_at) AS month,
        SUM(amount)                           AS revenue
      FROM data.transactions
     GROUP BY month
     ORDER BY month
""")
monthly = pyarrow.concat_tables(morsel.to_arrow() for morsel in morsels).to_pandas()

monthly.plot(x='month', y='revenue')

Push the aggregation into SQL and keep the result small — let Opteryx do the heavy scan and grouping, and let your plotting library draw the answer.

Working Row by Row

When you need Python-level logic per row rather than a DataFrame, iterate a morsel directly — each row comes back as a named tuple:

python
for morsel in session.execute_to_morsels("SELECT name, mass, gravity FROM $planets"):
    for row in morsel:
        print(row.name, row.mass, row.gravity)

Try It Against Sample Data

Opteryx Core ships with a built-in $planets relation, so you can confirm the engine works before pointing it at your own data:

python
for morsel in session.execute_to_morsels(
    "SELECT name, mass, gravity FROM $planets ORDER BY mass DESC"
):
    print(morsel.to_arrow())

$planets doesn't touch a workspace or connector, so it tells you the engine itself is working, independently of whether your data access is configured correctly.