Try Opteryx

Load and Query Data

This page covers getting data into opteryx.app, the hosted service, and reading it back out with SQL. If you're embedding the opteryx-core engine directly in a Python process instead, see Querying Local Data.

What You'll Be Able to Do

By the end of this page you'll be able to:

  • get a Parquet, CSV or NDJSON file into the hosted service, from the Studio upload flow or the Upload API
  • choose between the Jobs API, OData and Arrow Flight SQL for reading data back, and say what each is best at
  • know which of those need a token and which allow anonymous reads
  • find a public sample dataset to query before you've loaded anything of your own

Loading Data

Files are ingested through the Upload API: open a session, upload one or more files as parts (Parquet, CSV, or NDJSON), then commit the session into a table. Opteryx Studio also has a guided upload flow in the UI that wraps the same API, if you'd rather drag and drop a file than script it.

For a full walkthrough of the session/part/commit flow, including the Python SDK, see Upload client (Python SDK).

Reading Data

Once data is loaded, there are three ways to query it:

Method Auth Best for
Jobs API Required Submitting arbitrary SQL over plain HTTP/JSON from any language.
OData API Anonymous read on public datasets; a token for anything else Quick, filterable reads without writing any client code - see below.
Arrow Flight SQL Required Larger result sets in Python - data streams as Arrow, no JSON round trip.

See Running a Query via the API for the Jobs API request/response walkthrough, and Access and Permissions for how tokens and policies control what each method can see.

Try It: Public Sample Datasets

Datasets under the public workspace are readable by anyone signed in - useful for trying a query before you've loaded anything of your own. The geopolitics and security collections go further and allow anonymous reads over OData, with no account or token at all; astronomy and sales need you to be signed in:

Dataset Description
public.astronomy.planets Small example table of planets (good for quick queries and demos). Signed-in only.
public.geopolitics.countries Reference table of countries.
public.geopolitics.gdelt_events GDELT event records, ordered by date_added.
public.github.events GitHub event stream samples (event-level rows), ordered by created_at.
public.sales.orders Example sales orders table. Signed-in only.
public.sales.sales Example sales transactions table. Signed-in only.
public.security.cisa_kev CISA Known Exploited Vulnerabilities catalog.
public.security.epss Exploit Prediction Scoring System (EPSS) data.
public.security.exploit_db Exploit-DB entries.
public.security.ghsa_advisories GitHub Security Advisories, ordered by published_at.
public.security.ghsa_advisories_affected Packages/ecosystems affected by GHSA advisories, ordered by ecosystem.
public.security.nvd_vulnerabilities National Vulnerability Database (NVD) records.
public.security.exploited_vulnerabilities A view joining known-exploited and vulnerability data.
public.security.vulnerabilities_per_week A view summarizing vulnerability counts per week.