Try Opteryx

Convert Parquet to JSON Lines

JSON Lines — also called JSONL or NDJSON — is one JSON object per line. It is what log shippers, message queues, search indexes and many APIs accept, and it is easy to stream because a line is a complete record. Rugo writes it from a Parquet file with a single command.

What You'll Be Able to Do

By the end of this page you'll be able to:

  • convert a Parquet file to JSONL with one command
  • know what each line looks like — nulls, numbers, timestamps
  • write only some of the columns
  • split one Parquet file into several JSONL files, or combine many into one
  • know what is lost, and what happens when the JSONL is read back

Convert One File

bash
pip install rugo
rugo convert space_missions.parquet space_missions.jsonl
text
space_missions.parquet (parquet, 4630 rows) -> space_missions.jsonl (jsonl)

The example file is space_missions.parquet, 4,630 orbital launch attempts. Download it from the Opteryx repository.

What Each Line Looks Like

json
{"Company":"US Air Force","Location":"SLC-20, Cape Canaveral AFS, Florida, USA","Price":63.23,"Lauched_at":"1964-09-01T15:00:00+00:00","Rocket":"Titan IIIA","Rocket_Status":"Retired","Mission":"Transtage 1","Mission_Status":"Failure"}
  • Every line is one row, with the column names as keys.
  • Nulls are null. The earliest launches have "Price":null.
  • Numbers are JSON numbers, not strings: 63.23.
  • Timestamps are ISO-8601 text, with the UTC offset.

The key names are repeated on every line, which is why the file is large: 1.1 MB, against 93 KB for the Parquet.

What Is Lost

JSON has strings, numbers, booleans and null, and no other types. Dates and timestamps become text, and the column types, nullability and statistics in the Parquet footer aren't written.

Reading the JSONL back guesses the types from the first 5 records. Here those records have no price, so Price is read as text and not a number. If you're going to read the file again, Convert JSON Lines to Parquet shows how to declare the types.

Write Only Some of the Columns

The command writes every column. To choose, use Python, which reads just the columns you ask for — the rest are never decoded:

python
from rugo import parquet
from rugo.jsonl import write_jsonl

with parquet.read_parquet(
    "space_missions.parquet", columns=["Company", "Mission", "Price"]
) as reader:
    with open("missions_subset.jsonl", "wb") as f:
        for morsel in reader:
            f.write(write_jsonl(morsel))

Reading Parquet by column is also where the filtering options live — see How to Read a Parquet File with Python for predicates, which let you write only the matching rows.

Split One Parquet File into Several JSONL Files

split cuts by number of rows, and --format chooses what to write. To get four files, divide the row count by four, rounding up:

bash
rows=$(( ($(rugo count space_missions.parquet) + 3) / 4 ))
rugo split space_missions.parquet --rows $rows --format jsonl
text
space_missions.part0000.jsonl  (1158 rows)
space_missions.part0001.jsonl  (1158 rows)
space_missions.part0002.jsonl  (1158 rows)
space_missions.part0003.jsonl  (1156 rows)

Parts land beside the input and are numbered so a wildcard lists them in order. Every part holds exactly --rows rows except the last. This splits by position, not by value: it can't give you one file per company or per month.

Combine Many Parquet Files into One JSONL File

The last name on the line is the output, and its extension picks the format:

bash
rugo merge space_missions.part000*.parquet space_missions.jsonl
text
merged 3 files (4630 rows) -> space_missions.jsonl

Every input must have the same column names, in the same order, with the same types; see the schema rule.

Large Files

rugo convert builds the whole result in memory before writing it. The Python route above writes one batch at a time, so the JSONL can be larger than memory.