Processing Large JSON Files Without Running Out of Memory

A JSON document that parses fine at 5 MB will take down a process at 500 MB, and the usual advice - "use jq, it streams" - is wrong in a way that costs people a production incident. Plain jq parses the entire document into memory before your filter sees a single byte.

Below are measured numbers rather than folklore, then the approaches that actually hold memory flat.

The measurements

One document, 119.9 MB on disk: a top-level object with a 400,000 element orders array.

{
  "generatedAt": "2026-09-23",
  "orders": [
    { "orderId": "ord_00000000", "total": 10.99,
      "customer": { "id": 0, "city": "London", "country": { "code": "GB" } },
      "items": [ { "sku": "SKU-0000", "qty": 1, "price": 19.99 } ],
      "note": "..." },
    ...  400,000 of these
  ]
}

Peak resident memory and wall time for each approach. Measured on Linux with jq 1.7 and Python 3.12, taking ru_maxrss from a freshly spawned process per run so the figures do not contaminate each other.

Three things fall out of that table, and each one changes what you should do.

1. Plain jq is not a streaming tool

This is the misconception worth killing first. Both of these commands look like they process records one at a time. Neither does:

# Reads like streaming. Is not streaming.
jq -c '.orders[]' orders.json > lines.ndjson        # 1150 MB peak

# Also not streaming, despite touching one scalar at a time.
jq -r 'paths(scalars) | join(".")' orders.json      # 1101 MB peak

A nine-fold multiplier over file size is not a jq defect, it is how jq works: the filter operates on a parsed value, so the value has to exist first. The paths(scalars) case is the clearest illustration - it emits one tiny string per scalar, and still needs 1101 MB to do it.

Extrapolate before you run anything. At roughly 9x, an 8 GB machine will fail somewhere under a 1 GB input.

2. --stream is the real thing, and it costs about 3x in time

--stream switches jq to emitting [path, value] events as it reads, so nothing accumulates:

# --stream emits [path, value] events instead of building the document.
jq -cn --stream 'inputs' orders.json | head -3

[["generatedAt"],"2026-09-23"]
[["orders",0,"orderId"],"ord_00000000"]
[["orders",0,"total"],10.99]

Memory drops from 1101 MB to 12 MB. Wall time rises from 2.8s to 8.1s. That trade is almost always worth taking, because the alternative is not a slower run, it is a failed one.

Events are awkward to work with directly. fromstream and truncate_stream reassemble them into whole records:

# Rebuild whole records from the event stream, one at a time.
# truncate_stream's depth must match how deeply the array is nested:
# .orders[] sits two levels down, so the depth is 2.
jq -cn --stream '
  fromstream(2 | truncate_stream(inputs | select(.[0][0] == "orders")))
' orders.json > orders.ndjson

The depth argument to truncate_stream is where most people get stuck. It must equal the nesting depth of the array you want. For a top-level array the depth is 1; for .orders[] inside a wrapper object it is 2. Get it wrong and you do not get an error, you get the entire array rebuilt as a single record - which puts you straight back to 1 GB of memory. Check with head -2 before you let it run.

A key-path inventory without loading the file

This is the streaming equivalent of what this site's tool does, for files far past what a browser can hold:

# A key-path inventory of a file far larger than RAM. 12 MB peak.
jq -rn --stream '
  inputs | select(length == 2) | .[0]
  | reduce .[] as $p (
      "";
      if ($p | type) == "number"
      then . + "[]"
      else (if . == "" then $p else . + "." + $p end)
      end)
' orders.json | sort -u

generatedAt
orders[].customer.city
orders[].customer.country.code
orders[].customer.id
orders[].items[].price
orders[].items[].qty
orders[].items[].sku
orders[].note
orders[].orderId
orders[].total

Ten lines describing a 120 MB document, in 12 MB of memory. Array indices collapse to [], so the output is a schema rather than a record dump.

3. Python's ijson is nearly free

The most surprising row in the table. ijson used 13 MB against json.load's 650 MB, and took 2.6s against 2.5s - a 50x memory reduction for a 4% time cost.

import ijson

# Yields each order as a dict; only one is live at a time.
with open("orders.json", "rb") as fh:
    for order in ijson.items(fh, "orders.item"):
        process(order)

# Cheaper still: pull single fields without building the dict
with open("orders.json", "rb") as fh:
    for total in ijson.items(fh, "orders.item.total"):
        accumulate(total)

The usual reason to reach for json.load is that it is faster. At this size it is not meaningfully faster, so the habit is worth breaking whenever input size is not something you control.

Convert once, then stop streaming

Streaming is the right answer for a single pass. If you are going to query the same data repeatedly, pay the streaming cost once and convert to NDJSON - one JSON document per line:

import json

# Once the file is NDJSON, the stdlib is enough. One line, one document.
with open("orders.ndjson") as fh:
    for line in fh:
        order = json.loads(line)
        process(order)

Every subsequent pass is then both flat and fast: 12 MB and 2.0s, which beats --stream on time and matches it on memory. The conversion itself took 30s and 12 MB; after that the format stops fighting you.

NDJSON also makes the data trivially splittable, which a single JSON document never is:

# Shard a large NDJSON file for parallel work
split -l 50000 orders.ndjson chunk_

# Process the shards concurrently, 4 at a time
ls chunk_* | xargs -P 4 -I{} jq -r '.orderId' {} > ids.txt

If you control the producer, emitting NDJSON in the first place removes this entire class of problem. A 50 GB NDJSON file is unremarkable to process; a 50 GB JSON array is a research project.

On the JVM

The same rule holds: ObjectMapper.readValue, Gson.fromJson and JsonAdapter.fromJson all materialise the whole object graph. Jackson's JsonParser lets you bind one element at a time while the surrounding array stays on disk - see Jackson vs Gson vs Moshi for the streaming loop and the nesting-depth limits that matter on untrusted input.

Choosing

Caveats on these numbers

One file shape, one machine, one version of each tool. The multipliers move with record size, key count and how much of the document your filter touches - a document of many small records costs relatively more per byte than one of few large ones. Treat 5x to 10x as the order of magnitude to plan for, not a constant, and measure your own payload before sizing a container.

Further reading

• Jackson vs Gson vs Moshi - streaming on the JVM
• Working with JSON in Python
• How to extract all keys from a JSON object
• Understanding nested JSON objects and arrays