Is there a way to use jq to divide a JSON stream of objects into individual files according to the values of a specific object property?

Question

Is there a way to use jq to divide a JSON stream of objects into individual files according to the values of a specific object property?

Dealing with a hefty file (20GB+ compressed) named input.json, filled with a stream of JSON objects like so:

{
    "timestamp": "12345",
    "name": "Some name",
    "type": "typea"
}
{
    "timestamp": "12345",
    "name": "Some name",
    "type": "typea"
}
{
    "timestamp": "12345",
    "name": "Some name",
    "type": "typeb"
}

The goal is to split this giant file based on the property type: typea.json, typeb.json, etc., each containing its own set of JSON objects that match the respective type.

Tackling this challenge for smaller files wasn't an issue, but handling such a massive file causes memory overflow on my AWS instance. To keep memory consumption in check, I know I need to utilize --stream but figuring out how has me stumped.

cat input.json | jq -c --stream 'select(.[0][0]=="type") | .[1]'

retrieves the values of each type property, but how can I use this information to filter the objects accordingly?

Any guidance or insights would be immensely appreciated!

json bash stream jq partitioning

Answer 1

Answer №1

If the JSON objects in the file are relatively small, typically a few MB or less, you may not need to use the "--stream" command-line option, which can be rather complex and is usually reserved for handling huge JSON files.

However, there are still decisions to make. You can choose between a multi-pass approach involving multiple calls to jq (N or N+1 calls, where N is the number of output files), or a single-call approach followed by using a program like awk to divide the data into separate files. Each method has its own advantages and disadvantages, but if processing the input file multiple times is acceptable, the first approach might be more efficient.

To estimate the computational resources needed, consider measuring the resources used by running jq empty input.json.

Based on your description, it seems that the memory issue you encountered is likely related to unzipping the file.

Answer 2

If the JSON objects in the file are relatively small, typically a few MB or less, you may not need to use the "--stream" command-line option, which can be rather complex and is usually reserved for handling huge JSON files.

However, there are still decisions to make. You can choose between a multi-pass approach involving multiple calls to jq (N or N+1 calls, where N is the number of output files), or a single-call approach followed by using a program like awk to divide the data into separate files. Each method has its own advantages and disadvantages, but if processing the input file multiple times is acceptable, the first approach might be more efficient.

To estimate the computational resources needed, consider measuring the resources used by running jq empty input.json.

Based on your description, it seems that the memory issue you encountered is likely related to unzipping the file.

Answer 3

Answer №2

Utilizing the power of jq to separate a NUL-delimited stream containing pairs of (type, document), and harnessing the capabilities of native bash (4.1 or newer) to write these documents with a persistent group of file descriptors:

#!/usr/bin/env bash
case $BASH_VERSION in ''|[1-3].*|4.0*) echo "ERROR: Bash 4.1 required" >&2; exit 1;; esac

declare -A output_fds=( )

while IFS= read -r -d '' type && IFS= read -r -d '' content; do
  if [[ ${output_fds[$type]} ]]; then  # checking for existing file handle for this output file
    curr_fd=${output_fds[$type]}       # reusing it if available.
  else
    exec {curr_fd}>"$type.json"        # opening a new output file...
    output_fds[$type]=$curr_fd         # storing its file descriptor for later use.
  fi
  printf '%s\n' "$content" >&"$curr_fd"
done < <(jq -j '(.type) + "\u0000" + (. | tojson) + "\u0000"')

This method ensures that only a few records are processed at a time, limiting memory usage and enabling operation on large files with reasonable size records.

Answer 4

Utilizing the power of jq to separate a NUL-delimited stream containing pairs of (type, document), and harnessing the capabilities of native bash (4.1 or newer) to write these documents with a persistent group of file descriptors:

#!/usr/bin/env bash
case $BASH_VERSION in ''|[1-3].*|4.0*) echo "ERROR: Bash 4.1 required" >&2; exit 1;; esac

declare -A output_fds=( )

while IFS= read -r -d '' type && IFS= read -r -d '' content; do
  if [[ ${output_fds[$type]} ]]; then  # checking for existing file handle for this output file
    curr_fd=${output_fds[$type]}       # reusing it if available.
  else
    exec {curr_fd}>"$type.json"        # opening a new output file...
    output_fds[$type]=$curr_fd         # storing its file descriptor for later use.
  fi
  printf '%s\n' "$content" >&"$curr_fd"
done < <(jq -j '(.type) + "\u0000" + (. | tojson) + "\u0000"')

This method ensures that only a few records are processed at a time, limiting memory usage and enabling operation on large files with reasonable size records.

Is there a way to use jq to divide a JSON stream of objects into individual files according to the values of a specific object property?

Answer №1

Answer №2

Similar questions

The reason behind my unsuccessful attempt to utilize AJAX with the Google GeoChart API

Discovering key value pairs on a Linux server

Tips on renaming multiple JSON field names simultaneously

How to deserialize JSON with NewtonSoft when dealing with properties with the same name and arrays

Error: Trying to send FormData to ajax results in an Illegal Invocation TypeError being thrown

There was an issue parsing the query: An unexpected character "'"" was encountered in the Graph

Creating a multi-level JSON object from a string: A step-by-step guide

Enhance the editing capabilities of the Json data form

Displaying data from a JSON file in a table view in Swift can be challenging, especially if the type `ANY` is used and does

The current JSON array is unable to be deserialized (for example, [1,2,3])

Error encountered: "Invalid keyword 'using' while attempting to insert a new record into Smartsheet."

Saving a JSON data structure in a storage system

Developing a JSON object for an HTTP POST request

Utilizing the json_encode() function in PHP and JSON.parse() method in JavaScript for handling file data interchange

How can JavaScript/jQuery be used to update LocalStorage Objects when editing a form?

It is essential for Jquery to properly evaluate the first JSON result, as skipping

Unnecessary display of results caused by the use of CURLOPT_RETURNTRANSFER

Unable to display Klout topics using Ajax Json

Utilize Gson library to effectively load an object

Obtaining JSON data in a separate JavaScript file using PHP