Using JQ to structure logs in shell scripting

I've been looking to save some shell scripts into a file for collection by an agent like Fluentbit and sending them off to Cloudwatch and Datadog.

I came across this example online that works effectively with the use of jq.

__timestamp(){
  date "+%Y%m%dT%H%M%S"
}
__log(){
  log_level="$1"
  message="$2"
  echo '{}' | jq \
    --arg timestamp "$(__timestamp)"
    --arg log_level "$log_level" \
    --arg message "$message" \
    '.timestamp=$timestamp|.log_level=$log_level|.message=$message' >> logs.log
}
__log "INFO" "Hello, World!"

The only issue is that the output is formatting as a full JSON with newlines and tabs. It's easy to read visually, but not ideal for cloud logging services.

{
  "timestamp": "20220203T162320",
  "log_level": "INFO",
  "message": "Hello, World!"
}

How can I adjust it to format the output on a single line like this?

{"timestamp": "20220203T171908","log_level": "INFO","message": "Hello, World!"}

Answer №1

To achieve more concise output, utilize the --compact-output or -c option like so:

jq -c --arg … '…' >> logs.log

If you refer to the official manual:

--compact-output / -c:
    By default, JSON output in jq is nicely formatted. Activating this option will produce more condensed output by placing each JSON object on a single line.

After reviewing the piece on "Structured Logging", here is how I would have designed the __log function:

__log(){
  jq -Mcn --arg log_level "$1" --arg message "$2" \
    '{timestamp: now|todate, $log_level, $message}'
}
__log "INFO" "Hello, World!"

Answer №2

🤦 I can't believe I missed Step 3.0 The Final Steps from this amazing tutorial. It provided the perfect solution to my question.

If anyone else was wondering, you can find the solution here:

__timestamp(){
  date "+%Y%m%dT%H%M%S"
}
__log(){
  log_level="$1"
  message="$2"
  echo '{}' | \
  jq  --monochrome-output \
      --compact-output \
      --raw-output \
      --arg timestamp "$(__timestamp)" \
      --arg log_level "$log_level" \
      --arg message "$message" \
      '.timestamp=$timestamp|.log_level=$log_level|.message=$message'
}
__log "INFO" "Hello, World!"

This code will output:

{"timestamp":"20210812T191730","log_level":"INFO","message":"Hello, World!"}

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Error parsing JSON data with jQuery.parseJSON

I'm having trouble converting the following string to a JSON object. An error keeps popping up, but I can't figure out why. var jsonData = "{'firstName': 'John','lastName': 'Smith', 'age': 25, &a ...

Transform nested JSON key value pairs into a flat structure for easy conversion to a CSV file

I have a JSON file with some unique nested label/value pairs. Here's a snippet of the data: { "total_count":10000, "count_from":1, "count_to":1000, "contacts":[ { "contact_id":"ABC123", "contact_last_name":"Last nam ...

Using PHP to loop through SQL data and convert it to JSON format, mapping information from two tables based on their related IDs

I always wish for your good health. I am currently facing an issue trying to generate a JSON array from 2 tables. Here are the tables I have: Table 1: app_produk_ready Table 2: app_produk_ready_img Below is the PHP code I have written: <?php require_ ...

Combining arrays that run parallel in Jquery

Seeking a solution similar to looping through multiple arrays in parallel using jQuery, but this time to construct an HTML page. I have three sets of parallel objects with arrays, potentially forming a 2D array. {"images":["Bellevue\/images\/1. ...

Build a new shop using a section of data from a JSON database

Let's say I have a JSON store that is set up as shown below var subAccountStore = new Ext.data.JsonStore({ autoLoad: true, proxy: { type:'ajax', url : '/opUI/json/subaccount.action?name="ABC"' }, fields: ['acc ...

"Unraveling the mysteries of deserializing JSON with unfamiliar

Perhaps this may not be achievable. Check out the functioning code below: HttpResponseMessage playerResponse = await client.GetAsync("2018/export?TYPE=players&DETAILS=1&SINCE=&PLAYERS=9988%2C13604&JSON=1"); if (playerResponse.IsSuccessSta ...

What is the process for transmitting data in JSON format generated by Python to JavaScript?

Utilizing Python libraries cherrypy and Jinja, my web pages are being served by two Python files: Main.py (responsible for handling web pages) and search.py (containing server-side functions). I have implemented a dynamic dropdown list using JavaScript w ...

How can one transfer information from a client to a server and complete a form using JavaScript?

Can the information be transferred from a client to the server using just JS, then fill out a form on the server, and finally redirect the client to view a pre-filled form? I am considering utilizing AJAX to send a JSON object, but unsure if it will be s ...

Troubleshooting problem with jQuery's on() function on elements added dynamically

My webpage retrieves data from a JSON source and generates a product list with two different styles that the user can switch between by clicking a button. The styles are toggled by changing the parent ul to ul.fullview using CSS and a simple toggleClass() ...

Make sure that 'RegisterAPI' has a designated 'serializer_class' attribute, or if not, make sure to customize the 'get_serializer_class()' method

I am currently facing an issue while trying to display a register view. Despite making modifications to the views file, I am still encountering errors and unsure about how to resolve them. Below is the content of my views.py file: from django.shortcuts imp ...

Is it possible to retrieve a single attribute from a parent object using the Fast JSON API within Rails?

I am in the process of developing a travel application that utilizes a backend Rails API. To serialize my data, I have chosen to implement the Fast JSON API. Within my app, there exists a collection of countries, each containing numerous cities and attract ...

Obtaining the value of a JSON object in PHP when its key contains dots

I have a JSON data structure that looks like this: {"root":{ "version":"1", "lastAlarmID":"123", "proUser":"1", "password":"asd123##", "syncDate":"22-12-2014", ...

Is there a way to write numerous JSON files with distinct names during each iteration of a loop, such as incrementing by +1?

Is there a way to create new files for each json file created and dumped in a loop? Snippet of my code: (the specifics of the for-loop are not important, just to provide context). with open('my_json.json') as file: config = json.load(file) fo ...

Transforming JSON data into an Angular TypeScript object

Delving into the realm of Angular on my own has been quite an enlightening journey, but I'm currently facing a specific issue: My aim is to create a website using both Spring for the back end and Angular 7 for the front end. However, I've encoun ...

Using the Play framework to showcase data in an HTML table

I have a project where I am working with multiple tables, one of which is a Person table containing id and name attributes. I have successfully retrieved the data from the database using MySql and converted it into JSON. However, I am struggling to find a ...

Leverage boost library to extract data from a json array

Here's the JSON data that needs to be parsed: {"jsonrpc":"2.0","method":"subscription","params":{"channel":"book.BTC-PERPETUAL.raw","data":{"type":"change&q ...

Exporting JSON data to an Excel file using an array of objects with embedded arrays

I am currently developing a fitness application that allows users to create workout routines and download them as an excel file. The data is structured as an array of objects, with each object representing a workout date and containing details of the exerc ...

Retrofit encountered an unexpected object instead of an array at line 1, column 2, with a path of $

I am encountering an error with Retrofit. Despite trying some solutions found here, none seem to work for me. Below is the code I am working with: public void getMaterials() { RestAdapter adapter = new RestAdapter.Builder().setEndpoint(Root_ ...

Exploring Nodejs: Navigating through ServerResponse (or, Unraveling the mystery of client URL parameters)

User Interaction: Upon making a post request to the server, the user is redirected to Spotify's authorization endpoint where they grant access to our web application. After completion, the user is redirected back to our website with a URL that includ ...

Unique JSON binding feature in WinJS ListView

I'm having trouble connecting this JSON data to my list view. Nothing is showing up on the screen. Data.js (function () { "use strict"; var _list; WinJS.xhr({ url: 'http://pubapi.cryptsy.com/api.php?method=marketdatav2' }).t ...