Encountering a shell script parsing problem: Could it be an unexpected INVALID_CHARACTER due to Windows cmd shell quoting

What am I trying to accomplish? I have a JSON file called sonar-report.json. My goal is to iterate through the sonar-report.json in a shell script and extract values from the JSON file. To achieve this, I am using jq as my JSON parser ().

Below is the code snippet that I attempted to execute in my shell script:

alias jq=./jq-win64.exe
for key in $(jq '.issues | keys | .[]' sonar-report.json); do
    echo "$key"
    line=$(jq -r ".issues[$key].line" sonar-report.json)
done

Encountered Issue Upon executing the above script, I received the following error message:

jq: error: syntax error, unexpected INVALID_CHARACTER (Windows cmd shell quoting issues?) at <top-level>, line 1:

By providing static indices for the array elements, the script worked successfully:

alias jq=./jq-win64.exe

for key in $(jq '.issues | keys | .[]' sonar-report.json); do
    echo "$key"
    line0=$(jq -r ".issues[0].line" sonar-report.json)
    line1=$(jq -r ".issues[1].line" sonar-report.json)
done

Desired Outcome : My objective is to iterate through the values and display them on the console as follows:

alias jq=./jq-win64.exe
for key in $(jq '.issues | keys | .[]' sonar-report.json); do
    line=$(jq -r ".issues[$key].line" sonar-report.json)
    echo $line
done

Hence, the expected output should be:

15

This sample demonstrates the structure of my JSON file named sonar-report.json:

{
"issues": [
        {
            "key": "016B7970D27939AEBD",
            "component": "bits-and-bytes:src/main/java/com/catalystone/statusreview/handler/StatusReviewDecisionLedHandler.java",
            "line": 15,
            "startLine": 15,
            "startOffset": 12,
            "endLine": 15,
            "endOffset": 14,
            "message": "Use the \"equals\" method if value comparison was intended.",
            "severity": "MAJOR",
            "rule": "squid:S4973",
            "status": "OPEN",
            "isNew": true,
            "creationDate": "2019-06-21T15:19:18+0530"
        },
        {
            "key": "AWtqCc-jtovxS8PJjBiP",
            "component": "bits-and-bytes:src/test/java/com/catalystone/statusreview/service/StatusReviewInitiationSerivceTest.java",
            "message": "Fix failing unit tests on file \"src/test/java/com/catalystone/statusreview/service/StatusReviewInitiationSerivceTest.java\".",
            "severity": "MAJOR",
            "rule": "common-java:FailedUnitTests",
            "status": "OPEN",
            "isNew": false,
            "creationDate": "2019-06-18T15:32:08+0530"
        }
    ]
}

Your assistance is greatly appreciated. Thank you in advance.

Answer №1

It seems like the issue here is related to Windows/Unix line-ending compatibility problems, as mentioned in jq bugs 92 (for Cygwin) and 1870 (for MSYS2).

The solutions provided in those bug reports should resolve the issue. However, once the fix is included in the release binary (likely v1.7), using the new -b command-line option would be the simplest approach. (This option is available in recent jq preview builds; refer to the second bug report mentioned above):

for key in $(jq -b '.issues | keys | .[]' sonar-report.json); do
    line=$(jq -rb ".issues[$key].line" sonar-report.json)
    # Adding quotes for better style.
    echo "$line"
done

If you prefer not to upgrade to the next version of jq, or if it's not currently feasible, a workaround is to eliminate the CRs by using tr -d '\r' on the output of jq:

for key in $(jq -'.issues | keys | .[]' sonar-report.json | tr -d '\r'); do
    line=$(jq -r ".issues[$key].line" sonar-report.json | tr -d '\r')
    echo "$line"
done

Additionally, instead of iterating line-by-line in a shell loop, which can be inefficient due to multiple JSON input reparsing, you can utilize jq itself to iterate with this simpler command:

jq '.issues[].line' solar-response.json

This method will parse the JSON file just once and output each .line value. Consider using the -b command-line option or another workaround, depending on your specific requirements for the output.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Processing a JSON array of objects in AngularJS

When using Angular's fromJson function to parse a JSON string, I encountered an issue. If the JSON is a simple array like "[1, 2]", the code works fine. However, I need to work with an array of dictionaries instead. var str = "[{'title' ...

Utilizing Weather APIs to fetch JSON data

Trying to integrate with the Open Weather API: Check out this snippet of javascript code: $(document).ready(function() { if (navigator.geolocation) { navigator.geolocation.getCurrentPosition(function(position) { $(".ok").html("latitude: " + ...

What is the best way to craft an if/else statement for a situation when a twig variable is undefined?

When utilizing twig to declare a variable called user: <script type="text/javascript> {% if user is defined %} var user = { example: {{ userjson | raw }} }; {% endif %} </script> If a user is not logged in, an error message a ...

Implementing onClick functionality in RecyclerView post JSON data extraction

I recently implemented a RecyclerView in a fragment and successfully parsed JSON data from a website to display it in the RecyclerView following a helpful tutorial found at: Now, my next challenge is adding an onClick listener to the items in the Recycler ...

Tips for effectively parsing extensive nested JSON structures?

JSON Data on PasteBin I need assistance in converting this JSON data into an Object. It seems valid according to jsonlint, but I'm encountering issues with parsing it. Any help would be greatly appreciated. "Data":[{...},{...},] // structured like t ...

Extracting information from AJAX calls using a DataTable

When it comes to creating CRUD tables in school, I've always used scaffolding per page. However, I recently wanted to experiment with performing all operations without using Partial View. I decided to implement AJAX by following a tutorial on Everyth ...

extract the key identifier from the JSON reply

My JSON ResponseData example for form0 is provided below: { "MaterialType": "camera", "AssetID": 202773, "forms": [ { "release": "asyncCmd/accessCameraMulti", "action": "rest/Asset/202773/cameraAccessMultiple", ...

Utilizing JSON for live population of search filter results

I'm currently developing a search function for my website that will sift through a JSON Object using regular expressions. My goal is to have the results displayed in real time as the user types, similar to how Google shows search suggestions. However ...

A guide to dynamically extracting values from JSON objects using JavaScript

I have a JSON array with a key that changes dynamically (room number varies each time I run the code). My goal is to access the inner JSON array using this dynamic key. Here's what I've attempted so far, but it's throwing an error. Here is ...

Tips for Saving JSON Response from Fetch API into a JavaScript Object

I am facing an issue trying to store a Fetch API JSON as a JavaScript object in order to use it elsewhere. The console.log test is successful, however I am unable to access the data. The Following Works: It displays console entries with three to-do items: ...

retrieving JSON data within the controller

When I use the command console.log($scope.data), I am able to view my JSON file. Additionally, by using <div ng-repeat="item in data">, I can see all the items in the view. However, when I try console.log($scope.data[0]) or console.log($scope.data[0] ...

A guide on organizing nested JSON objects in JavaScript

I am currently using JavaScript to retrieve JSON data that looks like this: [{ "data": { "serialNumber": "12345678", "loopCount": 2, "temperature3": 22.74921781259558, "temperature2": 21.459065450414467, "temper ...

Implementing Node.JS ajax to update current JSON information

I am seeking assistance in updating data within a JSON file using NODE.JS. Currently, my method adds the data with the same ID as expected. However, upon receiving the data back, it eliminates the last duplicate because it encounters the old value first. I ...

What is the best method for encoding non-ASCII characters in JSON.stringify as ASCII-safe escaped characters (uXXXX) without the need for additional post-processing?

In order to send characters like ü to the server as unicode characters but in an ASCII-safe string format, I need them to be represented as \u00fc with 6 characters, rather than displaying the character itself. However, no matter what I try, after us ...

Exploring ways to retrieve nested values from JSON data using the Instagram API and Javascript

Trying to modify the script found at https://github.com/bigflannel/bigflannel-Instafeed in order to access Instagram photos on a website. Unfortunately, the script does not currently support displaying photo comments. I attempted to make modifications that ...

Runtime.UnhandledPromiseRejection - Oops! Looks like we're trying to read properties of something that doesn't exist (specifically 'headers')

I'm facing an unexpected error that has left me puzzled. Let me walk you through what I am trying to accomplish: The task at hand involves fetching data from one API and then transmitting it to another. This process is executed as a background-funct ...

What is the best way to transfer JSON data to a different controller in AngularJS?

Hello, I'm still learning AngularJS and facing an issue with the following code snippet. app.config(function($routeProvider) { $routeProvider .when('/', { templateUrl: "partials/home.html", controller: "mainControlle ...

Eliminate duplicate items using the reduce method in JavaScript

Working with a set of Json Objects, I use a javascript map function to list each field along with an array of its possible types. For example: birthDate, [Date, String, String, String, String] isMarried, [Boolean, Boolean, Boolean, Boolean, String] name, ...

Ensuring Node.js backend JavaScript waits for completion of my bash script before proceeding

Running three bash commands through a Node.js code snippet. Here's a portion of the script: exec(str, function(error, stdout, stderr){ console.log('stdout:'+stdout); console.log('stderr:'+stderr); if(error!=null){ ...

Arranging JSON information based on category

I am currently populating an HTML table with data retrieved from a JSON file. It currently displays all the data in the order it appears in the JSON file, but I would like to organize it into different tables based on the "group" category in the file such ...