I am currently developing a Next.js API page to extract data from a series of URLs. Interestingly, the URLs containing an apostrophe character ' fail to return any data, while those without it work perfectly fine. Surprisingly, when I execute the same code in Chrome's dev console or a RESTful Client, the URLs with the apostrophe do return data successfully. Both situations establish a successful connection (HTTP 200), but the discrepancy in outcomes occurs exclusively within the Next.js project. Any insights on what could be causing this behavior difference?

UPDATES:

1. I attempted using both encodeURI and encodeURIComponent, but unfortunately, they did not resolve the issue.
2. Even the substitution of AYR%27KA instead of AYR'KA did not rectify the problem.
3. Upon comparing the URL passed to the fetch function with the value retrieved from response.url, I noticed that in the browser, everything remains consistent. However, in Next.js, the AYR'KA transforms into AYR%27KA.
4. Collaborating with @Phil and thorough investigation highlighted Node.js's URL default component as the root cause. Here is the NPM link for reference and here is their GitHub repository link. It was discovered that when node-fetch constructs a Request object, they invoke URL.parse on the string provided. Exploring the file "url.js" on their GitHub, particularly lines 353-360 showcased how specific characters, including the apostrophe/tick mark ', are forcibly escaped despite being non-restrictive characters. (NOTE: This update reflects changes made on July 24th, 2021, and refers to the latest version of the code at the time.)

*NOTE: To avoid CORS issues, testing can be conducted via Browser Dev Tools directly from the website here

const worksEverywhereURL = "https://robertsspaceindustries.com/api/starmap/star-systems/STANTON",
    brokenInNodeURL = "https://robertsspaceindustries.com/api/starmap/star-systems/AYR'KA";

async function getJSONFrom(url){
    const options = {method: "POST"};
    let response, json;
    
    console.log("url: ",url)
    response = await fetch( url, {method:"POST"} );
    console.log("response url: ",response.url)
    json = response.json();

    return json;
}

function checkResponse(json) {
    try {
        return json.data.resultset.length > 0;
    } catch(err) {
        return false;
    }
}

const example1 = await getJSONFrom( worksEverywhereURL );
const example2 = await getJSONFrom( brokenInNodeURL );

console.log( checkResponse(example1) ); //"true" in chrome and "true" in next.js api page
console.log( checkResponse(example2) ); //"true" in chrome and "false" in next.js api page
//Ongoing output comparison in browser vs. command prompt detailed in original text.

Displaying below, the API page code snippet indicating an empty resultset being returned:

export default async function handler(req, res) {
    const worksEverywhereURL = "https://robertsspaceindustries.com/api/starmap/star-systems/STANTON",
        brokenInNodeURL = "https://robertsspaceindustries.com/api/starmap/star-systems/AYR'KA";

    async function getJSONFrom(url){
        const options = {method: "POST"};
        let response, json;

        response = await fetch( url, {method:"POST"} );
        json = response.json();

        return json;
    }

    function checkResponse(json) {
        try {
            return json.data.resultset.length > 0;
        } catch(err) {
            return false;
        }
    }

    const example1 = await getJSONFrom( worksEverywhereURL );
    const example2 = await getJSONFrom( brokenInNodeURL );

    console.log( checkResponse(example1) ); //"true" in chrome and "true" in next.js api page
    console.log( checkResponse(example2) ); //"true" in chrome and "false" in next.js api page
}

For cases where the response returns incorrectly, the structure resembles:

{
  "success": 1,
  "code": "OK",
  "msg": "OK",
  "data": {
    "rowcount": 0,
    "totalrows": 0,
    "estimatedrows": false,
    "pagesize": 0,
    "pagecount": null,
    "page": 1,
    "offset": 0,
    "startrow": 0,
    "resultset": []
  }
}

Whereas a correct response contains:

{
  "success": 1,
  "code": "OK",
  "msg": "OK",
  "data": {
    "rowcount": 1,
    "totalrows": 1,
    "estimatedrows": false,
    "pagesize": 0,
    "pagecount": null,
    "page": 1,
    "offset": 0,
    "startrow": 0,
    "resultset": [
      {
        "id": "398",
        "code": "AYR'KA",
        "description": "Primarily a military system focused on supporting, housing, and training Xi’an ground troops. Its close proximity to the Perry Line made it strategically important during the Human / Xi’an cold war. Once relations normalized, Xi’an forces from the Perry Line systems withdrew to here.  ",
        "frost_line": "179.10000000"
        ...
      }
    ]
  }
}

In Relation to Update 4: Provided are snippets extracted from url.js on GitHub signifying the intricate dynamics behind the unreachable target URL (disconnected portions originating from a dependency's dependency).

-Referencing Lines 56-70 alongside their explanatory comments:

/*
 * RFC 2396: characters reserved for delimiting URLs.
 * We actually just auto-escape these.
 */
delimiters = [
  '<', '>', '"', '`', ' ', '\r', '\n', '\t'
],

// RFC 2396: characters not allowed for various reasons.
unsafe = [
  '{', '}', '|', '\\', '^', '`'
].concat(delimiters),

// Compliant with RFCs yet susceptible to XSS attacks. Always warrant escaping.
autoEscapeChars = ['\''].concat(unsafe),

-Further exploration into lines 348-361 inclusive of insightful observations where the previously defined characters come into play:

/*
 * First and foremost, ensure all "autoEscape" characters undergo
 * proper escaping, regardless of whether encodeURIComponent detects
 * the necessity
 */
for (var index = 0, size = autoEscapeChars.length; index < size; index++) {
  var aechar = autoEscapeChars[index];
  if (remaining.indexOf(aechar) === -1) { continue; }
  var escChar = encodeURIComponent(aechar);
  if (escChar === aechar) {
    escChar = escape(aechar);
  }
  remaining = remaining.split(aechar).join(escChar);
}