Concurrency problem with multicore processing in Node.js

I need to ensure that the processing of the first item ('http://www.google.com') in array2 is completed before starting on the second item (''). However, the request is asynchronous so the current result is:

http://www.google.com
http://www.amazon.com
http://www.apple.com
1
2
.
.

What I actually want is:

http://www.google.com
1
2
3
.
.
http://www.amazon.com
1
2
.
.

Here is my code:

var async = require('async');
var request = require('request');

var array1 = ['http://www.google.com', 'http://www.amazon.com', 
'http://www.apple.com'];
var array2 = [1, 2, 3, 4, 5, 6, 7, 8, 9];

array1.each(function (item, callback) {

    console.log(item);
    aa(item, callback)

})

function aa(url) {

    request(url, function (err, res, html) {
        array2.forEach(function (item, callback) {
            //This part involves crawling the current page
            console.log(item);
            sleep(1000);
        });
    })
}

Can anyone suggest a way to achieve this?

Answer №1

You've successfully added the async module, but you haven't utilized it yet. To make use of it, consider using the async.eachSeries() function to meet your requirements. Below is a code snippet that might be helpful:

const async = require('async');
const request = require('request');

const array1 = ['http://www.google.com', 'http://www.amazon.com', 'http://www.apple.com'];
const array2 = [1, 2, 3, 4, 5, 6, 7, 8, 9];

function fetchData(url, callback) {

    request(url, function (err, res, html) {
        async.eachSeries(array2, function iteratee(item, callback2) {
            console.log(item);
            setTimeout(function() {
                callback2(null);
            }, 1000);
        }, function done() {
            callback(null)
        });
    })
}

async.eachSeries(array1, function iteratee(item, callback) {
    console.log(item);
    fetchData(item, callback)
}, function done() {
    console.log('Task completed');
});

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Creating a shared singleton instance in Typescript that can be accessed by multiple modules

Within my typescript application, there is a Database class set up as a singleton to ensure only one instance exists: export default class Database { private static instance: Database; //Actual class logic removed public static getInstance() ...

The application is unable to interpret parameterized queries when utilizing Express and PostgreSQL

Complete newbie over here, I'm attempting to revamp my endpoint in order to enable sorting the return table by multiple columns read in via variables 'sort_by' and 'order', which have default values. However, when running the test ...

Setting Collation and Charset in Node.js Models using Sequelize and Mysql

Currently, I am utilizing sequelize with node and node-mysql in my project. When creating models using the sequelize-cli, the resulting code appears as follows: 'use strict'; module.exports = function(sequelize, DataTypes) { let songs = seq ...

Debian on Node Express always serves index.html by default

I am currently facing an issue with my node express server. It works correctly on a Windows system, but when running on Debian, it always returns index.html. Whenever I try to access localhost:port/ in the browser, it displays index.html. The problem is t ...

Steps for transferring data from a POST response to the client in NodeJS/ExpressJS

I am currently in the process of setting up an EasyPost API and I need to save some data that is included in a response from a POST route. My goal is to send this data directly to the client for storage, but I am struggling to determine where or how to do ...

Tips on setting the appropriate route in Node Express for accessing bower_components?

I am having trouble with correctly defining the path for bower components in my file structure. Here is how it looks: projectName | - client/ | - app/ | - bower_components/ | - node_modules/ (grunt tasks) | - test ...

Having trouble with hanging and failing npm installation on Windows 10?

I keep encountering issues with my npm installations on Windows 10, even though I have the latest stable versions of node.js and npm. Every time I try to run 'npm install' in my project folder, which was initially set up with express following th ...

Utilizing AND and OR operators in Node.js for retrieving data

Hey there! I could use some assistance with the following issue: Objective: I am looking to filter data based on department content. My goal is to retrieve data from MongoDB based on your job title or department. For instance, if I belong to the Email Tea ...

When attempting to add or store data in MongoDB, it triggers a 500 server error

Greetings, I am currently working on developing a CRUD app using the MEAN stack. The Express application loads successfully and retrieves the "contactlist" collection from the database. However, when attempting to make a POST request to "/api/contacts", an ...

Tips for accessing private variables

After running this code in nodejs, I noticed that the 'Count' becomes negative for large values of 'count'. It made me wonder, is it the fault of 'count' or 'chain'? What would be the best approach to modify the &apo ...

Unable to find the module... designated for one of my packages

Within my codebase, I am utilizing a specific NPM package called my-dependency-package, which contains the module lib/utils/list-utils. Moreover, I have another package named my-package that relies on my-dependency-package. When attempting to build the pr ...

Mongoose encountered an error when trying to convert the value "me" to an ObjectId at the path "_id"

Although there have been multiple versions of this question, I couldn't find a helpful solution. function isAuthenticated() { return compose() // Validate jwt .use(function (req, res, next) { // allow access_token to be passed through ...

Do not use npm to install underscore libraries

How can I resolve the error I encountered while attempting to install packages using npm? Here is my packages file: "dependencies": { "express": "~3.3.6", "socket.io": "0.9.16", "jade": "~0.35.0", "less-middleware": "~0.1.12", "redis ...

What is the best method for transforming a string into JSON format using Node.js?

Looking for assistance in listing the data from a CSV file stored in S3 using Node.js code. Below is the provided code, but I am seeking help to achieve the expected output as shown: CODE: const AWS = require('aws-sdk'); const fs = require(&apo ...

Strategies for efficiently handling time in React and Node.js without being impacted by timezone issues

Despite exhausting all methods to assess time in my mern application, I have yet to find a solution. My search online has also been fruitless... ...

Is there an easy method for sorting through JSON data with various criteria?

After receiving a response from my web service containing an array of objects, I am looking to filter the data based on various combinations of values. What is the most efficient way to do this without impacting the performance of my web service? Data: Th ...

Having trouble downloading files in React and Node.js

I need to retrieve a specific file from the uploads folder and download it into my download folder. Both folders are located in the same directory, and the file's name is BasicUserFile-id.jpg. On my second attempt, I tried hardcoding the file name bu ...

Node error connecting to Mongo database

Having trouble connecting my node server to MongoDB, here is the code snippet I am using: var http = require("http"); var url = require("url"); var Router = require('node-simple-router'); var router = Router(); var qs = require('querystring ...

A guide to mocking req.session in Mocha/Chai API unit testing

When testing REST API unit tests with Mocha/Chai, I ran into an issue where I needed to mock req.session.someKey for certain endpoints. How can I successfully mock req.session? Currently, I am in the process of writing unit tests for a NodeJS Express app ...

Tips for re-translating images from another website without revealing the original source URL

Here's a link to an image: this or that. I want to display the same image on a different link such as http(s)://examplewebsite.com/john, without actually redirecting. I've attempted using express.static but it hasn't been successful. Thank ...