Utilizing Node and Socket.io to transmit data periodically via websockets from a CSV file

I am relatively new to working with Node.js and Express.js. My goal is to set up a websocket server that can send CSV data at irregular intervals stored within the file itself, line by line. The structure of the CSV looks something like this: [timeout [ms], data1, data2, data3 ...]

I have managed to create a functioning websocket server that communicates with the client.

My aim is to find the best solution for achieving the following tasks effectively: 1. Read a line from the CSV file 2. Send that line using WebSockets 3. Pause reading for the time specified in the first value of the row 4. Resume reading after the interval has passed, and repeat the process from step 1.

Although I have made some progress, my current code may not be correct as I'm still learning. It seems like the pause() function is not working as expected.

var $    = require('jquery')
,csv = require('csv');

exports.index = function(server){
  var io   = require('socket.io').listen(server);

  io.sockets.on('connection', function (socket) {

  socket.on('startTransmission', function(msg) {
    csv()
    .from.path('C:/dev/node_express/csv/test.csv', { delimiter: ',', escape: '"' })
    .on('record', function(row,index){
      var rowArray = $.parseJSON(JSON.stringify(row));
      var json = {},
          that = this;
        $.each(rowArray, function(i,value){
          json[keys[i]] = value;
        });
        socket.emit('transmitDataData', json);
        //this.pause(); //I guess around here is where I'd like to pause 
        // setTimeout(function(){
        //   that.resume();  //and resume here after the timeout, stored in the first value (rowArray[0])    
        // }, rowArray[0]);

    });
});
});
};

The commented out code unfortunately does not work - All data is sent immediately, row after row, the function doesn't pause

Answer №1

I encountered a similar issue with a different scenario. The problem arises when using pause() on the stream as it pauses the reading of the underlying stream but not the parsing of the csv records, causing the record event to be triggered with the remaining records from the last chunk read from the stream. In order to synchronize them, I adapted my approach like this:

var countRows = 0;
var countActions = 0;

stream.on('record', function(row, index) {

    countRows++;

    // Pause here and anticipate more record events until all raw data from the stream is processed
    stream.pause();

    runner.do(row, function(err, result) {
        if (countActions == countRows) {
            stream.resume();
        }
    });
});

In your situation, I recommend buffering the rows and releasing them with a timer. Here's a refactored snippet for illustration purposes:

var $ = require('jquery'),
    csv = require('csv');

exports.index = function(server){

  var io = require('socket.io').listen(server);
  io.sockets.on('connection', function (socket) {

      socket.on('startTransmission', function(msg) {

        var timer=null;
        var buffered=[];
        var stream=csv().from.path('C:/dev/node_express/csv/test.csv', { delimiter: ',', escape: '"' });

        function transmit(row) {        
           socket.emit('transmitDataData', row);                                     
        }       

        function drain(timeout) {                                                    
           if (!timer) {
               timer = setTimeout(function() {                                    
                   timer = null;
                   if (buffered.length <= 1) {
                       stream.resume();
                   } else {                        
                       var row = buffered.shift();
                       transmit(row);
                       drain(row[0]);                        
                   }

               }, timeout);               
           }                
        }

        stream.on('record', function(row, index){                        
            stream.pause();                                                                                   
            if (index == 0) {                            
                transmit(row);                                               
            } else {                            
                buffered.push(row);                                   
            }                                                       
            drain(row[0]);                                                               
        });

        stream.on('end', function() {
            // No more rows. Wait for buffer to empty before cleaning up.
        });

        stream.on('error', function() {
            // Handle errors.
        });

    });
};

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Attempting to grasp the concept of Google Cloud Function for SendGrid's inbound parsing feature

I'm currently working on setting up a sendgrid designated URL at www.mydomain.com/mail/post. My goal is to create a Google Cloud Function that will respond to emails coming in from the inbound parse API. However, I'm facing issues with triggering ...

Mysterious package found its way into my node_modules directory

Recently, I've been encountering a persistent package installation issue in my node_modules folder with a package named <a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="d8a8b0b9b6acb7b5b2abf5a8aabdbaadb1b4ac98eaf6e9f6e9ee"&g ...

Choosing a service endpoint based on URL patterns with Express-Gateway

I am faced with the challenge of consolidating multiple individual servers under the same domain behind a gateway. Currently, each server has different accessible names from the outside world. Our sales team wishes to offer customers unique URLs, leading t ...

MySQL query malfunctioning when attempting to perform a basic insert operation

https://i.stack.imgur.com/4S6X1.png const express = require('express') const app = express() const cors = require('cors'); const mysql = require('mysql'); const port = 3001 const db = mysql.createPool({ host: "localhos ...

The layout option is not specified within the ejs-mate package

error boilerplate I am baffled as to why the layout is not being defined. I have installed ejs-mate and ejs, yet it still gives this error. <% layout('layouts/boilerplate') %> <h1>All campgrounds </h1> <div> <a ...

Getting the name of parameters from Vue 3/Express

Greetings, I want to start by apologizing if the title of my question caused any confusion for those reading it. I struggled to find the right wording. The issue at hand involves a problem with sending a get request from my axios instance to my express ins ...

Is there a way to sequentially execute requests in a loop?

My goal is to extract a list of URLs from the request body, pass them to a request function (using the request module) to retrieve data from each URL, and then save that data to MongoDB. The response should be sent only after all requests are completed, in ...

What is the most effective method for cropping and uploading images using express?

Using the knox library, I have successfully uploaded an image to a S3 server through express. My goal is to allow users to crop their profile picture, similar to what Facebook offers. I am considering using Jcrop for the user interface and imagemagick fo ...

Utilizing the power of async/await in combination with the versatile Bluebird library to seamlessly

I'm currently developing a library that utilizes the power of async/await. My main concern is whether or not it's possible to integrate native modules like fs smoothly with async/await. As far as I understand, async/await is essentially just synt ...

Ways to fix the issue of an unspecified user error in authjs

Having trouble with creating a web token using passport LocalStrategy and passport-jwt. I keep getting a user undefined error in auth.js ****401 Unauthorized**** (if (!user) { return res.json(401, { error: 'message' });}). How can I fix this issu ...

The Nodejs server is up and running on Heroku, but unfortunately, there are

I recently launched a mini-forum app with features like authentication, posts, comments, and a gateway for routing requests to different endpoints. While everything works well locally, I encountered an issue after deploying the app to Heroku where only GET ...

Encountered a problem during the installation of Ionic on Ubuntu 18.04

Looking for guidance on the installation process of Ionic 4 on Ubuntu 18.04. Can anyone advise on the compatible versions of npm, Node.js, Cordova, and Android SDK required for a successful installation? I attempted the installation myself but encountere ...

The project is throwing an error: Unable to locate module './lib/md5-hex' in Ember JS

I attempted the following steps: uninstalling md5-hex using npm with the command: npm uninstall md5-hex --save reinstalling md5-hex using npm with the command: npm install md5-hex --save After performing these actions, I ran ember s but unfortunately, i ...

I need help setting up a personalized SMTP server for sending notification emails using Node.js

I have a need to send notification emails from my application to any email address, such as a Gmail account. I have explored modules like smtp-server, smtp-connection, and emailjs. This is what I have implemented so far: var SMTPServer = require('sm ...

A guide on implementing the MVC architecture in a web application with Node.js, Express, and PostgreSQL

I'm struggling with implementing the MVC architecture in my node web app, specifically when it comes to separating the model from the controller. Currently, I have the views properly organized (all my .ejs files) and I consider my app.js as the contr ...

What is preventing Node.js from executing my JavaScript code in the terminal?

I need help understanding why my JavaScript code isn't working properly. Can someone explain the Uncaught SyntaxError: Unexpected Identifier error message? ...

What is the best way to manage errors on my server to ensure it remains stable and never crashes?

Consider this server route example using expressjs: app.get('/cards', function(req, res) { anUndefinedVariable // Server doesn't crash dbClient.query('select * from cards', function(err, result) { anUndefinedVariab ...

The uploaded file exceeds the maximum size limit when utilizing Node.js API to upload files on AWS S3

Even though I have found numerous solutions on stackoverflow, I am still encountering errors in some devices when trying to upload more than 2 images using multipart in my node.js API. The error message that I receive is "Request entity too large". I am u ...

Fixing permission issues during the installation of Angular Client on MacOS: A comprehensive guide

As a beginner coder diving into Angular and Node through an Udemy tutorial, I've encountered some issues. While I have successfully installed Node.js version 16.15.1, my attempts to install the angular client have consistently failed (see screenshot a ...

Dynamic database switching in Node API REST services

Is there a way to dynamically switch the database configuration for my pool in Node/Express application? I am using a REST API with node/express and have multiple databases. What I need is for the API to select the appropriate database based on the user.c ...