Strategies for safeguarding data in Node.js in the event of a crash

In my express app, users have the ability to create subdomains which act as proxies for sites dynamically.

The challenge lies in potentially losing data if the app encounters errors, crashes, or restarts. This would result in all the unique proxies created by users being lost.

var app = require('express')();
var proxy = require('express-http-proxy');
var vhost = require('vhost');

app.get('/create', function(req, res){
    app.use(vhost('subdomain.mysite.com', proxy('http://example.com')));
    res.send('Proxy Created');
});

app.listen(8080);

While I could store these proxies in a database and recreate them individually upon restart, this method may not be efficient for managing potentially thousands of uniquely created proxies.

I am aware that the newly created routes are stored in the app variable (app._router). Is there a way to retrieve routes from an external source such as redis or mongo?

Alternatively, is there a way to persist this routing information to prevent data loss?

Are there any node management tools like PM2, forever, or supervisor, that can help prevent or recover from such situations?

If you have any advice or suggestions for a better solution, please feel free to share. Your input is greatly appreciated.

Answer №1

To ensure the persistence of this information even during server process crashes, it is necessary to save it in a way that survives such incidents. The typical method of achieving this is by storing the data in a persistent store, often on a disk drive.

From an architectural standpoint, there are two primary approaches you can take:

  1. Maintain your current architecture but incorporate a "save to disk" mechanism whenever the proxies' state is altered (added, removed, or modified). This involves saving the entire current state of routes to disk each time a change occurs. To implement this, introduce a new RAM-based data structure to hold the current state of all created routes. While you could attempt to read this from Express, it may be more beneficial to manage your own data structure, granting you control over which attributes are saved. By saving the state with every alteration, the most you risk losing is the operation that was being processed for saving at the moment. Ensure adequate concurrency protection in the save operation to prevent conflicts between simultaneous saves - this is a relatively straightforward task.

  2. Transition to a database-centric design where each modification triggers the writing of that particular change to a database equipped with a persistent store. Upon server restarts, retrieving the state from the database allows you to recreate the pre-restart status.

While a database solution typically offers better scalability, simply writing the whole state to disk after each change is usually simpler (avoiding the need for a database) - employing JSON file format might be the easiest option. Both methods can function effectively within a certain scale limit, beyond which the database approach becomes preferable (in terms of changes per second or total proxy count).

Storing these in a database and then retrieving them individually doesn't seem efficient for potentially handling thousands of uniquely generated proxies.

Consider reframing this perspective: a database setup is likely the more scalable choice for managing a large number of proxies, although dealing with thousands shouldn't pose a significant challenge using either of the mentioned techniques.

The newly created routes are stored in the app variable (app._router). Is it possible to pull routes from another source like Redis or MongoDB?

If I were handling this scenario, I would personally persist the data to a permanent storage repository each time a new proxy is added, removed, or updated, without overly relying on Express for tracking purposes.

Is there a way to ensure the continuity of routing information?

Express lacks built-in features for automatic route persistence. It's assumed in the express framework that the startup code or subsequent operations will cater to creating the required route handlers, leaving room for manual intervention in preserving this info.

Do any node management tools (PM2, forever, supervisor, etc.) safeguard against or recover from such issues?

To my knowledge, such tools mainly focus on process management rather than maintaining internal states within processes.

Is there an optimal alternative solution? Any advice would be highly appreciated.

Saving the data independently and restoring the state from the persisted storage upon server initialization, as outlined in the aforementioned strategies, appears to be a prudent course of action.


Here's a practical example of the first approach described above, illustrating the practice of saving data post every modification and reloading the stored state during server boot-up:

const app = require('express')();
const proxy = require('express-http-proxy');
const vhost = require('vhost');
const Promise = require('bluebird');
const fs = Promise.promisify(require('fs'));

let proxyData = [];
readProxyData();

app.get('/create', function(req, res){
    app.use(vhost('subdomain.mysite.com', proxy('http://example.com')));

    // save proxy data
    proxyData.push({subDomain: 'subdomain.mysite.com', userDomain: 'http://example.com'})
    saveProxyData();

    res.send('Created');
});

app.listen(8080);

// invoked whenever a new proxy is added 
// (after updating proxyData with proxy details)
function saveProxyData() {
     // leverage promises to sequence successive saves
     proxyData.promise = proxyData.promise.then(function() {
         return fs.writeFileAsync("proxyState.json", JSON.stringify(proxyData));
     }).catch(function(err) {
         console.err(err);
         return;
     });
}

// called only during server start
function readProxyData() {
    try {
        proxyData = require("proxyState.json");
    } catch(err) {
         console.err("Error reading proxyState.json - proceeding with no saved state: ", err);
    }
    proxyData.promise = Promise.resolve();

    // reinstate previous proxies stored in proxyData
    proxyData.forEach(function(item) {
        app.use(vhost(item.subDomain, proxy(item.userDomain)));
    });
}

Answer №2

One solution is to handle unhandled errors or exceptions in your code. By using the process and uncaughtException methods, you can ensure that your application does not exit unexpectedly when an error occurs.

process.on('uncaughtException', function (err) {
  console.log('Caught exception: ' + err);
});

Answer №3

When a process comes to an end, any information stored in its temporary memory is erased. To prevent data loss, it's recommended to save that data into a more durable system like a database or file storage as soon as it's captured by your server.

Recently, I developed a convenient npm package called cashola to simplify this task.

Here's how you can implement the same functionality using cashola:

var cashola = require('cashola');
var app = require('express')();
var proxy = require('express-http-proxy');
var vhost = require('vhost');

var state = cashola.rememberArraySync('state');

for (const item of state) {
    app.use(vhost(item.domain, proxy(item.proxy)));
}

app.get('/create', function(req, res){
    const vhostDomain = 'subdomain.mysite.com';
    const proxyDomain = 'http://example.com';
    state.push({ domain: vhostDomain, proxy: proxyDomain });
    app.use(vhost(vhostDomain, proxy(proxyDomain)));
    res.send('Created');
});

app.listen(8080);

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Handling an Excel file on a Node.js server

My EC2 nano instance is set up with an endpoint that retrieves data from my MongoDB, converts it to an Excel file, and provides a URL for download. However, when dealing with large amounts of data, the server crashes with a 502 error. Switching to an EC2 m ...

Access to an Express route in Node JS can only be granted upon clicking a button

Is it feasible to create an express route that can only be accessed once a button is clicked? I want to prevent users from entering the route directly in the URL bar, and instead require them to click a specific button first. I'm curious if this can b ...

What is the reason behind mongoose's utilization of schema, especially when MongoDB is known for its schema-less nature?

Just diving into the world of mongodb and feeling like a complete newbie. I've been utilizing mongoose to interact with mongodb through node.js, and while everything seems to be functioning properly, I can't help but wonder about the rationale be ...

After downloading Node.js version 14.4 from the website and installing it, I discovered that the Command Prompt still registers the older version 8.10

Need some help updating Node.js and npm. I've been using Node.js version 8.10.0 and npm version 3.5.2 for a while, but realized that there are newer versions available. I downloaded the latest Node.js version and added the path to Environment Variable ...

What are the best methods to improve the efficiency of inserting data into MongoDB using Node.js

Currently, we are performing data manipulation and insertion in MongoDB. However, the process of inserting one record into MongoDB takes approximately 28ms. Since I need to insert data twice per request, when dealing with 6000 requests simultaneously, th ...

Exploring the Dynamics between Koa, Co, Bluebird, Q, Generators, Promises, and Thunks in Node.js

Exploring the development of a web application using Koa has left me with questions about when and why to choose between various "making async easier" technologies. The abundance of online information on this topic has not provided clear guidance, especial ...

Injecting input values into a mongoose query within the initial parameter

Utilizing the MEAN stack, there exists a specific entry in my MongoDB database as seen below: { "_id" : ObjectId("5577467683f4716018db19ed"), "requestMatrix" : { "1698005072" : { "rideId" : "641719948", "status" :"accepted" },"1698005073" : { "rideId" : " ...

Which command in npm is used to generate a dist directory in Node.js?

I'm looking to set up my nodejs project (acting as a server for a react app) in a docker container, but I'm struggling to find an npm command that will generate a dist folder to include in the dockerfile. Any assistance would be greatly appreciat ...

What steps are needed to enable cross-domain access for a web service?

Recently, I developed an API using Node.js and Express: app.get('/api/v1/:search', function(req, res){ var response = {} res.contentType('application/json'); // process req.params['search'] ...

The authentication process using bcrypt and jwt is failing

I'm a newcomer to Node.js/Express and the world of web app development. I'm currently working on creating a user registration system where passwords are hashed with bcrypt before being saved to MongoDB. The login form allows users to log in by ch ...

Is there a way to activate an Electron API using a remote server?

Hello all, I am relatively new to Electron development and have an interesting idea that I want to implement: My goal is to send a request to an express server from a website outside of my electron app along with some data. I then want the express route t ...

Surprising outcome arising from simultaneous execution of numerous asynchronous operations on every individual object within an array

I'm fairly new to working with Node.js and I'm still trying to grasp the concept of callbacks and the asynchronous nature of Node.js. However, I've encountered a problem that I can't seem to solve. I've already searched extensively ...

Encountering issues with retrieving application setting variables from Azure App Service in a ReactJS TypeScript application resulting in '

My dilemma lies in my app setup which involves a Node.js runtime stack serving a ReactJs Typescript application. I have set some API URLs in the application settings, and attempted to access them in ReactJs components using process.env.REACT_APP_URL, only ...

Utilizing Node Sharp (LibVips) for Image Enhancement

I am looking to utilize VIPS for merging a collection of small images into one large image. The node package "sharp" leverages libvips. Is there a way to merge two images together using sharp? While VIPS has an "LRJOIN" function, I cannot seem to find a sh ...

Data modifications in polymer are not being accurately displayed

I need help with hiding/unhiding a UI element using a button in Polymer. Despite having the necessary elements and code set up, it is not working as expected: <button id="runPredictionButton"> <i>Button text</i> </button> <p ...

Mongoose encounters issues when trying to update a document using findOne() method, as it only

I have a set of fixtures assigned to a specific competitor, displayed in the following format: { "_id": { "$oid": "59dbdf6dbe628df3a80419bc" }, "timeOfEntrance": "1507581805813", "timeOfFinish": null, "competitor": { "$ ...

Exploring the magic of Node JS and integrating it with Twitter OAuth

I am new to working with Node JS and attempting to connect to Twitter using it. I have a JavaScript file located in the E:/ directory, but I'm unsure whether I should create a module folder or not with the provided code snippet. var express = req ...

Cross-Origin Resource Sharing policy: Missing 'Access-Control-Allow-Origin'

While working on a React website with Node.js backend, I encountered an error. Click here for more details. ...

npm installation encounters an error

I'm facing a problem with npm installation as it keeps failing. Despite this, I can see that the node-modules folder has been generated in my local path. How can I resolve this issue shown in the image above? Check out the installation error here. ...

There seems to be a syntax error lurking within npm.js, and for some reason npm insists on utilizing version 10.19.0 of Node.js despite my attempts to update it. The reason behind this behavior

Apologies if this question seems silly, but just a couple of days ago my code was running perfectly fine. Then today when I tried to load it, I encountered all sorts of errors. I am fairly new to node and npm, so I suspect it could be related to version ma ...