Receiving real-time information from Mongoose - "Cannot modify header information after it has been sent"

I am currently working on an application that utilizes Node, Express, Angular, and Mongoose. To retrieve a large dataset from MongoDB, I am attempting to stream the data. However, I encounter an issue after loading the initial chunk of data where I receive the error message: "Error: Can't set headers after they are sent." Below is a snippet of my code:

routes.js

module.exports = function(app) {
  var Org  = require('./models/org');
  var JSONStream = require('JSONStream');
  var stringify = require('stringify');
  app.get('/api/orgs', function(req, res){ 
        res.set('Content-Type', 'application/json'); 
        Org.find().stream().pipe(JSONStream.stringify()).pipe(res); 

        console.log('STATUS: ' + res.statusCode);
      })
    };

models/org.js

var mongoose = require('mongoose');
var Schema = mongoose.Schema;

module.exports = mongoose.model('Org', new Schema({
    ein : {type : Number},
}), 'irs9902013');

Answer №1

When utilizing the streaming interface, a data event is triggered for each document in the query result. This leads to multiple calls of res.json(data), which is incorrect because res.json internally invokes res.send and attempts to set the charset on the Content-Type header. Once the first document is written to the response body, it is no longer possible to set a header afterwards.

One solution is to directly pipe the query results to the response using JSONStream:


res.set('Content-Type', 'application/json');
Model.find().stream().pipe(JSONStream.stringify()).pipe(res);

An alternative approach would be to accumulate the documents from the data event and wait for the end event before calling res.json(accumulatedDocuments). However, this method goes against the purpose of streaming and can be simplified by:


Model.find().toArray(function (err, results) {
  res.json(results);
});

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

"Utilizing ExpressJS with Mongoose, what is the best method for choosing two column values and converting them

In my current schema setup found in Resource.js file: var mongoose = require("mongoose"), Schema = mongoose.Schema, objectId = mongoose.Schema.ObjectId; var labelShema = new Schema({ labelName: { type: String }, language: { type: String, }, resourceKey: { ...

What is the best way to integrate Bootstrap 5 with Next.js?

After adding Bootstrap to my project with npm install bootstrap, I included the style in /pages/_app.js like this: import 'bootstrap/dist/css/bootstrap.css'; export default function App({ Component, pageProps }) { return <Component {...pag ...

Why does Angular-CLI remove an old module when installing a new module using npm?

After adding the ng-sidebar module to my app, I decided to install a new module called ng2-d&d: npm install ng2-dnd --save However, after installing the new module, the old ng-sidebar module was removed from the app-module and an error occurred: C ...

Despite having a valid API route, an error occurred in the POST request using Axios

Currently, I am following a video tutorial (here) and specifically on part 8 which covers payment processing. The payment process involves using Stripe, Axios, and Node.js. However, whenever attempting to make a POST request to one of the API's routes ...

Incorporating an npm reference into a personalized node within Node-RED

As a novice in both the NodeRed and NodeJs/npm realms, I am embarking on the journey of creating a custom node for the first time. Despite my efforts to follow the official documentation on Creating your first node, I seem to have hit a roadblock. Everyth ...

What is the best way to merge three collections without using $unwind and generate a nested outcome depending on a specific condition?

People Database: [ { "_id": ObjectId("5f3258cfbaaccedaa5dd2c96"), "gender": "male", "name": { "title": "mr", "first": "victor", " ...

What causes the variation in output results between axios and express when using dot notation?

A geo tool application is in the works, built with reactjs. The concept involves users submitting a city name which then triggers a post request. This request searches through a city list.JSON file to find the corresponding city and returns its geolocation ...

"Optimize Your Data with PrimeNG's Table Filtering Feature

I'm currently working on implementing a filter table using PrimeNG, but I'm facing an issue with the JSON structure I receive, which has multiple nested levels. Here's an example: { "id": "123", "category": "nice", "place": { "ran ...

Error: Angular7 Unable to locate namespace 'google'

I am currently utilizing the import { MapsAPILoader } from '@agm/core'; package to enable auto-complete address search functionality. However, I have encountered an error message stating cannot find namespace 'google'. The error occu ...

Why is the Zip archive downloader not functioning properly when using Node.js and Archiver (Unexpected end of archive error)?

Looking to download multiple files using archiver with express. The server should respond to a post request from the client by sending a .zip file. However, there seems to be an issue where WinRAR displays an error message "! 98I9ZOCR.zip:Unexpected end of ...

We encountered an error while trying to locate the 'socket.io' view in the views directory

Having an issue with my nodejs server. Check out the code below: server.js global.jQuery = global.$ = require('jquery'); var express = require('express'), path = require('path'), menu = require("./routes/menu"); var ...

Is there a way for me to view the output of my TypeScript code in an HTML document?

This is my HTML *all the code has been modified <div class="testCenter"> <h1>{{changed()}}</h1> </div> This is my .ts code I am unsure about the functionality of the changed() function import { Component, OnInit } f ...

Observing data flow in NodeJS where readable stream sends events without any corresponding breaks in communication with the writable stream

Our production environment is experiencing significant memory usage due to high file sizes, stored in S3 and streamed to a local filesystem on EC2 instances. Some clients have files over 6GB, causing the node process to consume large amounts of memory and ...

execute npm postinstall a single time

Is there a way to ensure the postinstall script of npm only runs once, specifically after the initial npm install command is executed? I've searched through the documentation without success. Does anyone know of a method to achieve this without resor ...

Eliminate embedded document elements using $pull operation

Trying to implement a functionality in ExpressJS and Mongoose that involves removing items from subdocuments. Currently, only the first item is getting removed instead of all sub items. The goal is to delete "subitem 2" from the messages Array. Here is th ...

Is there a way to use npx with a parameter like `--registry https://registry.npmjs.org` during an npm install to bypass the need for artifactory?

Using --registry https://registry.npmjs.org has been a workaround to bypass an Artifactory-like proxy issue when installing with npm install. However, I am now facing a similar challenge with npm xxx, as I'm getting the following error: npm ERR! code ...

The creation of package.json did not involve the use of npm init command

When I run the npm init command within the terminal, the package.json file doesn't appear in the folder where I expected it to be located. However, when I use the dir command, it shows that the package.json file has actually been created. For more inf ...

The zlib.gunzipSync function is not supported in Node.js when using the meteorhacks:npm package

I recently encountered an issue while using meteorhacks:npm to import the npm module Telegram.link in Meteor. Everything seemed to be working fine until I attempted to call a specific function, which resulted in the following error message: W20160218-16:3 ...

Node.js: The error "req.session.save is not a function" occurred while trying to authenticate

I am currently facing an issue with PassportJS while trying to authenticate users in my application. Once a user logs in, the session is created, but upon redirection, the session seems to become undefined again as it has not been saved properly. I read on ...

Struggling to successfully upload a file using express and fileupload, but I am having trouble getting it to function properly

This is my first attempt at uploading files to the server, following a tutorial to build both frontend and backend in React. Unfortunately, I'm facing some difficulties getting it to work properly. I've made adjustments to my backend and tested ...