Fetching data in a sequential manner within a Server Component to manipulate a consistent data set in Next.js

Question

Fetching data in a sequential manner within a Server Component to manipulate a consistent data set in Next.js

Recently diving into the world of next.js (utilizing v13 with App Router), I find myself in search of guidance regarding a particular challenge I've been wrestling with.

My objective is to dynamically render data on a page and incrementally update it as "improved" data becomes available (e.g., more recent data, data from additional sources, etc.). Typically, there should be 2-3 updates until the data loading process is complete (similar to how options on a hotel booking website progressively update before the loading indicator disappears). I aim to handle this process primarily server-side.

Whenever an update is required, the API response from my Django backend contains a "next" entry along with a URL, for example:

{ "meta": {..., "next": "https://api.example.com/path/to/next/request"}, 
  "data": ... }

The official next.js documentation provides clarity on sequential data fetching, but does not elaborate on how to update the same dataset; instead, it focuses on progressively displaying different sets of data.

After experimenting with various approaches, I devised a solution that involves utilizing a recursive component, where a <Suspense> boundary is populated with the current data acting as the fallback.

import { Suspense } from 'react';

export default async function Stream() {
  return (
      <div className='m-auto h-screen w-1/2 p-3'>
        <h1>Welcome to the Stream page!</h1>
        <Results url={'http://127.0.0.1:3000/api/dummyData'} />
      </div>
  );
}

async function Results({ url }: { url?: string }) {
  const content = await fetch(url, {
    cache: 'no-store',
  }).then((res) => res.json());
  
  // if "next" URL is present in the response, include it in the recursive function
  if (content.next) {
    return (
      <Suspense fallback={<ResultsDisplayer content={content} />}>
        <Results url={content.next} />
      </Suspense>
    );
  } else {
    return <ResultsDisplayer content={content} />;
  }
}

async function ResultsDisplayer({ content }) {
  const isLoading = Boolean(content.next);
  return (
    <>
      <p>Loading: {isLoading ? 'YES' : 'NOPE'}</p>
      <br />
      <p>Data:</p>
      <pre>{JSON.stringify(content)}</pre>;
    </>
  );
}

Demonstration of the recursive solution (GIF)

This approach appears functional, yet I am interested in exploring a possibly more conventional or straightforward method to accomplish server-side data modifications prompted by linked hyperlinks. Thus far, I have not come across any comprehensive examples or documentation addressing this specific scenario.

Throughout my research, I explored alternative avenues, none of which seemed entirely suitable:

Client-side updates (e.g., employing useSWR and useEffect), though I aimed to steer clear of client-side fetching
Websocket implementation, deemed potentially excessive and seemingly only applicable for client-side consumption
ReadableStream utilization, as outlined in the next.js documentation on route handlers, unfortunately unsuccessful in my attempts (and did not appear conducive to server-side use)
Server Actions, leveraging the revalidateTag, still in an experimental phase and presented challenges conveying refresh signals to clients

Have any of you encountered a similar issue? If so, how did you navigate through it? Are there facets of next.js that may have eluded me or been misinterpreted?

Appreciate your insights! :)

next.js websocket streaming server-side-rendering mutation

Answer 1

Answer №1

This question brings up an intriguing idea. It seems like you're aiming to streamline the process by making just one HTTP request, send response data gradually, and have the client update as the data streams in.

Using streams both on the server and client side is essential for this task.

To start, you'll want to reply with a ReadableStream. The code for this may be complex, but you can check out this GitHub discussion on sending files with Route Handlers for different syntax examples.

On the client side, you can fetch this endpoint and consume the stream using a similar API. The fetch Response body will contain a ReadableStream with your data.

You can utilize stream event handlers to update a state variable that will trigger a rerender of your component. While there are potential solutions like <Supspense /> or utilizing incoming use hooks, sticking to event handlers instead of promises on the client side would suffice.

One thing to consider is how to separate each data chunk into standalone pieces (strings that can be JSON.stringify). One approach could be sending the chunk size along with the chunk itself (e.g., sending the size, then string, repeating until -1 is sent to close the stream).

In practice, one challenge you might face is that within Route Handlers, you must provide a Response object before Next.js initiates the HTTP response process. In contrast, in an API Route/Express endpoint, you can call res.send at any point and supply the stream.

The issue arises in understanding the stream's lifecycle: can you continue feeding the stream even after returning from the Route Handler function? Though it should theoretically work, testing it out is advisable.

Answer 2