Serve vast in-memory array using ESPAsyncWebServer

Question

Serve vast in-memory array using ESPAsyncWebServer

I am attempting to serve a sizable float array with 8192 values from the ESP32 Heap using the ESPAsyncWebServer library for ArduinoIDE. The microcontroller I'm using is an ESP32 devkit c, and my goal is to access the array through a web browser. Here is the code snippet that defines the array:

#include "AsyncJson.h"
#include "ArduinoJson.h"
#include <AsyncTCP.h>
#include <ESPAsyncWebServer.h>

#define SAMPLES 8192

static float * vReal;

void setup() {
  vReal = (float *) malloc(SAMPLES * sizeof(float));
  assert(vReal != NULL);
}

void loop() {
  //Add elements to the array
  for (int i = 0; i < SAMPLES; i++)
  {
    vReal[i] = 1.1;
  }
}

Currently, I'm utilizing the "ArduinoJson Basic Response" method to send the large array in chunks of 512 values. When attempting to access the array through the browser, using 1024 values leads to a stack overflow in the async_tcp task, so I've limited it to 512 values instead. Here's the corresponding code snippet:

server.on("/array1", HTTP_GET, [](AsyncWebServerRequest * request) {
AsyncResponseStream *response = request->beginResponseStream("application/json");
const size_t CAPACITY = JSON_ARRAY_SIZE(512); //Calculate array size
StaticJsonDocument<CAPACITY> vRealPart;
JsonArray array = vRealPart.to<JsonArray>();
for (int i = 0; i < 512; i++)
{
  vRealPart.add(vReal[i]);
}
serializeJson(vRealPart, *response); //Print to HTML
request->send(response);
});

I repeat this process 16 times to serve the entire array. Later on, I call the paths "/array1", "/array2", "/array3", and so on in JavaScript and parse the JSON data. This is a sample output when one of the paths is accessed via a web browser:

[0.334593,0.427480,0.181299,0.066654,0.271184,0.356220,0.374454,0.235625,...]

While this approach generally works, I find it quite cumbersome. It would be more convenient if there was just a single path for the entire array. A static file from SPIFFS can achieve this, like so:

server.serveStatic("/jsonArray1", SPIFFS, "/jsonArray1");

However, writing the complete array to flash takes too much time, even though reading from it is fast.

I also attempted to utilize the "ArduinoJson Advanced Response" method, but couldn't get it working with a JsonArray. Unfortunately, the examples on the GitHub page for ESPAsyncWebServer and ArduinoJson are deprecated due to syntax changes in the new version (v6).

In essence, what is the best way to serve large arrays from the ESP32 Heap using the ESPAsyncWebServer library? My objective is to process the array later with JavaScript in a web browser.

Thanks for your assistance!

Bentiedem

PS: If it helps: I'm performing a Fast Fourier Transform (FFT) using the arduinoFFT library on the motor current. I take measurements using an ADC and save the resulting 16384 values from the analog-to-digital converter in an array. This array is then passed to the FFT library, generating an output array with 8192 values. The result is stored in the heap and should be transmitted to my web interface to display the outcome. I aim to keep these arrays in RAM for faster processing. Each measurement yields a result array with 8192 values.

arrays json arduino esp32 embeddedwebserver

Answer 1

Answer №1

Check out this proposal for utilizing arrays compiled to flash:

const uint16_t MAXIMUM_SAMPLE_COUNT = 8196;
const uint8_t SAMPLE_NUMBER_SIZE = 8; // considering your numbers are 0.271184, requiring 8 characters

float measurementValues[MAXIMUM_SAMPLE_COUNT];
char valueArray[MAXIMUM_SAMPLE_COUNT * (SAMPLE_NUMBER_SIZE + 1) + 2 + 1] = {'\0'};
char numBuffer[SAMPLE_NUMBER_SIZE + 1] = {'\0'}; // Conversion helper array

Then you can add values using the following code:

server.on("/array", HTTP_GET, [](AsyncWebServerRequest * request) {
AsyncResponseStream *response = request->beginResponseStream("application/json");
  strcpy(valueArray, "[");
  for (uint16_t i = 0; i < MAXIMUM_SAMPLE_COUNT; i++) {
    if (i != 0) strcat(valueArray, ",");
    // float tempValue = functionToGetValues(); // directly fetched from a function
    float tempValue = measurementValues[i];
    // dtostrf(floatVariable, stringLengthIncludingDecimalPoint, numberVariablesAfterDecimal, charBuffer);
    dtostrf(tempValue, 2, 6, numBuffer);
    strcat(valueArray, numBuffer);
  }
  strcat(valueArray, "]");
  valueArray[strlen(valueArray)] = '\0'; // properly terminated
  Serial.println(valueArray);  // For debugging purposes only
 request->send(valueArray);
  valueArray[0] = '\0';  // Resetting the array for the next part
});

This code compiles successfully, but the sending part has not been tested yet. It might need to be processed this way or by using the chunked method. Usually, I only utilize such large arrays in the setup and read from the FileSystem. The handling is similar to how ArduinoJson works, but with fewer lines of code and no heap fragmentation. This ensures the app can run for a long time without any issues.
EDIT
My preferred solution would be continuously writing to a file in SPIFFS (refer to the SD example data logger or this post) and serving the file. For around 10,000 values, this is the most efficient approach. The code provided above demonstrates how to add brackets and commas so that the content of the file becomes one large JSON object that can be quickly processed through JavaScript. Additionally, by using multiple log files, you have the ability to recover in case of network errors or similar events.

Answer 2

Check out this proposal for utilizing arrays compiled to flash:

const uint16_t MAXIMUM_SAMPLE_COUNT = 8196;
const uint8_t SAMPLE_NUMBER_SIZE = 8; // considering your numbers are 0.271184, requiring 8 characters

float measurementValues[MAXIMUM_SAMPLE_COUNT];
char valueArray[MAXIMUM_SAMPLE_COUNT * (SAMPLE_NUMBER_SIZE + 1) + 2 + 1] = {'\0'};
char numBuffer[SAMPLE_NUMBER_SIZE + 1] = {'\0'}; // Conversion helper array

Then you can add values using the following code:

server.on("/array", HTTP_GET, [](AsyncWebServerRequest * request) {
AsyncResponseStream *response = request->beginResponseStream("application/json");
  strcpy(valueArray, "[");
  for (uint16_t i = 0; i < MAXIMUM_SAMPLE_COUNT; i++) {
    if (i != 0) strcat(valueArray, ",");
    // float tempValue = functionToGetValues(); // directly fetched from a function
    float tempValue = measurementValues[i];
    // dtostrf(floatVariable, stringLengthIncludingDecimalPoint, numberVariablesAfterDecimal, charBuffer);
    dtostrf(tempValue, 2, 6, numBuffer);
    strcat(valueArray, numBuffer);
  }
  strcat(valueArray, "]");
  valueArray[strlen(valueArray)] = '\0'; // properly terminated
  Serial.println(valueArray);  // For debugging purposes only
 request->send(valueArray);
  valueArray[0] = '\0';  // Resetting the array for the next part
});

This code compiles successfully, but the sending part has not been tested yet. It might need to be processed this way or by using the chunked method. Usually, I only utilize such large arrays in the setup and read from the FileSystem. The handling is similar to how ArduinoJson works, but with fewer lines of code and no heap fragmentation. This ensures the app can run for a long time without any issues.
EDIT
My preferred solution would be continuously writing to a file in SPIFFS (refer to the SD example data logger or this post) and serving the file. For around 10,000 values, this is the most efficient approach. The code provided above demonstrates how to add brackets and commas so that the content of the file becomes one large JSON object that can be quickly processed through JavaScript. Additionally, by using multiple log files, you have the ability to recover in case of network errors or similar events.

Answer 3

Answer №2

After much searching, I have finally discovered a new solution that utilizes the concept of chunked response. In addition, I no longer need to rely on ArduinoJson. To ensure stability, I had to reduce the buffer size (maxLen = maxLen >> 1;). Interestingly, when using maxLen, the library provides an incorrect length for the buffer, which might indicate a bug.

The newly implemented approach is significantly faster than my previous solution and runs without any microcontroller crashes. It successfully transfers an array containing 16384 values to the web browser in one seamless motion within only 422 milliseconds.

Below is the code snippet showcasing the server response:

server.on("/array", HTTP_GET, [](AsyncWebServerRequest * request) {
    indexvReal = 0;
    AsyncWebServerResponse* response = request->beginChunkedResponse(contentType,
                                       [](uint8_t* buffer, size_t maxLen, size_t index) {
      maxLen = maxLen >> 1;
      size_t len = 0;
      if (indexvReal == 0) {
        len += sprintf(((char *)buffer), "[%g", vReal[indexvReal]);
        indexvReal++;
      }
      while (len < (maxLen - FLOATSIZE) && indexvReal < LEN(vReal)) {
        len += sprintf(((char *)buffer + len), ",%g", vReal[indexvReal]);
        indexvReal++;
      }
      if (indexvReal == LEN(vReal)) {
        len += sprintf(((char *)buffer + len), "]");
        indexvReal++;
      }
      return len;
    });
    request->send(response);
  });

For testing purposes, a complete example is provided:

#include <WiFi.h>
#include <AsyncTCP.h>
#include <ESPAsyncWebServer.h>

const char *ssid = "...";
const char *password = "...";
int counter_wifi = 0;
AsyncWebServer server(80);

#define LEN(arr) ((int) (sizeof (arr) / sizeof (arr)[0]))
#define SAMPLES 16384
static float vReal[SAMPLES]; // Stack
uint16_t indexvReal = 0;

#define FLOATSIZE 20
const char *contentType = "application/json";
// const char *contentType = "text/plain";

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.println();
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
    counter_wifi++;
    if (counter_wifi >= 10){
      ESP.restart(); // Reset the board after 5 seconds of unsuccessful connection
    }
  }
  Serial.println("WiFi connected.");
  Serial.println("IP Address: ");
  Serial.println(WiFi.localIP());

  for (int i = 0; i < SAMPLES; i++) {
    vReal[i] = random(20) * 3.2357911;
  }

  server.on("/array", HTTP_GET, [](AsyncWebServerRequest * request) {
    indexvReal = 0;
    AsyncWebServerResponse* response = request->beginChunkedResponse(contentType,
                                       [](uint8_t* buffer, size_t maxLen, size_t index) {
      maxLen = maxLen >> 1;
      size_t len = 0;
      if (indexvReal == 0) {
        len += sprintf(((char *)buffer), "[%g", vReal[indexvReal]);
        indexvReal++;
      }
      while (len < (maxLen - FLOATSIZE) && indexvReal < LEN(vReal)) {
        len += sprintf(((char *)buffer + len), ",%g", vReal[indexvReal]);
        indexvReal++;
      }
      if (indexvReal == LEN(vReal)) {
        len += sprintf(((char *)buffer + len), "]");
        indexvReal++;
      }
      return len;
    });
    request->send(response);
  });

  DefaultHeaders::Instance().addHeader("Access-Control-Allow-Origin", "*");
  server.begin();
  Serial.println("HTTP server started");
}

void loop() {
  // Put your main code here, to run repeatedly:

}

Answer 4

After much searching, I have finally discovered a new solution that utilizes the concept of chunked response. In addition, I no longer need to rely on ArduinoJson. To ensure stability, I had to reduce the buffer size (maxLen = maxLen >> 1;). Interestingly, when using maxLen, the library provides an incorrect length for the buffer, which might indicate a bug.

The newly implemented approach is significantly faster than my previous solution and runs without any microcontroller crashes. It successfully transfers an array containing 16384 values to the web browser in one seamless motion within only 422 milliseconds.

Below is the code snippet showcasing the server response:

server.on("/array", HTTP_GET, [](AsyncWebServerRequest * request) {
    indexvReal = 0;
    AsyncWebServerResponse* response = request->beginChunkedResponse(contentType,
                                       [](uint8_t* buffer, size_t maxLen, size_t index) {
      maxLen = maxLen >> 1;
      size_t len = 0;
      if (indexvReal == 0) {
        len += sprintf(((char *)buffer), "[%g", vReal[indexvReal]);
        indexvReal++;
      }
      while (len < (maxLen - FLOATSIZE) && indexvReal < LEN(vReal)) {
        len += sprintf(((char *)buffer + len), ",%g", vReal[indexvReal]);
        indexvReal++;
      }
      if (indexvReal == LEN(vReal)) {
        len += sprintf(((char *)buffer + len), "]");
        indexvReal++;
      }
      return len;
    });
    request->send(response);
  });

For testing purposes, a complete example is provided:

#include <WiFi.h>
#include <AsyncTCP.h>
#include <ESPAsyncWebServer.h>

const char *ssid = "...";
const char *password = "...";
int counter_wifi = 0;
AsyncWebServer server(80);

#define LEN(arr) ((int) (sizeof (arr) / sizeof (arr)[0]))
#define SAMPLES 16384
static float vReal[SAMPLES]; // Stack
uint16_t indexvReal = 0;

#define FLOATSIZE 20
const char *contentType = "application/json";
// const char *contentType = "text/plain";

void setup() {
  Serial.begin(115200);
  Serial.println();
  Serial.println();
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
    counter_wifi++;
    if (counter_wifi >= 10){
      ESP.restart(); // Reset the board after 5 seconds of unsuccessful connection
    }
  }
  Serial.println("WiFi connected.");
  Serial.println("IP Address: ");
  Serial.println(WiFi.localIP());

  for (int i = 0; i < SAMPLES; i++) {
    vReal[i] = random(20) * 3.2357911;
  }

  server.on("/array", HTTP_GET, [](AsyncWebServerRequest * request) {
    indexvReal = 0;
    AsyncWebServerResponse* response = request->beginChunkedResponse(contentType,
                                       [](uint8_t* buffer, size_t maxLen, size_t index) {
      maxLen = maxLen >> 1;
      size_t len = 0;
      if (indexvReal == 0) {
        len += sprintf(((char *)buffer), "[%g", vReal[indexvReal]);
        indexvReal++;
      }
      while (len < (maxLen - FLOATSIZE) && indexvReal < LEN(vReal)) {
        len += sprintf(((char *)buffer + len), ",%g", vReal[indexvReal]);
        indexvReal++;
      }
      if (indexvReal == LEN(vReal)) {
        len += sprintf(((char *)buffer + len), "]");
        indexvReal++;
      }
      return len;
    });
    request->send(response);
  });

  DefaultHeaders::Instance().addHeader("Access-Control-Allow-Origin", "*");
  server.begin();
  Serial.println("HTTP server started");
}

void loop() {
  // Put your main code here, to run repeatedly:

}

Serve vast in-memory array using ESPAsyncWebServer

Answer №1

Answer №2

Similar questions

Building a structured JSON request using Jbuilder

Utilize AngularJS to enable the forward and back buttons to fetch JSON data, but ensure that the data is only retrieved if the item meets

No visible alterations were observed to the object following the invocation of JSONDecoder

Passing data in JSON format to PHP

What is the best way to fetch images from a JSON object in a React application?

Can you provide guidance on accessing the response API from Twitter?

Tips for preserving a collection of items in a thesaurus?

The Vue.js v-on:mouseover function is not functioning properly in displaying the menu

Guide to serializing JSON in C# without nested elements

Obtain the result of two "Synchronous" nested queries using Express and Mongoose

The variables do not align with the number of parameters specified in the prepared statement

display only the final outcome within the FOR loop of a Joomla module

Having difficulty interpreting an encrypted string retrieved from a Web Request using a StreamReader

A guide to accessing a specific element within a JSON object using Python

Using Node.js, I am utilizing a Javascript API to perform a POST request, reading data

What is the best way to add selected values from a multi-select dropdown into an array?

merging 4 arrays in a specified order, organized by ID

PHP - How to Efficiently Parse JSON Data

Unexpected value detected in D3 for translate function, refusing to accept variable

Attempting to construct a .csv file in PHP using an array