Stop Azure function from repeating its execution recursively

A custom Azure function has been developed to respond to an EventGrid subscription event triggered by the uploading of a new blob into Blob Storage.

This function is responsible for resizing the uploaded image and then re-uploading it back into the same container.

The challenge arises when the function triggers a new upload event from within itself, causing it to run again. Since both the original and resized images need to be stored in the same container, filtering the Event Subscription based on the container name won't suffice.

While the EventGrid Subscription allows for advanced filtering options based on the data object in the payload, there seems to be no straightforward way to add a custom field to the data object when uploading the resized image from the Azure function, making event filtering difficult.

https://i.stack.imgur.com/xeEmA.png

Function Implementation:

const stream = require('stream');
const Jimp = require('jimp');

const {
  Aborter,
  BlobServiceClient,
  StorageSharedKeyCredential,
} = require("@azure/storage-blob");

const ONE_MEGABYTE = 1024 * 1024;
const uploadOptions = { bufferSize: 4 * ONE_MEGABYTE, maxBuffers: 20 };

// const containerName = process.env.BLOB_CONTAINER_NAME;
const accountName = process.env.AZURE_STORAGE_ACCOUNT_NAME;
const accessKey = process.env.AZURE_STORAGE_ACCOUNT_ACCESS_KEY;

const sharedKeyCredential = new StorageSharedKeyCredential(
  accountName,
  accessKey);
const blobServiceClient = new BlobServiceClient(
  `https://${accountName}.blob.core.windows.net`,
  sharedKeyCredential
);

module.exports = async (context, event, inputBlob) => {  
  context.log(`Function started`);
  context.log(`event: ${JSON.stringify(event)}`);

  const widthInPixels = 100;
  const blobUrl = context.bindingData.data.url;
  const blobUrlArray = blobUrl.split("/");
  const blobFileName = blobUrlArray[blobUrlArray.length - 1];
  const blobExt = blobFileName.slice(blobFileName.lastIndexOf(".") + 1);
  const blobName = blobFileName.slice(0, blobFileName.lastIndexOf("."));
  const blobName_thumb = `${blobName}_t.${blobExt}`;

  const containerName = blobUrlArray[blobUrlArray.length - 2];

  const image = await Jimp.read(inputBlob);
  const thumbnail = image.resize(widthInPixels, Jimp.AUTO);
  const thumbnailBuffer = await thumbnail.getBufferAsync(Jimp.AUTO);
  const readStream = stream.PassThrough();
  readStream.end(thumbnailBuffer);

  const containerClient = blobServiceClient.getContainerClient(containerName);
  const blockBlobClient = containerClient.getBlockBlobClient(blobName_thumb);
  context.log(`blockBlobClient created`);
  try {
      await blockBlobClient.uploadStream(readStream,
            uploadOptions.bufferSize,
            uploadOptions.maxBuffers,
            { blobHTTPHeaders: { blobContentType: "image/jpeg" } });
      context.log(`File uploaded`);
  } catch (err) {

    context.log(err.message);

  } finally {

    context.done();

  }
};

Bindings:

{
  "bindings": [
    {
      "name": "event",
      "direction": "in",
      "type": "eventGridTrigger"
    },
    {
      "name": "inputBlob",
      "direction": "in",
      "type": "blob",
      "path": "{data.url}",
      "connection": "AzureWebJobsStorage",
      "dataType": "binary"
    }
  ]
}

Dealing with this issue highlights a common challenge when utilizing Azure functions to handle blob storage events, as it seems surprising that such a specific scenario lacks clear documentation within the extensive Azure resources!

Answer №1

When working with EventGrid Subscription, there is an option for advanced filtering to filter events based on the data object in the payload. However, it seems challenging to add custom fields to the data object when uploading a resized image from an Azure function.

It's true that adding anything to the data object is not possible as the event creation process is controlled by the Azure Platform. Filters can only be created based on the content of the event itself, as outlined in the documentation:

[{
  "topic": "/subscriptions/{subscription-id}/resourceGroups/Storage/providers/Microsoft.Storage/storageAccounts/my-storage-account",
  "subject": "/blobServices/default/containers/test-container/blobs/new-file.txt",
  "eventType": "Microsoft.Storage.BlobCreated",
  "eventTime": "2017-06-26T18:41:00.9584103Z",
  "id": "831e1650-001e-001b-66ab-eeb76e069631",
  "data": {
    ...
  },
  "dataVersion": "",
  "metadataVersion": "1"
}]

This situation presents two options. You can either utilize a staging container for uploaded files where resizing and final storage happens or implement a mechanism to prevent newly created thumbnails from being processed further.

One approach could involve inspecting the blob name in your function and terminating processing if it's identified as a thumbnail. Another solution could be to add metadata to the thumbnail blobs during creation and then use this metadata as a criteria during execution to determine whether to proceed with resizing or not.

Answer №2

If you want to enhance filtering in your event message, consider utilizing the optional property clientRequestId. Check out more details about this property here. To implement this, ensure that the upload client in your subscriber includes a client header x-ms-client-request-id with a specific value for filtering purposes. For example, I used ABCD-1234 in my testing.

In the simulated event message during blob creation via REST API PUT request, you'll see the x-ms-client-request-id header setup with the value ABCD-1234, as depicted below:

https://i.stack.imgur.com/6Iosx.png

This approach allows you to configure advanced filtering for subscription based on the unique clientRequestId value, as shown in the following:

https://i.stack.imgur.com/keZ1I.png

Implementing a distinct clientRequestId can help prevent unnecessary invocation of the subscriber.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Having trouble retrieving req.session variables in Express/NodeJS?

I have come across various versions of this particular question, however, none of them seem to address my issue directly. My current objective is to establish a Node.js server using Express. Below is my existing server configuration: var express = require ...

Having trouble establishing a basic websocket connection in NodeJS

I am currently following a tutorial on WebSocket protocol development from this link: . Upon visiting localhost:1337/index.html, I encountered the following error: This localhost page cannot be found. No webpage was found for the web address: http://loc ...

React: Exploring the intricacies of importing components in the component-oriented versus library-oriented approach

Someone mentioned to me that there are two distinct methods for importing components/modules. The component approach The library method Does anyone have insights on these concepts? ...

Leverage the Express JS .all() function to identify the specific HTTP method that was utilized

My next task involves creating an endpoint at /api that will blindly proxy requests and responses to a legacy RESTful API system built in Ruby and hosted on a different domain. This is just a temporary step to transition smoothly, so it needs to work seam ...

How to make a straightforward task list using ExpressJS

As a beginner, I am attempting to create a basic todo list using ExpressJS. Currently, my goal is to simply display some hardcoded todos that I have in my application. However, I seem to be struggling to identify the mistake in my code. Any assistance woul ...

Error in serving static files in React build through Express server

Hello everyone, I am fairly new to react and facing some challenges while trying to convert my MEAN stack app into an express and reactjs application. The main issue I am encountering is related to getting my build files to load correctly. It seems like my ...

Having trouble with the express message! I can't seem to access the template I created

I am looking to receive a notification similar to an alert, as described in this link: https://github.com/visionmedia/express-messages By default, I receive something like this https://i.stack.imgur.com/9XlA9.png If I use a template, I do not get any out ...

Postman issue: Your username and password combination is incorrect within the MEAN stack environment

I am new to mean stack development and facing some issues. When I try to run "api/users/login" in Postman, it shows an error saying "Username or password is invalid!". Additionally, when attempting to register using "register/users/register", it gives a me ...

How can you verify your identity with a private npm registry in Azure using a Dockerfile?

Below is my Dockerfile setup: FROM node:alpine WORKDIR ./usr/local/lib/node_modules/npm/ COPY .npmrc ./ COPY package.json ./ COPY package-lock.json ./ RUN npm ci COPY . . EXPOSE 3000 CMD ["npm", "run", "start-prod"] Th ...

Encountered an issue trying to access undefined properties in the mongoose response object

When utilizing the mongoose Schema in node.js, I have defined the following structure: mongoose.Schema({ name: { type: String, required: true }, userId: { type: String }, water: { type: Array }, fertilizer: { type: Array } }) Subsequently, ...

Issue with installing chromedriver and selenium-driver on Windows 7 using NPM arise

I have a relatively basic package.json file with some development dependencies included. Here is what it looks like: "devDependencies": { "cssnano": "3.3.2", "cucumber": "0.9.2", "diff": "2.2.0", "grunt": "0.4.5", "jit-grunt": "0.9.1", " ...

nodejs - pkg encountered an error: only one entry file or directory should be specified

I am currently facing issues with packing my simple cli node script using pkg. After running the command below: host:~ dev$ pkg /Users/dev/Desktop/myscript/ --target node14-macos-x64 node14-linux-x64 node14-win-x64 I encountered an error in the terminal: ...

Should I use express with NodeJS and socket.io?

Just a quick question. I'm currently in the process of creating an online browser game with phaser and socket.io. I know I'll be using socket.io quite a bit, but what about express? Once I start the server, what other tasks will I need to complet ...

Having trouble with npm installing bower?

Attempting to set up bower with npm. Following the steps and executed the command: npm install -g bower Changed the loglevel to "info" for better visibility. The process progresses until: npm info build \\bedcoll.local\san\Home&bso ...

Tips for managing errors when using .listen() in Express with Typescript

Currently in the process of transitioning my project to use Typescript. Previously, my code for launching Express in Node looked like this: server.listen(port, (error) => { if (error) throw error; console.info(`Ready on port ${port}`); }); However ...

Struggling to retrieve a response from the ListUsers endpoint in OKTA using the okta-sdk-nodejs Client's listUsers function

Code snippet: async fetchUsersByEmail(email) { try { return await Promise.all([ oktaClient.listUsers({ search: email, }), ]).then((response) => { console.log(response); }); } catch (error) { ...

Ways to retrieve all local variables in an Express view

I'm exploring options to access all the app.locals within views. Here's what I have in my app.js: app.locals.title = "Title" app.locals.page = "Page 1" These work fine, and I can easily use them in views like this: <%= title %&g ...

Struggling with dynamically updating fields based on user input in real time

I have a goal to update a field based on the inputs of two other fields. Currently, all the fields are manual input. Below is the code I'm using to try and make the "passThru" field update in real-time as data is entered into the other fields. The ma ...

When a process crashes, PM2 automatically launches a new daemon but fails to restart the processes

My application "prod" does not restore on SIGINT, resulting in PM2 crashing all other applications and restarting the daemon. If I manually kill other applications, they function properly by automatically restarting. The logs show: (process:3626): GLib- ...

A step-by-step guide on obtaining a reference to the Nextjs server

Incorporating ws (web socket) into my Nextjs application is something I'm working on. Instead of setting up a new server, my goal is to hand over the existing server object to initialize ws: const { Server } = require('ws'); wss = new Serve ...