Execute a Cron Job every half an hour following the onCreate event in Firestore

Question

Execute a Cron Job every half an hour following the onCreate event in Firestore

Looking to set up a cron job or scheduler that runs every 30 minutes following an onCreate event in Firestore. The goal is to execute a cloud function that retrieves documents created within the last 30 minutes, validates them against a JSON schema, and stores them in a separate collection. How can I programmatically create this scheduler? Additionally, what fail-safe measures can be implemented, as well as a queuing/tracking system for documents created prior to the execution of the cron job?

node.js firebase google-cloud-firestore google-cloud-functions google-cloud-scheduler

Answer 1

Answer №1

Creating a queue system with Firestore is straightforward and tailored perfectly to suit your specific needs. The concept involves adding tasks to a designated queue collection with set due dates for processing at the appropriate time.

Let's illustrate this with an example:

Upon triggering the initial onCreate event in your collection, insert a document containing the following data into a tasks collection:

    duedate: new Date() + 30 minutes
    type: 'yourjob'
    status: 'scheduled'
    data: '...' // <-- input any necessary task-specific data here

Have a worker regularly check for available tasks - for instance, every minute based on your requirements

// Define actions for each task type
const workers: Workers = {
  yourjob: (data) => db.collection('xyz').add({ foo: data }),
}


// The following schedule should be implemented:

export const checkQueue = functions.https.onRequest(async (req, res) => {
  // Consistent timestamp
  const now = admin.firestore.Timestamp.now();
  // Identify overdue tasks
  const query = db.collection('tasks').where('duedate', '<=', new Date()).where('status', '==', 'scheduled');
  const tasks = await query.get();
  // Execute tasks and update their status
  tasks.forEach(snapshot => {
    const { type, data } = snapshot.data();
    console.info('Executing job for task ' + JSON.stringify(type) + ' with data ' + JSON.stringify(data));
    const job = workers[type](data)
      // Update task doc with completion status or error
      .then(() => snapshot.ref.update({ status: 'complete' }))
      .catch((err) => {
        console.error('Error when executing worker', err);
        return snapshot.ref.update({ status: 'error' });
      });

    jobs.push(job);
  });
  return Promise.all(jobs).then(() => {
    res.send('ok');
    return true;
  }).catch((onError) => {
    console.error('Error', onError);
  });
});

You have various options for triggering the queue checks for pending tasks:

Utilize an HTTP callable function as shown above. This entails making periodic HTTP calls to the function to execute and review pending tasks. You can initiate these calls from your own server or use services like cron-job.org. Keep in mind that the HTTP callable function may be publicly accessible, but if your checking code is idempotent, security concerns should not arise.
Opt for the internal Firebase cron option which leverages Cloud Scheduler internally. This allows for direct scheduling of queue checks:

    export scheduledFunctionCrontab =
    functions.pubsub.schedule('* * * * *').onRun((context) => {
        console.log('This will be run every minute!');
        // Include code from checkQueue here from above
    });

Implementing such a queue system enhances the resilience of your application - in case of disruptions, tasks are not lost as they persist until processed by a dedicated worker. However, the successful recovery of tasks relies on the efficiency of your implementation.

Answer 2

Creating a queue system with Firestore is straightforward and tailored perfectly to suit your specific needs. The concept involves adding tasks to a designated queue collection with set due dates for processing at the appropriate time.

Let's illustrate this with an example:

Upon triggering the initial onCreate event in your collection, insert a document containing the following data into a tasks collection:

    duedate: new Date() + 30 minutes
    type: 'yourjob'
    status: 'scheduled'
    data: '...' // <-- input any necessary task-specific data here

Have a worker regularly check for available tasks - for instance, every minute based on your requirements

// Define actions for each task type
const workers: Workers = {
  yourjob: (data) => db.collection('xyz').add({ foo: data }),
}


// The following schedule should be implemented:

export const checkQueue = functions.https.onRequest(async (req, res) => {
  // Consistent timestamp
  const now = admin.firestore.Timestamp.now();
  // Identify overdue tasks
  const query = db.collection('tasks').where('duedate', '<=', new Date()).where('status', '==', 'scheduled');
  const tasks = await query.get();
  // Execute tasks and update their status
  tasks.forEach(snapshot => {
    const { type, data } = snapshot.data();
    console.info('Executing job for task ' + JSON.stringify(type) + ' with data ' + JSON.stringify(data));
    const job = workers[type](data)
      // Update task doc with completion status or error
      .then(() => snapshot.ref.update({ status: 'complete' }))
      .catch((err) => {
        console.error('Error when executing worker', err);
        return snapshot.ref.update({ status: 'error' });
      });

    jobs.push(job);
  });
  return Promise.all(jobs).then(() => {
    res.send('ok');
    return true;
  }).catch((onError) => {
    console.error('Error', onError);
  });
});

You have various options for triggering the queue checks for pending tasks:

Utilize an HTTP callable function as shown above. This entails making periodic HTTP calls to the function to execute and review pending tasks. You can initiate these calls from your own server or use services like cron-job.org. Keep in mind that the HTTP callable function may be publicly accessible, but if your checking code is idempotent, security concerns should not arise.
Opt for the internal Firebase cron option which leverages Cloud Scheduler internally. This allows for direct scheduling of queue checks:

    export scheduledFunctionCrontab =
    functions.pubsub.schedule('* * * * *').onRun((context) => {
        console.log('This will be run every minute!');
        // Include code from checkQueue here from above
    });

Implementing such a queue system enhances the resilience of your application - in case of disruptions, tasks are not lost as they persist until processed by a dedicated worker. However, the successful recovery of tasks relies on the efficiency of your implementation.

Answer 3

Answer №2

If you want to initiate a cloud function when a Firestore document is created, you can set up a process that schedules a Cloud Task 30 minutes after the event. This approach ensures there is a queuing system in place along with retry mechanisms for better reliability.

Answer 4

If you want to initiate a cloud function when a Firestore document is created, you can set up a process that schedules a Cloud Task 30 minutes after the event. This approach ensures there is a queuing system in place along with retry mechanisms for better reliability.

Answer 5

Answer №3

One simple solution is to include a created field with a timestamp, and then set up a scheduled function that runs at regular intervals (e.g., once per minute) to execute specific code for all records where the

created >= NOW - 31 mins AND created <= NOW - 30 mins

(pseudo code). For most cases, this method should suffice unless you have very precise time requirements.

If this approach doesn't meet your requirements, another option is to utilize Cloud Tasks from Google Cloud. More information on how to do this can be found in this informative article.

Answer 6

One simple solution is to include a created field with a timestamp, and then set up a scheduled function that runs at regular intervals (e.g., once per minute) to execute specific code for all records where the

created >= NOW - 31 mins AND created <= NOW - 30 mins

(pseudo code). For most cases, this method should suffice unless you have very precise time requirements.

If this approach doesn't meet your requirements, another option is to utilize Cloud Tasks from Google Cloud. More information on how to do this can be found in this informative article.

Execute a Cron Job every half an hour following the onCreate event in Firestore

Answer №1

Answer №2

Answer №3

Similar questions

The Express application is unable to locate the node_modules directory in order to load scripts

Unable to access the 'create' property from an undefined source

Retrieve information stored in cookies beyond the component's scope

Best practices for effectively managing errors within JSON web tokens

Sorry, the server cannot be reached at the moment. Please try again later

Strategies for resolving the issue of undefined req.body.password in express.js

"Embracing Progressive Enhancement through Node/Express routing and the innovative HIJAX Pattern. Exciting

running a Node JS script using PHP

Flowing Waterways and Transmission Control Protocol

I'm encountering an issue where the database table in Postgres is not updating correctly while working with Node

Oops! The mighty gulp-clean frowns upon attempts to eradicate files outside its domain

Extract website addresses from a text and store them in an array

Pass on Redis Pub/Sub messages to a designated client in socket.io through an Express server

GULP showing an error message "ACCESS DENIED"

Troubles encountered while attempting to properly mock a module in Jest

Rendering on the server with a focus on the header

Granting permission to the user

Navigating through async functions in an Express.js router

Steps for integrating a Facebook Messenger chatbot with a MongoDB database in order to submit requests and retrieve responses through conversations with the bot

How can I retrieve the GET parameters following the "?" in an Express application?