Which AWS service would be best suited for logging user application interaction events through Lambda; perhaps Cloudwatch could be the solution

Our team is currently working on developing an API server (using graphql, prisma, and node) that runs in a Lambda function deployed through Apex Up.

We have a need to log various user events, including:

  • Successful logins
  • Failed login attempts
  • User actions such as X

Currently, these events are logged in our app's Aurora DB through the main API's access. However, this approach results in a high volume of writes and rows due to the numerous user actions being tracked. This has raised concerns about scalability for us.

We are now considering whether it would be more efficient to store this data directly in Cloudwatch. We are specifically interested in knowing if Cloudwatch is suitable for categorizing and later analyzing/ querying the data based on user_id and event_type.

If there are alternative solutions better suited for AWS or elsewhere, we would like to explore those options as well.

node.jsamazon-web-servicesloggingaws-lambdaamazon-cloudwatch

Answer №1

Storing raw, unprocessed logs in CloudWatch is a practical choice. However, the limitation of not being able to query custom fields like user_id or event_type directly in CloudWatch poses a challenge.

To overcome this limitation, you can leverage CloudWatch subscription filters to route your logs to a Kinesis Stream for further analysis and processing. By subscribing a Lambda Function to your Kinesis Stream, you gain flexibility in handling log entries.

The Lambda function offers various possibilities for manipulating log data:

  • Archiving log entries in Parquet format on S3 for future querying with Redshift Spectrum
  • Parsing log entries and transferring them to a Redshift database
  • Analyzing log entries in an ElasticSearch cluster connected to Kibana for visualization
  • And more...

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Many of the middleware options that were once included with Express, such as favicon, now require separate installations

My attempts to resolve the issue included installing serve favicon into the directory of my app (C:\Users\I am\Backup\Work node install serve-favicon), but unfortunately, I encountered the same error. Check out the image for reference: ...

node.jsexpress

The Whatsapp webhook callback endpoint is experiencing a delay in being triggered

Currently, I am utilizing the Whatsapp cloud API and have successfully configured the webhook with a valid URL. While it is functioning correctly, there is an issue that arises when I receive a message from the business and immediately respond. The callbac ...

node.jsexpresswhatsapp

Error message: The 'run-android' command is not recognized after installing a new package in a React Native project

Progress on my React Native project was smooth until I attempted to add new packages. After running the project on an android emulator, I encountered an error stating Command run-android unrecognized. Make sure that you have run npm install and that you ar ...

androidnode.jswindowsreact-nativenpm

The response data from the API is filled with mysterious escape characters

Obtaining data from the API response looks like this: { "ORG_ID":"165", "DEPOT_NAME":"Pesto", "DEPOT_SHORT_NAME":"PSD", "PROD_ID":"709492", "DESCRIPTION":"EX CL (2X14) U17\SH36\5", "PRICE":"3708.55 ...

node.jsjsonsails.js

Streaming large files with Node.js can lead to significant memory consumption and potential memory errors like OOM

My current project involves using node.js to download large files (300MB) from a server and then piping the response to a file write stream. While I have a good understanding of how pipes work in Node.js, I am encountering an issue where the memory usage o ...

javascriptnode.jsexpressheap-memorynodejs-stream

Unable to prevent git from continuously keeping track of package-lock.json

After updating to npm v5, a new package-lock.json file is generated when running npm install. Although it is advised to commit this file, I am facing an issue where the content of this file differs between my development machine and the server. Even if I ...

node.jsgitnpm

I've been waiting forever for Product.find() to return some results, but it seems to

I have been encountering an issue where my code is supposed to return an empty object of a product but instead it just keeps loading forever. I have thoroughly checked through the code and explored every possible scenario where an error could be occurring, ...

javascriptnode.jsmongodbnext.js

Is there a way to consistently substitute a specific path parameter with a different value within node.js?

Snippet of my coding: router.get('/name/:name/height', (req,res) => { ... } router.get('/name/:name/weight', (req,res) => { ... } router.get('/age/:age/height', (req,res) => { ... } router.get('/ag ...

javascriptnode.jsexpress

Error: FileReader is not defined in Node.js (Nest.js) environment

I am working on converting my image to base64 using a custom function. However, when I try to execute the code, I encounter an error message stating ReferenceError: FileReader is not defined. This error has left me puzzled and unsure of its cause. Below i ...

javascriptnode.js

Tips for storing an array of objects in MongoDB with Mongoose

Here are some code snippets that involve objects: var s; var tab = []; var myarray= []; for(var i=0;i<=tab.length-1;i++){ s= "{\"id\":\"" + tab[i][0] + "\",\"ts\":\"" + tab[i][1] + "\",\"lat&bs ...

node.jsmongodbmongoose

ES6 npm package failing to bundle properly in webpack

In the project I'm working on package.json only includes babel related packages: "@babel/core": "7.0.0-beta.37", "@babel/plugin-syntax-dynamic-import": "7.0.0-beta.37", "@babel/register": "7.0.0-beta.37", "babel-eslint": "https://github.com/kesne/ba ...

node.jsnpmwebpackbabeljsbabel-loader

Is there a permanent solution to fixing the error code -4094 that is repeatedly occurring during React Native startup?

When attempting to execute react-native start, an error occurred which has not been encountered before. The error message is as follows: ERROR ENCOUNTERED Loading dependency graph...events.js:287 throw er; // Unhandled 'error' event ...

javascriptnode.jsreactjsreact-nativenpm

Creating unique message formats for communication between a python client and a socket.io node.js server

Currently, I am attempting to establish communication between a Python client and a Node.js server using Socket.io 0.7. My goal is to send a custom event from the client to the server. To achieve this, I have been referencing the Socket.io documentation o ...

pythonnode.jswebsocketsocket.io

Connection to MongoDB in Docker is abruptly terminated upon the receipt of metadata

I am currently working on setting up a docker application with two containers: mongo app While the mongo container is functioning properly, the app container is unable to establish a connection with mongo. Neither the node.js app nor mongostat are able ...

node.jsmongodbdockerdocker-compose

Connection error between frontend and backend was encountered

Whenever I try to register on my page and connect it to the database, I encounter an error after pressing the sign-in button... "Uncaught (in promise) TypeError: Converting circular structure to JSON --> starting at object with constructor &apo ...

mysqlnode.jsjsonreactjs

Choosing the right framework for implementing push notifications can be a critical decision. Should

I am currently working on a Java application that requires the server to send push notifications to the client every one second. To achieve this, I am using HTML5 server-sent events for one-way communication from the server to the client. However, my conce ...

javascripthtmlnode.jspush-notificationvert.x

Setting up package-lock.json as the definitive source of dependencies truth

I had a similar question to the one asked on Stack Overflow about package.json and package-lock.json (in summary; "what's the difference between package.json and package-lock.json?") and found some insightful answers there. However, I still have some ...

node.jsnpmpackage.jsondependency-managementpackage-lock.json

Using JavaScript to pass a newly created variable as an argument in a default

Currently, I am developing a node application that heavily utilizes promises. To enhance the readability of my code, I have been removing anonymous functions and storing them in a JavaScript object called "myCallbacks". Here's an example: let myCallb ...

javascriptnode.jses6-promiseasynccallback

The NestJS framework encountered an error due to a method being undefined in the

Encountering Error with NestJS Function create123: TypeError - Cannot read properties of undefined (reading 'create123') The constructor is displayed below \`export class AuthenticationService { constructor( private readonly usersServ ...

node.jstypescriptnestjs

Exploring the power of NestJS integration with Mongoose and GridFS

I am exploring the functionality of using mongoose with NestJs. Currently, I am leveraging the package @nestjs/mongoose as outlined in the informative documentation. So far, it has been functioning properly when working with standard models. However, my p ...

node.jsmongodbtypescriptmongoosenestjs