Currently, I am facing a situation where my server receives requests and I need a way to initiate the scraping process on a separate resource that is dynamically created. I would prefer not to conduct the scraping on my primary instance to prevent overloading it. Additionally, I do not want the secondary instance to continue running and costing me when it's not actively scraping data. Therefore, I am seeking a solution that allows me to request the start of the scraping process and automatically close the instance once the task is completed.
After exploring options such as Google Cloud Functions, which have a maximum runtime cap of 9 minutes per function - insufficient for my scraping needs, and AWS SDK, which enables the creation and closure of VMs but presents challenges with pushing API scripts onto the new AWS instance, I find myself in need of a more scalable and adaptable system. With multiple scripts dedicated to scraping different websites, I require a robust and flexible solution that can accommodate these various tasks.
I am flexible in terms of technology and open to any suggestions or assistance that could provide a suitable answer to this issue. Your help and guidance are highly appreciated. Thank you.