Recently, I created a Flask web application that utilizes Celery for task handling. Within this app, one of the tasks involves scraping approximately 200 pages using a custom Class derived from a selenium chrome driver.
@celery_app.task
def scrape_async():
driver = MyDriver(exec_path=os.environ.get('CHROMEDRIVER_PATH'), options=some_chrome_options)
# Update 'urls_to_scrape' attribute by finding the urls to scrape from a main url
driver.find_urls_to_scrape_from_main_url()
# Loop over each page and store the data in database
for url in driver.urls_to_scrape:
driver.scrape_page(url)
# Exit driver
driver.quit()
This process worked flawlessly both locally and in production until the number of pages to scrape increased. At that point, I encountered a memory usage error on Heroku and realized the high memory consumption involved in this task.
After conducting some research, I discovered that implementing a subtask within my loop (i.e., executing this subtask for each individual page) could be a more efficient approach, as illustrated below.
@celery_app.task
def subtask(url):
driver.scrape_page(url)
@celery_app.task
def scrape_async():
driver = MyDriver(exec_path=os.environ.get('CHROMEDRIVER_PATH'), options=some_chrome_options)
# Update 'urls_to_scrape' attribute by finding the urls to scrape from a main url
driver.find_urls_to_scrape_from_main_url()
# Loop over each page and store the data in database
for url in driver.urls_to_scrape:
subtask.delay(url)
# Exit driver
driver.quit()
I am now faced with the dilemma of how to maintain the driver object between the main task and the subtask. Although I came across this resource on instantiating Tasks, I struggled to figure out how to create one driver instance that spans multiple tasks.
Do you have any suggestions on how I can address this issue and achieve my objective?
Thank you