My web scraping process involves using both Scrapy and Selenium. When I run my spider, I notice the instance of my webdriver in htop.
I am unsure about when exactly I should close the webdriver in my code. Would it be best to do so after processing each link or at the end of the script?
def parse():
# I have all links in my array_links
for link in self.array_links:
self.driver.get(link)
# Here i Parse the products
item = MyTestItem()
item['test1'] = "test"
yield item
I considered adding this code snippet:
def __del__(self):
self.driver.quit()
to automatically close the webdriver at the end of the script. However, I still question if closing the webdriver after processing each link would be more efficient.
Any advice on this matter would be greatly appreciated. Thank you.