Guidelines for transferring Selenium WebDriver response to Scrapy's parse method

Question

Guidelines for transferring Selenium WebDriver response to Scrapy's parse method

I am currently facing two challenges that I need to overcome:

First: effectively locating and interacting with an element on a website using a driver. Second: passing the link generated from this interaction to a parse method or LinkExtractor.

ad. 1.

My task involves finding and clicking the "Load more" button in order to crawl the subsequent page.

<div class="col-sm-4 col-sm-offset-4 col-md-2 col-md-offset-5 col-xs-12 col-xs-offset-0">
            <button class="btn btn-secondary">Load more</button>
        </div>

ad.2.

I have set up LinkExtractor rules that function correctly on static websites, as well as a parse method. While I have come across similar examples in my research, I struggle to piece it all together seamlessly.

This is my latest attempt:

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from os.path import join as path_join
import json
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
import time

# Code shortened for readability...

<p>I have omitted a significant portion of the parse_page method for clarity (it does not impact functionality).</p>

<p>Here is another approach I experimented with, but did not achieve success:</p>

<pre><code>    driver = webdriver.Chrome(ChromeDriverManager().install())

    def parse_url(self, response):
        self.driver.get(response.url)

        while True:
            self.driver.implicitly_wait(2)
            next = self.driver.find_element_by_class_name('btn btn-secondary')

            try:
                next.click()
                time.sleep(2)
                # Extract and process data into Scrapy items
            except:
                break

        self.driver.close()

Any insights or guidance on either of these challenges would be greatly appreciated.

python selenium scrapy

Answer 1

Answer №1

To integrate Selenium requests and return a HtmlResponse, you can create a custom middleware as shown below:

from scrapy import Request

class SeleniumRequest(Request):
    pass

This subclass helps the middleware determine if Selenium should be utilized. Remember to close the driver when the spider closes.

class SeleniumMiddleware:
    def __init__(self):
        self.driver = webdriver.Chrome(ChromeDriverManager().install())

    @classmethod
    def from_crawler(cls, crawler):
        middleware = cls()
        crawler.signals.connect(middleware.spider_closed, signals.spider_closed)
        return middleware

    def process_request(self, request, spider):
        if not isinstance(request, SeleniumRequest):
            return None
        self.driver.get(request.url)

        # code snippet for clicking "load more" button on JS-page
        while True:
            try:
                button = WebDriverWait(self.driver, 10).until(
                    EC.presence_of_element_located((By.CLASS_NAME, 'btn btn-secondary'))
                )
                self.driver.execute_script("arguments[0].click();", button)
            except:
                break

        return HtmlResponse(
            self.driver.current_url,
            body=str.encode(self.driver.page_source),
            encoding='utf-8',
            request=request
        )

    def spider_closed(self):
        self.driver.quit()

Enable the middleware in your settings file:

DOWNLOADER_MIDDLEWARES = {
    'your_project.middleware_location.SeleniumMiddleware': 500
}

You can now use SeleniumRequest in your Spider's start_requests method:

def start_requests(self):
    yield SeleniumRequest(your_url, your_callback)

With this setup, all requests will be processed through Selenium, returning HtmlResponses. Make sure to utilize LinkExtractors and the parse method with these responses. For further guidance, check out this GitHub repository.

Answer 2

To integrate Selenium requests and return a HtmlResponse, you can create a custom middleware as shown below:

from scrapy import Request

class SeleniumRequest(Request):
    pass

This subclass helps the middleware determine if Selenium should be utilized. Remember to close the driver when the spider closes.

class SeleniumMiddleware:
    def __init__(self):
        self.driver = webdriver.Chrome(ChromeDriverManager().install())

    @classmethod
    def from_crawler(cls, crawler):
        middleware = cls()
        crawler.signals.connect(middleware.spider_closed, signals.spider_closed)
        return middleware

    def process_request(self, request, spider):
        if not isinstance(request, SeleniumRequest):
            return None
        self.driver.get(request.url)

        # code snippet for clicking "load more" button on JS-page
        while True:
            try:
                button = WebDriverWait(self.driver, 10).until(
                    EC.presence_of_element_located((By.CLASS_NAME, 'btn btn-secondary'))
                )
                self.driver.execute_script("arguments[0].click();", button)
            except:
                break

        return HtmlResponse(
            self.driver.current_url,
            body=str.encode(self.driver.page_source),
            encoding='utf-8',
            request=request
        )

    def spider_closed(self):
        self.driver.quit()

Enable the middleware in your settings file:

DOWNLOADER_MIDDLEWARES = {
    'your_project.middleware_location.SeleniumMiddleware': 500
}

You can now use SeleniumRequest in your Spider's start_requests method:

def start_requests(self):
    yield SeleniumRequest(your_url, your_callback)

With this setup, all requests will be processed through Selenium, returning HtmlResponses. Make sure to utilize LinkExtractors and the parse method with these responses. For further guidance, check out this GitHub repository.

Guidelines for transferring Selenium WebDriver response to Scrapy's parse method

Answer №1

Similar questions

Help! How can I prevent using Thread.sleep() multiple times in a selenium test?

An effective method for removing various stems from the end of a word is to utilize slicing techniques

Selenium encountered an error in retrieving the section id from the web page

What steps are involved in including the Gradle Kotlin dependency for the compile group 'org.seleniumhq.selenium'?

How can I update the code to remove the DeprecationWarning caused by using asyncio.get_event_loop()?

Customize the colors of a line plot for a NumPy array using matplotlib

A guide on incorporating background threads in Flask:

Selenium ChromeDriver encountered an issue loading a resource: net::ERR_CONNECTION_CLOSED

Choose a checkbox by targeting a specific column value with XPath

E2E tests for Internet Explorer using Selenium and Protractor

Error: The 'chromedriver' executable must be located in the PATH directory

Best method for releasing a string returned from a C function in CFFI?

Updating an object in Python multiprocessing

Effective methods for accessing and updating text editor values using Selenium

Capturing Screenshots as Numpy Arrays Using Selenium WebDriver in Python

The issue of using an import statement outside a module arises when executing Protractor

Selenium: The parent window automatically closes when a child window pops up

Leveraging the power of sqlite3 alongside the flexibility of k

Discovering elements in jQuery selected within Selenium

Struggling to concentrate on a recently opened Selenium window