The Failure of Scrapy Pagination

Question

The Failure of Scrapy Pagination

Hello everyone, I am excited to share my first post!

Currently, I am working on developing a Web Spider that is capable of following links on invia.cz and extracting all the hotel titles.

import scrapy

y=0
class invia(scrapy.Spider):
    name = 'Kreta'
    start_urls = ['https://dovolena.invia.cz/?d_start_from=13.01.2017&sort=nl_sell&page=1']

    def parse(self, response):

        for x in range(1, 9):
            yield {
             'titles':response.css("#main > div > div > div > div.col.col-content > div.product-list > div > ul > li:nth-child(%d)>div.head>h2>a>span.name::text"%(x)).extract() ,
             }

        if (response.css('#main > div > div > div > div.col.col-content >   
                            div.product-list > div > p > 
                            a.next').extract_first()):
         y=y+1
         go = ["https://dovolena.invia.cz/d_start_from=13.01.2017&sort=nl_sell&page=%d" % y] 
         print go
         yield scrapy.Request(
                response.urljoin(go),
                callback=self.parse
         )

I encountered an issue with web pages loading asynchronously due to AJAX. To address this, I manually update the URL value by incrementing it whenever the next button appears.

While testing in the scrapy shell, everything functions as expected. However, during actual spider execution, it only scrapes the initial page.

This project marks my debut into the world of web spiders, so any helpful advice is greatly appreciated!

For reference, here are the error logs: Error Log1, Error Log

ajax web-scraping scrapy

Answer 1

Answer №1

Your use of the "global" y variable is quite unusual and impractical.

Instead of relying on y to count parse calls, it's recommended to utilize the request.meta attribute within the function's scope:

def parse(self, response):
    y = response.meta.get('index', 1)  
    y += 1
    url = 'http://example.com/?p={}'.format(y)
    yield Request(url, self.parse, meta={'index':y})

In addressing the pagination issue, the css selector for the next page URL is incorrect due to missing href attributes in the selected <a> node:

def parse(self, response):
    next_page = response.css("a.next::attr(data-page)").extract_first()
    url = re.sub('page=\d+', 'page=' + next_page, response.url)
    yield Request(url, self.parse, meta={'index':y})

To ensure functionality, here is the revised spider code:

import scrapy
import re

class InviaSpider(scrapy.Spider):
    name = 'invia'
    start_urls = ['https://dovolena.invia.cz/?d_start_from=13.01.2017&sort=nl_sell&page=1']

    def parse(self, response):
        names = response.css('span.name::text').extract()
        for name in names:
            yield {'name': name}

        # navigating to the next page
        next_page = response.css("a.next::attr(data-page)").extract_first()
        url = re.sub('page=\d+', 'page=' + next_page, response.url)
        yield scrapy.Request(url, self.parse)

Answer 2

Your use of the "global" y variable is quite unusual and impractical.

Instead of relying on y to count parse calls, it's recommended to utilize the request.meta attribute within the function's scope:

def parse(self, response):
    y = response.meta.get('index', 1)  
    y += 1
    url = 'http://example.com/?p={}'.format(y)
    yield Request(url, self.parse, meta={'index':y})

In addressing the pagination issue, the css selector for the next page URL is incorrect due to missing href attributes in the selected <a> node:

def parse(self, response):
    next_page = response.css("a.next::attr(data-page)").extract_first()
    url = re.sub('page=\d+', 'page=' + next_page, response.url)
    yield Request(url, self.parse, meta={'index':y})

To ensure functionality, here is the revised spider code:

import scrapy
import re

class InviaSpider(scrapy.Spider):
    name = 'invia'
    start_urls = ['https://dovolena.invia.cz/?d_start_from=13.01.2017&sort=nl_sell&page=1']

    def parse(self, response):
        names = response.css('span.name::text').extract()
        for name in names:
            yield {'name': name}

        # navigating to the next page
        next_page = response.css("a.next::attr(data-page)").extract_first()
        url = re.sub('page=\d+', 'page=' + next_page, response.url)
        yield scrapy.Request(url, self.parse)

The Failure of Scrapy Pagination

Answer №1

Similar questions

Encountering AJAX Error 0 with jQueryUI Autocomplete upon pressing enter key

What is the most effective way to invoke a particular function/method within a Python script using Javascript (specifically, jquery/ajax)?

Refreshing certain sections of a webpage without the need to refresh the entire page

It is not possible to make a comparison between two Strings within a JSP file

Upgrade the WordPress light editor to the advanced version

Load the page before the images are displayed

"Troubleshooting: Inability to send emails through PHP mail script while using jQuery's

Send back alternate HTML content if the request is not made via AJAX

Rails offers a unique hybrid approach that falls between Ember and traditional JavaScript responses

Uncovering complete hyperlink text using Scrapy

Is there a way to automatically update the URL to include $_GET['data'] (media.php?data=123) when a selection is made from a dropdown list?

Sorting of tables does not function properly following an ajax request

Issue occurring while trying to select an item from the dynamically generated options using AJAX

Perform a bash command using PHP when an HTML button is clicked

The Ajax request functions flawlessly on Mozilla but encounters issues on Chrome, causing confusion as it occasionally works

Tips for updating the color of a table row when it is chosen and a row is inserted using ajax

What steps are involved in setting up a RESTful service for a web application?

jquery logic for iterating through all elements in a select menu encountering issues

"Is there a way to implement the functions of adding, editing, and deleting data in

Update the content within the Wicket AjaxLink