Utilizing Scrapy for Extracting Size Information obscured by Ajax Requests

Question

Utilizing Scrapy for Extracting Size Information obscured by Ajax Requests

I am currently trying to extract information about the sizes available for a specific product from this URL:

However, I am encountering difficulty in locating the details hidden within the Select Size Dropdown on the webpage (e.g., 7 - In Stock, 7.5 - In Stock 5+, etc.).

Even when inspecting the code in debugger mode, my attempts to scrape this information using the provided Xpath have been unsuccessful:

item["Sizes"] = sel.xpath("//select[@name='siz']/option/text()").extract()

I suspect that these details may be obscured by Ajax. Any assistance would be greatly appreciated.

python ajax xpath web-scraping scrapy

Answer 1

Answer №1

The issue arises from the fact that the sizes are contained within an iframe that is loaded from a separate URL. To resolve this, you first need to retrieve the URL from the src attribute of the iframe, then send a request to that URL in order to extract the sizes.

Here's a demonstration (using scrapy shell):

$ scrapy shell http://www.tennisexpress.com/k-swiss-mens-ultra-express-tennis-shoes-black-fade-and-electric-blue-38191
>>> from urlparse import urljoin
>>> url = 'http://www.tennisexpress.com/'
>>> path = response.xpath('//div[@id="prodPurchasing"]/iframe/@src').extract()[0]
>>> url = urljoin(url, path)
>>> fetch(url)
>>> response.xpath("//select[@name='siz']/option/text()").re(r'[\.0-9]+')
[u'7', u'7.5', u'8', u'8.5', u'9', u'9.5', u'10', u'10.5', u'11', u'11.5', u'12', u'13']

As a note, I am using re() to filter out legitimate sizes from the select options, as demonstrated below:

The pattern [\.0-9]+ will match one or more digits or dots.

Answer 2

The issue arises from the fact that the sizes are contained within an iframe that is loaded from a separate URL. To resolve this, you first need to retrieve the URL from the src attribute of the iframe, then send a request to that URL in order to extract the sizes.

Here's a demonstration (using scrapy shell):

$ scrapy shell http://www.tennisexpress.com/k-swiss-mens-ultra-express-tennis-shoes-black-fade-and-electric-blue-38191
>>> from urlparse import urljoin
>>> url = 'http://www.tennisexpress.com/'
>>> path = response.xpath('//div[@id="prodPurchasing"]/iframe/@src').extract()[0]
>>> url = urljoin(url, path)
>>> fetch(url)
>>> response.xpath("//select[@name='siz']/option/text()").re(r'[\.0-9]+')
[u'7', u'7.5', u'8', u'8.5', u'9', u'9.5', u'10', u'10.5', u'11', u'11.5', u'12', u'13']

As a note, I am using re() to filter out legitimate sizes from the select options, as demonstrated below:

The pattern [\.0-9]+ will match one or more digits or dots.

Utilizing Scrapy for Extracting Size Information obscured by Ajax Requests

Answer №1

Similar questions

Tips for detecting and eliminating outliers in a dataset containing a mixture of numerical and categorical data

What could be causing the parameters in my function to malfunction?

Alter the URL for Jquery AJAX post by utilizing radio buttons

Unable to convert JSON data into a datatable

What are the steps to modify the URL and template page post-login?

Retrieve another resource from the Flask API

Best practices for handling AJAX GET and POST requests in EmberJS

Using Django and mypy for type hinting with the ValuesQuerySet

Guide on how to dynamically add AJAX JSON array response to an HTML table

The PartialView is currently displaying the entire page instead of just rendering the form

What is the best way to perform vectorized calculations on a pandas dataframe, replacing values with the data from the previous row if a certain condition is not satisfied

Tips for extracting particular information from a website and displaying it within my application

"Create a notification pop-up in CSS that appears when a link is clicked, similar to

Exploring Python's list comprehension assignments

Updating array with user input using Ajax

Updating password in Django with AJAX

I am encountering an issue with implementing the (EC.presence_of_element_located(By.class, "")) function

"Enhance Your Form with Ajax Submission using NicEdit

Divergent behavior of Selenium xpath when used with python versus ruby

Issue with IE7: jQuery.getJSON callback not triggering