I'm looking to extract specific information from aliexpress using scrapy and selenium. However, I've encountered an issue where the HTML code appears differently when inspecting with Chrome compared to viewing the source.
It seems that the content is loaded asynchronously, which may be why I can't find the desired element on the page.
I attempted to use selenium to load the page first and then retrieve the necessary content, but was unsuccessful. My goal is to scroll down to access reviews section and obtain its content.
Could this be an advanced anti-bot measure implemented by AliExpress, or am I approaching the problem incorrectly?
This is my current code:
import scrapy
from selenium import webdriver
import logging
import time
logging.getLogger('scrapy').setLevel(logging.WARNING)
class MySpider(scrapy.Spider):
name = 'myspider'
start_urls = ['https://pl.aliexpress.com/item/32998115046.html']
def __init__(self):
self.driver = webdriver.Chrome()
def parse(self, response):
self.driver.get(response.url)
scroll_retries = 20
data = ''
while scroll_retries > 0:
try:
data = self.driver.find_element_by_class_name('feedback-list-wrap')
scroll_retries = 0
except:
self.scroll_down(500)
scroll_retries -= 1
print("----------")
print(data)
print("----------")
self.driver.close()
def scroll_down(self, pixels):
self.driver.execute_script("window.scrollTo(0, {});".format(pixels))
time.sleep(2)