Currently, I am diving into the world of web scraping using Selenium. To get some practical experience, I decided to extract promotions from this specific site:
https://i.stack.imgur.com/zKEDM.jpg
This is the code I have been working on:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
def get_promotion():
'''
Scraping data from Smiles promotions
'''
promotions = []
chrome_options = Options()
chrome_options.add_argument("--disable-notifications")
driver = webdriver.Chrome(options=chrome_options)
driver.implicitly_wait(10)
driver.get('https://www.smiles.com.br/home')
site_promotion = driver.find_elements(By.CLASS_NAME, 'swiper-slide')
for promotion in site_promotion:
promotions.append(
{
'destination': promotion.find_element(By.XPATH, f'./a/div/div/h3').text,
'origin': promotion.find_element(By.XPATH, f'./a/div/div/h4/span[2]').text,
'diamont_value': promotion.find_element(By.XPATH, f'./a/div/div/div/span[2]/p[3]').text,
'normal_value': promotion.find_element(By.XPATH, f'./a/div/div/div/span[2]/p[2]').text,
}
)
return promotions
The issue lies in the fact that the last 3 cards' information did not get extracted, as shown below:
[
{'destination': 'São Paulo (GRU)', 'origin': 'Maceió (MCZ)', 'diamont_value': '17.700', 'normal_value': '19.000'},
{'destination': 'Rio de Janeiro (GIG)', 'origin': 'Recife (REC)', 'diamont_value': '19.500', 'normal_value': '21.000'},
{'destination': 'Brasília (BSB)', 'origin': 'Recife (REC)', 'diamont_value': '17.700', 'normal_value': '19.000'},
{'destination': 'Porto Seguro (BPS)', 'origin': 'Belo Horizonte (CNF)', 'diamont_value': '13.300', 'normal_value': '14.500'},
{'destination': 'Goiânia (GYN)', 'origin': 'Palmas (PMW)', 'diamont_value': '11.500', 'normal_value': '12.500'},
{'destination': '', 'origin': '', 'diamont_value': '', 'normal_value': ''},
{'destination': '', 'origin': '', 'diamont_value': '', 'normal_value': ''},
{'destination': '', 'origin': '', 'diamont_value': '', 'normal_value': ''}
]
Upon further investigation, I noticed that when the browser is launched using Selenium, it fails to display the last 3 cards:
https://i.stack.imgur.com/5t5gx.png
The challenge does not seem to be related to the presence of elements. Upon debugging, I confirmed that the last 3 elements within the site_promotion are indeed present.
Could it be that Selenium is getting confused because the last 3 cards are not visible on the screen? If so, how can this issue be resolved?
Is there a way to retrieve text from these elements even if they are not in the current view?
I attempted adding
options.add_argument("--start-maximized")