I've been on a wild expedition through the depths of the internet in search of an answer to this particular enigma, but my efforts have been in vain so far.
Currently, I'm attempting to extract data from the last four pages of last.fm entries for "Jazz Metal" (check out the URL).
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.headless = True
driver = webdriver.Firefox(options=options)
driver.get('https://www.last.fm/tag/jazz+metal/artists?page=20')
super_list = []
wait = WebDriverWait(driver, 10)
while True:
try:
entries = wait.until(
EC.presence_of_element_located((By.CLASS_NAME, 'grid-items-section'))
)
grid = driver.find_element(By.CLASS_NAME, 'grid-items-section')
grid_children = grid.find_elements(By.TAG_NAME, 'li')
super_list.append(grid_children)
pagination = wait.until(
EC.presence_of_element_located((By.CLASS_NAME, 'pagination-next'))
)
pagination.click()
except:
break
The issue is that super_list.append(grid_children) doesn't provide much help because when the while loop ends and I try to access .text method outside of that scope using super_list, all I get is a list that's almost impossible to decipher.
<selenium.webdriver.firefox.webelement.FirefoxWebElement (session="11b49c8e-eec7-45f2->9e2a-e2034b93077a", element="ffe29b8e-5b65-4df3-985e-68e501e3a546")>
However, if I change super_list.append(grid_children) to
super_list.append([entry.text for entry in grid_children])super_list.append(grid_children) completely, it goes through every page (yes, it currently misses the last page entirely)!
To make matters more confusing, when I include
finally:
driver.quit()
only the first page gets visited. Can anyone shed some light on this mysterious behavior?