After previously seeking help on navigating multiple pages with static URLs at , I appreciate the assistance received so far! However, my current goal is to extract ethnicity information for every character listed by clicking on each name. Although I am able to navigate through all the pages, my code is stuck scraping data only from the initial page.
Here's what I have attempted:
url = 'https://ethnicelebs.com/all-celeb'
driver = webdriver.Chrome()
driver.get(url)
while True:
page = requests.post('https://ethnicelebs.com/all-celebs')
soup = BeautifulSoup(page.text, 'html.parser')
for href in soup.find_all('a', href=True)[18:]:
print('Found the URL:{}'.format(href['href']))
request_href = requests.get(href['href'])
soup2 = BeautifulSoup(request_href.content)
for each in soup2.find_all('strong')[:-1]:
print(each.text)
Next_button = (By.XPATH, "//*[@title='Go to next page']")
WebDriverWait(driver, 50).until(EC.element_to_be_clickable(Next_button)).click()
url = driver.current_url
time.sleep(5)
(Credit goes to @Sureshmani!)
My objective is for the code to scrape data from each subsequent page as it navigates instead of being limited to the first page. How can I ensure that the scraping process continues with each new page? Thank you!