Encountered an issue where the page parsing code consistently checks the same page every time, despite using it alongside selenium. Selenium has no problem opening new links, but the parsing only occurs on the initial page.
The frustrating part is that similar logic works correctly with another website in different code examples.
from bs4 import BeautifulSoup
from selenium import webdriver
keys_list = []
def start_browser(link):
profile = 'C:\\Users\\Crazy_MoT\\AppData\\Local\\Google\\Chrome\\User Data\\Default'
options = webdriver.ChromeOptions()
try:
options.add_argument(f"user-data-dir={profile}")
browser = webdriver.Chrome(options=options)
except:
print("Connect to profile... Error\n Opening new profile")
browser = webdriver.Chrome()
browser.quit()
#browser.get(link)
browser.get(link)
html = browser.page_source
soup = BeautifulSoup(html, 'html.parser')
author = soup.find("a", attrs={'data-qa': 'FileViewAuthorBox'}, href=True)
print(author["href"])
keywords = soup.find_all("span", class_="_oX66p")
for keys in keywords:
keys_list.append(keys.text)
print(keys_list)
def start(links):
for link in links:
start_browser(link)
links = ["https://ru.depositphotos.com/26182475/stock-photo-happy-birthday.html",
"https://ru.depositphotos.com/39273619/stock-photo-label-with-happy-birthday.html"]
start(links)
Attempting to gather data from various pages on the site, but information retrieval is limited to the first page and then repeats itself.