Leveraging Selenium for extracting data from a webpage containing JavaScript

Question

Leveraging Selenium for extracting data from a webpage containing JavaScript

I am trying to extract data from a Google Scholar page that has a 'show more' button. After researching, I found out that this page is not in HTML format but rather in JavaScript. There are different methods to scrape such pages and I attempted to use Selenium with the following code:

from selenium import webdriver
from bs4 import BeautifulSoup

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
chrome_path = r"....path....."
driver = webdriver.Chrome(chrome_path)

driver.get("https://scholar.google.com/citations?user=TBcgGIIAAAAJ&hl=en")

driver.find_element_by_xpath('/html/body/div/div[13]/div[2]/div/div[4]/form/div[2]/div/button/span/span[2]').click()

soup = BeautifulSoup(driver.page_source,'html.parser')

papers = soup.find_all('tr',{'class':'gsc_a_tr'})

for paper in papers:
    title = paper.find('a',{'class':'gsc_a_at'}).text
    author = paper.find('div',{'class':'gs_gray'}).text
    journal = [a.text for a in paper.select("td:nth-child(1) > div:nth-child(3)")]
    
       
    print('Paper Title:', title, '\nAuthor:', author, '\nJournal:', journal)

Even though the browser now clicks the 'show more' button and displays the complete page, I am only able to retrieve information for the first 20 papers. I am puzzled by this limitation and would appreciate any assistance to resolve it.

Thank you!

javascript python selenium web-scraping beautifulsoup

Answer 1

Answer №1

I suspect the issue lies in the timing of when your program checks for newly loaded elements on the website. Have you tried incorporating a time delay to allow all elements to fully load before scraping the data? Here's an example without the headless features:

from selenium import webdriver
import time
from bs4 import BeautifulSoup

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')

driver = webdriver.Chrome()

driver.get("https://scholar.google.com/citations?user=TBcgGIIAAAAJ&hl=en")
time.sleep(3)
driver.find_element_by_id("gsc_bpf_more").click()
time.sleep(4)
soup = BeautifulSoup(driver.page_source, 'html.parser')

papers = soup.find_all('tr', {'class': 'gsc_a_tr'})

for paper in papers:
    title = paper.find('a', {'class': 'gsc_a_at'}).text
    author = paper.find('div', {'class': 'gs_gray'}).text
    journal = [a.text for a in paper.select("td:nth-child(1) > div:nth-child(3)")]

    print('Paper Title:', title, '\nAuthor:', author, '\nJournal:', journal)

Answer 2

I suspect the issue lies in the timing of when your program checks for newly loaded elements on the website. Have you tried incorporating a time delay to allow all elements to fully load before scraping the data? Here's an example without the headless features:

from selenium import webdriver
import time
from bs4 import BeautifulSoup

options = webdriver.ChromeOptions()
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')

driver = webdriver.Chrome()

driver.get("https://scholar.google.com/citations?user=TBcgGIIAAAAJ&hl=en")
time.sleep(3)
driver.find_element_by_id("gsc_bpf_more").click()
time.sleep(4)
soup = BeautifulSoup(driver.page_source, 'html.parser')

papers = soup.find_all('tr', {'class': 'gsc_a_tr'})

for paper in papers:
    title = paper.find('a', {'class': 'gsc_a_at'}).text
    author = paper.find('div', {'class': 'gs_gray'}).text
    journal = [a.text for a in paper.select("td:nth-child(1) > div:nth-child(3)")]

    print('Paper Title:', title, '\nAuthor:', author, '\nJournal:', journal)

Answer 3

Answer №2

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.page_load_strategy = 'normal'
driver = webdriver.Chrome(options=options)

driver.get("https://scholar.google.com/citations?user=TBcgGIIAAAAJ&hl=en")

# Strategy for retrieving articles from webpage
for i in range(1, 3):
    driver.find_element_by_css_selector('#gsc_bpf_more').click()
    # wait for elements to load
    time.sleep(3)

# Extracting data from container
for result in driver.find_elements_by_css_selector('#gsc_a_b .gsc_a_t'):
    title = result.find_element_by_css_selector('.gsc_a_at').text
    authors = result.find_element_by_css_selector('.gsc_a_at+ .gs_gray').text
    publication = result.find_element_by_css_selector('.gs_gray+ .gs_gray').text
    print(title)
    print(authors)
    print(publication)
    # separating content
    print()

Sample of the retrieved output:

Tax/subsidy policies in the presence of environmentally aware consumers
S Bansal, S Gangopadhyay
Journal of Environmental Economics and Management 45 (2), 333-355

Choice and design of regulatory instruments in the presence of green consumers
S Bansal
Resource and Energy economics 30 (3), 345-368

Answer 4

import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.page_load_strategy = 'normal'
driver = webdriver.Chrome(options=options)

driver.get("https://scholar.google.com/citations?user=TBcgGIIAAAAJ&hl=en")

# Strategy for retrieving articles from webpage
for i in range(1, 3):
    driver.find_element_by_css_selector('#gsc_bpf_more').click()
    # wait for elements to load
    time.sleep(3)

# Extracting data from container
for result in driver.find_elements_by_css_selector('#gsc_a_b .gsc_a_t'):
    title = result.find_element_by_css_selector('.gsc_a_at').text
    authors = result.find_element_by_css_selector('.gsc_a_at+ .gs_gray').text
    publication = result.find_element_by_css_selector('.gs_gray+ .gs_gray').text
    print(title)
    print(authors)
    print(publication)
    # separating content
    print()

Sample of the retrieved output:

Tax/subsidy policies in the presence of environmentally aware consumers
S Bansal, S Gangopadhyay
Journal of Environmental Economics and Management 45 (2), 333-355

Choice and design of regulatory instruments in the presence of green consumers
S Bansal
Resource and Energy economics 30 (3), 345-368

Leveraging Selenium for extracting data from a webpage containing JavaScript

Answer №1

Answer №2

Similar questions

What should you do to move to the next line or block if a specific element cannot be located?

retrieve PHP function calls as an array using Ajax

A guide on customizing column names in MUI Datatables through object keys

The functionality of getAttribute has changed in Firefox 3.5 and IE8, no longer behaving as it did before

Combining JWT authentication with access control lists: a comprehensive guide

Analyze items in two arrays using JavaScript and add any items that are missing

Performing an ASync call to the GetData routine in MongoClient using NodeJS

Selenium: Mastering the Art of Dragging and Dropping

Using EJS to Render a Function Expression?

Flaskdance does not provide a secure HTTPS URI

Should a checkbox be added prior to the hyperlink?

The range selector on the x-axis of the Highstock chart is currently showing data only between January 8, 1970, and January 19,

Guidelines on centering an AJAX spinning animation within a parent div

Is there a way to modify a JSON dictionary at different levels?

The beforeCreate function is failing to execute when a new user is being created

Display a text field when the onclick event is triggered within a for

How can I design an avatar image within a button similar to Facebook's style?

What is the best way to transform a string representation of data into an array and then showcase it in

Converting a Base64 URL to an image object: A step-by-step guide

Tips for choosing the remaining items in a multiple selection process