The challenge of clicking the "ShowMore" button in an infinitely scrolling page is a common issue when using Selenium with Python

Question

The challenge of clicking the "ShowMore" button in an infinitely scrolling page is a common issue when using Selenium with Python

I am currently working on extracting Mobile Legend comment data from the site using web scraping. I need my bot to automatically scroll down and load as many comments as possible before scraping all of them.

However, I encountered an issue when the bot tries to click the "Showmore" button after infinite scrolling. It seems that after the second click on the "Showmore" button, an error is triggered ([7200:8128:0903/172837.024:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is disabled) causing the loop to break.

 
            // Python code for web scraping

This is a snippet of the terminal output:


            WebGL Activated
            [5296:3760:0903/174720.883:ERROR:device_event_log_impl.cc(214)] [17:47:20.883] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
            [5296:3760:0903/174720.889:ERROR:device_event_log_impl.cc(214)] [17:47:20.889] Bluetooth: bluetooth_adapter_winrt.cc:713 GetBluetoothAdapterStaticsActivationFactory failed: Class not registered (0x80040154)
            Click Showmore 0
            [9052:7864:0903/174913.082:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is disabled
            ------Scroll Finish-------
            Click ShowMore Counts = 1

python selenium selenium-webdriver web-scraping

Answer 1

Answer №1

My strategy involves using the END key 8 times and clicking on "show more" to reveal additional comments. There are quite a lot of comments, so let me know if you require assistance with scraping them.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.common.exceptions import NoSuchElementException
import time

url = "https://play.google.com/store/apps/details?id=com.mobile.legends&showAllReviews=true"

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(url)

while True:
    for _ in range(8):
        actions = ActionChains(driver)
        actions.send_keys(Keys.END).perform()
        time.sleep(1)
    
    try:
        driver.find_element_by_class_name("CwaK9").click()
    except NoSuchElementException:
        break

Answer 2

My strategy involves using the END key 8 times and clicking on "show more" to reveal additional comments. There are quite a lot of comments, so let me know if you require assistance with scraping them.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
from selenium.common.exceptions import NoSuchElementException
import time

url = "https://play.google.com/store/apps/details?id=com.mobile.legends&showAllReviews=true"

driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(url)

while True:
    for _ in range(8):
        actions = ActionChains(driver)
        actions.send_keys(Keys.END).perform()
        time.sleep(1)
    
    try:
        driver.find_element_by_class_name("CwaK9").click()
    except NoSuchElementException:
        break

Answer 3

Answer №2

This code is quite lengthy, but it will keep scrolling down as long as the showmore_count variable remains in place.

I have also included relevant comments to explain the code.

from selenium import webdriver
import time

driver = webdriver.Chrome(executable_path="path")
driver.maximize_window()
driver.implicitly_wait(10)
driver.get("https://play.google.com/store/apps/details?id=com.ucool.heroesarena&showAllReviews=true") # I had issues with the provided URL so I used this one instead.
time.sleep(5) # Wait for the page to fully load
j=0
showmore_count = 1 # This stops the script from continuous scrolling
try:
    while True:
        reviews = driver.find_elements_by_xpath("//div[@jsname='fk8dgd']/div")
        
        driver.execute_script("arguments[0].scrollIntoView(true);", reviews[j])
        driver.execute_script("window.scrollBy(0,-50)")
        print("{}: {}".format(j+1, reviews[j].find_element_by_xpath(".//span[@class='X43Kjb']").text)) # Display the reviewer's name
        j += 1
except IndexError: 
    while driver.find_element_by_xpath("//span[text()='Show More']").is_displayed() and showmore_count <=2:
        driver.find_element_by_xpath("//span[text()='Show More']").click()
        print("Clicked Show more {} time".format(showmore_count))
        showmore_count+=1
        time.sleep(5)
        try: 
            while True:
                reviews = driver.find_elements_by_xpath("//div[@jsname='fk8dgd']/div")
               
                driver.execute_script("arguments[0].scrollIntoView(true);", reviews[j])
                driver.execute_script("window.scrollBy(0,-50)")
                print("{}: {}".format(j, reviews[j].find_element_by_xpath(".//span[@class='X43Kjb']").text))
                j += 1
        except:
            pass
except Exception as e:
    print(e)

driver.quit()

The output of this script will look something like the following:

1: Reviewer name1
2: Reviewer name2
3: Reviewer name3
...
520: Reviewer name520

Answer 4

This code is quite lengthy, but it will keep scrolling down as long as the showmore_count variable remains in place.

I have also included relevant comments to explain the code.

from selenium import webdriver
import time

driver = webdriver.Chrome(executable_path="path")
driver.maximize_window()
driver.implicitly_wait(10)
driver.get("https://play.google.com/store/apps/details?id=com.ucool.heroesarena&showAllReviews=true") # I had issues with the provided URL so I used this one instead.
time.sleep(5) # Wait for the page to fully load
j=0
showmore_count = 1 # This stops the script from continuous scrolling
try:
    while True:
        reviews = driver.find_elements_by_xpath("//div[@jsname='fk8dgd']/div")
        
        driver.execute_script("arguments[0].scrollIntoView(true);", reviews[j])
        driver.execute_script("window.scrollBy(0,-50)")
        print("{}: {}".format(j+1, reviews[j].find_element_by_xpath(".//span[@class='X43Kjb']").text)) # Display the reviewer's name
        j += 1
except IndexError: 
    while driver.find_element_by_xpath("//span[text()='Show More']").is_displayed() and showmore_count <=2:
        driver.find_element_by_xpath("//span[text()='Show More']").click()
        print("Clicked Show more {} time".format(showmore_count))
        showmore_count+=1
        time.sleep(5)
        try: 
            while True:
                reviews = driver.find_elements_by_xpath("//div[@jsname='fk8dgd']/div")
               
                driver.execute_script("arguments[0].scrollIntoView(true);", reviews[j])
                driver.execute_script("window.scrollBy(0,-50)")
                print("{}: {}".format(j, reviews[j].find_element_by_xpath(".//span[@class='X43Kjb']").text))
                j += 1
        except:
            pass
except Exception as e:
    print(e)

driver.quit()

The output of this script will look something like the following:

1: Reviewer name1
2: Reviewer name2
3: Reviewer name3
...
520: Reviewer name520

The challenge of clicking the "ShowMore" button in an infinitely scrolling page is a common issue when using Selenium with Python

Answer №1

Answer №2

Similar questions

In Java with Selenium, I used the command list.clear() to empty the list that was previously storing the result I was trying to keep in a

Having difficulty capturing Selenium browser errors within a try/except block

How to deselect a single option in a dropdown menu with Python Selenium WebDriver

Retrieve an element using Selenium's find_element

Tips for troubleshooting a testcase in testng when utilizing a framework that includes page factory

Converting HDF5 data from MATLAB to a Pandas DataFrame using Python

Utilizing decorator functions to yield the return of a wrapper

Error in main thread: The program was unable to find the specified element

What is the most suitable data structure for storing an array of dictionaries?

Using Python to iterate through various pages on an API and extracting specific data before printing it

Issue with saving GIF files when using ImageIO/PIL

Utilizing Django fixtures: Importing HTML content as JSON into a Text Field

Issue encountered while attempting to import vincent library into Python 3

I require assistance with figuring out how to trigger a scroll up event

Can someone show me how to find and select the start button with selenium in python coding?

What is the best location to place the "get current URL" method while implementing page objects in Selenium?

employing a robotic tool to download a webpage under various file names

Learn the process of extracting keys and values from a response in Python and then adding them to a list

Verifying the date displayed using Java in Selenium

An effective method for removing various stems from the end of a word is to utilize slicing techniques