Having trouble locating the table element within a div element using Selenium in Python

My goal is to access values from a table within a div element using selenium.

Although I am able to see the table when inspecting the element, I cannot locate it in the source code while attempting to access it through selenium.

When Inspecting:

https://i.stack.imgur.com/yW8By.jpg

The following is the code snippet I have used:

totals = driver.find_element_by_id("totals") # gets the totals div element. Only one is there
labels = totals.find_elements_by_class_name("hasLabel") # should get the table rows, but it was empty
print(len(labels)) # returned 0

However, the results were empty as viewed in the source code. As part of automation, I need to export these values into a file.

Are there any alternative methods I can utilize to extract these values?

Answer №1

Your observation appears to be almost perfect. Based on the HTML code in the During Inspect snapshot you shared, it seems that the element with the text Taxes Due is highlighted.


However, in the While viewing the source code snapshot obtained using Selenium, it is possible that the Page Source was retrieved before all child elements were fully rendered within the DOM Tree:

This could explain why the child elements within

<div id='totals'></div>
are not present in the Page Source.


To accurately determine the total number of <tr> elements, providing the relevant HTML code in text format would have been beneficial for crafting a comprehensive answer. Nevertheless, one way to achieve this is by implementing an WebDriverWait for

visibility_of_all_elements_located()
using any of the suggested Locator Strategies:

  • Utilizing XPATH:

    print(len(WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table/tbody//tr[@class='hasLabel'][./td[@class='label']]")))))
    
  • Note: Remember to include the necessary imports:

    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support import expected_conditions as EC
    

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Dealing with Errors in Selenium Using Python 2.7

After transitioning from Python 3.5 to Python 2.7 due to py2exe compatibility issues, I encountered an error in my script. Can someone help me resolve this problem? Any assistance would be greatly appreciated. from selenium import webdriver import time ...

Path search for the location of the Selenium driver

I'm currently setting up selenium tests in nCrunch with the output of the tests going to a temporary folder specific to nCrunch. When I attempt to create the driver (new PhantomJSDriver()), I encounter the following error: OpenQA.Selenium.DriverSe ...

What is the process for setting up SSL for Angular e2e testing?

I'm working on an Angular sample project that includes end-to-end tests focusing on OAuth2 and OIDC flows. I've noticed that the behavior of browsers varies when SSL/TLS is enabled or disabled. To ensure consistency, I would like to run my end-to ...

Navigating infinite scroll pages with scrapy and selenium: a comprehensive guide

Struggling with using scrapy +selenium to extract data from a webpage that loads content dynamically as we scroll down? Take a look at the code snippet below where I encounter an issue with getting the page source and end up stuck in a loop. import scrap ...

"Flask gets stuck just before returning jsonify, resulting in a lack of output (error message

As a new developer, I have created a page that checks if the user is logged in on load. If the user is logged in, it replaces the login forms with data from the server. However, if the user is not logged in, it displays the login forms and waits for the us ...

Ensuring Page Load After Form Submission with Python Selenium

Currently, I am utilizing Python3 along with Selenium in Firefox to successfully submit a form and then retrieve the resulting URL where the users land. Here is how I have implemented it: inputElement.send_keys(postnumber) inputElement.submit() time.slee ...

Having trouble loading a page with Selenium WebDriver in Python on a Windows system?

After switching from running Python 3.5 code on my Mac to a Windows computer, I encountered an issue where the code wouldn't work as expected. On the Mac, everything ran smoothly without any problems, but on Windows, I ended up with a blank page inste ...

I'm having trouble receiving any output after initiating My Scraper

I've set up a scraper to gather information on Product name, Cat No, Size and Price. However, when I run the script, it doesn't produce any output or errors. I'm utilizing Jupyter Notebook for this task but I'm uncertain if that's ...

The art of using explicit waits with Selenium WebDriver

I am currently working on developing automated tests using the selenium chrome driver. I am attempting to create a method that can be reused to wait for elements to appear, and then I want to call this method in other classes. Despite what seems like a sim ...

What is the best way to retrieve HTML headers in PHPUnit when utilizing Selenium2?

public function checkHeaders() { $url = $this->getUrl('http://www.example.com/index.php'); $headers = get_headers($url); var_dump($headers); } Upon running the above code, I encountered an error: get_headers(): Filename cannot be ...

Is it possible to execute Selenium tests from a compiled jar file using TestNG in Selenium Testing?

Typically, our Selenium tests are executed using the TestNG xml runner. Here is an example of the XML: <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd"> <suite name="P1_jobSeeker" parallel="cl ...

What is the best approach to take when there are no identifying attributes like ID or name available, and using driver.find_elements_by_xpath is not a viable option?

I need to extract the exact time of the next moonrise at 12:05 am from the top section of this page. Despite using xpath, which has worked for me in the past, I am encountering an issue with this particular data point that lacks a name or ID: nextMoonRise ...

Having trouble getting Selenium to work in Eclipse with Java and BDD for Chrome or Firefox?

I am encountering an issue while attempting to use Selenium. I am currently on version 3.0.0-beta2 - any suggestions? Below is the error message: java.lang.IllegalStateException: The path to the driver executable must be set by the webdriver.gecko.driver ...

Ways to eliminate space within a list?

I am currently engaged in a project where I am extracting data from Indeed search results. At the moment, when I print the data I find, it appears with a space before the semicolon. Currently, my data prints like this: "Item 1 ; Item 2" I wan ...

Creating a custom selenium/standalone-chrome image with a customizable port (specifically port 4444)

I've been considering how to specify a different port number as an argument when launching the selenium container, instead of the default port (4444). Typically, I use: docker run --shm-size=2G -d --net=host -e TZ=UTC -e SCREEN_WIDTH=1920 -e SCREEN_ ...

Extract dropdown menu options

OBJECTIVE The aim is to explore every possible combination from this website: WHAT I'M SEEKING ADVICE ON I need some guidance on how to proceed with the task. ISSUE AT HAND I am facing a problem where I can retrieve a list of options for the first ...

Python - recursive function to update variable values during each call

Currently, I am attempting to develop a function that utilizes the Monte Carlo approximation method to determine the value of pi based on a specified accuracy (number of decimal places). My approach involves comparing the estimated value with the true valu ...

Selenium: Detecting the presence of specific text

Is there a method to confirm the presence of specific text within a larger string? For example, if the text reads "Warning: X23Y07 has not been activated," and 'X23Y07' can vary, how can I verify that the message is displayed? Using Selenium.IsT ...

Tips for clicking on unique links using Selenium

I'm facing an issue with a dynamic link that does not have a specific name or ID, but I know it is wrapped inside a span with a certain class. I attempted to retrieve a list of WebElements containing all the spans with the "more" class to access the s ...

Generate a script file that will execute the testng.xml file and initiate the execution of the selenium scripts

In need of assistance in creating a .bat file to execute testng.xml, which contains all the tests that need to be run. Following this source, I have crafted the .bat file with the following content: java -cp "C:\Program Files\Selenium Execution ...