What is preventing me from utilizing FindElementByXPath on the table from this website?

I am currently using VBA for web scraping with Selenium and ChromeDriver.

After logging into a website at , I encountered a table on the main page that contains a lot of information.

By inspecting the element, I found the table object highlighted in this image: print of the found element, the table is highlighted

When trying to copy the XPath, it gave me "//[@id="mCSB_67_container"]/table”

So, I set up the command like this:

Set table = driver.FindElementByXPath("//[@id="mCSB_67_container"]/table") 'Selects the table using Java

If table Is Nothing Then
    MsgBox "Not found"
Else
    table.AsTable().ToExcel (ActiveSheet.Cells(1, 1)) 'Pastes the table into the Excel worksheet where the button is located
End If

Unfortunately, nothing was printed on Excel. I have successfully scraped data from similar websites like solarweb.com, but those tables had specific IDs and classes unlike the one I'm dealing with now.

I understand if my question isn't very clear, please leave a comment below if you need more details and I will gladly edit my post.

vbaseleniumselenium-chromedriver

Answer №1

Uncover the mystery behind the dynamic ids hidden beneath the surface. Your mission is to uncover stable selectors that remain static despite the changes. Utilize these selectors to navigate through the DOM hierarchy until you reach your desired table. From the snippet provided, there doesn't seem to be a glaringly obvious unique id that remains constant. However, if there is only one visible table, you can simply locate it using the following code:

driver.findElement(By.cssSelector("table"));

Answer №2

mCSB_67_container appears to be a dynamically generated id, meaning it is not static.
Without seeing the actual webpage with developer tools, it is uncertain. You could attempt using this alternative XPath:

Set tabela = driver.FindElementByXPath("//div[contains(@class,'mCustomScrollBox']//table")

Answer №3

At times, finding the element can be challenging, especially if it is a static table with dynamically changing values on the page. It would be more effective to simply right click and copy the xpath provided by Chrome instead of writing complex looping logic.

view image description here

Answer №4

The attempt to target the table element in the question proved unsuccessful due to its dynamic nature, making it impossible to reference based on static attributes such as "id" and "class". Many thanks to @Prophet and @David M for guiding me towards finding a stable reference point within the site's interface, allowing me to successfully target the desired table.

I identified the division with an ID of "plantList" as the static element I could use for referencing. Referencing this element helped me navigate through other elements until I reached the table. Here is a screenshot illustrating how I located this critical element:

The red arrow indicates the table element, while the green arrow points to the static element used for referencing

Subsequently, I proceeded to locate and extract the table using the following code snippet:

Set canadian = driver.FindElementByXPath("//*[@id=""plantList""]/div[3]/div[1]/div[1]/div[2]/div[1]/div[1]/table") 'Selects the table via Java

If canadian Is Nothing Then
    MsgBox "Not Found"
Else
    canadian.AsTable().ToExcel (ActiveSheet.Cells(1, 1)) 'Pastes the table into the worksheet where the button is located
End If

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

An error occurred when trying to send an HTTP request to the remote WebDriver server for a URL, resulting in a null response being thrown

https://i.stack.imgur.com/s4RyZ.png Encountering an error when attempting to use EdgeDriver with Selenium. While ChromeDriver runs smoothly, I am required to use Edge for this project. Some sources suggest that the WebDriver version needs to match the Win ...

c#seleniumselenium-webdrivermicrosoft-edgeselenium-edgedriver

Extracting the number of likes from each post on a specific profile using Python scrapy

Here is the code snippet that I am currently working with: #IMPORT THESE PACKAGES import requests import selenium from selenium import webdriver import pandas as pd #OPTIONAL PACKAGE, BUY MAYBE NEEDED from webdriver_manager.chrome import ChromeDriverManage ...

pythonhtmlcssseleniumweb-scraping

I encountered a webdriver error while using Selenium in Google Colab

I attempted to find the solutions through Google, however, they proved to be ineffective. I am feeling quite disheartened at the moment. ...

pythonseleniumgoogle-colaboratory

Creating a user-defined function that returns an empty dataframe prior to executing a for loop

Although I have come across similar questions to mine multiple times, I have thoroughly reviewed them and still cannot solve my own code. Therefore, I am hoping someone might have the answer. The issue lies in a for loop inside a user-defined function tha ...

pythonpandasseleniumfor-loop

What steps are involved in including the Gradle Kotlin dependency for the compile group 'org.seleniumhq.selenium'?

Can anyone provide the syntax for adding Selenium as a dependency using the Gradle Kotlin DSL? Error: thufir@dur:~/NetBeansProjects/HelloKotlinWorld$ thufir@dur:~/NetBeansProjects/HelloKotlinWorld$ gradle clean > Configure project : e: /home/thufir/ ...

seleniumgradlekotlindependenciesgradle-kotlin-dsl

Acquiring a login session for future login bypass in Python using Chromedriver

The code I've written includes a curl request to obtain a login session token for a specific website. Once the token is generated, I aim to save the session details so that I can utilize chromedriver to access the website's dashboard page withou ...

pythonseleniumselenium-webdriverselenium-chromedriver

Is it possible to bypass detection using Python Selenium?

Attempting to extract data from this webpage - But facing detection as soon as the request is made, any workaround for this issue? Previously tried using Request but encountered detection. *Scrapy cannot be utilized in this particular project. Struggling ...

pythonseleniumgoogle-chromeselenium-webdriverselenium-chromedriver

Interacting with an iframe element using Selenium in Python

I have a webpage with an iframe embedded, and I'm using Selenium for test automation: <iframe class="wysihtml5-sandbox" security="restricted" allowtransparency="true" frameborder="0" width="0" height="0" marginwidth="0" marginheight="0" style="dis ...

javascriptpythonhtmlseleniumiframe

The partnership between Selenium and WhitePages has created an innovative solution

I am currently attempting to register on . The registration process requires entering my first name, last name, email address, and password. For the first name field, I attempted to locate the element by CSS using the syntax "input#name_fname", however, I ...

cssseleniumselenium-webdriver

Tips for handling select_input in Rails Cucumber step definitions:

I'm currently integrating cucumber with selenium chrome driver into my project. Currently, I am developing the scenario for creating a new post using CRUD operations. Within the step definition, I need to populate the form with both input and select ...

seleniumcucumberruby-on-rails-5

I am encountering an issue while trying to submit a form using the update button in Selenium Java. Could you please examine the

I attempted using classname, xpath with type and name but encountered no success. <div class="login-btn"> <button type="submit" class="hvr-shutter-out-horizontal">Update</button> </div> ...

seleniumselenium-webdriver

What is causing the capability error in my Python code that is prompting me to add a capability?

Please help me troubleshoot the error in my code. I am encountering an issue and need assistance, but unfortunately I am unable to include any images at this time due to being new here. The error occurs when I execute the code. [1]: https://i.stack.imgur ...

pythonseleniumautomationbots

Error encountered while executing test with cucumber

Having a bit of an issue that may seem trivial to some... I'm experiencing difficulty running my tests on the cucumber framework. Encountering the following error message: cucumber.runtime.CucumberException: Classes annotated with @RunWith(Cucumbe ...

javaseleniumautomationautomated-testscucumber

Exploring user names on a webpage with Python and Selenium: A step-by-step guide

I am a beginner in coding and looking for guidance to create a bot that can utilize the nextdoor website to search for posts and send promotions to users via their messaging system. I have successfully completed the login, search, and navigation to the Pos ...

pythonlistseleniumxpath

Running two different wdio.config.js files consecutively

Is it possible to run two wdio.config.js files with different configurations, one after another? Here is how the first configuration file is defined in the code: const { join } = require('path'); require('@babel/register') exports.co ...

javascripttypescriptseleniumappiumwebdriver-io

Extracting information from a database using Pandas and Beautiful Soup following authentication with Selenium

I'm currently in the process of scraping data from a paginated table that can only be accessed after logging into a user account. To accomplish this, I've opted to utilize Selenium for logging in and then plan to transfer the data into a Pandas D ...

pythonpandasseleniumselenium-webdriverbeautifulsoup

TestNG is experiencing issues with the order of test classes, causing them to run out of order and skip between

Encountering a problem with TestNG Selenium Webdriver 2.0 in Java has led to some perplexing behavior during debugging. Several class files contain groups of tests, each starting with the initialization of global variables for all tests within the class. ...

javaseleniumwebdrivertestng

Automating the management of alerts during page load with Selenium in Python

I'm looking to achieve the following: Clicking on the OK button inside an Alert using Selenium IDE However, the solutions provided in the linked question do not seem to work when alerts appear upon loading. How can I successfully click on the OK butt ...

pythonselenium

What is the best way for Cucumber to move on to the next annotation only after ensuring all async requests from the previous one have finished processing?

I am looking to set up a basic test using Selenium and Cucumber for logging into my web application and confirming that the main page is displayed correctly. Currently, all three tests are returning true even before the page is fully loaded. The issue ar ...

javascriptseleniumcross-browsercucumberchai

How can I resolve the issue of encountering the "Modal dialog present: If you navigate away from this page, any unsaved changes will be lost" message while using Internet Explorer with

Here's the code snippet I'm working with. While it successfully removes the alert box, it throws an error in the console: Exception in thread "main" org.openqa.selenium.UnhandledAlertException: Modal dialog present: If you leave this page, any c ...

javascriptjavaselenium