Finding an element with Python and Selenium by accessing innerHTML

I am currently navigating my way through Selenium and attempting to create my inaugural script using the Python package.

My setup includes:

  • Operating System: Windows 10
  • Python Version: 3.10.5
  • Selenium Version: 4.3.0

Up until now, I have managed to accomplish all tasks utilizing various selectors like ID, name, XPATH, and so on.

However, I have hit a roadblock where I must locate a specific element based on its innerHTML.

The challenge lies in identifying an element with the innerHTML-value of "Changed" within the provided HTML snippet below.

The primary hurdle is that this element lacks a unique identifier such as ID or name, and there are multiple occurrences of "dlx-treeview-node" objects/elements.

Another obstacle is that XPATH is unsuitable due to the element's varying position on the webpage, which results in different numbers of "dlx-treeview-node"-elements, leading to incorrect selections.

To validate that it's indeed innerHTML and not innerText, I successfully extracted the name using the given XPATH, "get_attribute," and console output. This information changes based on the webpage's location.

I would greatly appreciate any guidance offered to overcome this predicament while enhancing my knowledge of leveraging Selenium with Python.

Here is the section of code I'm working with:

select_filter_name = wait.until(EC.element_to_be_clickable((By.XPATH, "/html/body/div/app-root/dlx-select-filter-attribute-dialog/dlx-dialog-window/div/div[2]/div/div/div[5]/div/div/dlx-view-column-selector-component/div[1]/dlx-treeview/div/dlx-treeview-nodes/div/dlx-treeview-nodes/div/dlx-treeview-node[16]/div/div/div/div[2]/div/dlx-text-truncater/div")))
filter_name = select_filter_name.get_attribute("innerHTML")
print(filter_name)

Provided HTML excerpt:

<dlx-treeview-node _nghost-nrk-c188="" class="ng-star-inserted">
  <div _ngcontent-nrk-c188="" dlx-droppable="" dlx-draggable="" dlx-file-drop="" class="d-flex flex-column position-relative dlx-hover on-hover-show-expandable-menu bg-control-active bg-control-hover">
    <div _ngcontent-nrk-c188="" class="d-flex flex-row ml-2">
      <div _ngcontent-nrk-c188="" class="d-flex flex-row text-nowrap expand-horizontal" style="padding-left: 15px;">
        <!---->
        <div _ngcontent-nrk-c188="" class="d-flex align-self-center ng-star-inserted" style="min-width: 16px; margin-left: 3px;">
          <!---->
        </div>
        <!---->
        <div _ngcontent-nrk-c188="" class="d-flex flex-1 flex-no-overflow-x" style="padding: 3.5px 0px;">
          <div class="d-flex flex-row justify-content-start flex-no-overflow-x align-items-center expand-horizontal ng-star-inserted">
            <!---->
            <dlx-text-truncater class="overflow-hidden d-flex flex-no-overflow-x ng-star-inserted">
              <div class="text-truncate expand-horizontal ng-star-inserted">Changed</div>
              <!---->
              <!---->
            </dlx-text-truncater>
            <!---->
          </div>
          <!---->
          <!---->
          <!---->
        </div>
      </div>
      <!---->
      <!---->
    </div>
  </div>
  <!---->
  <dlx-attachment-content _ngcontent-nrk-c188="">
    <div style="position: fixed; z-index: 10001; left: -10000px; top: -10000px; pointer-events: auto;">
      <!---->
      <!---->
    </div>
  </dlx-attachment-content>
</dlx-treeview-node>

Edit-1:

NOTE: Unsure if my HTML terminologies are accurate, corrections are welcome.

A follow-up question has surfaced:

How can I search for the specified text solely within the "dlx-treeview-node" elements (approximately 100)? Essentially targeting the "children" within these nodes specifically.

This query arises from discovering other elements containing the same text elsewhere.

Edit-2/Solution:

Prior to receiving responses, I managed to devise an answer myself – documenting it here for potential assistance to others. The response marked as "answer" was nearest to what I required.

The final code came together as follows (initially searching nodes then examining children for the targeted innerHTML):

select_filter_name = wait.until(EC.element_to_be_clickable((By.XPATH, "//dlx-treeview-node[.//div[text()='Changed']]")))

Answer №1

When assuming the innerText of the <div> element to be a unique text in the HTML DOM for locating the element with innerHTML as Modified, you have the option to use any of the following xpath based locating techniques:

  • Using xpath and text():

    element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[text()='Modified']")))
    
  • Using xpath and contains():

    element = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//div[contains(., 'Modified')]")))
    

Answer №2

Simply run this code snippet on your webpage to generate an array of all elements that are div with the value of Changed.

# Defining the XPath Function for later use
driver.execute_script("function getXPathOfElement(elt) {var path = "";for (; elt && elt.nodeType == 1; elt = elt.parentNode) { idx = getElementIdx(elt); xname = elt.tagName; if (idx > 1) xname += "[" + idx + "]"; path = "/" + xname + path;} return path;}")

# Retrieve XPaths for all nodes that are divs with the text 'changed'
xpaths = driver.execute_script("return Array.from(document.querySelectorAll(\"div\")).find(el => el.textContent.includes('Changed')).map((node)=>{ return getXPathOfElement(node)});');

Description

  • The first execute function adds a JavaScript function called getXPathOfElement to the DOM. This function takes an HTML node element as input and returns the XPath string for that node.

  • The second execute function retrieves all div elements with the text 'Changed'. It then iterates through each element, providing an array of strings where each string represents an XPath derived using the getXPathOfElement function on each node.

The JavaScript code is straightforward and safe to use.

Tips and Suggestions

  • Verify if the length of xpaths is greater than or equal to 1
  • Access specific indexes in xpaths such as xpaths[0], or utilize loops to implement changes
  • You now have an XPath that can be utilized as a regular selector

Best of luck!

Update 1

Learn more about execute_script() which synchronously executes JavaScript in the current window/frame.

For additional information, visit this link

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Prevent ChromeDriver from generating backtrace logs in cases of Selenium test failures

I am facing an issue with excessive logging by the ChromeDriver Whenever a Selenium test fails, in addition to the regular error logs in the terminal, I receive backtrace messages like: Stacktrace: Backtrace: Ordinal0 [0x00B378B3+2193587] Ordinal0 ...

Python script to read and write JSON files on the Google Cloud Storage platform

Recently, I came across a fascinating scenario involving a JSON file stored in a Cloud Storage bucket. Is there an effective approach to read and potentially modify the data within using Python? @app.route("/myform/", methods=('GET', 'POST& ...

I'm attempting to retrieve text from an <a> tag with a ::before selector using selenium in python

<ul class="ipc-inline-list ipc-inline-list--show-dividers ipc-inline-list--inline ipc-metadata-list-item__list-content baseAlt" role="presentation"> <li role="presentation" class="ipc-inline-list__item"><a class="ipc-metadata ...

The WebDriver URL Timeout feature appears to be malfunctioning

In the test class of my Java code, I have implemented the following snippet which is supposed to set a timeout of 5 seconds for page loading. However, it doesn't seem to work as expected. Even when tested on slow connections, the page load waits indef ...

Encountering issues when using a proxy with Selenium Webdriver in Python

After experimenting with a Selenium Python script for web scraping, I encountered an issue once my IP was detected. To overcome this, I decided to explore using a proxy server. I attempted two different methods to implement this. First approach: PROXY = & ...

To construct a well-balanced Binary Search Tree in Python, simply provide a sorted list as input to the sorting function

Looking to print the preorder of a Balanced Binary Search Tree based on a sorted list in ascending order. The method must be a part of the TreeNode class and accept the sorted list as an argument rather than individual elements. The sorted_array_to_bst() ...

The file data/mscoco_label_map.pbtxt cannot be found

Seeking Assistance! Thank You in Advance for your help. I am currently working on creating an object detector using Python in Google Colab. I'm facing some issues and would greatly appreciate your guidance. Could it be a module version error or perha ...

What is the process for providing arguments to a class at the time of object instantiation?

I am in the process of developing a graphical user interface that features two buttons, "Choose Input File" and "Execute". Upon clicking on "Open Input File", users have the ability to select a file from their computer containing URLs in a single column. S ...

Utilizing pandas and numpy to import a CSV file and transform it from a two-dimensional vector to a one-dimensional vector while excluding any diagonal elements

Here is the content of my csv file: 0 |0.1|0.2|0.4| 0.1|0 |0.5|0.6| 0.2|0.5|0 |0.9| 0.4|0.6|0.9|0 | I am attempting to read it row by row, excluding the diagonal values, and converting it into a single long column: 0.1 0.2 0.4 0.1 0.5 0.6 0.2 0.5 0.9 ...

Running Selenium WebDriver instances in parallel can lead to interruptions in other tests when attempting to close one WebDriver

Having some difficulty with managing RemoteWebDriver sessions in parallel. This issue arises when multiple instances of RemoteWebDriver are running simultaneously and one instance calls the close method. Here's an example scenario: Test A begins an ...

Guide on clicking the SIGN IN button with Selenium and C#

Currently, I am working on creating a test scenario where users will go to a specific URL, input their login details, and then proceed to sign in by clicking a button. Everything is functioning correctly except for the button click action. I have attempted ...

Having trouble selecting a dynamically changing div element in Selenium with Java using a click action

Every time I navigate to www.reddit.com and enter a query in the Search field, then press enter to go to the first valid link within a subreddit, sorting options are presented. By default, it is set to BEST but I wish to change it to TOP. My test class cod ...

Executing tests with Selenium webdriver when the browser is minimized can be done by adjusting the browser window

I encountered an issue while running this program. It performs well with codes initially, but when I tried to minimize the browser, an error popped up indicating that the program couldn't find the button. Is there a solution available to address this ...

What is the reason that a combination of two types in a list does not result in a list of combinations of these two types?

I encountered some difficulties with using mypy in scenarios where List and Union types are combined. Although I have found the solution, I wanted to share my discoveries here. The core question that arose was: why does a list of a union of two types not ...

Steps to select an option from a drop-down menu that does not have an ID assigned

How can I interact with a drop-down element that appears after clicking on an item? <div class="field loan-selection"> <label class="field__body"> <div class="field__label">Nettokreditbetrag <!-- -->&nbs ...

Setting values for identical rows in a pandas dataframe

Is there a way to identify duplicates in a dataframe based on the 'Customer' column and replace the 'Score' with NA? Customer Score 3a62-4799 500 3a62-4799 NA 3a62-1234 450 3a62-1234 NA I attempted the following approach: X[& ...

How to retrieve data as a dictionary instead of a tuple using MySQL's fetchall() method

I have a table with the columns id, col1, col2, and col3. When I ran the query, this is how it was executed: cur.execute(query) row = cur.fetchall() Is there a way to store the data in the variable row inside a dictionary so that I can pass this result t ...

Error: Unable to locate elements due to a null XPath expression. This issue arises while attempting to input keys using a Method

Encountering an error when using Sendkeys in methods. public static void inputTask(String task) throws Exception { // Entering task name GUIFunctions.typeTxtboxValue(driver,By.xpath(ObjRepoProp.getProperty("inputTaskName_XPATH")),task); Thread ...

The 'selenium.common' module could not be found

I encountered an issue while attempting to execute this code on a Windows 64-bit system. from selenium.common.exceptions import NoSuchElementException, ElementClickInterceptedException from selenium import webdriver The specific error message I received w ...

Why is a cast required to the generic type in this particular method, whereas similar methods do not require it?

I encountered an issue with a method that requires a generic return type, resulting in the following error: Description Resource Path Location Type Type mismatch: cannot convert from PageTypeOne to P SecuredPage.java To resolve this error, I ha ...