What is the best way to locate an element on a website that consistently changes its label ID buried deep within nested DIVs?

I'm facing an issue with my website tracking system, where I am trying to extract shipment details using Selenium in Python and save them to an Excel file. However, I am encountering difficulties with getting Selenium to function properly. It keeps showing me errors indicating that the element cannot be located. Furthermore, these elements are buried deep within nested DIV structures.

https://i.stack.imgur.com/0y5HA.png

Initially, I attempted the following:

driver.find_element(By.XPATH,"//*[@id='trackItNowForm:j_idt529:0:j_idt535']").text

I quickly realized that the ID portion changes constantly (e.g., from 'idt529:0:j_idt535' to 519, 509, etc.), making it challenging due to lack of a consistent structure. As a result, using By.ID is not a viable solution.

Currently, I find myself at a loss for what steps to take next. It's possible that I haven't correctly input the syntax required. Since I am new to Selenium, many aspects of it still confuse me (such as writing XPATH expressions). Therefore, any guidance provided would be greatly appreciated.

pythonseleniumweb-scrapingxpathcss-selectors

Answer №1

To find that specific element, you can use its TrackingNumber class name in combination with other known attribute values.
Give this a try:

driver.find_element(By.XPATH,"//label[contains(@id,'trackItNowForm') and(contains(@class,'TrackingNumber'))]").text

You may need to include a delay to ensure the element is visible. In such cases, using WebDriverWait expected conditions is the best approach:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

wait = WebDriverWait(browser, 20)

wait.until(EC.visibility_of_element_located((By.XPATH, "//label[contains(@id,'trackItNowForm') and(contains(@class,'TrackingNumber'))]"))).text

Answer №2

If you're looking to pinpoint a specific element using CSS selector, consider the following options:

Option 1:

driver.find_element(By.CSS_SELECTOR,"[id^='trackItNowForm'][id*='id']").text

Option 2:

driver.find_element(By.CSS_SELECTOR,"[id^='trackItNowForm'][id*='id'][class*='TrackingNumber']").text

Here, ^ represents starts-with and * indicates contains.

In Option 1, it validates that the id attribute must start with 'trackItNowForm' and should also contain 'id'

Answer №3

Understanding how xpath functions is crucial.
Take the code snippet you provided as an example:

driver.find_element(By.XPATH,"//div[@class='col-sm-10']/label").text

This line of code searches for the first instance of a label within a div element with the class attribute set to col-sm-10, and then retrieves its text.

Helpful resources:

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Exploring ways to programmatically click on a button located beneath a text using the Selenium library in Python

I am interested in creating a bot that can automatically like comments on a web page. Each comment has a like and dislike button attached to it. The xPath for the comment is: //*[@id="commentText-40538697"]/span The xPath for the like button is: //*[@id ...

pythonhtmlseleniumxpathweb-scraping

Python3 requires a bytes-like object to be encoded in Base64

Here is the code snippet: _cmd = "|| echo " + base64.b64encode(args.cmd) + "|base64 -d|bash" p.update({"form_284": _cmd}) The error encountered is as follows: Traceback (most recent call last): File "openemr_rce.py& ...

python

Creating Load Duration Curves Using Matplotlib

Currently, I am working on creating a load duration curve using matplotlib. For those unfamiliar, a load duration curve is essentially an organized list of numbers in descending order on the y-axis with an index typically representing time in hours on the ...

pythonmatplotlibgraphstatistics

Unable to insert object into space in pymunk framework

Currently, I am working on a program that involves using pymunk and pygame together. The program itself features a car driving with obstacles in its path. My main goal is to use pymunk to detect collisions between the car and any obstacle present. If a col ...

pythonpymunk

Choose a text file from a given list and use Selenium with Python to extract and read all of

I'm looking to randomly select one file from a list and read all text. Does anyone have any suggestions or references on how to do this? fcaption = open('caption-1.txt','caption-2.txt','caption-3.txt','caption-4.txt& ...

pythonselenium

Using Python and Selenium, you can locate an element inside another element

I've got the xpath for an element: /html/body/div[1]/div/div[2]/div[3]/div/div/div/div[2]/div[1]/div[2]/div[6]/div/div/div/div[1] Inside that div, there's another element: /html/body/div[1]/div/div[2]/div[3]/div/div/div/div[2]/div[1]/div[2]/div[ ...

pythonhtmlseleniumselenium-webdriverselenium-chromedriver

Unable to Retrieve Information from Section Element Using Selenium

I am currently faced with the challenge of counting and interacting with multiple buttons on a web page. The obstacle is that these buttons are nested within an iframe, some generic layers in the form of div elements, and a region layer denoted by section ...

pythonseleniumselenium-webdriverwebdriver

making adjacent violin plots in seaborn

I've been experimenting with creating side-by-side violin plots using Matplotlib and Seaborn, but I'm facing challenges in getting them to display correctly. Instead of being superimposed, I want to compare the average Phast scores for different ...

pythonmatplotlibplotseaborn

Ways to stop flask from automatically opening links in a new tab

I created a header that contains different links to various sections of my web application: <div class="collapse navbar-collapse" id="navbarSupportedContent"> <!-- Left --> <ul class="navbar-nav mr-auto"> <li ...

pythonhtmlflask

Utilizing the Correct (secondary) CTRL Key in Sikuli

Seeking a solution for automating a unique mainframe application that utilizes the Right CTRL and Right ENTER keys. Is it possible to achieve this using Sikuli without the ability to install additional remapper software such as autoHotKey? ...

pythonsikulisikuli-script

Using Python's Selenium to capture elements that are loaded lazily

Struggling to click a button using Selenium, but encountering an error: NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"AGREE"} This is the button I am attempting to click. https://i.stack.i ...

pythonseleniumweb-scraping

Is it feasible for the new window in selenium to be automatically focused as the primary window on your screen?

As a newcomer to selenium, I have been exploring various solutions on stack overflow in search of an answer. Some of the threads I've looked at include: Get focus on the new window in Selenium Webdriver and Python and Make sure that browser opened ...

pythonselenium

Issue with passing arguments for entry_points in Setuptools

I have developed a Python script and I am using setuptools Once the installation is complete, I run: $ megazord -i input -d database -v xx-xx -w yy-yy Just like how I would run it as ./like_this However, I encounter the following error: Traceback (mos ...

pythonsetuptoolsdistutils

Utilizing Visual Basic and C# in Selenium to manipulate CSS selectors with variables

Just beginning my journey with C#, and I've been piecing together code with help from forums like this. However, I've hit a roadblock when it comes to using a variable that's been giving me some trouble. List<IWebElement> elementList ...

c#seleniumcss-selectors

Setting the ItemFlag in a QTreeView or QListView: A Step-by-Step Guide

UPDATE2: Implementing a QTreeView using QFileSystemModel. Is there a way to set specific flags on a tree item? Especially in a subclass of QFileSystemModel? class CustomTreeModel(QFileSystemModel): def __init__(self, parent=None): super(Custo ...

pythonpyqt5pyside2

What is the best method for defining specific kwargs for a URL outside of a loop?

I have added some links in my template. <ul> {% for cat in cats %} <li><a href="{% url 'c_index' cat.slug %}">{{ cat.name }}</a> {% endear %} </ul> <ul> {% for type in types %} < ...

pythondjango

Python script unable to detect Chromedriver

Having trouble running a selenium script and encountering issues with detecting the chromedriver. Can anyone assist me with this problem? When attempting to run the script, an error message indicates that the chromedriver is not present in the specified pa ...

pythonseleniumselenium-chromedriverundetected-chromedriver

Dynamic audio blending with Python in real time

I am experimenting with using scapy and pyaudio to generate a unique sound every time a packet is transmitted or received. The pitch of the sound is determined by the IP address of the sender. sniff(prn = makeSound) In this snippet of code, the function ...

pythonscapypyaudio

Python newbie feeling completely baffled, receiving an error when trying to implicitly convert an integer object to a string

Recently, I attempted to create a simple code that converts your age into lion years (essentially multiplying your age by 5). However, I encountered an issue where I couldn't get it to accept the age as an integer input and print the "lionage" value c ...

pythonpython-3.x

Incorporating a delete button onto an image using JavaScript

I am currently developing a BlogApp and facing difficulty in attempting to include a button on an image. The image is being retrieved from the Django database in JavaScript (HTML). My goal is to have a clickable button overlaid on the image. views.py def ...

javascriptpythonhtmldjangodjango-views