My Python script utilizing Selenium is failing to execute

Why is the loop not functioning correctly after opening the initial element based on the xpath? I keep receiving the following exception:

 raise exception_class(message, screen, stacktrace)
 selenium.common.exceptions.NoSuchElementException: Message: Unable to
 locate element:{"method":"xpath","selector":"//[@id='searchresults']/tbody/tr[2]/td[1]"}
 Stacktrace:
at FirefoxDriver.prototype.findElementInternal_ (file:///c:/users/home/appdata/local/temp/tmpeglp49/extensions/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="b7d1cfd3c5dec1d2c5f7d0d8d8d0dbd2d4d8d3d299d4d8da">[email protected]</a>/components/driver-component.js:10723)
at FirefoxDriver.prototype.findElement (file:///c:/users/home/appdata/local/temp/tmpeglp49/extensions/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="42243a26302b34273002252d2d252e27212d26276c212d2f">[email protected]</a>/components/driver-component.js:10732)
at DelayedCommand.prototype.executeInternal_/h (file:///c:/users/home/appdata/local/temp/tmpeglp49/extensions/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="62041a06100b14071022050d0d050e07010d06074c010d0f">[email protected]</a>/components/command-processor.js:12614)
at DelayedCommand.prototype.executeInternal_ (file:///c:/users/home/appdata/local/temp/tmpeglp49/extensions/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="15736d71677c63706755727a7a727970767a71703b767a78">[email protected]</a>/components/command-processor.js:12619)
at DelayedCommand.prototype.execute/< (file:///c:/users/home/appdata/local/temp/tmpeglp49/extensions/<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="ec8a94889e859a899eac8b83838b80898f838889c28f8381">[email protected]</a>/components/command-processor.js:12561)

Code:

from selenium import webdriver
from texttable import len
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
driver=webdriver.Firefox()
driver.get('https://jobs.ericsson.com/search/')
driver.maximize_window()
driver.find_element_by_css_selector('[type="text"][id="keywordsearch-q"]').send_keys('Python')
driver.find_element_by_css_selector('[class="btn"][type="submit"]').click()
i=len("//*[@id='searchresults']/tbody/tr/td")
for j in range(1,i+1):
   driver.find_element_by_xpath("//*[@id='searchresults']/tbody/tr[%d]/td[1]"%j).click()
   print driver.find_element_by_id("job-title").text
   driver.back()
   continue

Question 2: Why does the list length display as 12 when there are only 5 'href' elements present?

from selenium import webdriver
from texttable import len
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
driver=webdriver.Firefox()
driver.delete_all_cookies()
driver.get('https://jobs.ericsson.com/search/')
driver.maximize_window()
driver.find_element_by_css_selector('[type="text"][id="keywordsearch-q"]').send_keys('Python')
driver.find_element_by_css_selector('[class="btn"][type="submit"]').click()
#currenturl = driver.current_url
pages=driver.find_elements_by_css_selector('a[rel="nofollow"]')
print pages
print 'Its working'
pages1=[]
for page1 in pages:
  pages1.append(page1.get_attribute('href'))
  print int(len(pages1))

Question 3: How do I retrieve the elements within HTML tags?

a. How can I extract the values "1-25" and "104" separately using the 'b' tag?

Please refer to the URL: (result section displayed at the bottom of the page)

<div class="paginationShell clearfix" lang="en_US" xml:lang="en_US">
<div class="pagination well well-small">
<span class="pagination-label-row">
<span class="paginationLabel">
 Results 
 <b>1 – 25</b>
   of 
 <b>104</b>
 </span>

b. How can I extract the Job ID from the HTML?

 <div class="job">
<span itemprop="description">
<b>Req ID:</b>
 128378
<br/>
<br/>
pythonpython-2.7seleniumselenium-webdriver

Answer №1

Here are the steps to follow:

for position in range(len(driver.find_elements_by_class_name('jobTitle-link'))):
    driver.implicitly_wait(5)
    driver.find_elements_by_class_name('jobTitle-link')[position].click()
    print(driver.find_element_by_id("job-title").text)
    driver.back()

Answer №2

In my personal experience, I have found that one common reason for encountering this error is when a webpage has not fully loaded yet. One possible solution that often works for me is to introduce a small delay using time.sleep(1) before attempting to locate the element.

import time

# Your other code goes here

for j in range(1,i+1):
   time.sleep(1)
   driver.find_element_by_xpath("//*[@id='searchresults']/tbody/tr[%d]/td[1]"%j).click()
   print driver.find_element_by_id("job-title").text
   driver.back()
   continue

Answer №3

Implementing a different approach that avoids clicking on each link individually but rather storing the URLs in a list to navigate to them:

from selenium import webdriver
from texttable import len
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait

driver=webdriver.Firefox()
driver.get('https://jobs.ericsson.com/search/')
driver.maximize_window()
driver.find_element_by_css_selector('[type="text"][id="keywordsearch-q"]').send_keys('Python')
driver.find_element_by_css_selector('[class="btn"][type="submit"]').click()

#Preserve the current url for further processing    
currenturl = driver.current_url
#Retrieve all elements by class name
jobs  = driver.find_elements_by_class_name('jobTitle-link')
jobslink = []
#Extract hyperlink urls from the job elements
#This method prevents the need to click each link and backtrack to the previous page
for job in jobs:
    jobslink.append(job.get_attribute('href'))
#Access each job page  
for job in jobslink:
    driver.get(job)
    print(driver.find_element_by_id("job-title").text)

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

What steps can I take to ensure that axios in NodeJS functions similarly to Python requests?

My goal is to acquire essential cookies for future use as a registered user of the website called toloka.to by sending a POST request to with all necessary body parameters. When using Python, everything functions smoothly, and I receive plain HTML where ...

pythonnode.jsweb-scrapingcookiesrequest

What is the method for retrieving text from a textarea that lacks a 'value' attribute?

Is there a way to retrieve text from a textarea element that does not have a value attribute set? <textarea name="comment_txt" class="comment_class"></textarea> If the textarea had a value attribute, I could simply use this code: driver.find ...

javaselenium

Verifying a parameter from a request URL using Selenium - A step-by-step guide

I found myself on the following webpage: http://localhost:8080/login?error=true How do I verify if the error variable contains the value true? https://i.stack.imgur.com/F5TkF.png This is what I have tried so far and failed: https://i.stack.imgur.com/3 ...

seleniumselenium-idekatalon-recorder

Looking for a reliable source to download a permanent link for the IE WebDriver Tool specifically built for Internet Explorer 11

I am interested in utilizing Microsoft's webdriver tool for conducting automated end-to-end tests on a web application. As a part of our build process, we have automated the downloading and installation of necessary webdriver components for different ...

internet-explorerseleniuminternet-explorer-11

Is it important to verify that all of my web application code is in UTF-8 format?

I am in the process of localizing my Django site, which currently only contains English language strings. I have not specified any file encoding options yet. Should I convert all my Python code to UTF-8 for this localization process? Is it considered a bes ...

pythondjangoutf-8file-encodings

An instructional guide on utilizing Python to extract the URLs of companies from a LinkedIn search

I'm having an issue extracting company profile URLs from a LinkedIn search as I keep getting a "not found" message. Despite my code running smoothly, the problem persists. Here is my code: import requests import csv import time import numpy from bs4 ...

pythonhtmlselenium-webdriverbeautifulsoupspring-social-linkedin

Verify the changing text within a Span tag with the use of Selenium in Java

Can anyone assist me in creating a logic to verify a dynamic text? The text within the tag below is constantly changing (to 6 distinct words), and I need to validate if those 6 unique words match the expected text. Is there a method to do this verification ...

javascriptjavaseleniumselenium-webdriverselenium-ide

Is there a way to programmatically simulate clicking on the "Cancel search" button?

I have a text input field with type "search". In order to perform UI testing, I need to simulate clicking on the "cancel search" button: The code for this specific input field is as follows: <input type="search" value="user"> Although the cancel b ...

javascripthtmlselenium

Initial attempt at sending data to an inactive socket is successful

When a client in a TCP client-server communication tries to send data to the server after the server has disconnected, the first attempt to use the send() function may succeed, but all subsequent attempts will fail with the error message [Errno 32] Broke ...

pythonsocketstcp

Exploring the Efficiency of Selenium for Testing Enterprise Web Applications

As a newcomer to Selenium Automation in a DevOPS environment, I am comfortable using the framework and automating some basic tests. However, I am unsure of the extent of functionality that can be automated in an Enterprise Web Application. Our test automat ...

selenium

How does list[::] differ from just list?

I am trying to understand why I can't use the following code to rotate a matrix 90 degrees clockwise: matrix = zip(*matrix[::-1]) Instead, I found that this code works: class Solution: def rotate(self, matrix): """ :type matrix: ...

pythonarrayspython-3.xlist

Robot Framework allows for the invocation of keywords from another file by referencing the keyword along with any required

Good morning! Currently, I am working on automating tests using Robot Framework with the Sikulli Library. I have encountered a challenge that I am seeking assistance with - creating a reusable keyword in a file that combines a Wait Until Screen Contain c ...

pythontestingautomationrobotframework

Need to transform a column within a Pyspark dataframe that consists of arrays of dictionaries so that each key from the dictionaries becomes its own

I currently have a dataset structured like this: +-------+-------+-------+-------+ | Index |values_in_dicts | +-------+-------+-------+-------+ | 1 |[{"a":4, "b":5}, | | |{"a":7, "b":9}] | +----- ...

pythondataframeapache-spark-sqlpyspark

Using JSON.load with a one-liner JSON isn't going to give you the desired result

I am dealing with JSON data that comes in two different formats - one is a single line, the other is formatted nicely. JSON A: {"id":1, "name":"BoxH", "readOnly":true, "children":[{ "id":100, "name":"Box1", "readOnly":true, "children":[ { "id":1003, "nam ...

pythonjson

When using StreamReader.readline(), it will not return as soon as it encounters a newline character

Recently, I delved into the world of Streams in Python and decided to experiment with some code. Here's what I came up with in ServiceSubscription.py: class ServiceSubscription(): def __init__(self) -> None: self.subscriber_connections ...

pythonpython-3.xpython-asyncio

Getting the response page after authentication in Selenium requires implementing the appropriate code that handles the authentication

After submitting a form, I am looking to receive a response on the same page with empty form fields. I attempted to use time.sleep(), but unfortunately it did not resolve the issue. from selenium.webdriver.common.desired_capabilities import DesiredCapab ...

pythonselenium

Error occurs when trying to choose an option by using selectByVisibleText and a NoSuchElementException is thrown

While I have successfully implemented this method for all select elements in my project, unfortunately I am encountering an issue with a particular select element. The problem lies in selecting the option "PIT - (111310001)" using the selectByVisibleText m ...

javaselenium

What could be causing Python to return a None value after performing a calculation?

Recently, I needed a tool to generate a multiplication table, so I came up with the following code: def createMultiplicationTable(num): for i in range(1, 11): result = num * i print(f'{num} X {i} = {result}') table = createM ...

python

Why does my Selenium Python code fail to retrieve the image URL?

My quest is to extract all image URLs from the gallery section of this webpage. Each product on the page contains multiple images. Below is the code snippet I've written in an attempt to retrieve the image URLs: img = driver.find_elements_by_css_selec ...

pythonpython-3.xseleniumselenium-webdriverweb-scraping

Engaging with online forms on websites

Having trouble automating the login process on a school website using Selenium. Initially attempted with Splinter but encountered similar issues. Unable to interact with username and password fields, realized it is within an iframe. Currently working with: ...

pythonseleniumpython-3.xselenium-webdriver