Retrieve all search results displayed on the Google search results page

I need assistance in using Selenium to retrieve search results from Google after entering a specific query. Currently, I have created the following code:

WebDriver driver = new ChromeDriver();
        driver.get("http://www.google.com/xhtml");
        Thread.sleep(5000);
        WebElement searchBox = driver.findElement(By.name("q"));
        searchBox.sendKeys("ChromeDriver");
        searchBox.submit();

        System.out.println("Current Url: " + driver.getCurrentUrl());

        List<WebElement> results = driver.findElements(By.xpath("//h3[@class=r]/a"));

        for(int i=0; i<results.size(); i++){
            System.out.println(">>>>> results " + results.get(i).getText());
        }

        // second attempt

        List<WebElement> allSearchResults = driver.findElements(By.cssSelector("ol li h3>a"));

        //iterate the above list to get all the search titles & links from that page
        for (WebElement eachResult : allSearchResults)
        {
            System.out.println("Title : " + eachResult.getText() + ", Link : " + eachResult.getAttribute("href"));
        }

Seeking guidance on how to improve this solution.

Answer №1

Make sure to give time for the results to show up once you submit the search form:

WebDriverWait wait = new WebDriverWait(webDriver, timeoutInSeconds);
wait.until(ExpectedConditions.presenceOfElementLocated(By.cssSelector("h3.r a")));

Answer №2

It appears that you are headed in the right direction, but there seems to be an issue with your xpath. It looks like you are missing a single quote next to 'r'.

List<WebElement> results = driver.findElements(By.xpath("//h3[@class=r]/a"));

The correct version should be:

 List<WebElement> results = driver.findElements(By.xpath("//h3[@class='r']/a"));

Check out this screenshot of a Google search for reference

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

How do I prevent third-party scripts from running during my Selenium tests?

Recently, my Selenium test performance has been hindered by unnecessary third-party scripts. I'm looking for a way to block these scripts, ideally restricting requests to only localhost. ...

Ways to verify the content within two separate p elements

<p> expanded state:</p> <P> true </p> Merging the text from both tags into a single output. Final Output: expanded state: true ...

Tips for showing Japanese characters received through an ajax request on a jsp webpage

I'm facing an issue with my .jsp page where it makes an ajax request to another .jsp page. The query parameter contains Japanese characters and I've confirmed that they are passed correctly in the request. However, the string is not being receive ...

"During a recent HTML test, an error was encountered when attempting to import stringIO, even though

Greetings, I am currently utilizing the code below to incorporate an HTML test runner into my project: import HtmlTestRunner import unittest from io import StringIO class TestStringMethods(unittest.TestCase): """ Example test for HtmlRunner. """ ...

Automated UI - Self-Repair Mechanism

Currently, I am working on UI automation using Selenium in Java. I am facing challenges with tests failing due to changes in XPATH or CSS selectors in different environments. To tackle this issue, I have decided to create a utility script that can automat ...

Extract initial odds data from Oddsportal webpage

I am currently attempting to scrape the initial odds from a specific page on Oddsportal website: https://www.oddsportal.com/soccer/brazil/serie-b/gremio-cruzeiro-WhMMpc7f/ My approach involves utilizing the following code: from selenium.webdriver.common.b ...

How to extract text from HTML body using Python with Selenium without the tags?

Can anyone help me extract a specific sentence using Selenium and Python? <h2 id='PO-PF2' class="section">Program Information</h2> Length: Two-year Ontario College Graduate Certificate program <br />Delivery Sequence:<br /&g ...

Unable to parse the configuration file in Selenium Java due to an issue with Log4j

Could someone tell me how to configure log4j to read a properties file? I am currently working on a Java project for testing with selenium and I would like to implement log4j. However, I keep encountering an error stating that Log4j could not read the con ...

Scraping a dynamic webpage that necessitates scrolling with Selenium using Java

I am trying to extract and display the names of all the backgrounds available on this specific page: The issue I'm facing with my program is that it's unable to scroll down to the bottom of the page. I have attempted various methods including: ...

Exploring the Keys of Multiple HashMaps

I am looking to compare the keys of four hashmaps. All of the keys in the hashmaps are the same. However, I want to assess each hashmap to compare the keys of all four hashmaps. In the provided code snippet, I am using 4 nested loops and a logical AND cond ...

When using Selenium, Python opens Chrome with the keyword "Data;"

When using python selenium for web scraping, I found that after running the code below, Chrome launches but does not load the website as expected. Instead, it displays 'data;' in the URL bar. Can anyone offer assistance with this issue? Thank yo ...

How to deselect a single option in a dropdown menu with Python Selenium WebDriver

Having trouble deselecting a selected option after using webdriver to select it? I'm encountering an error that says NotImplementedError("You may only deselect options of a multi-select") NotImplementedError: You may only deselect options of a multi-s ...

Protractor and Cucumber working together

I'm attempting to create a basic cucumber example using protractor, but I'm encountering errors in the feature file. Here is my code: protractor.conf.js file var prefix = 'src/test/javascript/'.replace(/[^/]+/g,'..'); expor ...

Unusual actions observed on element while implementing Selenium with Python

I've been struggling to find a specific element on the webpage. The troublesome element is a < ul > that pops up when I right-click the page, acting as a menu similar to the desktop context menu in Windows. Here's the code snippet that runs a ...

Selenium not running smoothly on Mac operating system

Embarking on my journey to learn web scraping with selenium, I've encountered some errors that are puzzling me. Below is an image of the issue I am facing. The code snippet below appears to be malfunctioning. chrome_path = r"\\Users\&b ...

Using Selenium to Determine if a WebElement is Currently Focused

Initially, I hoped for a more straightforward solution like WebElement.isfocus(), but instead I came across the :focus pseudo class. Is this task really so uncommon that there is not an abundance of information available? It seems surprising to me. I am ...

Navigating from the final column to the second column in a table with Selenium using XPath: a guide

Within the UI, there is a table with a varying number of rows. At any given time, only one row contains an icon that signifies completing a task by clicking it. Once the task is done, the icon moves to another row. I am looking to extract the task name fr ...

Tips for navigating through dropdown options and choosing concealed elements with selenium automation tool

I am currently working on the Redbus.in website and I am trying to select a random travel checkbox. However, when attempting to select a checkbox that is not visible on the screen, an ElementNotVisibleException occurs. This happens because only the first 4 ...

Guide on establishing a connection to a broker API using Java and sockets?

I've been struggling to establish a connection with the XTB API and it's becoming quite frustrating. I have very limited knowledge when it comes to sockets, so I'm basically learning as I go along. My current challenge involves sending a JSO ...

error message on a MacBook using Python: "File or directory not found: [Errno 2]"

I encountered an issue with [Errno 2] stating there is no such file or directory. This problem is unfamiliar to me in Windows Visual Studio. Below, you can find the code where the problem arises: from selenium import webdriver from selenium.webdriver.comm ...