How can I specifically extract data from certain columns in an HTML table using BeautifulSoup? Currently, my code is pulling data from every column

I need help extracting specific data from an HTML table. I want to extract the information from the columns with the text "Total" and the value "93" only, rather than extracting all column data as my current code does.

For example, the current output is:

Total
93
93
0
0

But I would like the output to be:

Total 93

Here is a snippet of my code:

def extract_total_from_report_htmltestrunner(): 
    filename = (
    r"C:\test_runners 2 edit project\selenium_regression_test\TestReport\ClearCore_Automated_GUI_Regression_TestReport.html")
    html_report_part = open(filename, 'r')
    soup = BeautifulSoup(html_report_part, "html.parser")
    tr_total_row = soup.find('tr', {'id': 'total_row'})
    tr_total_row.find(text=True, recursive=False)
    print tr_total_row.text
    return tr_total_row.text

And here is the relevant HTML snippet:

<table id='result_table'>
    <tr id='total_row'>
        <td>Total</td>
        <td>93</td>
        <td>93</td>
        <td>0</td>
        <td>0</td>
        <td>&nbsp;</td>
    </tr>
</table>

Can anyone suggest how I can specifically extract and print out "Total" and "93" in the same line?

Thank you, Riaz

Answer №1

To extract data, you can utilize the find_all() method and then manipulate the results by slicing:

" ".join(td.get_text(strip=True) for td in tr_total_row.find_all("td")[:2])

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

When utilizing the JHipster DevBox, npm does not generate a symbolic link for the webdriver-manager

At my workplace, I am attempting to integrate JHipster with the DevBox using a proxy. Despite configuring everything correctly, I am encountering issues in utilizing JHipster effectively. My goal is to create a microservice gateway with Protractor, but wh ...

Discovering the art of clicking at random on a webpage using Selenium in Python, along with mastering the techniques of zooming in on a webpage

Is there a way to click randomly within a specific area on a webpage using selenium? I am also curious about how to zoom in on a designated portion of a page, rather than the entire page. For example, if a page is x by y in dimensions, how can one zoom int ...

Selenium and Docker: Struggling to load a personalized Chrome user profile

Hey there, I posted an issue a couple of days ago, but it seems like it's not really considered an issue. However, my problem still persists, and I'm hoping someone can help me figure out what's going on :D You can find the detailed discus ...

Automated data collection with selenium

Hey there! I'm currently working on scraping a website and initially used Bs4 to extract certain elements like sector and name. However, I'm facing difficulty in retrieving the financial data using it. In the page source below, the "-" should be ...

Encountering an error in the response body while attempting to send body data from an external file using J

I am facing an issue where I need to send data in the request body from an external file within jmeter, but every time I attempt to do this, I receive an ERR in the response body. Below is my post method request body: ${__FileToString(/Users/public/${__eva ...

The JavaScript function getSelection is malfunctioning, whereas getElementById is functioning perfectly

I am encountering a peculiar situation while trying to input text into a textbox using a JavaScript command. The CSS locator does not seem to update the text, whereas the ID locator is able to do so. URL: Below are screenshots of the browser console disp ...

Adjusting the size of browser windows to half of the screen with the help of Selenium

Is there a method available to resize two windows to half of the screen size with Selenium? If not, any alternative suggestions are welcome. I am currently working with Python and Chromium, but willing to consider other options. ...

Can phantomJS be used to interact with elements in protractor by clicking on them?

While attempting to click a button using PhantomJS as my browser of choice, I encountered numerous errors. On my first try, simply clicking the button: var button = $('#protractorTest'); button.click(); This resulted in the error: Element is ...

Tips on selectively crawling specific URLs from a CSV document using Python

I am working with a CSV file that contains various URLs with different domain extensions such as .com, .eu, .org, and more. My goal is to only crawl domains with the .nl extension by using the condition if '.nl' in row: in Python 2.7. from selen ...

Error: Can const be used in strict mode?

Currently, I am attempting to log in to facebook.com using selenium-webdriver. var webdriver = require('selenium-webdriver'), By = require('selenium-webdriver').By, until = require('selenium-webdriver').until; var dr ...

Getting started with Codeception may not always be straightforward: [PHPUnitFrameworkException] Error encountered - Element index not defined

After following the Codeception Quick Start instructions diligently, I proceeded to run the initial example test using PhpBrowser... # Codeception Test Suite Configuration # # [further comments omitted] # actor: AcceptanceTester modules: enabled: ...

Tips for waiting for a button to be clicked by the user in Selenium web-driver with Python?

I have a Login form with a username, password, captcha text field, and SIGN-IN button. The elements are clickable and visible from the beginning. https://i.stack.imgur.com/rHbms.png With Selenium, I am able to provide input for the username and password ...

Automated test does not trigger Selenium Webdriver modal popup

Currently, I am developing automation scripts using Selenium Webdriver in Java to test an application designed for Point of Sale systems. Upon the initial launch of the application, a modal dialog box is displayed for the user to make a selection. This mo ...

Searching for an element using Python's Selenium library by referencing the value of its sibling element

This is an example of HTML code structure: <div> <span>Title</span> <input value="a"> </div> <div> <span>Price</span> <input value=""> </div> The task at hand is to ch ...

The issue with Python Selenium driver.get() function not functioning properly when used within a for loop

Here is some code that logs into a YouTube account and then visits a few YouTube videos. The problem arises in the following scenarios: If I use a direct link like this, it works fine driver.get('https://www.youtube.com/watch?v=FFDDN1C1MEQ') ...

utilizing the process builder in order to retrieve the stream from this particular process

When attempting to use the BufferedReader class to fetch input stream, I encountered an issue. Even though the command console opens with the application running, the variables remain empty and the program gets stuck at the readline() step. This is probabl ...

Updating the download destination in Microsoft Edge with Python and Selenium

Is there a way to change the download directory for Python Selenium in Chrome? DownloadPath = (f'C:\\Users\\{EmplyID}\\Desktop\\General Orders\\{Newfolder}') chromeOptions = Options() chro ...

Unable to select dropdown menu on Safari browser using MacBook Pro

I am just starting out with automation and I'm having trouble clicking on a dropdown in my application using Safari Browser. The script keeps failing with an error message. Can someone please assist me? Error : An unexpected server-side error occurr ...

When you click on the bottom of the list, you access the element at the top of the list

Currently, for my test using Java and Selenium, I need to click on an element at the bottom of a list within the application. The code snippet I am using is as follows: wait.until(ExpectedConditions.visibilityOfElementLocated(By.xpath("//span[text()=&apos ...

Converting a table into a collection of dictionaries using BeautifulSoup

I am encountering some difficulties in scraping a specific table and converting it into a list of dictionaries. The table I am interested in scraping can be found on this page My objective is to scrape only the "Batting" table. Below is the code I have b ...