Is there a way to extract information from a <span> element with restricted access?

Question

Is there a way to extract information from a <span> element with restricted access?

I'm currently utilizing bs4 and urllib2 to extract data from a website.

Check out the webpage here.

The goal is to retrieve the remaining digits of the telephone number 3610...... but beforehand, it's necessary to click on a button to reveal the full phone number.

https://i.stack.imgur.com/IvevI.png

This specific information can be found within this designated div:

<div class="telefones">
        Phone(s): <span id="telefones">3610...
        <span><input type="button" id="verTel" value="ver telefone completo"/></span></span>
</div>

Would it be feasible to accomplish this task using bs4 along with urllib2?

python html web-scraping beautifulsoup

Answer 1

Answer №1

To retrieve the phone number, you can fetch it from the JSON response of a request made to the following URL:

http://www.ribeiraosaude.com.br/home/GetTelefone/<id>

. Use the Python library requests to make this request and extract the phone number.

import requests
from bs4 import BeautifulSoup

page_id = 937
with requests.Session() as session:  # maintaining web-scraping session
    response = session.get("http://www.ribeiraosaude.com.br/detalhe/%d" % page_id)
    soup = BeautifulSoup(response.content, "html.parser")

    phone_number = session.get("http://www.ribeiraosaude.com.br/home/GetTelefone/%d" % page_id).json()["telefone"]
    print(phone_number)

Answer 2

To retrieve the phone number, you can fetch it from the JSON response of a request made to the following URL:

http://www.ribeiraosaude.com.br/home/GetTelefone/<id>

. Use the Python library requests to make this request and extract the phone number.

import requests
from bs4 import BeautifulSoup

page_id = 937
with requests.Session() as session:  # maintaining web-scraping session
    response = session.get("http://www.ribeiraosaude.com.br/detalhe/%d" % page_id)
    soup = BeautifulSoup(response.content, "html.parser")

    phone_number = session.get("http://www.ribeiraosaude.com.br/home/GetTelefone/%d" % page_id).json()["telefone"]
    print(phone_number)

Answer 3

Answer №2

To approach this situation, you need to consider how the button behaves when clicked.

If clicking the button triggers JavaScript that reveals a number, you can extract the relevant JavaScript code associated with the button.

For example:

function displayNumber(){ document.body.yourSpan.innerHTML = 'NUMBER'

On the other hand, if clicking the button results in an ajax request, you can simulate the page's actions using Python's built-in request library to communicate directly with the server.

For instance:

phone_number = session.get("http://www.ribeiraosaude.com.br/home/GetTelefone/%d" % page_id).json()["telefone"]

(Credit: alecxe)

Regardless of how the button functions, there is another alternative. Many people also opt for using Selenium, which simulates browser behavior and can be controlled via Python. For guidance on clicking buttons using Selenium, refer to this answer

Answer 4

To approach this situation, you need to consider how the button behaves when clicked.

If clicking the button triggers JavaScript that reveals a number, you can extract the relevant JavaScript code associated with the button.

For example:

function displayNumber(){ document.body.yourSpan.innerHTML = 'NUMBER'

On the other hand, if clicking the button results in an ajax request, you can simulate the page's actions using Python's built-in request library to communicate directly with the server.

For instance:

phone_number = session.get("http://www.ribeiraosaude.com.br/home/GetTelefone/%d" % page_id).json()["telefone"]

(Credit: alecxe)

Regardless of how the button functions, there is another alternative. Many people also opt for using Selenium, which simulates browser behavior and can be controlled via Python. For guidance on clicking buttons using Selenium, refer to this answer

Is there a way to extract information from a <span> element with restricted access?

Answer №1

Answer №2

Similar questions

``The background color will dynamically change according to the result of a function

gem5 is throwing an "ImportError: No module named six" error, despite the fact that the module six has already been successfully

Animated Gradient Header with CSS

CSS navigation bar (background gradient problem)

Utilizing regular expressions or BeautifulSoup to locate a specific word or numerical value following a designated keyword

Disallow negative numbers but allow decimals in HTML input

Is it possible to create a button that can bring in a .css file?

Exploring the potential of HTML5 canvas for speedy pixel manipulation

Disable the outer div scrolling in VueJS, but re-enable it once the inner div has reached the bottom of

What is the best way to choose the next adjacent element using a CSS selector with Python Selenium?

Examining data within individual groups in a Python dataframe and making comparisons

Combining various postponed JavaScript file imports in the HTML header into a single group

Ways to incorporate vertical alignment while adjusting the size of a <div> block

Is the presence of the selenium webdriver being registered?

Tips for choosing one row at a time using Python and SQLite?

Requesting HTML content from a PHP file using JQuery Ajax callback

Leveraging the Power of Ajax Button for Django Model Filtering

Ensure that a distinct cross button is included within the text or search input and positioned to float alongside the text for

Methods for deleting all instances of a substring within a collection of lists

When the click event is triggered, the second function guess() does not execute