Discovering a deeply nested div or class using Beautiful Soup

Recently, I came across this URL:

All I want to do is extract the zestimate and include it in a list.

The specific class where it's located is:

class="Text-c11n-8-65-2__sc-aiai24-0 eUxMDw"
.

I attempted to target it at a higher level in the HTML with:

for div in soup.findAll('div', attrs={'id':'.Home-value '}):
    print (div.text)

I also experimented with using the select method:

items = [item.text.strip() for item in soup.select('.ds-home-values.' )]
items

Furthermore, I tried extracting it directly:

for t in soup.select("pfs-upsell"):
    t.extract()

Despite all my efforts, I consistently end up with an empty list. It seems like there might be something about this type of HTML that I'm not grasping completely.

Answer №1

The website is interactive and constantly changing. Your element selection process yields no results, but that's not the only issue at play. The primary reason for difficulty in retrieving data is that it is stored in JSON format within the HTML DOM as a comment.

import requests
import re
import json
url='https://www.zillow.com/homes/for_sale/2_p/?searchQueryState=%7B%22pagination%22%3A%7B%22currentPage%22%3A2%7D%2C%22mapBounds%22%3A%7B%22west%22%3A-97.88981437683105%2C%22east%22%3A-97.78407096862793%2C%22south%22%3A30.136145838104586%2C%22north%22%3A30.208863801102932%7D%2C%22isMapVisible%22%3Atrue%2C%22filterState%22%3A%7B%22sort%22%3A%7B%22value%22%3A%22days%22%7D%2C%22ah%22%3A%7B%22value%22%3Atrue%7D%7D%2C%22isListVisible%22%3Atrue%2C%22mapZoom%22%3A13%7D'
r = requests.get(url,headers = {'User-Agent':'Mozilla/5.0'})

data = json.loads(re.search(r'!--(\{"queryState".*?)-->', r.text).group(1))

for item in data['cat1']['searchResults']['listResults']:
    price=item['unformattedPrice']
    print(price)
    listing_url='https://www.zillow.com'+item['detailUrl']
    print(listing_url)

Output:

470000
https://www.zillow.comhttps://www.zillow.com/homedetails/7337-Menchaca-Rd-36-Austin-TX-78745/119617777_zpid/
539500
https://www.zillow.comhttps://www.zillow.com/homedetails/3214-Barnsley-Dr-Austin-TX-78745/29488852_zpid/
530000
https://www.zillow.comhttps://www.zillow.com/homedetails/10041-Aly-May-Dr-Austin-TX-78748/119624978_zpid/
659000
https://www.zillow.comhttps://www.zillow.com/homedetails/9831-Briar-Ridge-Dr-Austin-TX-78748/29509925_zpid/
499000
https://www.zillow.comhttps://www.zillow.com/homedetails/10209-Brantley-Bnd-Austin-TX-78748/29511458_zpid/
550... (truncated)

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

The nested CSS grid is failing to show its nested elements on the page

Having trouble with a nested CSS GRID - the child grid items are not appearing correctly despite multiple checks on the markup. The first grid row should span three columns, as it does, and is set to display:grid with 2 child grid items. However, the chil ...

Guide on grabbing characters/words typed next to # or @ within a div element

Within a div element, I have enabled the contenteditable property. My goal is to capture any text input by the user after typing '#' or '@' until the spacebar key is pressed. This functionality will allow me to fetch suggestions from a ...

Guidelines for incorporating JS in Framework7

I am developing an application using the framework7. I am facing a challenge where I need to execute some javascript in my page-content, but it is not running as expected. <div class="pages"> <div class="page close-panel" data-page="item"> ...

Incorporate a background image into input fields with Autofill on mobile browsers to override the default yellow background that obscures them

It is common knowledge that modern browsers apply a less than appealing yellow background to text fields when Autofill is used by visitors. A helpful workaround is known to override the default text and background colors, and even incorporate a background ...

BeautifulSoup does not recognize circular HTML pages

Encountered an issue where the page parsing code consistently checks the same page every time, despite using it alongside selenium. Selenium has no problem opening new links, but the parsing only occurs on the initial page. The frustrating part is that si ...

The website is failing to adapt properly to smaller screen sizes

I would share some code here, but it might be easier for you to just check out the URL instead. The issue is that the website was responsive across different screen sizes at first, but after making some changes in the CSS and HTML, it's not working pr ...

Improve the way you manage the active selection of a button

ts isDayClicked: { [key: number]: boolean } = {}; constructor() { } setSelectedDay(day: string, index: number): void { switch (index) { case 0: this.isDayClicked[0] = true; this.isDayClicked[1] = false; this.isDay ...

Dynamically populate 7 select boxes with options using JQuery

I have a webpage that includes 14 different dropdown menus, one for each day of the week (Monday to Sunday). Each day has two dropdowns: one for opening time and one for closing time. I used jQuery to populate all 14 dropdowns with a pre-defined list of ho ...

How can we stop the jumping of images in the grid? Is there a way to eliminate the jump effect?

Here is a script I am working with: <script src="http://static.tumblr.com/mviqmwg/XyYn59y3a/jquery.photoset-grid.min.js"></script> <script> $(document).ready(function() { $('.photoset-grid').photose ...

What are the steps to testing a webpage designed for Retina display?

I have designed a webpage optimized for retina display, but I don't own a monitor with that capability. Is there a simulation tool available to test websites on retina displays? Alternatively, are there non-Apple monitors comparable to Apple's ...

A technique for horizontally centering an image while simultaneously ensuring that the bottom is pushed down, even at an unknown resolution, and maintaining center alignment if

Imagine a scenario where I have an image with unknown resolution. My goal is to horizontally center it, even if the window's width is smaller than the picture, while also adjusting the bottom of the window to match the height of this picture. Is ther ...

Using the HTML <span> and svg elements together does not allow them to be inline with each other

In my current scenario, I have a fieldset that is inline with another one on the page. The specific fieldset in question contains an SVG circle and some text. My objective is to have both elements inline with each other, while allowing the fieldset to adj ...

There seems to be an issue with configuring placeholders in Tailwind CSS

After deciding to switch to using only tailwind CSS, I noticed that on a particular page there were inputs with transitions but no icon on the inner left side. Adjusting the colors and position of the placeholder helped prevent collisions between the icon ...

Is it possible to maintain a fixed footer while utilizing async/ajax functions?

Looking for a reliable solution to have a fixed footer that adjusts based on the page content? I've tested multiple samples, but they all fall short when it comes to incorporating AJAX elements. Is there a fixed footer out there that truly works seaml ...

Navigating through JavaScript links in Selenium WebDriver using Python

I'm encountering an issue with Selenium Webdriver in Python where I am unable to activate a javascript button. My goal is to click the Go to Previous Month button twice so that I can land on August 2014. After that, I need to select one of the days f ...

What is the process for making the functions within a module accessible to a class?

Let's discuss a scenario: I have a module containing various function definitions. I am looking to create a class that can access these functions. Which approach would you recommend: Option 1 or Option 2? import ModuleWithFunctions class MyClass(o ...

update the markers on a leafletjs map

I am aiming to create a dynamic map with markers that update every 10 minutes. The marker positions are stored in a spreadsheet document that is regularly updated. After successfully retrieving the data from the spreadsheet and positioning the markers in ...

Steps to transfer text from one input field to another by clicking a button using JavaScript

Hello, I am new to Javascript so please forgive me if my question seems silly. I have been tasked with creating a form containing two input fields and a button. When the button is clicked, the text entered in the first field should move to the second field ...

Video Background, Adjusting Scale to Fit Height and Cropping Width as Necessary

Forgive me if this question has already been posed, but my search through various related discussions has not yielded the exact solution I'm looking for. My dilemma involves a 1280x720 video that I want to use as the background on my webpage. I need ...

What is it about PHP7 that enables it to outperform Python3 in running this basic loop?

Running a simple benchmark test, I decided to compare the execution times of the same code in PHP 7.0.19-1 and Python 3.5.3 on my Raspberry Pi 3 model B. To my surprise, Python's performance was significantly slower than PHP's (74 seconds vs 1.4 ...