Waiting for Elements to be Added to Parent in a Lazy Loading Website with Selenium and Python

When working with Selenium's expected conditions for wait, such as those mentioned here: https://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.support.expected_conditions, I find myself unsure about which one to use or if that is even necessary.

My current task involves extracting reviews from various recipes, like the ones found on this page:

I am aiming to collect all reviews by continuously clicking the "show more" button until it disappears.

Currently, I have been using a sleep(2) delay between iterations to ensure the next set of reviews loads properly. However, I acknowledge that this approach is not efficient and may not always be sufficient.

I would greatly appreciate any guidance or suggestions you may have. Thank you in advance!

pythonseleniumweb-scrapinglazy-loadingwebdriverwait

Answer №1

If you're looking to gather all the reviews from the recipe located at 

https://www.foodnetwork.com/recipes/tyler-florence/a-gooey-decadent-chocolate-cake-recipe-2102926
, here is a solution that might help:

  • Code Block:

      To obtain reviews:
  - Utilize Selenium for web scraping
  - Navigate to the specified URL
  - Scroll and load additional comments if applicable
  - Extract and print the text content of each review
  
  The code provided above demonstrates how this process can be achieved.

  • Console Output:

      Various user reviews are displayed in the console output, sharing experiences with baking the chocolate cake recipe. Opinions range from positive feedback on taste to critiques regarding sweetness levels, texture, and consistency of the frosting. Users also provide tips and modifications based on their personal baking results.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Protractor Throws Element Visibility Issue

When attempting to check a checkbox in the UI, I encounter an "ElementNotVisibleError: element not visible" error. Strangely, I am able to successfully capture and click the checkbox using the console in Chrome developer tools. Has anyone else experience ...

javascriptseleniumautomationprotractorelement

The content becomes empty once the initial dict() is called

Seeking Help: When working with Django, I encountered an issue related to dictionaries. After calling dict(q_a) for the first time, the second call returns null. Can anyone explain how Python operates in this scenario? if request.method == "POST": q_ ...

pythondjangodictionary

Tips for executing a Python function from JavaScript, receiving input from an HTML text box

Currently, I am facing an issue with passing input from an HTML text box to a JavaScript variable. Once the input is stored in the JavaScript variable, it needs to be passed to a Python function for execution. Can someone provide assistance with this pro ...

javascriptpythonhtmlrobotframework

Can this code snippet be transformed into a list comprehension? If so, how can it be done?

x = sum(...)

Appreciate your assistance, pf.me ...

pythonlist-comprehension

Django skillfully crops and isolates just the facial feature within the image field with the help of opencv

Here is a question that I need help with regarding Django and OpenCV2: Django cut and put only the face in the picture field using opencv2 The issue I am facing includes: Even when I update the image, the processing causes distortions. The image size is ...

pythondjangoopencv

I'm curious, is it possible to customize the number of threads, blocks, and grids for CuPy computation? If so, how can this

I have a code snippet that uses Cupy: import cupy as cp vals, vecs = cp.linalg.eigh(Array) I am looking to configure the number of threads in the code. How can I do this? Specifically, how can I set it to execute with 100 threads? ...

pythoncupy

Unable to map 1024 bytes due to memory allocation issues, despite having sufficient RAM available

I am currently immersed in the creation of a seminar paper focused on natural language processing (NLP) and the summarization of source code function documentation. To achieve this, I have meticulously curated my own dataset consisting of approximately 640 ...

pythonpytorchmmapsimpletransformers

Separating email addresses from a list of emails in Python: Splitting them based on commas and spaces

My input is email_list = "<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="d9b4b8b0b5e899b4b8b0b5f7bab6b4">[email protected]</a>,<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="f598949c99c ...

pythonstringlist

Issues with executing a Selenium project in Java using Maven - Troubles with the Surefire plugin

I encountered the following issue: While running my Selenium-Maven-Java project, I faced a Surefire plugin error. When building my project on Circle CI, I received this build error. I would appreciate any suggestions on how to rectify this. This is ...

mavenseleniumcontinuous-integrationgeometry

Guide to saving a Python docx file on the client side with the help of Ajax

I am currently using Python 3.6 with Django, and my code snippet is shown below: from docx import Document document = Document() document.add_heading('My docx', 0) document.save('myFile.docx') return HttpResponse(document, content_typ ...

pythonhtmldjangoajaxpython-docx

Issues were encountered in the Suite Class as it failed to trigger the methods annotated with @BeforeClass and

I have a collection of test cases that require the use of @BeforeClass and @AfterClass annotations in JUnit to set up and tear down resources before and after execution. While these methods work correctly when running the suite class through JUnit, they ...

mavenseleniumselenium-webdriverjunitsurefire

Ensure to verify the presence of a file before attempting to read its contents. Wait patiently

I am facing an issue where I need to read a file only after it has been created. The code snippet below is what I have, but unfortunately, it doesn't seem to be working: import os.path if os.path.isfile(file_path): read file in else: wait Ca ...

python

What is the best way to include special characters in a function without altering them?

Is there a way to prevent \t from being interpreted as a real tab in a function that asks for a delimiter? For example, consider the following code snippet: def example(dataToBeSplit, delimiter): return dataToBeSplit.split(delimiter) example(&apo ...

pythonfunctiontabsspecial-characters

Creating sprites in proximity to a specific location: A step-by-step guide

Creating a game with enemy ships that shoot lasers at the player sprite presents some challenges. These enemies need to rotate in order to face the player and then fire a projectile aimed at colliding with the player ship. The current issue is that the la ...

pythonpygame

Is there a way to transmit the Ctrl key to the terminal using subprocess commands?

Hey there, I'm currently working on a script to automate the process of opening tmux and splitting windows in my terminal. The key combination needed for this operation is sending tmux along with Ctrl + b and %. def create_window(): subprocess.c ...

pythonsubprocess

Is there a way to set up a virtual environment using virtualenv for Python 3.7 on Ubuntu 16.04.6 LTS x64, without the need to download and install Python 3.7 on my system?

Is there a way to set up a virtual environment using virtualenv for Python 3.7 on Ubuntu 16.04.6 LTS x64 without actually installing Python 3.7 directly on the computer? virtualenv -p python3.7 /mnt/ilcompn0d1/user/dernonco/pyenv/codetest When I try this ...

pythonubuntuvirtualenv

Separate the dictionaries extracted from API data into individual columns

Starting off, I must admit that I am quite the novice in this field. So, any specific and detailed advice or guidance would be greatly appreciated. I've managed to pull data from Canvas API using the code snippet below: result = requests.get('h ...

pythonjsonpandas

Simple steps to retrieve a value in Python: Let's say you have the following HTML code: <span class="label">Google</span

How can I retrieve the value of "Google"? I attempted using the get.attribute method, but it returned "None". Strangely, when I used get.attribute('innerHTML'), it displayed the entire element: <span class="label">Google</spa ...

pythonselenium-webdriver

TCP allows files to be transferred without needing to close the socket

I developed a basic TCP server - client file sharing system, but I've encountered an issue where it only works when the socket is closed in the client script. I attempted to eliminate the 's.close' command from the client python script, whi ...

pythonsockets

Is it possible to create a JSON object with a hierarchical structure using a recursive

  As much as I hate to admit it, my computer science skills are letting me down on this one :( Dealing with an API that spits out JSON responses like the following is throwing me off: // hit /api/get/200 { id : 200, name : 'France', childNod ...

pythonjsonalgorithmrecursion