Questions tagged [python-requests]
Exclusively designed for the Python Requests library, this tool offers a comprehensive set of features and a user-friendly interface for handling HTTP requests in Python.
import requests from bs4 import BeautifulSoup import time import sys url = "https://www.doviz.com/" response = requests.get(url) html_content = response.content soup = BeautifulSoup(html_content,"html.parser") names = soup.find ...
I'm attempting to scrape this specific website. The following code appears to be functional: import requests header = { 'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:32.0) Gecko/20100101 Firefox/32.0', } r = requests.get('http://www.m ...
Is there a way to use Selenium with Python and Requests to check if a specific profile has the message function? https://i.stack.imgur.com/0Q4RB.png ...
I am facing an issue with this specific piece of code: import re from lxml import html from bs4 import BeautifulSoup as BS from selenium import webdriver from selenium.webdriver.firefox.firefox_binary import FirefoxBinary import requests import sys import ...
Attempting to gather search results from Walmart has been a challenge for me. For instance, let's navigate to the website "" Then, try to extract only the text from the element identified by the class name search-product-result, using python. ...
I'm working on a script that interacts with by sending a string and receiving one or all of the "fancy text" variations provided by the site. I am struggling to identify the input area within the HTML structure, especially since I aim to use requests for ...
Currently, I am in the process of developing a Python script that navigates to a specific website. The objective is to scan all the results of coins on the webpage, click on each result, and verify if there is a Telegram link available. If a Telegram link ...
view image description hereWhen attempting to submit form data on the , I receive a response, but it doesn't reflect the dates I provided in my post. Below is the code snippet I am using: import requests.sessions url = "https://seffaflik.epias.com.tr ...
Currently attempting to extract specific data by parsing the content inside a script tag. Utilizing a legitimate Xbox Live account for this purpose. from selenium import webdriver from selenium.webdriver.common.keys import Keys import requests import time ...
Trying to extract text from a specific webpage, I encountered an issue. Despite using the following code snippet: url = "http://www.koeri.boun.edu.tr/sismo/2/latest-earthquakes/list-of-latest-events/" response = requests.get(url) html = response ...
I am encountering an issue while attempting to make a request call to Elasticsearch. I can successfully execute the same call from a basic script. However, when I convert a list into a string and pass it to the payload, I receive the following stack trace: ...
I'm feeling puzzled as this code seems to work inconsistently. It utilizes the beautiful soup module and I'm curious why it functions in certain situations but fails at other times. from bs4 import BeautifulSoup import requests import lxml import ...
I'm struggling with a simple request and can't figure out what's wrong data1 = df.loc[N, 'online_raw_json'] print(type(data1)) print(data1) data1 = json.dumps(data1) print(type(data1)) print(data1) response = requests.post("http://127.0.0.1:5000/pred ...
While working in Python, I am utilizing the requests library to make a POST request to a specific URL. However, upon clicking the button, it seems that nothing is happening as per Chrome Developer Tools. No XHR requests are being made and no data is being ...
As I embark on my journey with Python Flask, I have encountered a hurdle while coding using the requests and flask modules. My current project revolves around utilizing the web API provided by the Panther platform. The project initially demonstrated an ex ...
My scenario is similar to the one described in this Stack Overflow post. However, I want to log an error message if the URL cannot be reached. Here's my attempt: # Defining the URL and checking the connection by printing the status url = 'https ...
Experiencing a 400 status code error currently. Feeling a bit stuck on what steps to take next. The 400 status code typically relates to syntax issues. How can I properly format my output into JSON file structure? import requests, json, urllib3 urllib3.d ...
After receiving a response using urllib3 (also tested with requests), I decoded it and then used json.loads on it. The data, post-decoding with decode('utf-8'), that I would like to convert into JSON is as follows: json_thing = b'{\n& ...
I have a unique person class that provides detailed information about individuals, as shown below: Datatracker.py class @dataclass class Person: resource_uri: str id: int name: str name_from_draft: str ascii: str ascii_short: Opti ...
I'm having issues running my Python script as it keeps displaying the same error message: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it. Note that I have disabled my f ...
I have been troubleshooting this issue and reviewed similar threads, yet despite verifying all options, the problem persists. My code includes: application.py @app.route("/Booksearch/<int:book_id>", methods=["GET", "POST"]) def Bookdetails(book_id ...
What distinguishes the data from the json parameters in Python's Requests library? The documentation available at this link does not provide a clear answer. Consider this code snippet: import requests import json d = {'a': 1} response = requests.post(url ...
I've been attempting to extract some email addresses from mdpi.com, but these emails are only accessible to users who are logged in. Unfortunately, my attempts have not been successful so far. When I try to do this while logged out, the following issue ari ...
Currently, I am using Python along with the requests library for web-scraping. I've encountered an issue regarding the loading of a page; I would like to implement a delay before receiving the result from requests.get(). I have come across some indiv ...
While attempting to extract website information from firms listed on Bloomberg using XPath, I encountered a challenge as the results came back empty. After conducting multiple tests, it became evident that I was unable to locate any elements on the webpage ...
I've been working on a Python script to make a post request, but I'm not receiving any response. Everything seems correct in my code, so I suspect there might be an issue with the service itself causing the lack of response. Can anyone spot if th ...
In my program written in Python, I am utilizing a basic script to retrieve reservation results for a specific CID using the file simple.py: data = {"minorRev":"current minorRev #","cid":"xxx","apiKey":"xxx","customerIpAddress":" ","creationDateStart":"03 ...
Having an issue with logging into Facebook using selenium and transferring cookies to the requests module in order to collect profile names from specific URLs. The profile names on these URLs are static but require authentication. While my script successf ...
My goal is to monitor a page for updates while maintaining the same session and cookies, without sending a completely new request. How can I determine if there are HTML updates within my current request? The page will redirect but keep the same URL. This ...
Using Python libraries like requests and BeautifulSoup, I am attempting to scrape the tables from the following Wikipedia page: https://en.wikipedia.org/wiki/Mobile_country_code. While I am able to retrieve all the data within the tables, my goal now is to ...
I'm attempting to access a website using python 3, requests, and lxml for logging in. However, despite sending a post request to the login page, I am unable to access the pages that are meant to be visible after logging in. What could be the issue? i ...
My approach to streamline my program and enhance user experience was by creating an executable file that consolidates all functions, including the download of various applications and their installation. However, I encountered a challenge where the link is ...
I am currently exploring the requests module as an alternative to Selenium for web scraping. Below is the code snippet I have been working on that extracts a table from a webpage. I'm struggling to optimize this code using requests in a more efficient an ...
I am working on a project that involves periodically sending an Array of doubles from a Python service to another. Despite being a newcomer to Python, I have delved into topics like Flask and requests. Below is a basic code example that I put together jus ...
Greetings, I am a novice programmer facing difficulties with sending data to a site using an API. Unfortunately, things did not go as planned and now I find myself in trouble. Here is the code snippet I have been working on: data = { "indus ...
I'm currently learning from this insightful guide Utilizing session.mount, I have the capability to instruct requests to perform multiple retries. However, it appears challenging to regulate the time interval between each request. Currently, I am re ...
I have developed a Python code that checks the availability of web pages stored in a tuple by browsing them using Selenium. The objective is to regularly check if these pages are online and responding. Here is a snippet of the code: from selenium import w ...
I am trying to retrieve the website's activity and uptime data, but I'm only able to get the activity status and not the uptime information. I have targeted the exact class of the website, but it shows null. Can someone please assist me in modifying my cod ...
My application is encountering issues with the python-requests library, leading to a traceback that looks like this: Traceback (most recent call last): File "/usr/lib/python3.2/http/client.py", line 529, in _read_chunked chunk_left = int(line, 16) V ...
Here's a snippet of code I've been working on that involves sending a request to an API: request_parameters = { "headers": myheaders, "params": { "myparam": True } } response = requests.get(myurl, **request_pa ...
The Print Job Dilemma My current challenge involves sending a print job from a python script to a printer that is registered in 'Google Cloud Print'. Required Components for Success A valid google account Registration of Google cloud API, ...
{ "listing":{ "deliveryOption":"option", "event":{ "date":"date", "name":"name of event", "venue":"venue" }, "externalListingId":"000000000", "inhandDate":"inhand date", "pricePerTicke ...
I am currently working on scraping data from a table located at import pandas as pd import requests import json headers = {'Host': 'stats.nba.com','User-Agent': 'Firefox/55.0','Accept': 'application/ ...
I am encountering an issue while attempting to retrieve data from the API. Specifically, I am looking to obtain the block height from this particular API. import requests import json fetch_json_data = requests.get('https://api.minaexplorer.com/blocks?l ...
In my Python script, I am attempting to make multiple API calls using the requests library. Currently, my code involves connecting to different API URLs and writing the JSON responses to separate files for further data manipulation. The connection and writ ...
While attempting to crawl a website, I copied the Request Headers information from Chrome directly. However, after using requests.get, the returned content is empty. Surprisingly, the header printed from requests is correct. Any idea why this might be happ ...
Currently, I am in need of sending a JSON object as input to an API in order to receive another JSON response. My usual method for accessing an API using requests is shown below: response=requests.get('url',auth=uauth,headers=headers).json() B ...
Whilst conducting tests, I stumbled upon a curious phenomenon: url = ' http://wi312.rockdizfile.com/d/uclf2kr7fp4r2ge47pcuihdpky2chcsjur5nrds2hx53f26qgxnrktew/Kimbra%20-%20Love%20in%20High%20Places.mp3' When this URL is entered into a browser, the file d ...
Indeed, selenium can fulfill my current requirement, but it comes with certain limitations: it is slow and necessitates the opening of a real browser on the user's machine. My approach involves using selenium to log in and fetch some state data, whic ...
I'm working on extracting a specific element from JSON data and organizing it into a separate column within my pandas DataFrame. This is the code that I currently have: #Import necessary libraries import json import requests from IPython.display impo ...
When I run curl command, I receive back a unique hash value: $ curl "http://127.0.0.1:8002/hash/" -X POST -d 'data={"key":"value"}' { "hash": "9724c1e20e6e3e4d7f57ed25f9d4efb006e508590d528c90da597f6a775c13e5" } However, when using Python requests, I en ...
I am currently utilizing React on the frontend and Fastapi on the backend main.py from typing import Union from fastapi import FastAPI from fastapi.encoders import jsonable_encoder from fastapi.middleware.cors import CORSMiddleware from pydantic import ...
I attempted to extract an image from a Reddit post using this code snippet. However, when I execute the code, it sometimes displays an HTML snippet and other times it prints None without any errors. Can anyone explain why this is happening? Below is the ...
As I work on a program to send data to a specific URL and receive a response, I encountered an issue while using Python. The problem arose when trying to include a token in the request as required by Postman, resulting in a [401] response. The main obstac ...
I need to extract the value of the id, which is "id": 59 from a JSON output obtained using curl command. The JSON output looks like this: [{"id":59,"description":"This is a demo project","name":"Demo_Project","name_with_namespace":"sam / Demo_Project","pa ...
How can I include a boolean value in the request body using the Python requests library? I attempted: request_body = {'someBooleanValue': true}. => NameError: name 'true' is not defined request_body = {'someBooleanValue': ...
For my project, I am working on capturing network traffic using Selenium Webdriver with Python. In order to achieve this, I need to utilize a proxy such as BrowserMobProxy. Everything works smoothly when using webdriver.Chrome: from browsermobproxy impor ...
Referencing the example provided in this link - How to extract html links with a matching word from a website using python I developed a web scraping script to search for specific keywords in both recent and cached versions of a local newspaper. from bs4 ...
Currently, I am working on a Python script to generate a daily COVID-19 dashboard for my country and state. Unfortunately, I have hit a roadblock in downloading one of the required files. To download this file, I need to visit the website () and click on ...
Currently, I am utilizing Python (BeautifulSoup) to extract data from various websites. However, there are instances where accessing search results can be challenging. For example: import requests from bs4 import BeautifulSoup url1 = 'https://auto.ria.co ...
I'm currently utilizing the REST API found at this link. Here is an example code snippet for reference: server = "https://rest.ensembl.org" ext = "/vep/human/hgvs/ENSP00000401091.1:p.Tyr124Cys?" r = requests.get(server+ext, header ...
I'm struggling with scraping the contents of a specific web page. Here is an example of my Python code: response = requests.post('http://a836-acris.nyc.gov/bblsearch/bblsearch.asp?borough=1&block=733&lot=66',{'User-Agent' ...
import requests from bs4 import BeautifulSoup # Fetch data from Google Search response = requests.get("https://www.google.com/search?q=book") soup = BeautifulSoup(response.content, 'html.parser') phrase_extract = soup.find_all(id="result-stats") ...
Hello there, Lately, I've been experiencing timeouts when making requests to the ebay website. Here is the simple code that I am using: import requests headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; ...
Trying to extract the latest stock price using Fidelity's screener. For instance, the current market value of AAPL stands at $165.02, accessible via this link. Upon inspecting the webpage, the price is displayed within this tag: <div _ngcontent-cx ...
My attempt at this solution didn't yield the desired results. Interestingly, it did work with a different JSON file (). I suspect the issue lies in the differences at the beginning and end of the current file, as there are some additional parts before ...
Has anyone successfully retrieved ABS data using the pandasdmx library? Below is an example code that retrieves data from the European Central Bank (ECB) and works as expected. from pandasdmx import Request ecb = Request('ECB') flow_response ...
My goal is to send all files located in a specific folder on my local disk to a designated web address using Requests and Glob. Each time I upload a new file to the URL, I intend to update a dictionary with a new entry consisting of the "file name (key)" a ...
My current project involves scraping German zip codes (PLZ) based on a specific street in a particular city using Python's requests library. The data I am working with can be found on this server, and I'm applying techniques learned from here. The goal is ...
<PIL.WebPImagePlugin.WebPImageFile image mode=RGB size=1600x1600 at 0x1F4E779BA00> is the type of file I'm working with. When I make a post request, I encounter this error: TypeError: a bytes-like object is required, not 'WebPImageFile'. Any sugges ...
My current challenge involves attempting to send a POST request to the USGS EarthExplorer inventory API, beginning with a straightforward log-in process. They provide a handy test page at: This test page showcases some formatting examples. When exploring ...
Currently, I am in the process of developing a script that interacts with the Jenkins plugin API to retrieve a list of plugin dependencies. To achieve this, I have utilized the Python requests module. However, I am encountering an issue where the response ...
I'm dealing with some code that involves making a requests.get() call, which could potentially fail in different ways. My goal is to catch any exceptions related to requests, without necessarily needing to know the specific reason for the failure. I ...
While attempting to extract data from a particular website using the requests.get method, I encountered an issue. The information retrieved from the website seems to be inconsistent and does not align with the actual data displayed on the site. For instan ...
My current setup allows me to send text emails with the Mailgun Python API without any issues: def send_simple_message(email_text, attachment_filename=""): requests.post("https://api.mailgun.net/v3/mydomain.in/messages", auth=("api", "key-1234"), fi ...