Tips for using Selenium to download .csv files in Python

Utilizing Selenium to browse through this particular website:

https://apps1.eere.energy.gov/sled/#/

I am interested in obtaining data for a city such as Boston: here is my process:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary

driver = webdriver.Firefox()
driver.get("https://apps1.eere.energy.gov/sled/#/")
search = driver.find_element_by_class_name('sr-only')
search.send_keys("Boston")
search.send_keys(Keys.RETURN)
time.sleep(5)
tmp = driver.find_element_by_css_selector("a.csv-link")

Following this method leads me to the desired webpage:

My next step is to download a csv file related to 

State and National Retail Electricity Rate Trends
by clicking on Download Chart and then Download Chart/Data CSV.

Attempting to click on the button, I encountered an error:

ElementNotInteractableException: Message:

Upon inspection, here is a screenshot of the button (Download Chart/Data) that I aim to interact with in Firefox.

https://i.stack.imgur.com/IaJAf.png

pythoncsvselenium

Answer №1

To successfully download the file from the designated page, you will need to follow two simple steps. First, click on the drop-down button. Next, select the csv data option. To automate this process, use the respective xpath for each element.

driver.find_element_by_xpath('//*[@id="historical-electric-download"]/a[@class="no-underline"]').click()
driver.find_element_by_xpath('//*[@id="historical-electric-download"]/div/a[@class="csv-link"]').click()

Answer №2

'a.csv-link' does not refer to a class. You may want to consider using the id that is available on the page instead.

You can attempt the following:

content = driver.find_element_by_id('csv-link')

Answer №3

First, make sure to click on "Download chat" before attempting to download the CSV file by clicking on "Download Chart/Data CSV." The issue may lie with the locator being used.

If you are trying to click based on the element's class name, but the locator does not contain a class attribute.

After clicking on "Download Chat," try using the following xpath:

//a[@id='csv-link']

Answer №4

The locator <code>a.csv-link is specifically for identifying elements using CSS, therefore the correct method to use in this case would be 

driver.find_element_by_css_selector

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

rearrange the format of JSON in Python

I have a collection of json files with the following structure: [ { "INVOICE": { "label": { "box": { "left": 102, ...

pythonjson

When trying to locate the element using `driver.find_elements_by_id('fakebox-input')` and entering the search query `site:linkedin.com/in/ AND "python developer" AND "London"`, an attribute error was displayed

Issue with driver.find_elements_by_id('fakebox-input') displaying attribute error While executing this in anaconda ipython prompt, I encounter the following error - AttributeError: 'list' object has no attribute 'send_keys' ...

pythonseleniumselenium-chromedriver

Is there a way to successfully run Python in a QwikLabs Linux VM using Windows PuTTY, as it currently does not work? Interestingly, it does work when using a Mac

Recently, I started a course that requires accessing a Qwiklabs Linux virtual machine via SSH to write and run Python scripts. Surprisingly, when using PuTTY on Windows 10, everything works smoothly until it's time to execute the script - then I encou ...

pythonvirtual-machine

The structure of Django, NPM, and Node packages

When considering the architecture of the project I will be working on, it's important to note the structure of the node_packages: |- Django project |-- app1 |-- app2 |-- node_modules |--- foundation-sites |--- grunt |-- static |--- css |--- images | ...

pythonnode.jsdjangonpmarchitecture

Utilize designated columns to create a new column by associating it with JSON data

Currently, I am working with a data frame that contains the following information: A B C 1 3 6 My objective is to extract columns A and C and combine them to create column D, which should look like {"A":"1", "C":"6}. Th ...

pythonjsonpandasdataframe

Having difficulty extracting table data with selenium and beautiful soup

I've hit a wall in trying to extract data from a table on Yahoo's daily fantasy webpage. Despite scouring StackOverflow for solutions, I can't seem to get the desired information. The table either appears empty or the elements within it are ...

pythonselenium-webdriverweb-scrapingbeautifulsoup

"Obtaining the price of a product based on its various sizes using Selenium with

I need to extract product details from this website. The product has three different sizes, and the price changes based on the selected size from the dropdown menu. Currently, my scraper is only able to fetch the default price of $35 for 1kg upon initial ...

pythonselenium

Using Python to Automate Chromium with Selenium

I have developed a Python script that uses Selenium to gather data from various websites. It works perfectly on my Windows PC, but I'm now faced with the challenge of making it run on my Raspberry Pi. Since Chrome is not supported on the Pi, I need to ...

pythonseleniumselenium-webdriverraspberry-pichromium

Python and Scrapy encounter issues with locating website links

I've been searching for the URLs of all events listed on this page: https://www.eventshigh.com/delhi/food?src=exp However, I can only locate the URL in JSON format: { "@context":"http://schema.org", "@type":"Event", "name":"DANDIYA NIG ...

pythonjsonscrapyweb-crawler

Newbie Python Programmer: How to Send Selenium WebElement Object to PHP

I am currently exploring the possibility of extracting data from a website and then passing it for use in PHP. After researching various methods, one suggestion that stood out to me was to serialize the Python data before transferring it. However, I seem ...

phppythonseleniumautomation

Locating data points in close proximity to the classifier's decision boundary

Greetings, I am new to this field so please excuse me if my question seems basic. I have recently trained an XGboost classifier using Python and now I am wondering how I can identify the samples in my training data that fall within a certain distance from ...

pythonclassificationxgboost

Converting HTML table to pandas dataframe: Extracting data from HTML elements

After extracting data from a large table on the web using requests and BeautifulSoup, I encountered an issue with specific parts of the information. Here is a snippet of the table: <table> <tbody> <tr> <td>265</td> <td> ...

pythonpandasbeautifulsoup

Configuring Proxies in Scrapy

I'm having trouble configuring the proxy in my Scrapy project. Every time I try to run it, I encounter the following error: Error 2019-05-09 19:36:50 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2019-05-09 19:36:50 [sc ...

pythonscrapy

Transform the multidimensional dictionary output from Yahoo Finance into a structured dataframe

Currently in the process of developing a stock screener that focuses on fundamental metrics using the yahoofinancials module. The code provided generates output in multidimensional dictionary format, which I'm finding challenging to convert into a da ...

pythonpandasdataframedictionary

Having trouble locating the table element within a div element using Selenium in Python

My goal is to access values from a table within a div element using selenium. Although I am able to see the table when inspecting the element, I cannot locate it in the source code while attempting to access it through selenium. When Inspecting: https:/ ...

python-3.xseleniumselenium-chromedriver

Discover the XPATH for selenium in VBA programming

I am currently facing a challenge with a web page hosted on a secure website. I have attached a snapshot of the specific section I am troubleshooting. The XPATH that identifies the rows of a table (totaling 13 rows) is: //div[@id='Section3'] How ...

excelvbaseleniumxpath

Is there a way to load JSON data into an OrderedDict structure?

Is it possible to use an OrderedDict as an output in json.dump? I know it can be used as an input, but I'm curious about maintaining key order when loading into an OrderedDict. If it's not directly possible, are there any alternative solutions o ...

pythonjsonloadordereddictionary

How can the linewidth and facecolor of an AnchoredOffsetbox be adjusted?

Can you customize the appearance of an AnchoredOffsetbox in matplotlib? I've implemented it to display variables next to my plot, with the '=' symbols aligned vertically. It serves as an additional legend, but I'm unable to find a way ...

pythonmatplotlib

Ways to ensure that a FILE type object is able to be serialized as

When attempting to send a JSON response in Django, I encountered the following error: TypeError: Object of type File is not JSON serializable This error was caused by utilizing type <class 'django.core.files.base.File'> in my JSON respon ...

pythonjsondjango

Create a new column in a pandas dataframe that calculates the proportions of a specified column, grouped by another column

My task involves handling some data in the following manner: Grouping the data by a set of columns G Finding the proportion of a specific column within each group Returning the original data with an additional column for proportions I'm currently ex ...

pythonpandaspandas-groupby