Exploring DataFrames with interrows() and writing them out as CSV files with .to_csv:

I am using the following script to perform the following actions:

  • Apply a function to a column in each row of a DataFrame
  • Write the returns from that function into two new columns of a DataFrame
  • Continuously write the DataFrame into a *.csv

I am interested in finding out if there is an improved method for executing this computation:

df = a DataFrame with 500 rows and 20 columns

for index, row in df.iterrows():
    df.loc[index, 'words'], df.loc[index, 'count'] = transcribe(df.loc[index, 'text'])
    df.to_csv('out.csv', encoding='utf-8', index=False)

Currently, the script outputs the full df dataframe as *.csv every time (for each row), including the added values for the computed rows "words" and "counts" up to that point. I would like to know if it is possible to only output complete lines in the csv file, rather than the entire dataframe.

Thank you!

pythonpandasloopscsv

Answer №1

It puzzles me why you prefer to process the dataframe row by row instead of writing it out in one go, but here's a solution tailored to your request: save slices of the dataframe (i.e. the current row) using append mode, including the header only for the initial row:

is_first_row = True
for index, row in df.iterrows():
    df.loc[index, 'words'], df.loc[index, 'count'] = transcribe(df.loc[index, 'text'])
    df.loc[index:index].to_csv('out.csv', encoding='utf-8', index=False, mode='a', header=is_first_row)
    is_first_row = False


Update based on comment that script could be interrupted:
If interruptions are a concern, you can decide whether to include the header by checking if the file exists or is new:

with open('out.csv', encoding='utf-8', mode='a') as f:
    for index, row in df.iterrows():
        df.loc[index, 'words'], df.loc[index, 'count'] = transcribe(df.loc[index, 'text'])
        df.loc[index:index].to_csv(f, index=False, header=f.tell()==0)

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Using JSON.load with a one-liner JSON isn't going to give you the desired result

I am dealing with JSON data that comes in two different formats - one is a single line, the other is formatted nicely. JSON A: {"id":1, "name":"BoxH", "readOnly":true, "children":[{ "id":100, "name":"Box1", "readOnly":true, "children":[ { "id":1003, "nam ...

pythonjson

Tips for launching and controlling new tabs using selenium

How can I open a new tab with the 'https://www.gmail.com' url, extract some information, and then return to the original page using Python 3.8.5? I am currently opening the new tab with CTRL + t command, but I'm unsure how to switch between ...

pythonseleniumwebdriverselenium-chromedriver

HTML/JavaScript: Embrace the Power of Dynamic Page

I have a unique element in my HTML code: <image src="http://..." style='...'> Using Python-Flask, I pass on a dynamic source address and save it as window.dynamicEmbedding. Now, during page load, I want to change the image's ...

javascriptpythonhtmldynamicembedding

Guide to eliminating any negative numbers from a list using Python's lambda functions

After successfully implementing a lambda function to sort a list, I am now looking to remove all the negative objects from the list using lambda functions. dto_list.sort(key=lambda x: x.count, reverse=True) Is there anyone who knows how to write the lamb ...

pythonlistlambda

Is it possible to apply a CSS property to an XPath locator in order to guarantee the visibility of an element?

One of the challenges I'm facing involves a series of div blocks that can be clicked to reveal hidden content in an accordion format. Using Selenium, my goal is to cycle through these blocks, opening each one before capturing a screenshot. However, I ...

pythonxpathseleniumwebdriver

Refine the pandas Dataframe with a filter on a JavaScript-enabled website

I recently inherited a large software project using Python/Flask on the backend and HTML/Javascript on the frontend. I'm now looking to add some interactivity to one of the websites. I have successfully passed a dataframe to the webpage and can displa ...

javascriptpythonhtmlpandasflask

Tips for providing a dynamic value in an XPATH element with Python Selenium

game = "Chess" driver.find_element(By.XPATH, "//div[@title = {game}]").click() I'm having trouble making my code function properly. Can you provide guidance on how to insert a dynamic variable into the XPATH? ...

pythonseleniumxpath

Tips for accessing an IP camera with OpenCV

Having trouble reading an IP camera stream. Currently, I can only view the video through Internet Explorer because of the ActiveX plugin requirement. The camera is located at 192.168.0.8:8000. Take a look at the image below https://i.stack.imgur.com/kvmA9. ...

pythonopencvip-camera

"Utilizing Python to extract data from JSON and determine the

I have extracted a .json file from Wireshark containing the following instance: "_source": { "layers": { "frame": { "frame.encap_type": "1", "frame.time": "Jan 23, 2018 10:3 ...

pythonjsonparsingwireshark

Nodejs installation failure detected

I am dealing with a package node and I am looking to install it using the following command: ./configure --prefix=/path/NODEJS/node_installation && make && make install However, when I run this command, it returns the error below: File ...

pythonnode.js

Exploring the process of file documentation using the statement 'import * from x'

Is it possible to utilize sphinx's automodule:: and other automatic features to document modules that contain from x import * statements without including all of the documentation from the imported modules? UPDATE: According to mzjn's observatio ...

pythonpython-sphinxmatplotlib

matching parentheses in python with stack algorithm

I'm working on implementing a stack to validate parenthesis, but I'm having trouble getting the correct output with the code below. Despite reviewing it multiple times, I can't seem to identify the mistakes. Any suggestions or assistance wou ...

pythonstacksingly-linked-list

Using Python Selenium to Download an Image to a Local Directory

What is the best way to transfer an image from a source to a local folder using Python with Selenium? I could use some assistance, thank you in advance. ...

pythonseleniumselenium-webdriverselenium-ideselenium-rc

"Access Denied": PyCharm's Struggle with Setting Up an Anaconda Environment

Even after granting PyCharm root permissions, I am still experiencing a "permission denied" error message when trying to create a Conda environment. This issue has persisted despite my initial success in setting up the environment. ...

pythonpycharmanacondaconda

Passing JSON data from template to view in Django

I'm facing an issue with sending JSON data via AJAX from the template to the view and storing it directly in the database. The problem lies in the data not reaching the view successfully. When I try to insert my JSON data (json_data['x'], js ...

jquerypythonajaxjsondjango

Reducing the number of features of a single image during the inference process

I am currently working on training a SVM classifier with scikit-learn. During the training process, I need to decrease the dimension of the feature vector. To achieve this, I have utilized PCA for dimensionality reduction. pp = PCA(n_components=400).fit(fe ...

pythonscikit-learnpcafeature-extraction

Using Python to display data stored in a JSON object's "key" field

I've run into a bit of an issue with my scraping code: import urllib import re import json htmltext = urllib.urlopen("http://dx.com/p/GetProductInfoRealTime?skus=48616") htmltext = json.load(htmltext) print htmltext The output looks like this: { ...

pythonjsonkeyweb-scraping

Guide for retrieving input from a URL using Python

I have a hyperlink that includes email addresses and I am interested in counting the number of emails with the same domain. The input needs to be taken from the URL provided. import requests def finddomains(input_): domain_frequency = dict() ...

pythonautomation

Can you provide me with instructions on how to navigate through the JSON response in order to access a

I've included a sample of the JSON structure for reference. My goal is to display information about each driver individually, but I'm only able to access the "Drivers" key. { "MRData": { "xmlns": "http://ergas ...

pythonjsondictionary

Could this site be inhibiting my scraping efforts using BeautifulSoup?

For the past few years, I've been utilizing BeautifulSoup to extract TopCashBack website links. However, when I attempt to change the URL to a Screwfix link, I am not able to retrieve any data. s = requests.get("https://www.screwfix.com/p/128hf&q ...

pythonweb-scrapingbeautifulsoup