Python code to verify if a specific condition is satisfied within a certain time period

Seeking to determine if certain conditions are met over a period of time. The data is structured as follows:

Datetime Valve1 Valve2
01/01/2020 11:00:01 1 0

The condition being evaluated is: (Valve1=1 for 1h) and (Valve-0 for 1h) Utilizing rolling sum technique:

data = data.set_index('Datetime', drop=True)
data.loc[((data.Valve1.rolling('1h').sum())==?) & ((data.Valve2.rolling('1h').sum())==0), 'alarm'] = 'Yes'

Data should not be resampled or contain interpolated values. [Note]: Missing datetimes will inherit the Valve1 and Valve2 from the previous available datetime.

The resulting table would look like this:

Datetime Valve1 Valve2 Alarm
01/01/2020 11:00:01 1 0
pythonpandasdatetime

Answer №1

If you previously mentioned that data resampling was not possible, here is a way to achieve it successfully:

>>> temp_df = pd.concat([df.copy().iloc[0, :].to_frame().T, df.copy()], axis=0, ignore_index=True)
# Additional line for initialization effect
>>> temp_df.loc[0, ['Valve1', 'Valve2']] = [0, 1]
>>> temp_df['alarm'] = temp_df.Valve1.eq(1) & temp_df.Valve2.eq(0)
>>> df['alarm'] = temp_df.set_index('Datetime').rolling('1h').agg({'alarm': pd.Series.product}).replace({1: 'Yes', 0: 'No'})[1:].values
>>> df

              Datetime  Valve1  Valve2 alarm
0  2020-01-01 11:00:01       1       0    No
1  2020-01-01 11:00:15       1       0    No
2  2020-01-01 11:30:00       1       0    No
3  2020-01-01 11:30:45       1       0    No
4  2020-01-01 12:00:10       1       0   Yes
5  2020-01-01 12:15:00       1       1    No
6  2020-01-01 12:15:30       1       0    No
7  2020-01-01 12:16:45       1       0    No
8  2020-01-01 13:17:00       1       0   Yes
9  2020-01-01 13:20:15       1       0   Yes
10 2020-01-01 13:21:30       1       0   Yes
11 2020-01-01 13:45:08       1       0   Yes
12 2020-01-01 14:00:00       0       1    No
13 2020-01-01 14:01:15       0       1    No
14 2020-01-01 14:30:00       0       1    No

Give this method a try.

Answer №2

  • Utilize the groupby() method with date() and hour
  • Implement logical operations to obtain a boolean result
  • Perform a merge() operation back to your dataframe
df = pd.read_csv(io.StringIO("""Datetime    Valve1  Valve2
01/01/2020 11:00:01 1   0
01/01/2020 11:00:15 1   0
01/01/2020 11:30:00 1   0
01/01/2020 11:30:45 1   0
01/01/2020 12:00:10 1   1
01/01/2020 12:15:00 1   1
01/01/2020 12:15:30 1   1
01/01/2020 12:16:45 0   1
01/01/2020 13:17:00 1   0
01/01/2020 13:20:15 1   0
01/01/2020 13:21:30 1   0
01/01/2020 13:45:08 1   0
01/01/2020 14:00:00 0   1
01/01/2020 14:01:15 0   1
01/01/2020 14:30:00 0   1
"""), sep="\t")

df.Datetime = pd.to_datetime(df.Datetime)

dfr = df.groupby([df.Datetime.dt.date, df.Datetime.dt.hour]).apply(lambda dfa: ((dfa.Valve1==1) & (dfa.Valve1==1)).all())

df = (df.merge(dfr.to_frame(), left_on=[df.Datetime.dt.date, df.Datetime.dt.hour], right_index=True)
 .drop(columns=["key_0","key_1"])
 .rename(columns={0:"Cond"})
)
Datetime Valve1 Valve2 Cond
0 2020-01-01 11:00:01 1 0 True
1 2020-01-01 11:00:15 1 0 True
2 2020-01-01 11:30:00 1 0 True
3 2020-01-01 11:30:45 1 0 True
4 2020-01-01 12:00:10 1 1 False
5 2020-01-01 12:15:00 1 1 False
6 2020-01-01 12:15:30 1 1 False
7 2020-01-01 12:16:45 0 1 False
8 2020-01-01 13:17:00 1 0 True
9 2020-01-01 13:20:15 1 0 True
10 2020-01-01 13:21:30 1 0 True
11 2020-01-01 13:45:08 1 0 True
12 2020-01-01 14:00:00 0 1 False
13 2020-01-01 14:01:15 0 1 False
14 2020-01-01 14:30:00 0 1 False

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Select the optimal algorithm with Sklearn while efficiently managing memory concerns

My dataset consists of a .csv file with 36,000 columns (0 or 1) and 26,500 rows representing the input for my training set. Additionally, I have another .csv file with one column and 26,500 rows representing the output (0 or 1). To train and validate my m ...

pythonmachine-learningscikit-learnout-of-memoryclassification

What is the best way to manage my Python IRC bot using Twisted in an interactive manner?

Currently, I am working on a simple IRC bot using Twisted's IRC client. The code can be found at the following link: http://pastebin.com/jjMSM64n I am wondering how I could integrate the bot with the command line interface so that I can control it th ...

pythontwistedbotsirccommand-line-interface

Efficiently passing parameters to AJAX URLs in Django

When trying to pass a parameter to the URL path to specify which user's role I want to change, I keep encountering an error. NoReverseMatch at /manageuser/ Reverse for 'editrole' with keyword arguments '{'user_name': &apo ...

pythonhtmldjangoajax

Automate the process of replacing strings in a dataframe using Pandas

In my code, I have implemented a function to identify duplicates within a dataframe: def checkForDuplicates(): database_dup_first = df.drop_duplicates(subset=['name','slug','id'], keep='first') df = database_ ...

pythonpandas

Normalizing 2-dimensional input arrays using Keras

Having recently ventured into the realm of machine learning, I'm faced with a challenge in applying it to my specific problem. My training dataset consists of 44000 rows of features with a shape of 6 by 25. My goal is to construct a sequential model, ...

pythontensorflowmachine-learningkerasnormalization

A guide on incorporating recursion to nest dictionaries within current data records

Currently, I am in the process of converting XML data from Open Street Map into JSON format to upload it into a database. Due to the size of the data, I am using iterparse for parsing. However, I have encountered some tags that have a specific structure li ...

pythonjsondictionaryrecursionopenstreetmap

"Detailed ConnectionError handling is a key feature of Python requests libraries

After running the following code snippet: try: r = requests.get('http://example.com') except requests.exceptions.ConnectionError as e: print(e) I received the following output: ('Connection aborted.', RemoteDisconnected(&apos ...

pythonexceptionpython-requests

Transforming cloud-init logs into json format with a custom conversion script

I'm trying to convert the cloud-init logs into JSON format so that filebeat can collect them and send them to Kibana. I am looking to achieve this using either a shell script or a Python script. Is there an existing script that can accomplish this con ...

pythonjsonpython-3.xbash

Python2 has the site-packages folder, while Python3 does not have it

When I am logged in as the root user, I noticed the following: root@5d6f29f1d4e9:/usr/local/lib/python2.7# ls -a . .. dist-packages site-packages root@5d6f29f1d4e9:/usr/local/lib/python3.6# ls -a . .. dist-packages Upon running this command: find / ...

pythonpython-3.xubuntu

Encountering a Django issue when attempting to save data in a DateTimeField

I am facing an issue while trying to input a new entry into my Django model which includes a field of type DateTimeField. The problem arises from the fact that I am utilizing Angular on the frontend, resulting in the date format being posted to my Django V ...

pythonangularjsdjango

Encountering a "buffer too small" error when attempting to retrieve a numpy image from a gstreamer appsink

I'm attempting to extract a numpy array from a gstreamer appsink buffer, but the size of the buffer is causing issues with fitting it into an array. I came across a snippet of code on Stack Overflow that seemed promising. I am using videotestsource w ...

pythonnumpybuffergstreamer

Exploring Additional Feedback with BeautifulSoup

I am currently working on web-scraping reviews from an IMDB movie link in order to extract usernames associated with each review. However, I am facing a limitation where only 25 usernames are being retrieved due to the fact that the page displays this nu ...

pythonseleniumbeautifulsouprequest

The 'selectExpr' attribute cannot be found in the 'NoneType' object

import logging import findspark from cassandra.cluster import Cluster from pyspark.sql import SparkSession from pyspark.sql.types import StructType, StructField, StringType def setup_keyspace(session): session.execute(""" CREA ...

pythonpysparkapache-kafka

What could be causing the 500 internal server error when attempting to add users to a Google Sheet using the Google Drive API in Python?

Currently, I am in the process of developing a Python script that utilizes the Google Sheets API. This script is designed to read data from a spreadsheet, write it to a new file, share that file with a specified email address, and then return the ID of the ...

pythongoogle-apigoogle-api-python-clientgoogle-sheets-api

decrease the sum by one for each group

Here's a scenario with a dataframe: import pandas as pd import numpy as np ycap = [2015, 2016, 2017] df = pd.DataFrame({'a': np.repeat(ycap, 5), 'b': np.random.randn(15)}) a b 0 2015 0.43696 ...

pythonpandas

Guide to determining the quantity of distinct values within each group within the past n days

Below is a pandas dataframe that I am working with: groupId date value 1 2023-01-01 A 1 2023-01-05 B 1 2023-01-17 C 2 2023-01-01 A 2 2023-01-20 B 3 2023-01-01 A 3 2023-01-10 B 3 2023-01-12 C I want to perform a groupby operation and ...

pythonpandasdataframe

Using Foreign Key in AJAX POST method with Django

I'm encountering difficulties with my web app as I try to specify a user in my Ajax request. Whenever I attempt to add a new item, an error pops up: TypeError: Field 'id' expected a number but got . Is there any solution to this issue? Her ...

pythondjangoajaxdjango-views

Having trouble selecting content in an HTML document using Xpath and response.css with Scrapy?

Something is quite puzzling me and I've been pondering over it for almost a week now. Perhaps the solution is staring right at me and I just can't see it clearly... Any hints for alternative approaches would be greatly appreciated. I have n ...

pythonhtmlcssxpathscrapy

Python script for deleting empty lines from txt/srt files in a directory and its subfolders

I have a collection of subtitle files that follow this specific format. 1 00:00:01,000 --> 00:00:02,008 some sample text 2 00:00:02,008 --> 00:00:05,006 some sample text some sample text 3 00:00:05,006 --> 00:00:08,008 some sample text some s ...

pythontext-processing

Python fastAPI and MongoDB environment allows the return of a tuple in the BaseModel list

Currently, my setup includes env with python311, pydantic, fastapi, and mongod. return membercollection(members=await c_members.find(queryparam).to_list(1000)) This code snippet retrieves the following information: members=[membermodel(id='65b3908a77 ...

pythonmongodbfastapipydantic