When comparing timestamps for the same date, Python's datetime and pandas may produce varying results

from datetime import datetime
import pandas as pd

date="2020-02-07T16:05:16.000000000"

#Using datetime to convert date format
t1=datetime.strptime(date[:-3],'%Y-%m-%dT%H:%M:%S.%f')

#Converting date using Pandas
t2=pd.to_datetime(date)

#Calculating the difference between dates
print(t1-t2)

#Calculating the difference between timestamps
print(t1.timestamp()-t2.timestamp())

This code snippet demonstrates converting dates with both the `datetime` and `pandas` libraries while assuming timezone-naive dates. However, a discrepancy is observed where the difference between the dates is zero, but the difference between the timestamps shows variation due to timezone offsets from GMT.

Answer â„–1

When dealing with naive datetime objects in Python which belong to the datetime.datetime class, it's essential to remember that they represent local time. While this might seem straightforward, it can still be puzzling at times. Interestingly, calling the timestamp method on a naive datetime object returns a POSIX timestamp related to UTC (seconds since the epoch).

The behavior of a naive pandas.Timestamp may not be as intuitive when compared to the Python datetime object. If derived from a tz-naive string in a similar manner, a pandas.Timestamp will actually represent UTC rather than local time. This distinction becomes evident when you localize the datetime object to UTC:

from datetime import datetime, timezone
import pandas as pd

date = "2020-02-07T16:05:16.000000000"

t1 = datetime.strptime(date[:-3], '%Y-%m-%dT%H:%M:%S.%f')
t2 = pd.to_datetime(date)

print(t1.replace(tzinfo=timezone.utc).timestamp() - t2.timestamp())
# 0.0

Conversely, you have the option to make a pandas.Timestamp timezone-aware, like so:

t3 = pd.to_datetime(t1.astimezone())
# e.g. Timestamp('2020-02-07 16:05:16+0100', tz='Mitteleuropäische Zeit')

# confirming that both t1 and t3 now represent my local time:
print(t1.timestamp() - t3.timestamp())
# 0.0

In conclusion, if you are certain about the time zones represented by your timestamps, it is advisable to work with timezone-aware datetime objects. For instance, for handling timestamps in UTC:

import pytz # using pytz since pandas relies on it internally

t1 = datetime.strptime(date[:-3], '%Y-%m-%dT%H:%M:%S.%f').replace(tzinfo=pytz.UTC)
t2 = pd.to_datetime(date, utc=True)

print(t1 == t2)
# True
print(t1-t2)
# 0 days 00:00:00
print(t1.timestamp()-t2.timestamp())
# 0.0

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Establishing the framework for djangoappengine

I've been attempting to configure djangoappengine. After cloning the code from git / bitbucket according to the instructions provided here (following various links on Google), I duplicated django-testapp and modified it for my own project. However, w ...

Multiple subplots showcasing the union of two dataframes

I am working on creating 5 different subplots, each representing a year's worth of data. These subplots will display the DEMs and REPs for each county within that specific year. The code I have come up with so far is as follows: fig = plt.figure(figs ...

What are some effective ways to extract additional information from lxml error messages?

Due to the inability to use an XSL IDE, I have developed a simple Python script utilizing lxml in order to convert XML files using a specified XSL transformation and save the output to a new file. Here is the code snippet (excerpted): p = XMLParser(huge_t ...

The CSV writer in Python has a tendency to include extraneous quotation marks

I'm struggling to generate a CSV file with the desired output format: 14897,40.50891,-81.03926,168.19999 However, despite my efforts, the CSV writer keeps enclosing the values in quotes like this: '14897,40.50891,-81.03926,168.19999' Alt ...

What is the method for using a Python turtle to illustrate the Collatz sequence as a line graph?

def generate_sequence(number): if number <= 0: print("Zero or negative numbers are not even, nor Odd.", "Enter number >", number) else: print(int(number)) while number != 1: #number is e ...

Is it possible to achieve compatibility between Python and PHP in web development?

I am currently engaged in a legacy PHP project that relies heavily on PHP for its backend operations. However, I have a strong interest in scripting and developing fun and useful features using Python. My question is: Is there a way to incorporate Python ...

Every 10 minutes, the Linux pip command seems to incur unusually high CPU usage

After noticing a spike in Google Cloud Server's CPU usage every 10 minutes, I decided to investigate using the top command. Check out the Server CPU Usage here The culprit seems to be the pip command causing high CPU usage at regular intervals. I a ...

Flask encountering issues when parsing correct arguments in HTTP request

I am currently facing an issue with my flask server running through gunicorn and nginx where it is not parsing certain characters like the plus sign (+) correctly in the arguments of the http messages. For instance, when I check the nginx log, I see: [04 ...

What is the best way to divide the Flask directories within app.route?

Confusion Alert: The title might not make sense, but I'm stumped on how to put it into words. I've set up an app.route in Flask for handling some assets. @app.route('/assets/css/<path:path>') def send_css(css_file): return send ...

Utilizing Selenium WebDriver in Python to locate and interact with WebElements

Currently, I am utilizing Selenium for real-time web scraping. However, I am encountering issues searching for a specific WebElement even though the documentation indicates that it is feasible. while True: try: member = self.pdriver.find_all(" ...

What is the reason behind the recursion caused by calling list() on an instance that has an __iter__() method?

Currently, I am coding in Python 3.7 and facing an issue with the following code snippet: class Foo: def __iter__(self): yield from self.some_sequence def __len__(self): return len(list(self)) >>> len(Foo()) Traceback ( ...

Automating the process of expanding all dropdown menus using Selenium

I am trying to open all the dropdown menus on this webpage in order to access the href links for the different subcategory pages. My main obstacle is figuring out how to do this systematically because of the dynamic nature of the site and the difficulty I& ...

Should I employ Scrapy Selenium to scrape the initial request page?

I have successfully implemented a solution using scrapy_selenium to scrape a website that uses JavaScript for loading content. In the code snippet provided below, you can see that I am using SeleniumRequest when yielding detailPage with parseDetails. Howe ...

What could be causing my code to only recognize the first condition and overlook the second one?

Can someone help me understand why my code is only evaluating the first condition and not checking for the second condition as well? def identifyWeapon(): if keyPress.positive: if raySensor.hitObject['primaryWeapon']: ...

Query to retrieve all IDs contained within the list stored in TinyDB

In my python project, I am utilizing TinyDB to enhance the logic. To improve efficiency, I need to be able to retrieve multiple objects in a single query, preferably using a list. listOfIDs = ['123', '456'] I am working with the lates ...

Ways to eliminate tuples from a tuple list?

As I contemplate the cards in my hand, a sense of disappointment washes over me, urging me to abandon them and draw new ones. But how does one go about achieving this task? I find myself unable to dispose of these troublesome tuples - removing them with d ...

Incorporating a color-coded legend onto a Folium map for

When creating a map in Folium with multiple layers, each containing shaded areas (utilizing GeoJSON) colored by a colormap, I encountered an issue with adding legends to my layers. An initial solution was found here, but it posed problems as the legend rem ...

I recently created a Python3 script utilizing selenium to automate message sending, but unfortunately, it is not functioning as expected

I've written a script (https://pastebin.com/dWLFvirn) that I'm trying to run through the terminal, but it's giving me an error. Here's a screenshot of the error (https://ibb.co/NSfLgL8). I would like to use "" as the link to make the s ...

Guide for setting the executable path of the Chrome driver to match the operating system. My goal is to create a generic executable path for the Selenium driver using Python

What is the best way to define the executable path of the Chrome driver to match the operating system's path? I am aiming to create a universal executable path that works across all systems for the Selenium with Python driver. ...

Notification: The specific HTML element identified as <div id="tabber_obj_0_div_3" class="ptr"> cannot be smoothly brought into view by scrolling

I am currently working on extracting book reviews from a specific URL using web scraping techniques. My goal is to gather the book name and each review in separate rows. Here's the code snippet I have written so far, which utilizes both selenium and b ...