Python Selenium encountering a UnicodeEncodeError issue

Whenever I write and execute my code in PyCharm, everything runs smoothly. However, when I try to open the code in standard Python, an error occurs. Strangely, it works perfectly fine in PyCharm.

from selenium import webdriver
import time
import random

print("\n")
user_input = input("Username: ")

##########################################################
path = r"C:\Users\John\Desktop\chromedriver.exe"
driver = webdriver.Chrome(path)
##########################################################

text_file = open(user_input +  str(random.random()) + ".txt", "w")
text_file.write("GoogleSearch:\n\n")
##########################################################
print("Google results:\n")
driver.get("https://www.google.com/#q=" + user_input)
for n in range(20):
    try:
        driver.find_element_by_xpath("""//*[@id="pnnext"]/span[2]""").click()
    except: print("out of pages")
    pass
    time.sleep(2)
    posts2 = driver.find_elements_by_class_name("_Rm")
    for post2 in posts2:
        print(post2.text)
        text_file.write(post2.text + "\n\n")


print("\n")
print("Pipl results:\n\n")
text_file.write("\n\n")
text_file.write("Pipl results:\n\n")
driver.get("https://pipl.com/search/?q=" + user_input + "&l=&sloc=&in=5")
posts1 = driver.find_elements_by_class_name("line1")
for post1 in posts1:
    print(post1.text)
    text_file.write(post1.text + "\n")

time.sleep(1)
driver.close()
Traceback (most recent call last):
  File "C:\Users\John\Desktop\peopleSearchTool.py", line 32, in <module>
    print(post2.text)
  File "C:\Users\John\AppData\Local\Programs\Python\Python35-32\lib\encodings\cp437.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u203a' in position 23: character maps to <undefined>

Answer №1

It's likely that the issue stems from a codepage discrepancy. PyCharm runs your program in a console using a Unicode codepage, while a standalone run operates in a console with a codepage 437 (which dates back to the 1980s). The Unicode codepage includes a character with the code #203a, whereas codepage 437 does not as it is limited to only 256 characters.

In order to display a string with characters above FF, you will need to encode/decode your string or adjust your console's codepage. Be aware that both methods may present various challenges.

The ideal solution is to stick with Unicode and avoid printing to non-Unicode consoles altogether.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

TimeoutException raised with message, screen, and stacktrace

I am a beginner when it comes to Python and Selenium, and I was attempting to try out an example code that I found on YouTube. Here is the code snippet: from selenium import webdriver from selenium.webdriver.common.keys import Keys from selenium.webdriver ...

Scraping with selenium: The element could not be found due to the absence of such element error

I need help with scraping data from a sports betting site for my analysis. I keep encountering the error "no such element: Unable to locate element". Is it possible that betting sites prohibit scraping their data? Some suggestions mention issues with ifram ...

Employing the split() method within a pandas dataframe

Here is the dataframe I am working with: https://i.stack.imgur.com/NgDRe.png To remove the percentage signs, I attempted to use a function on the Democrat and Republican columns by splitting them at the percentage sign. This is the code snippet I used: ...

Python: Exploring JSON Data

Is there a way to extract data specific to a district (such as Nicobars, North and Middle Andaman...) from ? I am trying to extract the Active cases by simply searching for the district name without having to specify the state every time. Currently, I&apos ...

Discover how to extract JSON data from a webpage even when the response is "None"

I need to retrieve data maps from their API, which are in JSON format. This is the code I have written: r = requests.get(url) if r.ok: soup = BeautifulSoup(r.content, 'lxml') soup.status_code j = json.loads(str(soup)) However, an e ...

Failure during the installation of a Python package due to an ImportError

I followed the installation guide for django-wiki from However, when I run 'python manage.py migrate', I encounter this error: Error message here... Strangely, when I import it using the python shell... >>> from html5lib.constants im ...

Tips for accessing the bottom of a div element in Selenium using C#

I am currently working on scraping reviews from a specific location, let's say this link. The challenge I am facing is that I need to scroll down to the bottom of the div in order to load all the reviews before extracting them into the database. IJa ...

Selenium: A guide to specifying CSS for webpages with multiple elements sharing the same name

Looking for assistance with CSS selection: <table id="userPlaylistTable" cellspacing="0" cellpadding="0"> <div class="numCellWrapper"> ... When defining a css selector, how can I target a specific element among multiple elements of the same ...

extracting information with beautifulsoup

Currently, I am diving into BS4 to enhance my skills and expertise. My goal is to scrape various tables, lists, and other elements from well-known websites in order to grasp the syntax. However, I am encountering difficulties when it comes to formatting a ...

Assigning a value to a text field in Selenium results in an error message stating that the element cannot be found: NoSuchElementException - "Unable to locate element

I am diving into the world of Web scraping and Python for the first time. My task involves setting the value in a text box named "rcdate" on a specific URL using Selenium, and then extracting the filtered values. When trying to run this code snippet, an ex ...

Selenium is unable to locate the XPath, yet it functions properly with jQuery

After encountering an issue with locating and clicking on a specific element using XPath in WebDriver, I discovered an interesting difference between the console in Chrome DevTools and the Find function. The XPath that was causing trouble was: //table[@t ...

Dealing with changing elements using cucumber and capybara

(Click for image) I've been working on a project where I need to create a scenario to test the login feature. However, I've encountered an issue with Capybara not being able to access dynamic elements. To Reproduce the Issue: 1) Go to Redfin.co ...

Conflicting submissions

Can anyone help me with a JavaScript issue I'm facing? On a "submit" event, my code triggers an AJAX call that runs a Python script. The problem is, if one submit event is already in progress and someone else clicks the submit button, I need the AJAX ...

TestNG priority annotations were disregarded

Within the testNG.xml file, there are over 10 test classes grouped under a test-suite tag for regression testing. Each test class has been assigned priorities using the priority=xxx annotation in @Test to determine the order of execution. The priority valu ...

Sorting numbers based on their relationships in Python using clustering techniques

I am looking for a solution that is similar to the one discussed here, but with a different approach: Instead of calculating the maximum gap between numbers in a list (like 10 in the example provided), I want to specify how many groups I would like the nu ...

Guidelines for requesting integer input from the user with Python's tkinter library

I'm facing an issue with getting the correct integer value input from a user in Python tkinter. I've tried using a=n.get() and then converting it to an integer using a=int(a), but it's not working as expected. Below is the code snippet I use ...

Looking to retrieve information from a dataframe in a specific row

Here is the data that I currently have: Year Month 2003 06 2003 09 I need to extract this data in the following format: (('2003', '06'),('2003','09')) ...

The problem with my Chrome driver is that it is malfunctioning due to an undetectable Chrome browser

I recently encountered an issue with my script that was running smoothly until I decided to use the undetected chrome browser. After installing it, I received the error message: "WebDriver.init() got an unexpected keyword argument 'chrome_options&apos ...

When should you invoke django.setup()?

Can someone help me understand the purpose of django.setup()? Should I include it in my settings file? I encountered an issue when calling apps.get_models() and received the following error message: django.core.exceptions.AppRegistryNotReady: Models aren ...

Before the completion of the initial test, the second selenium test has already been initiated

I am facing an issue while running two Selenium tests sequentially. I have implemented waiters in the tests to wait for elements to become visible before interacting with them. Sometimes, the second test starts executing while the first test is still waiti ...