I am trying to access all correios agency information in Brazil by web scraping using Selenium. However, I am encountering some issues with utf-8 encoding in my code. How can I resolve this and set everything to utf-8 format? Here is the code snippet I am working on:
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.select import Select
import time
chrome_path = r"D:\Users\gbenites\Desktop\Inovação\arquivos py\WebDriver\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("http://www2.correios.com.br/sistemas/agencias/")
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.ID, 'municipioAgencia')))
driver.maximize_window()
time.sleep(3) #this time allows the page load all the elements
estadoList = Select(driver.find_element_by_name('estadoAgencia'))
for index in range(1,len(estadoList.options)):
select = Select(driver.find_element_by_name('estadoAgencia'))
print("selecting Estado",select.options[index].text)
select.select_by_index(index)
time.sleep(2)
municipioList = Select(driver.find_element_by_name('municipioAgencia'))
for index_b in range(1,len(municipioList.options)):
select_b = Select(driver.find_element_by_name('municipioAgencia'))
print("...selecting Municipio",select_b.options[index_b].text)
select_b.select_by_index(index_b)
time.sleep(0.05)
Below is the error message I am facing:
selecting Estado ACRE
...selecting Municipio ACRELANDIA
...selecting Municipio ASSIS BRASIL
[Decode error - output not utf-8]
...selecting Municipio AGUA BRANCA
...selecting Municipio ANADIA
...selecting Municipio ARAPIRACA
Whenever a State with utf-8 error occurs, it repeats the counties. How can I ensure that the State info is encoded in utf-8?