Rodeo encounters a version conflict between the worker and driver when running pySpark

When executing this basic script in Pyspark from the terminal, it runs without any issues:

import pyspark

sc = pyspark.SparkContext()
foo = sc.parallelize([1,2])
foo.foreach(print)

However, running the same script in Rodeo results in an error message, with a key line stating:

Exception: Python in worker has different version 2.7 than that in driver 3.5, PySpark cannot run with different minor versions

You can view the complete error output by following this link: http://pastebin.com/raw/unGuGLhq

I have configured my$SPARK_HOME/conf/spark-env.sh file with the following lines:

export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=python3

Despite these settings and also adding them to ~/.bashrc, the issue persists.

Rodeo version: 1.3.0

Spark version: 1.6.1

Operating System: Linux

pythonapache-sparkpysparkrodeo

Answer №1

There seems to be an issue that is related to the one discussed in a previous topic.

When using Rodeo as a desktop application, it struggles to work with shell environment variables. To solve this, instead of declaring variables in the usual spark-env.sh file, it is recommended to add them to Rodeo's .rodeoprofile file using the os module. In this particular case, adding the following lines to the .rodeoprofile file resolved the issue:

os.environ["PYSPARK_PYTHON"]="python3"
os.environ["PYSPARK_DRIVER_PYTHON"]="python3"

(even though the second line may be considered redundant since the driver already used Python 3.5)

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Issue with WTForms form submission: validation errors not displayed despite form not being properly submitted

I'm having issues trying to implement file uploads using flask-uploads. I've encountered some obstacles while working on it and I need some assistance. Below, I will display my flask view function along with the HTML code, in hopes that someone c ...

pythonflaskwtforms

Unpacking a lengthy array in Psycopg2 for efficient insertion into a column in a Postgres table

I am currently managing a local postgres database. The challenge I am facing involves an array that looks like this: [1,2,3,...2600] Because the array is extremely long, manually typing out each element for insertion is not a practical option. That' ...

pythonsqlpostgresqlpsycopg2

The error message "TypeError: 'bool' object is not iterable while attempting to use all()" is indicating that a boolean

I wrote a code to determine if a list is monotonically increasing or decreasing. However, I encountered the following error message: TypeError Traceback (most recent call last) <ipython-input-64-63c9df4c ...

pythonpython-3.x

Send a JSON payload to an API endpoint using the requests module

Currently, I am in need of sending a JSON object as input to an API in order to receive another JSON response. My usual method for accessing an API using requests is shown below: response=requests.get('url',auth=uauth,headers=headers).json() B ...

pythonjsonpython-requests

Is there a way to verify the status of a checkbox in Selenium using Python?

When attempting to determine the status of a checkbox using a method that returns a boolean value, I am encountering a problem where it always returns false regardless of whether the checkbox is checked or unchecked. def checkbox_status(self): ch ...

pythonpython-3.xselenium-webdriver

Mastering the art of selecting a web element using xpath with Python and Selenium

Struggling with web scraping on a site featuring this specific structure. <div> <div class = “class1” > <div class = “class2” > <div class = “class3” > <div style = “clear: both; ” > </div&g ...

pythonseleniumxpathweb-scrapingsiblings

PyInstaller reminds you that you have the option to load I/O plugins by using the `skimage.io.use_plugin` function

When using Pyinstaller to package a Python script Pyinstaller version 3.2 Operating System: Ubuntu Traceback (most recent call last): File "<string>", line 57, in <module> File "<string>", line 29, in feature_extract File "caffe/ ...

pythonpyinstaller

Tips for modifying an LpVariable using an integer

I am currently utilizing pulp, a third-party Python library, for linear programming. One of the constraints I encountered is regarding the use of the modulus operator with LpVariable: from pulp import * x1 = LpVariable('x1', 1, 5, cat='Int ...

pythonlinear-programmingpulpinteger-programming

Automate button clicking on web pages using Python with Selenium

While I am able to access the tag below through css_selector, I am unsure how to perform a button click. <a title="Sign In" onclick="toggleLogin('signInBlock','regBenefitsBlock');" href="javascript:void(0);">Sign In</a> I ...

pythonseleniumwebdriver

Could there be a more efficient method for accessing the file referenced in a Python stack trace?

When faced with a Python stack trace in either Output or Terminal, how can one quickly navigate to the specific file or line where the error occurred? I've noticed that Ctrl-click doesn't work for this purpose. I've been using ctrl-e and man ...

pythonvisual-studio-code

Modifying a single character in a string using Python

Do strings have mutability in Python? Trying to assign a value like someString[3] = "a" results in an error message: TypeError: 'str' object does not support item assignment I understand why this isn't allowed (as doing something like ...

python

Locate child elements within a parent class that includes the term "hijack" using

Check out this page: Extract data from: <tr> <td class="event_type">Possible Hijack</td> <td class="country"> </td> <td class="asn">; <i>Expected Origin AS:</i ...

pythonparsingweb-scrapingbeautifulsoup

Error: The host is experiencing an AttributeError because the 'Service' object does not have the attribute 'process' available

Looking to scrape a website using Python on a host without GUI. Unfortunately, the webpage I want to scrape ( ) does not allow scraping with the requests library, so I have to resort to using Selenium with a headless webdriver. (Apologies for any language ...

pythonseleniumcpanelhostheadless-browser

Changing the color of the plot border

Does anyone know how to change the border color from dark gray to white? I've attempted a few methods without success. Below is the code and image: Image here import numpy as np import matplotlib.pyplot as plt plt.style.use('classic') from ...

pythonmatplotlibhistogramfigure

Is there a way to determine the quantity of boxes based on their specific placement?

Currently working on a python memory game, I am struggling with determining the number of boxes that the user clicks based on the cursor's position. This is what I have so far: number = ev.pos[y]//boxsize*numboxsx+ev.pos[x]//boxsize (this calculati ...

pythonpygame

A situation where a complex query involves multiple variables being reused and formatted using the %s string format

I stumbled upon a helpful thread on SO which got me thinking. Currently, I'm utilizing psycopg2 to run SQL queries using strings: import credentials as creds import psycopg2 start_date = '2020-01-01' end_date = '2020-01-02' anothe ...

pythonpython-3.xformattingpsycopg2

What causes the discrepancy in height options for tkinter buttons between OSX and Windows operating systems?

I'm encountering an issue with my code that involves the following line: self.button = Button(frame, text="CALCOLA", width=28, height=2, command=callback) When running the code on OSX, the button appears as a single line separated above and below fr ...

pythonwindowsmacosbuttontkinter

What is the best way to extract all labels from a column that has been one hot encoded?

Converting One Hot Encoded Columns to Multi-labeled Data Representation. I am looking to transform over 20 one hot encoded columns into a single column with label names, while also considering the fact that the data is multi-labeled. I aim for the label co ...

pythondataframenlpdataset

Use SQLite3 in Python to remove rows that match the specific date and time in the format yyyy/mm/dd hh:mm:ss

I have a large SQLITE3 database with the following structure: https://i.stack.imgur.com/G9GQX.jpg The "date_" field in the database is formatted as YYYY/MM/DD HH:MM:SS I am looking to remove all rows before the year 2020. I attempted to achieve this by ...

pythonsqlitedatetimesql-deletepython-datetime

Emphasize the cells of a table with bold text

Here is the code I am working with: table = document.add_table(rows=1, cols=8) hdr_cells = table.rows[0].cells hdr_cells[0].text = 'Name' hdr_cells[1].text = 'Surname' hdr_cells[2].text = 'Telephone' (...) I'm wondering ...

pythonpython-docx