Currently, I am utilizing python with pyspark for my projects.
For testing purposes, I operate a standalone cluster on docker.
I found this repository of code to be very useful.
It is important to note that before running the code, you must execute this command in order to log into it properly:
docker network create --gateway 10.5.0.1 --subnet 10.5.0.0/24 spark_master
As I interact with both the worker and master, I noticed that when I check the Python version using the command:
which python
The Python versions appear similar (3.5).
However, when executing a basic pyspark code outside of the containers like this:
from pyspark import SparkContext, SparkConf
conf = SparkConf().setAppName('appName').setMaster('spark://0.0.0.0:7077')
sc = SparkContext(conf=conf)
rdd = sc.parallelize([1, 2, 3, 4])
rdd.map(lambda x: x**2).collect()
An error consistently occurs which states:
Exception: Python in worker has different version 3.5 than that in driver 3.7, PySpark cannot run with different minor versions. Please ensure that environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
After consulting resources online, such as this link, it was recommended to make sure the driver and worker Python versions match. Despite setting up a new conda environment with Python 3.5, the issue persisted.
Moreover, I attempted to adjust environment variables using os.environ by adding the following lines to my Python code:
os.environ["PYSPARK_PYTHON"] = "/usr/bin/python3.5"
os.environ["PYSPARK_DRIVER_PYTHON"] = "/usr/bin/python3.5"
os.environ["PYSPARK_WORKER_PYTHON"] = "/usr/bin/python3.5"
Unfortunately, the error remains unchanged. It seems that while trying to include a missing path, a new error emerged indicating "no such file or directory." This suggests that the code runs within the cluster but does not address the root problem.