To start an interactive Spark shell use:
| Code Block |
|---|
$ module use -a /lrz/sys/share/modules/extfiles $ module load python or module load python/3.5_intel $ module load spark $ pyspark |
PySpark in a Jupyter Notebook
| Code Block |
|---|
$ module use -a /lrz/sys/share/modules/extfiles $ module load python or module load python/3.5_intel $ module load spark $ export PYSPARK_DRIVER_PYTHON=jupyter $ export PYSPARK_DRIVER_PYTHON_OPTS='notebook' $ pyspark |
Now, this command should start a Jupyter Notebook in your web browser. Create a new notebook by clicking on ‘New’ > ‘Notebooks Python [default]’.