Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

To start an interactive Spark shell use:


Code Block
$ module use -a  /lrz/sys/share/modules/extfiles
$ module load python or module load python/3.5_intel
$ module load spark
$ pyspark



PySpark in a Jupyter Notebook


Code Block
$ module use -a  /lrz/sys/share/modules/extfiles
$ module load python or module load python/3.5_intel
$ module load spark
$ export PYSPARK_DRIVER_PYTHON=jupyter
$ export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
$ pyspark


Now, this command should start a Jupyter Notebook in your web browser. Create a new notebook by clicking on ‘New’ > ‘Notebooks Python [default]’.