Use Spark With the BeakerX Scala Kernel

Things under legendu.net/outdated are outdated technologies that the author does not plan to update any more. Please look for better alternatives.

1. Open a JupyterLab notebook with the BeakerX Scala kernel from the launcher.

2. Download Spark (say, 2.3.1) dependencies.

   %%classpath add mvn
   org.apache.spark spark-core_2.11 2.3.1
   org.apache.spark spark-sql_2.11 2.3.1
   

3. Create a SparkSession object.

   import org.apache.spark.sql.SparkSession
   import org.apache.spark.sql.functions._
   
   val spark = SparkSession.builder()
       .master("local[2]")
       .appName("Spark Example")
       .config("spark.some.config.option", "some-value")
       .getOrCreate()
   
   import spark.implicits._
   

4. Use Spark as usual.

   val df = Range(0, 10).toDF
   df.show