Thursday, November 5, 2015

Building Spark Notebook for MapR 5.0

Env : MapR 5.0
Spark : 1.4.1

git clone https://github.com/andypetrella/spark-notebook.git

// add mapr repo to sbt proxy repo
vim ~/.sbt/repositories
[repositories]
local

maven-central

typesafe: http://repo.typesafe.com/typesafe/releases/

typesafe-snapshots: http://repo.typesafe.com/typesafe/snapshots/

mapr: http://repository.mapr.com/maven

cd spark-notebook
sbt -Dspark.version=1.4.1 -Dhadoop.version=2.7.0-mapr-1506 -Dwith.hive=true -Dwith.parquet=true clean dist

add env variables

$ export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
$ export EXTRA_CLASSPATH=/opt/mapr/lib/commons-configuration-1.6.jar:/opt/mapr/lib/hadoop-auth-2.7.0.jar:/opt/mapr/lib/maprfs-5.0.0-mapr.jar:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/zookeeper-3.4.5-mapr-1503.jar

copy spark assembly jar to the /apps/spark
hadoop fs -put /opt/mapr/spark/spark-1.4.1/lib/spark-assembly-1.4.1-hadoop2.5.1-mapr-1501.jar /apps/spark/spark-assembly.jar


now start notebook using sbt run
[info] play - Listening for HTTP on /0:0:0:0:0:0:0:0:9000

(Server started, use Ctrl+D to stop and go back to the console...)
the message indicate that everything went smooth and notebook has started on port 9000. to access notebook open a browser and type address http://:9000 and u are good to go
Post a Comment