Apache Spark : JDBC connectivity with MySQL

ENV: Apache Spark-1.4.1 with spark-shell

scala> def getConnection(){
     | Class.forName("com.mysql.jdbc.Driver").newInstance();
     | DriverManager.getConnection("jdbc:mysql://localhost:3306/test?user=root");
     | }
scala> def convertRS(rs:ResutlSet){
     | (rs.getInt(1),rs.getString(2))
     | }
scala> val result = new JdbcRDD(sc,getConnection,"SELECT * FROM emp WHERE ? > empid AND empid <= ?",0,20,2,maprow=convertRS)
scala> result.collect().toList

Apache Spark : Reading and Writing Sequence Files

Writing a Sequence file:

scala> val data = sc.parallelize(List(("key1", 1), ("Kay2", 2), ("Key3", 2)))
data: org.apache.spark.rdd.RDD[(String, Int)] = ParallelCollectionRDD[7] at parallelize at :27
scala> data.saveAsSequenceFile("/tmp/seq-output")

The output can be verified using hadoop ls command

[root@maprdemo sample-data]# hadoop fs -lsr /tmp/seq-output
lsr: DEPRECATED: Please use 'ls -R' instead.
-rwxr-xr-x   1 root root          0 2015-10-02 01:12 /tmp/seq-output/_SUCCESS
-rw-r--r--   1 root root        102 2015-10-02 01:12 /tmp/seq-output/part-00000
-rw-r--r--   1 root root        119 2015-10-02 01:12 /tmp/seq-output/part-00001
[root@maprdemo sample-data]# hadoop fs -text /tmp/seq-output/part-00001
Kay2 2
Key3 2

Reading Sequence file

scala> import org.apache.hadoop.io.Text
import org.apache.hadoop.io.Text
scala> import org.apache.hadoop.io.IntWritable
import org.apache.hadoop.io.IntWritable
val result = sc.sequenceFile("/tmp/seq-output/part-00001", classOf[Text], classOf[IntWritable]). map{case (x, y) => (x.toString, y.get())}
scala> val result = sc.sequenceFile("/tmp/seq-output/part-00001", classOf[Text], classOf[IntWritable]). map{case (x, y) => (x.toString, y.get())}
result: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[15] at map at :29

scala> result.collect
res14: Array[(String, Int)] = Array((Kay2,2), (Key3,2))

Apache Spark and HBase Integration on MapR Distribution

Env: Spark-1.4.1,HBase 0.98.12 on MapR-5.0

Problems: you can see lots of problem like this
*java.lang.RuntimeException: Error occurred while instantiating com.mapr.fs.hbase.MapRTableMappingRules
*java.lang.ClassNotFoundException: org.apache.hadoop.hbase.client.mapr.BaseTableMappingRules
*error: object HBaseAdmin is not a member of package org.apache.hadoop.hbase.client
*error: object HBaseConfiguration is not a member of package org.apache.hadoop.hbase

to resolve these issue and submit a job from spark-shell you need to change few things.

1. edit spark-defaults.conf and add spark.executor.extraClassPath as follows

spark.executor.extraClassPath /opt/mapr/hbase/hbase-0.98.12/lib/hbase-client-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/hbase-common-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/hbase-hadoop2-compat-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/hbase-it-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/hbase-protocol-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/hbase-server-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/htrace-core-2.04.jar:/opt/mapr/lib/mapr-hbase-5.0.0-mapr.jar

2. start the spark shell with the --driver-class-path as follows
bin/spark-shell --driver-class-path /opt/mapr/hbase/hbase-0.98.12/lib/hbase-client-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/hbase-common-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/hbase-hadoop2-compat-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/hbase-it-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/hbase-protocol-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/hbase-server-0.98.12-mapr-1506.jar:/opt/mapr/hbase/hbase-0.98.12/lib/htrace-core-2.04.jar:/opt/mapr/lib/mapr-hbase-5.0.0-mapr.jar

Utility to create lots of parquet files using hive

Create table in hive

CREATE TABLE partition_table_par(id INT, username string)
 PARTITIONED BY(year STRING, month STRING,day STRING,eventtype STRING,varfunction STRING,varname STRING)
 STORED AS PARQUET;

Bash Script to pump the data into the table which will store it in the parquet files

#!/bin/bash

for i in {1..100}
do
echo $i
year=`expr $i + 1996`
yy=\'$year\'
echo $yy
month=`expr $i % 12`
mm=\'$month\'
day=`expr $i % 30 `
dd=\'$day\'
eventtype=eventtype$i
et=\'$eventtype\'
varfunction=varfunction$i
vf=\'$varfunction\'
varname=varname$i
v=\'$varname\'


hive -e 'insert into table partition_table_par PARTITION(year='$yy',month='$mm',day='$dd',eventtype='$et',varfunction='$vf',varname='$v') select 1,'$yy' from test_table limit 1;'
#sleep 1m
done

Java Program to use JPam Authentication

JPam is a Java-PAM bridge. PAM, or Pluggable Authentication Modules, is a standard security architecture used on Linux, Mac OS X, Solaris, HP-UX and other Unix systems. JPam is the missing link between the two.

JPAM permits the use of PAM authentication facilities by Java applications running on those platforms.

These facilities include:

account
auth
password
session

In this example I will demonstrate how to use Jpam in your java application.
1. Download the JPam from http://sourceforge.net/projects/jpam/files/
2. Extract the tar on the disk
3. change directory to the extracted folder and copy the net-sf-jpam file to the /etc/pam.d
4. Create a sample java program-make sure you have JPam-1.1.jar in the classpath

import java.util.Arrays;
import java.util.Arrays;
import java.util.List;

import net.sf.jpam.Pam;


public class PamAuthenticator {
        public static void main(String[] args) {

                authenticate(args[0],args[1],args[2]);
        }

        public static void authenticate(String user, String password,String profile)  {
              Pam pam = new Pam(profile);
              System.out.println("Starting auth with username "+user+" passwd "+password+" for profile "+profile);
              if (!pam.authenticateSuccessful(user, password)) {
                throw new RuntimeException(String.format("PAM profile '%s' validation failed", profile));
              }
              System.out.println("done with "+profile);
          }
}

5. compile and run the test class as follows

java -cp .:* -Djava.library.path=`pwd` PamAuthenticator mapr mapr login

output:
Starting auth with username mapr passwd mapr for profile login
done with login

Apache Zookeeper : monitoring the zookeeper using four letters commands

here are few 4 words commands you can use with zookeeper to know the stats and status of zookeeper

[root@ip-10-0-0-251 ~]# 
[root@ip-10-0-0-251 ~]# echo ruok | nc `hostname` 5181
imok[root@ip-10-0-0-251 ~]# 
[root@ip-10-0-0-251 ~]# echo stat | nc `hostname` 5181
Zookeeper version: 3.4.5-mapr-1406--1, built on 01/19/2015 22:57 GMT
Clients:
 /10.0.0.251:57652[0](queued=0,recved=1,sent=0)
 /10.0.0.251:57623[1](queued=0,recved=9151,sent=9158)
 /10.0.0.251:57643[1](queued=0,recved=9120,sent=9125)
 /10.0.0.251:57622[1](queued=0,recved=9156,sent=9164)
 /10.0.0.251:57624[1](queued=0,recved=12974,sent=12974)
 /10.0.0.251:57620[1](queued=0,recved=9150,sent=9157)
 /10.0.0.251:57644[1](queued=0,recved=7708,sent=7966)
 /10.0.0.251:57626[1](queued=0,recved=9195,sent=9206)
 /10.0.0.251:57619[1](queued=0,recved=9185,sent=9186)
 /10.0.0.251:57621[1](queued=0,recved=9143,sent=9149)

Latency min/avg/max: 0/0/63
Received: 85043
Sent: 85345
Connections: 10
Outstanding: 0
Zxid: 0x136
Mode: standalone
Node count: 59
[root@ip-10-0-0-251 ~]# 
[root@ip-10-0-0-251 ~]# 
[root@ip-10-0-0-251 ~]# echo conf | nc `hostname` 5181
clientPort=5181
dataDir=/opt/mapr/zkdata/version-2
dataLogDir=/opt/mapr/zkdata/version-2
tickTime=2000
maxClientCnxns=100
minSessionTimeout=4000
maxSessionTimeout=40000
serverId=0
[root@ip-10-0-0-251 ~]# echo cons | nc `hostname` 5181
 /10.0.0.251:57623[1](queued=0,recved=9162,sent=9169,sid=0x14f1efc29d70004,lop=PING,est=1439333921242,to=30000,lcxid=0x1d,lzxid=0x136,lresp=1439425306327,llat=0,minlat=0,avglat=0,maxlat=2)
 /10.0.0.251:57643[1](queued=0,recved=9131,sent=9136,sid=0x14f1efc29d70008,lop=PING,est=1439334130760,to=30000,lcxid=0x13,lzxid=0x136,lresp=1439425306958,llat=0,minlat=0,avglat=0,maxlat=2)
 /10.0.0.251:57654[0](queued=0,recved=1,sent=0)
 /10.0.0.251:57622[1](queued=0,recved=9166,sent=9174,sid=0x14f1efc29d70003,lop=PING,est=1439333921221,to=30000,lcxid=0x22,lzxid=0x136,lresp=1439425302064,llat=0,minlat=0,avglat=0,maxlat=5)
 /10.0.0.251:57624[1](queued=0,recved=12989,sent=12989,sid=0x14f1efc29d70005,lop=GETC,est=1439333936044,to=30000,lcxid=0x1aef,lzxid=0x136,lresp=1439425303437,llat=0,minlat=0,avglat=0,maxlat=6)
 /10.0.0.251:57620[1](queued=0,recved=9161,sent=9168,sid=0x14f1efc29d70001,lop=PING,est=1439333921154,to=30000,lcxid=0x1d,lzxid=0x136,lresp=1439425307676,llat=0,minlat=0,avglat=0,maxlat=4)
 /10.0.0.251:57644[1](queued=0,recved=7716,sent=7974,sid=0x14f1efc29d70009,lop=PING,est=1439334142657,to=40000,lcxid=0x394,lzxid=0x136,lresp=1439425303832,llat=0,minlat=0,avglat=0,maxlat=16)
 /10.0.0.251:57626[1](queued=0,recved=9206,sent=9217,sid=0x14f1efc29d70007,lop=PING,est=1439333966309,to=30000,lcxid=0x4f,lzxid=0x136,lresp=1439425306675,llat=0,minlat=0,avglat=0,maxlat=6)
 /10.0.0.251:57619[1](queued=0,recved=9196,sent=9197,sid=0x14f1efc29d70000,lop=PING,est=1439333920963,to=30000,lcxid=0x41,lzxid=0x136,lresp=1439425304945,llat=0,minlat=0,avglat=0,maxlat=63)
 /10.0.0.251:57621[1](queued=0,recved=9153,sent=9159,sid=0x14f1efc29d70002,lop=PING,est=1439333921194,to=30000,lcxid=0x15,lzxid=0x136,lresp=1439425302145,llat=0,minlat=0,avglat=0,maxlat=6)

[root@ip-10-0-0-251 ~]# 
[root@ip-10-0-0-251 ~]# 
[root@ip-10-0-0-251 ~]# echo envi | nc `hostname` 5181
Environment:
zookeeper.version=3.4.5-mapr-1406--1, built on 01/19/2015 22:57 GMT
host.name=ip-10-0-0-251
java.version=1.7.0_45
java.vendor=Oracle Corporation
java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.45.x86_64/jre
java.class.path=/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../build/classes:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../build/lib/*.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../lib/slf4j-api-1.6.1.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../lib/netty-3.2.2.Final.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../lib/log4j-1.2.15.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../lib/jline-0.9.94.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../zookeeper-3.4.5-mapr-1406.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/bin/../src/java/lib/*.jar:/opt/mapr/zookeeper/zookeeper-3.4.5/conf::/opt/mapr/lib/maprfs-4.0.2-mapr.jar:/opt/mapr/lib/protobuf-java-2.5.0.jar:/opt/mapr/lib/libprotodefs-4.0.2-mapr.jar:/opt/mapr/lib/baseutils-4.0.2-mapr.jar:/opt/mapr/lib/json-20080701.jar:/opt/mapr/lib/flexjson-2.1.jar:/opt/mapr/lib/commons-codec-1.5.jar
java.library.path=/opt/mapr/lib
java.io.tmpdir=/tmp
java.compiler=
os.name=Linux
os.arch=amd64
os.version=2.6.32-431.el6.x86_64
user.name=mapr
user.home=/home/mapr
user.dir=/
[root@ip-10-0-0-251 ~]# 
[root@ip-10-0-0-251 ~]# 
[root@ip-10-0-0-251 ~]# echo wchc | nc `hostname` 5181
0x14f1efc29d70004
 /services/kvstore/ip-10-0-0-251
 /services/hoststats/ip-10-0-0-251
 /services/hoststats/master
 /nodes/ip-10-0-0-251/services/hoststats
0x14f1efc29d70008
 /nodes/ip-10-0-0-251/services/drill-bits
 /services/drill-bits/ip-10-0-0-251
 /services/drill-bits/master
0x14f1efc29d70003
 /nodes/ip-10-0-0-251/services/cldb
 /services/kvstore/ip-10-0-0-251
 /services/cldb/ip-10-0-0-251
 /datacenter/controlnodes/cldb/active/CLDBRunningMaster
 /services/cldb/master
0x14f1efc29d70005
 /datacenter/controlnodes/cldb/active/CLDBMaster
0x14f1efc29d70009
 /drill/sys.storage_plugins/hive
 /drill/sys.storage_plugins/cp
 /drill/sys.storage_plugins/mongo
 /drill/MyDrillCluster-402-drillbits/9d62d3ba-f9fa-4ad4-9b9c-684b854d41fe
 /drill/sys.storage_plugins/hbase
 /drill/sys.storage_plugins/dfs
0x14f1efc29d70001
 /services/webserver/master
 /nodes/ip-10-0-0-251/services/webserver
 /services/webserver/ip-10-0-0-251
 /services/cldb/master
0x14f1efc29d70007
 /services/drill-bits/master
 /services_config/kvstore/ip-10-0-0-251
 /services/kvstore/ip-10-0-0-251
 /services/services/drill-bits
 /services/services/webserver
 /services_config/drill-bits/ip-10-0-0-251
 /services/cldb/master
 /services/webserver/master
 /services/drill-bits/ip-10-0-0-251
 /services_config/cldb/ip-10-0-0-251
 /datacenter/controlnodes/cldb/active/CLDBMaster
 /services/services/cldb
 /services/services/kvstore
 /services/hoststats/ip-10-0-0-251
 /services/services/hoststats
 /services/cldb/ip-10-0-0-251
 /services_config/hoststats/ip-10-0-0-251
 /services/hoststats/master
 /services/webserver/ip-10-0-0-251
 /services/kvstore/master
 /services_config/webserver/ip-10-0-0-251
0x14f1efc29d70000
 /nodes/ip-10-0-0-251/services/nfs/start
 /nodes/ip-10-0-0-251/services/hbmaster/start
 /nodes/ip-10-0-0-251/services/historyserver/stop
 /nodes/ip-10-0-0-251/sdump
 /nodes/ip-10-0-0-251/services/hbmaster/stop
 /nodes/ip-10-0-0-251/services/nfs/stop
 /nodes/ip-10-0-0-251/services/tasktracker/stop
 /nodes/ip-10-0-0-251/services/fileserver/stop
 /nodes/ip-10-0-0-251/services/resourcemanager/stop
 /nodes/ip-10-0-0-251/services/fileserver/start
 /servers/ip-10-0-0-251
 /nodes/ip-10-0-0-251/services/jobtracker/stop
 /nodes/ip-10-0-0-251/services/nodemanager/start
 /nodes/ip-10-0-0-251/services/nodemanager/stop
 /nodes/ip-10-0-0-251/services/hbregionserver/stop
 /nodes/ip-10-0-0-251/services/historyserver/start
 /nodes/ip-10-0-0-251/services/jobtracker/start
 /nodes/ip-10-0-0-251/services/tasktracker/start
 /datacenter/controlnodes/cldb/active/CLDBRunningMaster
 /nodes/ip-10-0-0-251/services/hbregionserver/start
 /nodes/ip-10-0-0-251/stop
 /nodes/ip-10-0-0-251/services/resourcemanager/start
0x14f1efc29d70002
 /datacenter/license/m7/enabled
 /services/kvstore/ip-10-0-0-251
 /nodes/ip-10-0-0-251/services/kvstore
 /services/kvstore/master

Zookeeper : create,list,delete zknodes using java

quick and dirty way but working, to test and doing create,list and delete zknodes using java