Thursday, January 7, 2016

SparkThrift Server Configuration in yarn mode

Env: MapR,Spark 1.4.1
Start spark thrift server on mapr cluster as follows (-- start thrift server non root user)
/opt/mapr/spark/spark-1.4.1/sbin/start-thriftserver.sh --master yarn --hiveconf hive.server2.thrift.bind.host `hostname` --hiveconf hive.server2.trift.port 10000

check for any error exception in the logs(in my case it is tailf /opt/mapr/spark/spark-1.4.1/logs/spark-mapr-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-ip-10-0-0-233.out)
if there is no error exception in the logs try connecting using the beeline ship with the spark distribution

/opt/mapr/spark/spark-1.4.1/bin/beeline 
Beeline version 1.4.1 by Apache Hive
beeline> !connect jdbc:hive2://10.0.0.233:10000/default;auth=noSasl
scan complete in 1ms
Connecting to jdbc:hive2://10.0.0.233:10000/default;auth=noSasl
Enter username for jdbc:hive2://10.0.0.233:10000/default;auth=noSasl: 
Enter password for jdbc:hive2://10.0.0.233:10000/default;auth=noSasl: 
Connected to: Spark SQL (version 1.4.1)
Driver: Spark Project Core (version 1.4.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://10.0.0.233:10000/default> show tables;
+------------------------------+--------------+
|          tableName           | isTemporary  |
+------------------------------+--------------+
| b1                           | false        |
| b2                           | false        |
| bob                          | false        |
| ct                           | false        |

Post a Comment