How to Configure and Run Storm AutoHDFS plugin (sample Application)

Add these configuration to custom storm-site.

nimbus.autocredential.plugins.classes ["org.apache.storm.hdfs.common.security.AutoHDFS"]
nimbus.credential.renewers.classes ["org.apache.storm.hdfs.common.security.AutoHDFS"]
hdfs.keytab.file  /etc/security/keytabs/hdfs.headless.keytab
hdfs.kerberos.principal hdfs-s253_kerb@LAB.HORTONWORKS.NET
nimbus.credential.renewers.freq.secs 518400

nimbus.childopts -Xmx1024m _JAAS_PLACEHOLDER -javaagent:/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8649,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-nimbus/contrib/storm-jmxetric/conf/jmxetric-conf.xml:/etc/hadoop/conf/hdfs-site.xml:/etc/hadoop/conf/core-site.xml:/etc/hbase/conf/hbase-site.xml,process=Nimbus_JVM

add the following in storm-env

export STORM_EXT_CLASSPATH=/usr/hdp/current/hbase-client/lib/:/usr/hdp/current/hadoop-mapreduce-client/:/usr/hdp/current/hadoop-client

remove the hadoop-aws.*.jar from classpath.

update core-site,core-default and hdfs-site.xml to storm-hdfs jar

add following snippet in core-site.xml

<property>
<name>fs.hdfs.impl</name>
<value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
</property>

[root@r253secure contrib]# jar uvf storm-hdfs/storm-hdfs-1.0.1.2.5.3.0-37.jar core-site.xml 
[root@r253secure contrib]# jar uvf storm-hdfs/storm-hdfs-1.0.1.2.5.3.0-37.jar hdfs-site.xml
[root@r253secure contrib]# jar uvf storm-hdfs/storm-hdfs-1.0.1.2.5.3.0-37.jar core-default.xml

jar tvf storm-hdfs/storm-hdfs-1.0.1.2.5.3.0-37.jar | egrep 'core-site|hdfs-site'
  5538 Wed May 03 14:25:04 UTC 2017 core-site.xml
  8047 Tue Jan 17 14:14:10 UTC 2017 hdfs-site.xml

copy updated storm-hdfs to storm lib directory

[root@r253secure contrib]# cp storm-hdfs/storm-hdfs-1.0.1.2.5.3.0-37.jar ../lib/

copy core-site.xml and hdfs-site.xml to lib folder

copy hadoop-hdfs.jar to storm lib folder.

Sample code to test your topology

https://github.com/rajkrrsingh/sample-storm-hdfs-app

view raw Storm_AutoHDFS_configuration.md hosted with ❤ by GitHub