Kafka Mirror Maker - from source non-kerberized cluster to kerberized cluster

Kafka Mirror Maker - from source non-kerberized cluster to target (kerberized) cluster

Env:

source cluster:
HDP242
un-secure
hostname: rksnode1

destination cluster:
HDP242
Kerberos enabled
hostname: rksnode2

Source cluster consumer settings

cat /tmp/sourceClusConsumer.properties 
zookeeper.connect=rksnode1:2181
bootstrap.servers=rksnode1:6667
zk.connectiontimeout.ms=1000000
consumer.timeout.ms=-1
group.id=mirror-maker-group
shallow.iterator.enable=true
mirror.topics.whitelist=app_log
enable.auto.commit=true
#security.protocol=PLAINTEXT

Destination cluster producer settings

cat /tmp/targetClusProducer.properties 
bootstrap.servers=rksnode2.hdp.local:6667
#zk.connect=rksnode2:2181
producer.type=async
compression.codec=0
serializer.class=kafka.serializer.DefaultEncoder
max.message.size=10000000
queue.time=1000
queue.enqueueTimeout.ms=-1
security.protocol=PLAINTEXTSASL

Running mirror maker

now login into the target cluster(rksnode2 in this case) and run the mirror maker as follows
//get the kerberos ticket
kinit -kt /etc/security/keytabs/kafka.service.keytab kafka/rksnode2.hdp.local@EXAMPLE.COM
//run mirror maker
/usr/hdp/current/kafka-broker/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config /tmp/sourceClusConsumer.properties  --producer.config /tmp/targetClusProducer.properties --whitelist test --new.consumer

view raw Kafka-MirrorMaker-Set-Up.md hosted with ❤ by GitHub