Contains a kafka transforms plugin that converts confluent avro bytes to schema included JSON format. It is meant to be used in combination with MirrorMaker2. I couldn't find any public resource that is capable of providing this functionality which was required for our project, so we decided to build our own.
Thanks to the ING Dora team for providing me with a sample code and repository to start with. This really helped me get a jumpstart on this project. Turns out this sample code was taken from https://github.com/OneCricketeer/schema-registry-transfer-smt, many thanks to OneCricketeer!
-
Configure MirrorMaker
Make sure that the plugin path is included in the CLASS_PATH variable that is used for MirrorMaker2. For example: CLASSPATH="/your/path/kafkaconnect-mirrormaker/plugins/*"
# Requires that records are entirely byte-arrays. These can go in the worker or connector configuration. key.converter=org.apache.kafka.connect.converters.ByteArrayConverter value.converter=org.apache.kafka.connect.converters.ByteArrayConverter #Setup the SMT {{ source_cluster_alias }}->{{ destination_cluster_alias }}.transforms=AvroSchemaTransfer #Implementation of the connect Transformation {{ source_cluster_alias }}->{{ destination_cluster_alias }}.transforms.AvroSchemaTransfer.type=com.ing.eventbus.connect.schema.converters.AvroTransform #Source and target Schema Registry urls {{ source_cluster_alias }}->{{ destination_cluster_alias }}.transforms.AvroSchemaTransfer.src.schema.registry.url=http://schema-reg-url:8081 # List of avro topics for which this SMT will be applied. The boolean flag sets if the key also has an avro schema {{ source_cluster_alias }}->{{ destination_cluster_alias }}.transforms.AvroSchemaTransfer.avro.topics="test-topic:false"
-
Build the project
mvn clean package -
Copy the JAR from target to all Kafka Connect workers under a directory set by CLASS_PATH.
-
Restart MirrorMaker2