From d4b223d86077706beccf41331bd1fa3810abd237 Mon Sep 17 00:00:00 2001 From: Junwei Zhao Date: Fri, 22 Apr 2022 12:20:01 +1000 Subject: [PATCH] Update kafaka --- categories/queues/kafka.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/categories/queues/kafka.md b/categories/queues/kafka.md index f7144f8..56d56eb 100644 --- a/categories/queues/kafka.md +++ b/categories/queues/kafka.md @@ -8,15 +8,18 @@ Apache Kafka is an open-source distributed event streaming platform ## How to migrate self hosted Kafka to Some hosting vendor -1. Setup various replicators to replicate schemas and events in kafka. Before replicate schemas, make sure the schema registry in target cluster is empty and in `IMPORT_ONLY` mode. -2. Start migrate consumers including apps and sink connectors - 1. Reset consumer offset in the target Kafka cluster for a consumer(As the offsert will be different in most cases, unless the no events from the source cluster ever been deleted.) - 2. Update Kafka configurations and scema registries for the consumer to the new cluster - 3. Restarted consumer to consume events from the new cluster -3. Stop schema replicator before migrating publisher, otherwise new publisher won't be able to publish schemas as the schema registry is in IMPORT mode. Once updating target schema registry to `READ_WRITE` mode, then schema replicator won't work anymore. -4. Once all the consumers migerated, then migrated publichers (apps and source connectors) - 1. Update Kafka configurations for Kafka clusters and shema registries - 2. Restart publishers +1. Setup target Kafka cluster +2. Setup various replicators to replicate schemas and events in kafka. Before replicate schemas, make sure the schema registry in target cluster is empty and in `IMPORT_ONLY` mode. +3. Setup users, roles for applications in the target clusters +4. Start migrate consumers including apps and sink connectors + 1. Reset consumer offset in the target Kafka cluster for a consumer(As the offsert will be different in most cases, unless the no events from the source cluster ever been deleted.) + 2. Update Kafka configurations and scema registries for the consumer to the new cluster + 3. Restarted consumer to consume events from the new cluster +5. Stop schema replicator before migrating publisher, otherwise new publisher won't be able to publish schemas as the schema registry is in IMPORT mode. Once updating target schema registry to `READ_WRITE` mode, then schema replicator won't work anymore. +6. Once all the consumers migerated, then migrated publichers (apps and source connectors) + 1. Update Kafka configurations for Kafka clusters and shema registries + 2. Restart publishers +7. Stop replcators and source Kafka clusters ## A typical Kafka workflow with schema registry