home/knowledge
2
0
Fork 0
mirror of https://github.com/wahyd4/knowledge.git synced 2026-08-09 05:06:28 +10:00
Code Issues Packages Projects Releases Wiki Activity

Update kafaka

Browse Source
This commit is contained in:
junv
2022-04-22 12:20:01 +10:00
committed by GitHub
parent 8de19c2b67
commit d4b223d860
1 changed files with 12 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
categories/queues/kafka.md
+12 -9
View File
@@ -8,15 +8,18 @@ Apache Kafka is an open-source distributed event streaming platform
## How to migrate self hosted Kafka to Some hosting vendor
1. Setup various replicators to replicate schemas and events in kafka. Before replicate schemas, make sure the schema registry in target cluster is empty and in `IMPORT_ONLY` mode.
2. Start migrate consumers including apps and sink connectors
1. Reset consumer offset in the target Kafka cluster for a consumer(As the offsert will be different in most cases, unless the no events from the source cluster ever been deleted.)
2. Update Kafka configurations and scema registries for the consumer to the new cluster
3. Restarted consumer to consume events from the new cluster
3. Stop schema replicator before migrating publisher, otherwise new publisher won't be able to publish schemas as the schema registry is in IMPORT mode. Once updating target schema registry to `READ_WRITE` mode, then schema replicator won't work anymore.
4. Once all the consumers migerated, then migrated publichers (apps and source connectors)
1. Update Kafka configurations for Kafka clusters and shema registries
2. Restart publishers
1. Setup target Kafka cluster
2. Setup various replicators to replicate schemas and events in kafka. Before replicate schemas, make sure the schema registry in target cluster is empty and in `IMPORT_ONLY` mode.
3. Setup users, roles for applications in the target clusters
4. Start migrate consumers including apps and sink connectors
1. Reset consumer offset in the target Kafka cluster for a consumer(As the offsert will be different in most cases, unless the no events from the source cluster ever been deleted.)
2. Update Kafka configurations and scema registries for the consumer to the new cluster
3. Restarted consumer to consume events from the new cluster
5. Stop schema replicator before migrating publisher, otherwise new publisher won't be able to publish schemas as the schema registry is in IMPORT mode. Once updating target schema registry to `READ_WRITE` mode, then schema replicator won't work anymore.
6. Once all the consumers migerated, then migrated publichers (apps and source connectors)
1. Update Kafka configurations for Kafka clusters and shema registries
2. Restart publishers
7. Stop replcators and source Kafka clusters
## A typical Kafka workflow with schema registry