diff --git a/_sidebar.md b/_sidebar.md index a777f5f..1dff9b6 100644 --- a/_sidebar.md +++ b/_sidebar.md @@ -9,6 +9,7 @@ - [Interview](categories/interviews/interview.md) - [Software Design](categories/software-design.md) - [Message Queue](categories/message-queue.md) + - [Kafka](categories/queues/kafka.md) - [Database](categories/database.md) - [Application](categories/application.md) - [Web Scraping](categories/web-scraping.md) diff --git a/categories/message-queue.md b/categories/message-queue.md index fd2be54..e721602 100644 --- a/categories/message-queue.md +++ b/categories/message-queue.md @@ -5,8 +5,6 @@ ## Redis -## Kafka - # MQTT A lightweight messaging protocol for small sensors and mobile devices, optimized for high-latency or unreliable networks. It's basically a machine-to-machine `protocol` diff --git a/categories/queues/kafka.md b/categories/queues/kafka.md new file mode 100644 index 0000000..f7144f8 --- /dev/null +++ b/categories/queues/kafka.md @@ -0,0 +1,34 @@ +# Kafka + +# What is Kafka + + + +Apache Kafka is an open-source distributed event streaming platform + +## How to migrate self hosted Kafka to Some hosting vendor + +1. Setup various replicators to replicate schemas and events in kafka. Before replicate schemas, make sure the schema registry in target cluster is empty and in `IMPORT_ONLY` mode. +2. Start migrate consumers including apps and sink connectors + 1. Reset consumer offset in the target Kafka cluster for a consumer(As the offsert will be different in most cases, unless the no events from the source cluster ever been deleted.) + 2. Update Kafka configurations and scema registries for the consumer to the new cluster + 3. Restarted consumer to consume events from the new cluster +3. Stop schema replicator before migrating publisher, otherwise new publisher won't be able to publish schemas as the schema registry is in IMPORT mode. Once updating target schema registry to `READ_WRITE` mode, then schema replicator won't work anymore. +4. Once all the consumers migerated, then migrated publichers (apps and source connectors) + 1. Update Kafka configurations for Kafka clusters and shema registries + 2. Restart publishers + +## A typical Kafka workflow with schema registry + +![typical workflow](https://docs.confluent.io/platform/current/_images/schema-registry-and-kafka.png) + +Note: image from https://docs.confluent.io/platform/current/schema-registry/index.html + + +## Some facts and best practices + +* Set proper partitions for topics, would be better to start with 3 or more depends on the numbers of events and type of events in the topic +* Enable RBAC for topics, which means a Kafka user/service account can only access what he supposed to access. Minimal access scope +* Use Debezium related source connnectors to achieve at least once delivery guarantee. +* Use various sink connectors to export data via Restful API/S3 bucket and so on. +* Set proper schema compatibility levels and tests to make sure existing events can be still consumed by new version of consumers.