Initial import: links posts + github knowledge

github-knowledge/apps/elasticsearch.md

@@ -0,0 +1,26 @@
+---
+title: Elasticsearch
+created: 2022-04-30
+updated: 2022-04-30
+type: summary
+tags: [tech, reference]
+external: https://github.com/wahyd4/knowledge/blob/master/categories/apps/elasticsearch.md
+---
+
+# Elasticsearch
+
+Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
+
+
+## Tips
+
+## Calculate number of replicas
+
+> N >= R + 1
+
+N : number of nodes
+R: Number of replicas
+
+### How to resolve unassigned shards
+
+https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/#reason-3-you-need-to-reenable-shard-allocation

github-knowledge/apps/etcd.md

@@ -0,0 +1,44 @@
+---
+title: Etcd
+created: 2022-06-10
+updated: 2022-06-10
+type: summary
+tags: [tech, reference]
+external: https://github.com/wahyd4/knowledge/blob/master/categories/apps/etcd.md
+---
+
+# etcd
+
+etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. It gracefully handles leader elections during network partitions and can tolerate machine failure, even in the leader node.
+
+## Set up a ETCD cluster
+
+https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/07-bootstrapping-etcd.md
+
+##  Tips
+
+### Make the k8s embedded single etcd to be a cluster
+
+Unfortunately, there is no straightfoward way to change the existing embedded etcd to a cluster.
+But we can leverage etcd `make-mirror` command to replicate the existing etcd to a new cluster.
+The steps are:
+
+1. Follow the guide to setup a new etcd cluster
+2. Run the following command to replicate k8s emmbedded etcd data to the new cluster
+
+```
+sudo ETCDCTL_API=3 etcdctl make-mirror https://new_etcd_node:2379 --endpoints=https://192.168.1.2:2379  --cacert=/k8s_etcd/ca.crt   --cert=/k8s_etcd/server.crt   --key=/k8s_etcd/server.key   --dest-cacert=/new_etcd/ca.pem   --dest-cert=/new_etcd/etcd.pem   --dest-key=/new_etcd/etcd-key.pem
+```
+3. On one of the new cluster node, to verify the data
+```
+sudo ETCDCTL_API=3 etcdctl get --prefix=true "" -w json  --endpoints=https://127.0.0.1:2379   --cacert=/new_etcd/ca.pem   --cert=/new_etcd/etcd.pem   --key=/new_etcd/etcd-key.pem
+```
+
+### Join another existing cluster
+
+1. Run the join command on an existing node, for instnace
+```
+sudo ETCDCTL_API=3 etcdctl member add server-2 --peer-urls=https://192.168.1.xxx:2380  --endpoints=https://192.168.1.xxx:2379,https://192.168.1.xxx:2379   --cacert=/etc/etcd/ca.pem   --cert=/etc/etcd/etcd.pem   --key=/etc/etcd/etcd-key.pem
+```
+2. Prepare for the configursations for the new node, and remember to set `--initial-cluster-state` to existing, add put all the existing nodes to `--initial-cluster` field
+3. Start the etcd service of the new node

github-knowledge/apps/kafka.md

@@ -0,0 +1,46 @@
+---
+title: Kafka
+created: 2022-04-24
+updated: 2022-04-24
+type: summary
+tags: [tech, reference]
+external: https://github.com/wahyd4/knowledge/blob/master/categories/apps/kafka.md
+---
+
+# Kafka
+
+# What is Kafka
+
+<iframe width="560" height="315" src="https://www.youtube.com/embed/FKgi3n-FyNU" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+
+Apache Kafka is an open-source distributed event streaming platform
+
+## How to migrate self hosted Kafka to Some hosting vendor
+
+1. Setup target Kafka cluster
+2. Setup various replicators to replicate schemas and events in kafka. Before replicate schemas, make sure the schema registry in target cluster is empty and in `IMPORT_ONLY` mode.
+3. Setup users, roles for applications in the target clusters
+4. Start migrate consumers including apps and sink connectors
+    1. Reset consumer offset in the target Kafka cluster for a consumer(As the offsert will be different in most cases, unless the no events from the source cluster ever been deleted.)
+    2. Update Kafka configurations and scema registries for the consumer to the new cluster
+    3. Restarted consumer to consume events from the new cluster
+5. Stop schema replicator before migrating publisher, otherwise new publisher won't be able to publish schemas as the schema registry is in IMPORT mode. Once updating target schema registry to `READ_WRITE` mode, then schema replicator won't work anymore.
+6. Once all the consumers migerated, then migrated publichers (apps and source connectors)
+    1. Update Kafka configurations for Kafka clusters and shema registries
+    2. Restart publishers
+7. Stop replcators and source Kafka clusters
+
+## A typical Kafka workflow with schema registry
+
+![typical workflow](https://docs.confluent.io/platform/current/_images/schema-registry-and-kafka.png)
+
+Note: image from https://docs.confluent.io/platform/current/schema-registry/index.html
+
+
+## Some facts and best practices
+
+* Set proper partitions for topics, would be better to start with 3 or more depends on the numbers of events and type of events in the topic
+* Enable RBAC for topics, which means a Kafka user/service account can only access what he supposed to access. Minimal access scope
+* Use Debezium related source connnectors to achieve at least once delivery guarantee.
+* Use various sink connectors to export data via Restful API/S3 bucket and so on.
+* Set proper schema compatibility levels and tests to make sure existing events can be still consumed by new version of consumers.

github-knowledge/apps/mqtt.md

@@ -0,0 +1,12 @@
+---
+title: Mqtt
+created: 2022-04-24
+updated: 2022-04-24
+type: summary
+tags: [tech, reference]
+external: https://github.com/wahyd4/knowledge/blob/master/categories/apps/mqtt.md
+---
+
+## MQTT
+
+A lightweight messaging protocol for small sensors and mobile devices, optimized for high-latency or unreliable networks. It's basically a machine-to-machine `protocol`