diff --git a/CNAME b/CNAME new file mode 100644 index 0000000..903de5e --- /dev/null +++ b/CNAME @@ -0,0 +1 @@ +wiki.toozhao.com \ No newline at end of file diff --git a/_sidebar.md b/_sidebar.md index 912054b..ba3164b 100644 --- a/_sidebar.md +++ b/_sidebar.md @@ -17,8 +17,12 @@ - [Kubernetes](categories/devops/kubernetes.md) - [Incident Management](categories/devops/incident.md) - [Linux](categories/devops/linux.md) + - [Google Cloud Platform](categories/devops/gcp.md) + - [AWS](categories/devops/aws.md) - [Web](categories/web.md) - [IoT](categories/iot.md) +- [Fintech](categories/fintech.md) +- [Microservice](categories/microservice.md) - [Success Team Work](categories/team-work.md) - [Uncategorised](categories/uncategorised.md) - English diff --git a/assets/images/circuitbreaker.png b/assets/images/circuitbreaker.png new file mode 100644 index 0000000..e9bc103 Binary files /dev/null and b/assets/images/circuitbreaker.png differ diff --git a/assets/images/scs.png b/assets/images/scs.png new file mode 100644 index 0000000..bc0b1b0 Binary files /dev/null and b/assets/images/scs.png differ diff --git a/assets/images/service-mesh.png b/assets/images/service-mesh.png new file mode 100644 index 0000000..b5ff8db Binary files /dev/null and b/assets/images/service-mesh.png differ diff --git a/categories/database.md b/categories/database.md index 55069c7..bffd8f6 100644 --- a/categories/database.md +++ b/categories/database.md @@ -32,6 +32,41 @@ - will impacted by time zone - index by timestamp will be faster than datetime due to 4 bytes. +## Postgres trigger example + +Giving we have a table called users, and the columns are: +```bash +id name age +1 tom 10 +2 Dave 17 +``` +there is a requirement which is when some user's age turn to `18` move that person to `adults` table. +The adults table have the same fields with `users`. + +The potential trigger implementation can be + +```sql + +CREATE OR REPLACE FUNCTION to_adults () + RETURNS TRIGGER + AS $$ +BEGIN + INSERT INTO adults VALUES (new.*); + DELETE FROM users WHERE age >= 18; + RETURN NEW; +END; +$$ +LANGUAGE plpgsql; + +CREATE TRIGGER to_adults_trigger + AFTER UPDATE ON users + FOR EACH ROW + WHEN (NEW.age >= 18) + EXECUTE PROCEDURE to_adults(); +``` + + + ## SQL join diagram ![sql join diagram](https://raw.githubusercontent.com/wahyd4/knowledge-mind-mapping/master/Knowledge.mindnode/resources/FC67BE77-F837-4207-B3C4-45205F7C9C40.png) diff --git a/categories/devops.md b/categories/devops.md index 2ef68d2..2d9052c 100644 --- a/categories/devops.md +++ b/categories/devops.md @@ -153,5 +153,3 @@ terraform apply # Apply changes to cluster - vscode ## Security - -# diff --git a/categories/devops/aws.md b/categories/devops/aws.md new file mode 100644 index 0000000..cbc685e --- /dev/null +++ b/categories/devops/aws.md @@ -0,0 +1,6 @@ +# EC2 +Virtual machines + +# ECS + +Running and managing containers without worry about infrastructure diff --git a/categories/devops/docker.md b/categories/devops/docker.md index b4e3512..e7b8383 100644 --- a/categories/devops/docker.md +++ b/categories/devops/docker.md @@ -4,10 +4,6 @@ Easy to use, just drag and drop things. -## [Kubernetes](categories/devops/kubernetes.md) - -Which is the actual standard orchestration platform. - ## Benefits of using Docker - escape the app dependency matrix diff --git a/categories/devops/gcp.md b/categories/devops/gcp.md new file mode 100644 index 0000000..73024a2 --- /dev/null +++ b/categories/devops/gcp.md @@ -0,0 +1,27 @@ +# Google Compute Engine + +# Kubernetes Engine + +# Cloud Storage + +# Cloud SQL + +# IAM + +# Stackdriver + +## Logging + +## Log based metrics + +## Debug + +## Trace + +# App Engine + +# Cloud Functions + +# Cloud Run + +Run Containers without caring about infrastructure diff --git a/categories/devops/kubernetes.md b/categories/devops/kubernetes.md index 0faacba..70c0e6f 100644 --- a/categories/devops/kubernetes.md +++ b/categories/devops/kubernetes.md @@ -63,15 +63,18 @@ spec: ## Some useful commands ```bash - kubectl config set-context context-name --namespace default-namespace-name #default active context kubectl config view #view config kubectl get nodes #get nodes kubectl get namespaces #get all namespaces kubectl get pods #get pods kubectl get deployments #get deployments +kubectl logs -f --tail 200 # tail logs from some pod kubectl get services #get services kubectl get ingress #get ingresses +kubectl get hpa # get horizontal auto scaling policies +kubectl get all # get all kinds of units +kubectl get secrets -o yaml #view a secret details with yaml format, fields are encrypted with base64 ``` @@ -169,3 +172,115 @@ template: https://blog.hasura.io/draft-vs-gitkube-vs-helm-vs-ksonnet-vs-metaparticle-vs-skaffold-f5aa9561f948/ +## Horizontal scaling + +### V1 +```yaml +apiVersion: autoscaling/v1 +kind: HorizontalPodAutoscaler +metadata: + name: worker-auto-scaling + namespace: x-prod +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: + minReplicas: 1 + maxReplicas: 2 + targetCPUUtilizationPercentage: 75 # trigger point +``` + +### V2 +```yaml +apiVersion: autoscaling/v2beta2 +kind: HorizontalPodAutoscaler +metadata: + name: php-apache + namespace: default +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: php-apache + minReplicas: 1 + maxReplicas: 10 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + averageUtilization: 50 + - type: Pods + pods: + metric: + name: packets-per-second + target: + type: AverageValue + averageValue: 1k + - type: Object + object: + metric: + name: requests-per-second + describedObject: + apiVersion: networking.k8s.io/v1beta1 + kind: Ingress + name: main-route + target: + type: Value + value: 10k +``` +## Limit CPU and memory for pods + +```yaml +ports: +- name: db-port + containerPort: 2345 + protocol: TCP +resources: +requests: + cpu: 30m + memory: 64Mi +limits: + cpu: 200m + memory: 256Mi +``` + +## Kubernetes Operator + +A Kubernetes Operator is an abstraction for deploying non-trivial applications on Kubernetes. It wraps the logic for deploying and operating an application using Kubernetes constructs. As an example, the `etcd` operator provides an `etcd` cluster as a first-class object. + +### An example Operator + +- deploying an application on demand +- taking and restoring backups of that application’s state +-handling upgrades of the application code alongside related changes such as database schemas or extra configuration settings +- publishing a Service to applications that don’t support Kubernetes APIs to discover them +- simulating failure in all or part of your cluster to test its resilience +- choosing a leader for a distributed application without an internal member election process + +### Use Operator + +``` +kubectl get SampleDB # find configured databases + +kubectl edit SampleDB/example-database # manually change some settings +``` + +### Some 3rd party operators + +- Custom operators list: https://gist.github.com/philips/a97a143546c87b86b870a82a753db14c +- Prometheus operator: https://coreos.com/blog/the-prometheus-operator.html +- + +## Tools + +### Kubectx and kubens + +Switch faster between clusters and namespaces in kubectl https + +- kubectx for switching contexts +- kubens for switching namespaces + +Github page: https://github.com/ahmetb/kubectx diff --git a/categories/devops/linux.md b/categories/devops/linux.md index a17c124..247d1e4 100644 --- a/categories/devops/linux.md +++ b/categories/devops/linux.md @@ -105,17 +105,59 @@ scp your_username@remotehost.edu:foobar.txt /local/dir scp /path/to/local/file user@server:/path/to/remote/file ``` -## Curl +### Merge video files into one -Curl is a very popular and powerful HTTP client, there is also another very good tool called `httpie` writen in Python which is even eaiser to use. +```bash +ffmpeg -f concat -safe 0 -i ./files.txt -c copy output.mp4 +``` + +In the `files.txt` you should list all the video clips like the following + +```txt +file 1.flv +file 2.flv +file 3.flv +``` + + +### Curl + +Curl is a very popular and powerful HTTP client, there is also another very good tool called `httpie` written in Python which is even eaiser to use. The following are some useful commands: -### Post file +#### Post request with file ```bash curl --data "@$(pwd)/fixure/sample.json" --user "foo:bar" localhost:8080/api/abc ``` +### Mount + +#### Mount NFS(Network file system) to local + +```bash + +mkdir /to-local-folder +# mount +mount -t nfs -O user=abc,pass=pass 192.168.1.2:/nfs-folder /to-local-folder + +# remount +mount -o remount /to-local-folder +``` + +#### Mount folder permanently + +In the example above, the mounted folder will lost the connection when you restart your system, in order to permanently mount the folder, you will need to edit `/etc/fstab` by adding one more line + +``` +192.168.1.2:/nfs-folder /to-local-folder nfs defaults 0 0 +``` +You can specify more options + +``` +192.168.0.216:/nfs-folder /to-local-folder nfs defaults,proto=tcp,port=2049 0 0 +``` + ## Package manager * [snap](https://snapcraft.io/) diff --git a/categories/fintech.md b/categories/fintech.md new file mode 100644 index 0000000..74ec63b --- /dev/null +++ b/categories/fintech.md @@ -0,0 +1,45 @@ +# Fintech(Financial technology) + +## ISO8583 message + +### A example + +``` +02004200040000000002161234567890123456060 +9173030123456789ABC1000123456789012345678 +90123456789012345678901234567890123456789 +0123456789012345678901234567890123456789 +``` +Where: + +``` + 0200 - MTI (Message Type Indicator), + 4200040000000002 - primary bitmap, + 161234567890123456 - field 2, the first 2 digits: 16, is length indicator + 0609173030 - field 7, + 123456789ABC - field 22, + 012345678901234567890123456789012345678901234567890\ + 1234567890123456789012345678901234567890123456789 - field 63. +``` + +### Definition + +ISO8583 message is used in bank or financial institutes card-originated transactions. But sometime, those not card-originated transactions also use ISO8583. + +### Typical Parts + +Typical Fields an ISO8583 message +A typical message might include such fields: +``` +MTI - Message Type Indicator. +Bit 2 - Primary Account Number (PAN) +Bit 3 - Processing Code +Bit 7 - Date and Time of transmission +Bit 11 - Audit Code +Bit 12 - Local time transaction +Bit 13 - Local transaction date +Bit 32 - Identification code of the institution of acquisition +Bit 38 - The authorization code in response +Bit 39 - Response Code +Bit 49 - Currency code of transaction +``` diff --git a/categories/go.md b/categories/go.md index 0c27964..78a8cff 100644 --- a/categories/go.md +++ b/categories/go.md @@ -1,4 +1,4 @@ -# Language +# Go ## pass pointer or value - value: Variable must not be modified @@ -20,18 +20,22 @@ new(T) -> *T ## array ```go -[]string -[]string{"a", "b"} -[...]string{"a","b"} +var a [1]int ``` Array has a exactly length, can't be modified. ## slice -- Auto increment length +```go +var a []string +[]string{"a", "b"} +[...]string{"a","b"} +``` +1. Auto increment length ```go new([]int) make([]int, 2, 5) ``` +2. nil is a valid slice which length is `0` ## Go routine @@ -83,6 +87,23 @@ m.Store("a", "b") value, ok := m.Load("a") ``` +### Defer to Clean Up + +Use defer to clean up resources such as files and locks. +```go +p.Lock() +defer p.Unlock() + +if p.count < 10 { + return p.count +} + +p.count++ +return p.count + +// more readable +``` + # Frameworks ## db @@ -98,19 +119,30 @@ value, ok := m.Load("a") - go kit - full stack micro service framework like spring boot -## tools +## Tools -- profiler +### profiler - go-wrk(wrk) - a http benchmark tool - go-torch - Stochastic flame graph profiler -- test +### test - Testify - Ginkgo +### Linter + - Golangci-lint https://github.com/golangci/golangci-lint # Tips +## Sort slice + +```go +// sort users by user age ASC +sort.Slice(users, func(i, j int) bool { + return users[i].age < planets[j].age +}) +``` + ## slice - byte* array //actual data @@ -120,7 +152,7 @@ value, ok := m.Load("a") ## map * implement by hash table -* slice can't be the key of a map, but sized array could. `var a map[[2]int]string` +* slice can't be the key of a map, but sized array could. e.g. `var a map[[2]int]string` ## Go has no generics @@ -151,14 +183,31 @@ value, ok := m.Load("a") ## error handling -- error type assertion +### check error type + ```go - if serr, ok := err.(*json.SyntaxError); ok {} +ErrorSample := errors.New("some error") +if errors.Is(err, ErrorSample) { + // something wasn't found +} ``` -- better error handling +### error type assertion +```go + if serr, ok := err.(*json.SyntaxError); ok {} +//or + +// var e *QueryError +if errors.As(err, &e) { + // err is a *QueryError, and e is set to the error's value +} + +``` + +### better error handling + +#### custom error type - - custom error type ```go type appError struct { Error error @@ -166,11 +215,13 @@ type appError struct { Code int } ``` - - concat error check +#### concat error check + ```go if err1() != nil || err2() != nil {} ``` - - some error constants +#### some error constants + ```go errNotFound = errors.New("Item not found") switch err { @@ -245,3 +296,8 @@ REPL stands for read eval print loop, basically it just like the irb in Ruby. Wiki: * gore + +# Useful links +* [Go best practice](https://github.com/golang/go/wiki/CodeReviewComments) +* [Code Review Comments](https://github.com/golang/go/wiki/CodeReviewComments) +* [Uber Go code style](https://github.com/uber-go/guide/blob/master/style.md#pointers-to-interfaces) diff --git a/categories/microservice.md b/categories/microservice.md new file mode 100644 index 0000000..acbca17 --- /dev/null +++ b/categories/microservice.md @@ -0,0 +1,62 @@ +# Microservice + +- benefits + + - Each application is relatively small + - Easy to scale + - Improved fault isolation + - One service fails won’t influence others + - You can use different stack for each service + - Each service can be deployed independently +- drawbacks + + - Increased memory consumption + - Developers must deal with extra complexity of creating a distributed system + - Deployment complexity + - To deploy different types of services + +## Service Mesh + +The term service mesh is used to describe the network of microservices that make up such applications and the interactions between them. As a service mesh grows in size and complexity, it can become harder to understand and manage. Its requirements can include discovery, load balancing, failure recovery, metrics, and monitoring. A service mesh also often has more complex operational requirements, like A/B testing, canary rollouts, rate limiting, access control, and end-to-end authentication. + +![service mesh](https://raw.githubusercontent.com/wahyd4/knowledge-mind-mapping/master/assets/images/service-mesh.png) + + +### Opensource Projects + +* [Istio](https://istio.io/) +* [Consul](https://www.consul.io/) + +## CircuitBreaker Design + +### Why we need CircuitBreaker + +In case we have serviceB down, serviceA should still try to recover from this and try to do one of the followings: +- **Custom fallback**: Try to get the same data from some other source. If not possible, use its own cache value. +- **Fail fast**: If serviceA knows that serviceB is down, there is no point waiting for the timeout and consuming its own resources. It should return ASAP “knowing” that serviceB is down +- **Don’t crash**: As we saw in this case, serviceA should not have crashed. +- **Heal automatic**: Periodically check if serviceB is working again. +- **Other APIs should work**: All other APIs should continue to work. + +### What is circuit breaker design? +The idea behind is simple: +- Once serviceA “knows” that serviceB is down, there is no need to make request to serviceB. serviceA should return cached data or timeout error as soon as it can. This is the OPEN state of the circuit +- Once serviceA “knows” that serviceB is up, we can CLOSE the circuit so that request can be made to serviceB again. +- Periodically make fresh calls to serviceB to see if it is successfully returning the result. This state is HALF-OPEN. + +![CircuitBreaker](https://raw.githubusercontent.com/wahyd4/knowledge-mind-mapping/master/assets/images/circuitbreaker.png) + + +### More links + +- +- + + +### Self-contained System (SCS) + +The Self-contained System (SCS) approach is an architecture that focuses on a separation of the functionality into many independent systems, making the complete logical system a collaboration of many smaller software systems. This avoids the problem of large monoliths that grow constantly and eventually become unmaintainable. Over the past few years, we have seen its benefits in many mid-sized and large-scale projects. + +The idea is to break a large system apart into several smaller self-contained systems, or SCSs, that follow certain rules. + +![Self contained system](https://raw.githubusercontent.com/wahyd4/knowledge-mind-mapping/master/assets/images/scs.png) diff --git a/categories/software-design.md b/categories/software-design.md index 23dbe56..f10c263 100644 --- a/categories/software-design.md +++ b/categories/software-design.md @@ -11,32 +11,6 @@ - temporary variable - coupling -# Microservice - -- benefits - - - Each application is relatively small - - Easy to scale - - Improved fault isolation - - One service fails won’t influence others - - You can use different stack for each service - - Each service can be deployed independently -- drawbacks - - - Increased memory consumption - - Developers must deal with extra complexity of creating a distributed system - - Deployment complexity - - To deploy different types of services - -## Service Mesh - -A service mesh is a way to control how different parts of an application share data with one another. Unlike other systems for managing this communication, a service mesh is a dedicated infrastructure layer built right into an app. This visible infrastructure layer can document how well (or not) different parts of an app interact, so it becomes easier to optimize communication and avoid downtime as an app grows. - -### Opensource Projects - -* [Istio](https://istio.io/) -* [Consul](https://www.consul.io/) - # OOP - Polymorphism