Initial import: links posts + github knowledge

github-knowledge/devops/incident.md

@@ -0,0 +1,49 @@
+---
+title: Incident
+created: 2018-03-27
+updated: 2018-03-27
+type: summary
+tags: [tech, reference]
+external: https://github.com/wahyd4/knowledge/blob/master/categories/devops/incident.md
+---
+
+# Incident Management
+
+## Incident dealing
+
+It's important for a company to have a formal process about how do people dealing with those scenarios. There should some places to list all the common issues and its' solutions.
+During the process, we should records the steps we have, so we can learn from this after the incident.
+Company should have a on call rotation rules to have every engineers on the list and well rotated.
+
+### Some tools
+
+* Pagerduty - on call scheduling and incident resolution
+* Datadog - metrics
+* Stackdriver - logging
+
+## Incident Training
+
+### The purpose for doing this
+
+* It's better to have some training before people get on call
+* We can learn something from the incident training, so we can find some potential issues in production
+* It's a good opportunities to have people from different teams work together to solve the issue
+* Customer first. We should let customer know what happened if the issue has some impact to the customers.
+
+### How to organise an incident training
+
+* Have some check lists for people prepared before training
+* Have a almost production like environment to do this
+* Organiser triggers a issue, then ask trainees to fix the issue and investigate the root clause
+* Let customers knows what happened. The other way round, the customer can give us some more information and also give us some complaint about his loss due to the incident.
+* The on-call team should solve the issue as well as write down some notes to record how to deal with this.
+* Escalate the severity if necessary to involve more people to help to solve this
+* Give enough information to the next people when you are going to hand over it to.
+
+## After incident
+
+* Run a post incident review
+* Write down the incident events timeline
+* Provide a problem statement
+* List the root cause
+* List the ways of how do we prevent this happen again