diff --git a/_sidebar.md b/_sidebar.md index dc95467..fe9d6da 100644 --- a/_sidebar.md +++ b/_sidebar.md @@ -12,7 +12,8 @@ - [Application](categories/application.md) - [Web Scraping](categories/web-scraping.md) - [Devops](categories/devops.md) - - [Docker](categories/devops/docker.md) + - [Docker](categories/devops/docker.md) + - [Incident Management](categories/devops/incident.md) - [Web](categories/web.md) - [Success Team Work](categories/team-work.md) - English diff --git a/categories/devops/incident.md b/categories/devops/incident.md new file mode 100644 index 0000000..b3b70fc --- /dev/null +++ b/categories/devops/incident.md @@ -0,0 +1,40 @@ +# Incident Management + +## Incident dealing + +It's important for a company to have a formal process about how do people dealing with those scenarios. There should some places to list all the common issues and its' solutions. +During the process, we should records the steps we have, so we can learn from this after the incident. +Company should have a on call rotation rules to have every engineers on the list and well rotated. + +### Some tools + +* Pagerduty - on call scheduling and incident resolution +* Datadog - metrics +* Stackdriver - logging + +## Incident Training + +### The purpose for doing this + +* It's better to have some training before people get on call +* We can learn something from the incident training, so we can find some potential issues in production +* It's a good opportunities to have people from different teams work together to solve the issue +* Customer first. We should let customer know what happened if the issue has some impact to the customers. + +### How to organise an incident training + +* Have some check lists for people prepared before training +* Have a almost production like environment to do this +* Organiser triggers a issue, then ask trainees to fix the issue and investigate the root clause +* Let customers knows what happened. The other way round, the customer can give us some more information and also give us some complaint about his loss due to the incident. +* The on-call team should solve the issue as well as write down some notes to record how to deal with this. +* Escalate the severity if necessary to involve more people to help to solve this +* Give enough information to the next people when you are going to hand over it to. + +## After incident + +* Run a post incident review +* Write down the incident events timeline +* Provide a problem statement +* List the root cause +* List the ways of how do we prevent this happen again \ No newline at end of file