The existing behavior of the backups is to overwrite whatever data
was still there in the scratch directories. This broke when we added
a 'gzip' step because 'gzip database.sql' will fail if 'database.sql.gz'
already exists. Doing 'rm -f database.sql.gz' before the 'gzip'
avoids this failure.
Recalculating graph_log took minutes on a large repository since each
of the 6000 commits would need to calculate the diff. Since graph_log
does not appear to be used, remove it from the list of things to update.
For the Linux kernel repository, this reduced the ProjectCacheWorker time
from 8+ minutes to 16 seconds.
Closes#2016
Merge branch 'advanced-cache' into 'master'
Backport to stable from merge request !986
- - -
Advanced cache
Fixes#1993
* Build missing cache values in background job after each push
* Store commit_count in project table. Refresh in background job
* moved repository size calculation in background job
## Advantages
Every time push is triggered - we build cache for project even without user visiting project page.
That means first visit is as fast as others now. This is especially good for active projects where people have some requests fast because of cache and some slow - because cache was build in their request.
Between cache expired and cache built we we had gap when Linux repo can give 502 error because calculation commits count takes 30 seconds or even more.
Using value from database fix this problem. Before cache is updated you see old value from database. After - you see new one.
Basically this merge request is super win to GitLab. We don't do some heavy operations in user request but instead in background job.
## Temporary problem
After this migration all projects will have `0` commits in database. It fill be replaced with real value on next push. I did not add recalculation to migration because it will last forever on big instances. Can be fixed by adding rake task which will go in background on live instance without downtime
See merge request !989
Advanced cache
Fixes#1993
* Build missing cache values in background job after each push
* Store commit_count in project table. Refresh in background job
* moved repository size calculation in background job
## Advantages
Every time push is triggered - we build cache for project even without user visiting project page.
That means first visit is as fast as others now. This is especially good for active projects where people have some requests fast because of cache and some slow - because cache was build in their request.
Between cache expired and cache built we we had gap when Linux repo can give 502 error because calculation commits count takes 30 seconds or even more.
Using value from database fix this problem. Before cache is updated you see old value from database. After - you see new one.
Basically this merge request is super win to GitLab. We don't do some heavy operations in user request but instead in background job.
## Temporary problem
After this migration all projects will have `0` commits in database. It fill be replaced with real value on next push. I did not add recalculation to migration because it will last forever on big instances. Can be fixed by adding rake task which will go in background on live instance without downtime
See merge request !986
fixed notifications title
I fixed the title and link to this document because I believe that it's more clear and easier to find if it has "emails" in the name.
See merge request !1897
Valery Sizov
Valery Sizov
Douwe MaanandValery Sizov
Achilleas PipinellisandValery Sizov
Marin JankovskiandValery Sizov
Hannes RosenöggerandValery Sizov
Jacob VosmaerandValery Sizov
Dmitriy ZaporozhetsandValery Sizov
Stan HuandValery Sizov
Robert SpeicherandValery Sizov
Sytse SijbrandijandValery Sizov
Job van der Voort
Dmitriy Zaporozhets