Allow customising of queries used for `update_column_in_batches`
This MR makes two changes to `add_column_with_default` and `update_column_in_batches`:
1. `add_column_with_default` no longer wraps the entire set of updates in a single transaction, preventing any locks from sticking around for the duration of the entire transaction
2. `update_column_in_batches` now takes a block which can be used to customise the queries. This uses Arel as messing with raw SQL strings is a total pain
In !4381 there's a need for updating existing rows/columns in a table in batches using a custom `WHERE` condition. Without the changes in this MR this would not be possible.
See merge request !4680
Fix permission checks in member row (backport from gitlab-org/gitlab-ee!460)
## What does this MR do?
It improves the check we use to display or not the members' access and controls in the members list.
## Are there points in the code the reviewer needs to double check?
No, I replaced an helper with just a permission check so I think it's a better solution.
## Why was this MR needed?
There were a spec failure in gitlab-org/gitlab-ee!460 because of the refactor done in the "request access" MR.
## What are the relevant issue numbers?
None.
## Does this MR meet the acceptance criteria?
- No CHANGELOG needed
- [x] Tests
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !4670
Added API endpoint for Sidekiq Metrics.
## What does this MR do?
It adds an API endpoint to gather metrics about Sidekiq, it's jobs, queues, and processes.
## Why was this MR needed?
There was no API endpoint for Sidekiq information.
## What are the relevant issue numbers?
Fixes#7171
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] API support added
- [x] Tests
- [x] Added for this feature/bug
- [x] All builds are passing
See merge request !4653
Add sorting dropdown to tag page
## What does this MR do?
Adds a sorting dropdown to the tags page just like the one on the branches page.
## Are there points in the code the reviewer needs to double check?
No
## Why was this MR needed?
Clients were asking for this
## What are the relevant issue numbers?
Closes#14918
## Screenshots (if relevant)
See merge request !4423
Fixed ordering in Project.find_with_namespace
This MR fixes the ordering of `Project.find_with_namespace` to ensure that it returns rows that match literally first.
Closes#18603
See merge request !4682
Remove JiraIssue model and replace references with ExternalIssue
This MR Removes unused JiraIssue class and replaces references with ExternalIssue
Closes#18203
See merge request !4659
Fair usage of Shared Runners
## What does this MR do?
Introduces a fair usage scheduler for shared runners.
It tries to assign builds to shared runner from projects that have the lowest number of builds currently running on shared runners.
**Example 1**:
```
We have following builds in queue:
build 1 for project 1
build 2 for project 1
build 3 for project 1
build 4 for project 2
build 5 for project 2
build 6 for project 3
With the new algorithm we will assign builds in following order:
- We choose build 1, because project 1 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
- We choose build 4, because project 2 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
- We choose build 6, because project 3 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
- We choose build 2, because project 1 as other it runs 1 build,
- We choose build 5, because project 2 runs 1 build, where project 1 runs 2 builds now,
- We choose build 3, because project 1 and runs 2 builds.
```
**Example 2**:
```
We have following builds in queue:
build 1 for project 1
build 2 for project 1
build 3 for project 1
build 4 for project 2
build 5 for project 2
build 6 for project 3
With the new algorithm we will assign builds in following order:
- We choose build 1, because project 1 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
- We finish build 1,
- We choose build 2, because project 1 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
- We choose build 4, because project 2 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
- We finish build 4,
- We choose build 5, because project 2 doesn't run currently any builds and has the lowest build number from projects that doesn't run builds,
- We choose build 6, because project 3 doesn't run currently any builds,
- We choose build 3, because project 1, 2 and 3 runs exactly one build now,
```
## Why was this MR needed?
Currently, we are scheduling builds using FIFO. This is catastrophic if there are projects that create a 100-300 jobs, this basically eats most of available shared runners.
## Performance
All this logic is implemented with the help of SQL queries, because this is the fastest way to process 1k-2k pending builds in queue.
It's not the fastest SQL query, because it sorts based on number of running_builds, and this forces to calculate a number of running builds for all dependent projects. However, since we have one/two shared runners that asks every few seconds for builds this should have minimal impact on DB performance.
```
explain analyze SELECT "ci_builds".* FROM "ci_builds" JOIN (SELECT "ci_builds"."gl_project_id", count(case when status = 'running' AND runner_id = (SELECT "ci_runners"."id" FROM "ci_runners" WHERE "ci_runners"."is_shared" = 't') then 1 end) as running_builds FROM "ci_builds" INNER JOIN "projects" ON "projects"."id" = "ci_builds"."gl_project_id" AND "projects"."pending_delete" = 'f' WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" IN ('running', 'pending') AND "projects"."builds_enabled" = 't' AND "projects"."shared_runners_enabled" = 't' GROUP BY "ci_builds"."gl_project_id") AS projects ON ci_builds.gl_project_id=projects.gl_project_id WHERE "ci_builds"."type" IN ('Ci::Build') AND "ci_builds"."status" = 'pending' AND "ci_builds"."runner_id" IS NULL ORDER BY projects.running_builds ASC, ci_builds.id ASC;
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------
Sort (cost=64777.28..64777.29 rows=1 width=1010) (actual time=301.794..302.535 rows=1537 loops=1)
Sort Key: (count(CASE WHEN (((public.ci_builds.status)::text = 'running'::text) AND (public.ci_builds.runner_id = $0)) THEN 1 ELSE NULL::integer END)), public.ci
_builds.id
Sort Method: quicksort Memory: 1423kB
-> Nested Loop (cost=63279.78..64777.27 rows=1 width=1010) (actual time=66.384..298.724 rows=1537 loops=1)
-> HashAggregate (cost=63177.15..63177.30 rows=15 width=15) (actual time=65.641..65.851 rows=187 loops=1)
InitPlan 1 (returns $0)
-> Seq Scan on ci_runners (cost=0.00..26963.66 rows=1 width=4) (actual time=1.145..34.381 rows=1 loops=1)
Filter: is_shared
Rows Removed by Filter: 6965
-> Nested Loop (cost=0.00..36186.34 rows=2715 width=15) (actual time=0.065..29.717 rows=1710 loops=1)
-> Index Scan using index_ci_builds_on_status on ci_builds (cost=0.00..8913.95 rows=3577 width=15) (actual time=0.051..12.012 rows=2583 loops
=1)
Index Cond: ((status)::text = ANY ('{running,pending}'::text[]))
Filter: ((type)::text = 'Ci::Build'::text)
Rows Removed by Filter: 1219
-> Index Scan using projects_pkey on projects (cost=0.00..7.61 rows=1 width=4) (actual time=0.003..0.004 rows=1 loops=2583)
Index Cond: (id = public.ci_builds.gl_project_id)
Filter: ((NOT pending_delete) AND builds_enabled AND shared_runners_enabled)
Rows Removed by Filter: 0
-> Bitmap Heap Scan on ci_builds (cost=102.63..106.64 rows=1 width=1002) (actual time=1.216..1.231 rows=8 loops=187)
Recheck Cond: ((gl_project_id = public.ci_builds.gl_project_id) AND ((status)::text = 'pending'::text))
Filter: ((runner_id IS NULL) AND ((type)::text = 'Ci::Build'::text))
-> BitmapAnd (cost=102.63..102.63 rows=1 width=0) (actual time=1.201..1.201 rows=0 loops=187)
-> Bitmap Index Scan on index_ci_builds_on_gl_project_id (cost=0.00..10.52 rows=241 width=0) (actual time=0.406..0.406 rows=1944 loops=187)
Index Cond: (gl_project_id = public.ci_builds.gl_project_id)
-> Bitmap Index Scan on index_ci_builds_on_status (cost=0.00..91.78 rows=3089 width=0) (actual time=0.652..0.652 rows=3362 loops=187)
Index Cond: ((status)::text = 'pending'::text)
Total runtime: 303.832 ms
```
## Specific runners
It doesn't affect the specific runners which still serve builds FIFO.
@stanhu @markpundsack @yorickpeterse What do you think?
See merge request !4634
Reduce SQL query counts in IssueReferenceFilter
## What does this MR do?
This MR adds a preparation phase for reference filters that allows them to prepare/create data structures used while iterating over HTML nodes. In this particular case the preparation phase is used for issue references to greatly cut down the amount of queries executed to get projects/issues for Markdown references.
## Are there points in the code the reviewer needs to double check?
No.
## Why was this MR needed?
Rendering Markdown containing issue references would run at most two queries for every issue reference: one to get the project and one to get the issue from said project. When rendering Markdown with lots of issue references this would result in _a lot_ of queries being executed.
## What are the relevant issue numbers?
#18042
See merge request !4410
Improve pipeline status in case that pipeline has no jobs
## What does this MR do?
This MR resolves problem with pipeline status when there are no build in pipeline.
This can happen when builds were skipped - for example - by using `only`/`except` keyword in `.gitlab-ci.yml`.
## What are the relevant issue numbers?
Closes#17977
See merge request !4403
Turn Group#owners into a has_many association
## What does this MR do?
This turns the regular method `Group#owners` into a `has_many` association.
## Are there points in the code the reviewer needs to double check?
As far as I can tell there's no way to do this without using an intermediate association, but perhaps I'm missing something. The reason an intermediate association is needed is because the supplied Proc is applied to the _final_ association (the one returning users), this means that when using a single `has_many` you can't filter out any intermediate rows (e.g. group members).
## Why was this MR needed?
This code being a regular method would prevent eager loading of the owners of a Group, turning it into a `has_many` association resolves this problem. This was discovered in !4410.
## What are the relevant issue numbers?
None.
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] ~~[Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)~~
- [x] ~~API support added~~
- [ ] Tests
- [x] Added for this feature/bug
- [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !4676
Show categorised search queries in the search autocomplete
Fixes#5885
It works in 3 categories, Dashboard, Groups and Project.
## Dashboard context
## Group context
## Project context
## Screencast
See merge request !4499
Fixed issue with bold in issuable sidebar
## What does this MR do?
~~The sidebar contained both normal font weight & bold font weight, so this standardises them to bold.~~
After looking at the designs, i've updated the font weights & colors throughout the sidebar to correctly match them
## Screenshots (if relevant)
See merge request !4398
Resolve "Change admin navigation to match new UI"
## What does this MR do?
Moves admin navigation to layout nav to match Project, Groups, and Profile navigation
## Are there points in the code the reviewer needs to double check?
## Why was this MR needed?
## What are the relevant issue numbers?
Closes#18338
## Screenshots (if relevant)
## Does this MR meet the acceptance criteria?
- [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- [ ] Tests
- [ ] Added for this feature/bug
- [ ] All builds are passing
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !4650
Add support for Docker Registry manifest v1
## What does this MR do?
Adds support for Manifest V1 generated by older versions of Docker (before 1.10).
## What are the relevant issue numbers?
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/18609
See merge request !4669
Add environments and deployments
This MR is a continuation of https://gitlab.com/gitlab-org/gitlab-ce/issues/17009.
The current implementation is as follow:
1. We have two new tables: `environments` and `deployments`.
2. We have a new tab: `Environments` under `Pipelines` where you can see all you environments and add a new one.
3. We add a new option to `.gitlab-ci.yml` to track where we should create a deployment for environment.
4. If environment in `.gitlab-ci.yml` is specified it will create a deployment. **If environment does not exist it will be created.** (this got changed)
5. The deployment is always successful and shows the time of the action, in that case a build that presumably should do deployment. In the future we could extend deployment with statuses: success, failure. We could extend deployments with information that this is partial or full deployment.
6. User have to create environments that he will track first.
7. User can remove environments.
8. User can retry/rollback past deployment (in that case we retry past build). The new build when succeeds it will create a new deployment.
9. Currently environment have only one parameter: `name`. In the future it should have: `variables`, `credentials` and possibly `runners` and maybe other resources.
10. Currently deployment have this parameters: `sha`, `ref`, `deployable (in this case a build)`, `user (who triggered a deployment)`, `created_at`.
The `.gitlab-ci.yml`:
```
deploy to production:
stage: deploy
script: dpl travis...
environment: production
```
What needs to be done:
- [x] Write initial implementation
- [x] Improve implementation (@ayufan)
- [x] Write tests (@ayufan)
- [x] Improve UX of the forms (cc @markpundsack) - reviewed by @markpundsack
- [x] Improve implementation of the views (cc @jschatz1) - done by @iamphill
- [x] Write .gitlab-ci.yml documentation for `environments` - done by @ayufan
- [ ] Write user documentation (@ayufan and @markpundsack)
See merge request !4605
Fixed issue with move dropdown not being searchable
## What does this MR do?
Adds the ability to search the project move dropdown.
## What are the relevant issue numbers?
Closes#18634
## Screenshots (if relevant)
See merge request !4666
Don't allow U2F set up unless an authenticator app is set up
Closes#17333
# TODO
- [ ] #17333 Authenticator should be set up before enabling U2F
- [x] Implementation
- [x] Fix/add tests
- [x] Refactor
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/964c98a3c427cac6e3de88ddc74a9f172ee9742d/builds) to pass
- [x] Assign to endboss for review
- [x] Address @DouweM's comments
- [x] No need for `javascript:void(0)`
- [x] Add screenshots
- [ ] Wait for merge
# Screenshots
See merge request !4585
Resolve "Track the number of new Redis connections per transaction"
## What does this MR do?
Add a new metric counter, `new_redis_connections`, that contains the number of calls to `Redis::Client#connect` in the current transaction.
## Are there points in the code the reviewer needs to double check?
Not sure. I tested this in kind of a brute-force way:
1. Add a debugger in the monkey-patched `connect` method.
2. With metrics enabled, start the app and load a page.
3. The first Redis connection is created by `Rack::Attack` and isn't in a transaction, but still works fine.
4. The second Redis connection is within a transaction (the page load), and increments the counter.
5. If I reload the page, neither debugger is hit.
6. If I use a Redis client and do `CLIENT KILL` on my two existing clients, then reload the page, I get 3 and 4 again.
7. If I disable metrics collection, the debugger never gets hit.
## Why was this MR needed?
We may have a Redis connection leak somewhere, so adding metrics will let us track this.
## What are the relevant issue numbers?
Closes#18451.
## Screenshots (if relevant)
Hahaha nope, not relevant.
## Does this MR meet the acceptance criteria?
- [ ] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [ ] API support added
- [ ] Tests
- [ ] Added for this feature/bug
- [ ] All builds are passing
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [ ] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
cc @yorickpeterse
See merge request !4649
Don't fail builds for projects that are deleted when they are stuck
## What does this MR do?
Solves when dropping stuck connection.
```
Dropping stuck pending build 1545510 for runner
NoMethodError: undefined method `origin_merge_requests' for nil:NilClass
from /opt/gitlab/embedded/service/gitlab-rails/app/services/merge_requests/base_service.rb:50:in `merge_request_from'
from /opt/gitlab/embedded/service/gitlab-rails/app/services/merge_requests/base_service.rb:57:in `each_merge_request'
from /opt/gitlab/embedded/service/gitlab-rails/app/services/merge_requests/add_todo_when_build_fails_service.rb:5:in `execute'
from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/instrumentation.rb:164:in `execute'
from /opt/gitlab/embedded/service/gitlab-rails/app/models/commit_status.rb:51:in `block (2 levels) in <class:CommitStatus>'
from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/state_machines-0.4.0/lib/state_machines/eval_helpers.rb:79:in `call'
...
```
This happens, because that `default_scope` of `Projects` filters returns the projects that are not deleted, where `Ci::Build` doesn't take into account.
See merge request !4609
Manually create todo for issuable
## What does this MR do?
Adds a button to the sidebar in issues & merge requests to allow users to manually create a todo item themselves.
## What are the relevant issue numbers?
Closes#15045
## Screenshots (if relevant)
See merge request !4502
We can't do a lot with classes without names as we can't filter by them,
have no idea where they come from, etc. As such it's best to just ignore
these.
Allow users to request access to projects and groups
#### TODO:
- [x] Factorize the projects & groups members list
- [x] Factorize `request_access` and `approve` controller methods into a concern
- [x] Factorize the project & group dropdowns
- [x] Factorize access request routes
- [x] Factorize mailers
#### Will be done in separate MRs
- [ ] Documentation
- [ ] API
#### Access request to project
- [x] Design project access request button to spec (defined in #13948)
- [x] Request access to a project as a logged in user
- [x] Withdraw access request to a project as a logged in user
- [x] Do not treat project member request as actual membership (done partly in Ability)
- [x] Display access requests on project's members page (+ allow to change level, approve, and deny request)
- [x] Email requesting user at notification email address when accepted/denied
- [x] Email all project admins at notification email addresses when new access request
- [x] Unit specs to cover new functionality: `AccessRequestable` is missing specs
- [x] Controller specs to cover new functionality
- [x] Mailer specs to cover new functionality
- [x] Features specs to cover new UI and interactions
- [x] Regression specs still passing
#### Access request to group
- [x] Design group access request button to spec (defined in #13948)
- [x] Request access to a group as a logged in user
- [x] Withdraw access request to a group as a logged in user
- [x] Do not treat group member request as actual membership
- [x] Display access requests on group's members page (+ allow to change level, approve, and deny request)
- [x] Email requesting user at notification email address when accepted/denied
- [x] Email all project admins at notification email addresses when new access request
- [x] Unit specs to cover new functionality
- [x] Controller specs to cover new functionality
- [x] Mailer specs to cover new functionality
- [x] Features specs to cover new UI and interactions
- [x] Regression specs still passing
Closes#13948.
### Requesting access to a project
#### Responsive UI
#### Request access link
#### Flash once access is requested
#### Email sent to project admins on new access requests
##### HTML Email
##### Plain Email
### Withdrawing access request to a project
#### Withdraw access request link
#### Withdraw access request confirmation dialog box
#### Flash once access has been withdrawn
### Access requests management page
#### Deny access request confirmation dialog box
#### Email sent to requester on denied access requests
##### HTML Email
##### Plain Email
#### Email sent to requester on granted access requests
##### HTML Email
##### Plain Email
----------------------------------
----------------------------------
### Requesting access to a group
#### Responsive UI
#### Request access link
#### Flash once access is requested
#### Email sent to project admins on new access requests
##### HTML Email
##### Plain Email
### Withdrawing access request to a project
#### Withdraw access request link
#### Withdraw access request confirmation dialog box
#### Flash once access has been withdrawn
### Access requests management page
#### Deny access request confirmation dialog box
#### Email sent to requester on denied access requests
##### HTML Email
##### Plain Email
#### Email sent to requester on granted access requests
##### HTML Email
##### Plain Email
See merge request !3798
By default instrumentation will instrument public,
protected and private methods, because usually
heavy work is done on private method or at least
that’s what facts is showing
Rémy CoutableandRobert Speicher
Robert SpeicherandRobert Speicher
Douwe MaanandRobert Speicher
Achilleas PipinellisandRobert Speicher
Yorick PeterseandRobert Speicher
Stan HuandRobert Speicher
Jacob SchatzandRobert Speicher
Dmitriy Zaporozhets
Fatih AcetandJacob Schatz
Douwe Maan
Phil Hughes
Kamil Trzcinski
Timothy Andrew
Douglas Barbosa Alexandre
Alfredo Sumaran
Sean McGivern
Paco Guzman