Allow specifying protected branches using wildcards
Closes#18627
# Tasks
- [ ] #18627!4665 Allow specifying protected branches using wildcards
- [x] Find existing usages of protected branches
- Protecting branches
- `ProtectedBranchesController` is used to mark a branch protected/unprotected
- `API::Branches` can be used to mark a branch protected/unprotected
- Enforcing branch protection
- `Gitlab::GitAccess` has helpers (`can_push_to_branch?`, `check`) that are used to deny pushes if a branch is protected
- Over SSH: `gitlab-shell` receives a push, and calls `/allowed` on the GitLab API, which calls `GitAccess.check`
- Over HTTP:
- `gitlab-workhorse` receives the request, and forwards it to rails
- Rails (in the `GitHttpController#git-recieve-pack`) runs basic checks (is the user logged in, not protected branch checks) and returns ok with `GL_ID` and `RepoPath`
- `gitlab-workhorse` looks at the response, and calls the relevant `gitlab-shell` action from `git-http/handlePostRPC`
- Rest of this flow is the same as the SSH flow above
- [x] Implementation
- [x] Backend
- [x] Change `project#protected_branch?` to look at wildcard protected branches
- [x] Change `project#developers_can_push_to_protected_branch?`
- [x] Change `project#open_branches`
- [x] Better error message when creating a disallowed branch from the Web UI
- [x] Frontend
- [x] Protected branches page should allow typing out a wildcard pattern
- [x] Add help text explaining the use of wildcards
- [x] Show matching branches for each protected branch
- [x] ~~On the index page~~
- [x] On a show page
- [x] Index?
- [x] Can't have the "last commit" column for wildcard protected branches
- [x] Fix / write tests
- [x] What happens if a hook is missing in dev?
- [x] Refactor
- [x] Test workflows
- Create a branch matching a wildcard pattern
- Push to a branch matching a wildcard pattern
- Force push to a branch matching a wildcard pattern
- Delete a branch matching a wildcard pattern
- [x] Test using Web UI
- [x] Test over SSH
- [x] Test over HTTP
- [x] Test as developer and master
- [x] Investigate performance
- [x] Test with a large number of protected branches / branches
- [x] Paginate list of protected branches
- [x] ~~Possibly rewrite `open_branches`~~
- [x] Add `iid`s to existing `ProtectedBranch`es
- [x] Add documentation
- [x] Add CHANGELOG entry
- [x] Add screenshots
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/2f753e3ed2ce681b4444944d521f4419e8ed37f7/builds) passes
- [x] Assign to endboss for review
- [x] Address @DouweM's comments
- [x] `protected_branch_params`
- [x] `exact_match` instead of `explicit_match`
- [x] When would self.name be blank?
- [x] Move `protected_branches.each` to a partial
- [x] Move `matching_branches.each` to a partial
- [x] If the branch is in @matching_branches, it's not been removed
- [x] move this regex to a method and memoize it
- [x] `commit_sha` directly for exact matches
- [x] Number of matches for wildcard matches, with a link
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/43f9ce0e88194b8f719bb1c1e656b7fc13278d56/builds) to pass
- [x] Respond to @DouweM's comments
- [x] Don't use iid
- [x] Controller should use `@project.protected_branches.new`
- [x] move the memoization to `def wildcard_regex`
- [x] render with `collection: @protected_branches`
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/f7beedf122fa0c7aa89e86181fe7499321fb10ca/builds) to pass
- [x] Wait for @DouweM's review
- [x] Wait for @jschatz1's review
- [x] Respond to @jschatz1's comments
- [x] Use the new dropdown style
- [x] description should be moved to the description section without the styling
- [x] Protect button should be disabled when no branch is selected
- [x] Update screenshots
- [x] Merge conflicts
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/20f3cfe8d5540eab64c2ba548043d600b28c61ba/builds) passes
- [ ] Revisit performance, possibly with staging/production data
- [ ] Get a dump of staging / run against staging live
- [ ] Get SSH access to staging
- [ ] Wait for review/merge
# Screenshots
## Creating wildcard protected branches
### Using the `GLDropdown` component
## Enforcing wildcard protected branches
### From the Web UI
### Over SSH
### Over HTTPS
## Listing matching branches
See merge request !4665
1. The `open_branches` method is used to provide a list of branches
while creating a protected branch.
2. It makes sense to include branches which are matched by one or more
wildcard protected branches, since the user might want to make exact
protected branches from these as well.
3. This also provides a large performance improvement. On my machine, in
a project with 5000 branches and 2000 protected branches, the
`ProtectedBranches#index` page went from a 40 seconds load time to 4
seconds (10x speedup).
Fixing URL validation for import_url on projects
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/17536
This MR fixes problems related to bypassing `import_url` validation on projects. This makes sure the URL is properly validated so we don't enter crap and fail while running workers that handle this URL.
It also adds a migration to fix current invalid `import_url`s
See merge request !4753
1. The main implementation is in the `ProtectedBranch` model. The
wildcard is converted to a Regex and compared. This has been tested
thoroughly.
- While `Project#protected_branch?` is the main entry point,
`project#open_branches` and
`project#developers_can_push_to_protected_branch?`
have also been modified to work with wildcard protected branches.
- The regex is memoized (within the `ProtectedBranch` instance)
2. Improve the performance of `Project#protected_branch?`
- This method is called from `Project#open_branches` once _per branch_
in the project, to check if that branch is protected or not.
- Before, `#protected_branch?` was making a database call every
time it was invoked (in the above case, that amounts to once
per branch), which is expensive.
- This commit caches the list of protected branches in memory, which
reduces the number of database calls down to 1.
- A downside to this approach is that `#protected_branch?` _could_
return a stale value (due to the caching), but this is
an acceptable tradeoff.
3. Remove the (now) unused `Project#protected_branch_names` method.
- This was previously used to check for protected branch status.
This ensures that Project.find_with_namespace returns a row matching
literally as the first value, instead of returning a random value.
The ordering here is _only_ applied to Project.find_with_namespace and
_not_ Project.where_paths_in as currently there's no code that requires
Project.where_paths_in to return rows in a certain order. Since this
method also returns all rows that match there's no real harm in not
setting a specific order either. Another reason is that generating all
the "WHEN" arms for multiple values in Project.where_paths_in becomes
really messy.
On MySQL we have to use the "BINARY" operator to turn a "WHERE" into a
case-sensitive WHERE as otherwise MySQL may still end up returning rows
in an unpredictable order.
Fixesgitlab-org/gitlab-ce#18603
Add environments and deployments
This MR is a continuation of https://gitlab.com/gitlab-org/gitlab-ce/issues/17009.
The current implementation is as follow:
1. We have two new tables: `environments` and `deployments`.
2. We have a new tab: `Environments` under `Pipelines` where you can see all you environments and add a new one.
3. We add a new option to `.gitlab-ci.yml` to track where we should create a deployment for environment.
4. If environment in `.gitlab-ci.yml` is specified it will create a deployment. **If environment does not exist it will be created.** (this got changed)
5. The deployment is always successful and shows the time of the action, in that case a build that presumably should do deployment. In the future we could extend deployment with statuses: success, failure. We could extend deployments with information that this is partial or full deployment.
6. User have to create environments that he will track first.
7. User can remove environments.
8. User can retry/rollback past deployment (in that case we retry past build). The new build when succeeds it will create a new deployment.
9. Currently environment have only one parameter: `name`. In the future it should have: `variables`, `credentials` and possibly `runners` and maybe other resources.
10. Currently deployment have this parameters: `sha`, `ref`, `deployable (in this case a build)`, `user (who triggered a deployment)`, `created_at`.
The `.gitlab-ci.yml`:
```
deploy to production:
stage: deploy
script: dpl travis...
environment: production
```
What needs to be done:
- [x] Write initial implementation
- [x] Improve implementation (@ayufan)
- [x] Write tests (@ayufan)
- [x] Improve UX of the forms (cc @markpundsack) - reviewed by @markpundsack
- [x] Improve implementation of the views (cc @jschatz1) - done by @iamphill
- [x] Write .gitlab-ci.yml documentation for `environments` - done by @ayufan
- [ ] Write user documentation (@ayufan and @markpundsack)
See merge request !4605
+ Move 'Edit Project/Group' out of membership-related partial
+ Show the access request buttons only to logged-in users
+ Put the request access buttons out of in a more visible button
+ Improve the copy in the #remove_member_message helper
Signed-off-by: Rémy Coutable <remy@rymai.me>
Add Project.where_paths_in
In https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4410 I'm working on reducing the SQL queries needed to render Markdown. One reason for the large amount of queries is executing a large number of queries needed to find projects and related data. Basically `Project.find_with_namespace` is called in a loop and then any relations have to be retrieved separately.
By using `Project.where_paths_in` we can work around this by doing something like:
```ruby
project_paths = [...] # populated by some method
projects = Project.where_paths_in(project_paths).includes(:namespace, ...)
```
Ref: https://gitlab.com/gitlab-org/gitlab-ce/issues/18042
See merge request !4535
This method can be used to find multiple projects for multiple paths.
For example, take this snippet:
Project.where_paths_in(%w{gitlab-org/gitlab-ce gitlab-org/gitlab-ee})
This will return an ActiveRecord::Relation containing the GitLab CE and
GitLab EE projects.
This method takes care of matching rows both case-sensitively and
case-insensitively where needed.
Project.find_with_namespace in turn has been modified to use
Project.where_paths_in without nuking any scoping (instead it uses
reorder(nil)). This means that any default scopes (e.g. those used for
"pending_delete" stay intact).
The method Project.where_paths_in was added so the various Markdown
filters can use a single query to grab all the projects referenced in a
set of documents, something Project.find_with_namespace didn't allow.
Updated error message when project limit is zero
When the project limit is zero & the user tries to create a project, the
error displayed is now 'Personal project creation is not allowed. Please
contact your administrator with questions'
Closes#17691
See merge request !4280
In 8278b763d9 the default behaviour of annotation
has changes, which was causing a lot of noise in diffs. We decided in #17382
that it is better to get rid of the whole annotate gem, and instead let people
look at schema.rb for the columns in a table.
Fixes: #17382
This changes Project#protected_branch? to use a query to check if a
branch is protected, instead of loading all ProtectedBranch records into
memory just to check if the list of names includes a given branch name.
This caches the output of Repository#exists? in Redis while making sure
it's flushed properly when creating new repositories, deleting them,
etc.
For the ProjectWiki tests to work I had to make ProjectWiki#create_repo!
public as testing private methods in RSpec is a bit of a pain.
Douwe Maan
Timothy Andrew
Douwe Maan
Robert Speicher
Rémy Coutable
James Lopez
Alejandro Rodríguez
Ruben Davila
Yorick Peterse
Kamil Trzcinski
Douglas Barbosa Alexandre
Paco GuzmanandYorick Peterse
Phil Hughes
Robert Speicher
Jeroen van Baarsen
Stan Hu
Zeger-Jan van de Weg
Felipe Artur