Added setting to set new users by default as external
## What does this MR do?
This implements the feature request #14508. It adds an option in the application settings to set new users by default as external.
## Are there points in the code the reviewer needs to double check?
Everything. Like I mentioned in the discussion of the issue my knowledge of Ruby basically doesn't exists. I tested it on my machine and it seems to work, but as I am very unexperienced in Ruby I highly recommend to take a close look at the code.
## Why was this MR needed?
It was requested by @DouweM to work on the issue with the proposed changes by me.
## What are the relevant issue numbers?
This MR is for the issue #14508 that followed up after the implementation of #4009.
See merge request !4545
Allow specifying protected branches using wildcards
Closes#18627
# Tasks
- [ ] #18627!4665 Allow specifying protected branches using wildcards
- [x] Find existing usages of protected branches
- Protecting branches
- `ProtectedBranchesController` is used to mark a branch protected/unprotected
- `API::Branches` can be used to mark a branch protected/unprotected
- Enforcing branch protection
- `Gitlab::GitAccess` has helpers (`can_push_to_branch?`, `check`) that are used to deny pushes if a branch is protected
- Over SSH: `gitlab-shell` receives a push, and calls `/allowed` on the GitLab API, which calls `GitAccess.check`
- Over HTTP:
- `gitlab-workhorse` receives the request, and forwards it to rails
- Rails (in the `GitHttpController#git-recieve-pack`) runs basic checks (is the user logged in, not protected branch checks) and returns ok with `GL_ID` and `RepoPath`
- `gitlab-workhorse` looks at the response, and calls the relevant `gitlab-shell` action from `git-http/handlePostRPC`
- Rest of this flow is the same as the SSH flow above
- [x] Implementation
- [x] Backend
- [x] Change `project#protected_branch?` to look at wildcard protected branches
- [x] Change `project#developers_can_push_to_protected_branch?`
- [x] Change `project#open_branches`
- [x] Better error message when creating a disallowed branch from the Web UI
- [x] Frontend
- [x] Protected branches page should allow typing out a wildcard pattern
- [x] Add help text explaining the use of wildcards
- [x] Show matching branches for each protected branch
- [x] ~~On the index page~~
- [x] On a show page
- [x] Index?
- [x] Can't have the "last commit" column for wildcard protected branches
- [x] Fix / write tests
- [x] What happens if a hook is missing in dev?
- [x] Refactor
- [x] Test workflows
- Create a branch matching a wildcard pattern
- Push to a branch matching a wildcard pattern
- Force push to a branch matching a wildcard pattern
- Delete a branch matching a wildcard pattern
- [x] Test using Web UI
- [x] Test over SSH
- [x] Test over HTTP
- [x] Test as developer and master
- [x] Investigate performance
- [x] Test with a large number of protected branches / branches
- [x] Paginate list of protected branches
- [x] ~~Possibly rewrite `open_branches`~~
- [x] Add `iid`s to existing `ProtectedBranch`es
- [x] Add documentation
- [x] Add CHANGELOG entry
- [x] Add screenshots
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/2f753e3ed2ce681b4444944d521f4419e8ed37f7/builds) passes
- [x] Assign to endboss for review
- [x] Address @DouweM's comments
- [x] `protected_branch_params`
- [x] `exact_match` instead of `explicit_match`
- [x] When would self.name be blank?
- [x] Move `protected_branches.each` to a partial
- [x] Move `matching_branches.each` to a partial
- [x] If the branch is in @matching_branches, it's not been removed
- [x] move this regex to a method and memoize it
- [x] `commit_sha` directly for exact matches
- [x] Number of matches for wildcard matches, with a link
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/43f9ce0e88194b8f719bb1c1e656b7fc13278d56/builds) to pass
- [x] Respond to @DouweM's comments
- [x] Don't use iid
- [x] Controller should use `@project.protected_branches.new`
- [x] move the memoization to `def wildcard_regex`
- [x] render with `collection: @protected_branches`
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/f7beedf122fa0c7aa89e86181fe7499321fb10ca/builds) to pass
- [x] Wait for @DouweM's review
- [x] Wait for @jschatz1's review
- [x] Respond to @jschatz1's comments
- [x] Use the new dropdown style
- [x] description should be moved to the description section without the styling
- [x] Protect button should be disabled when no branch is selected
- [x] Update screenshots
- [x] Merge conflicts
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/20f3cfe8d5540eab64c2ba548043d600b28c61ba/builds) passes
- [ ] Revisit performance, possibly with staging/production data
- [ ] Get a dump of staging / run against staging live
- [ ] Get SSH access to staging
- [ ] Wait for review/merge
# Screenshots
## Creating wildcard protected branches
### Using the `GLDropdown` component
## Enforcing wildcard protected branches
### From the Web UI
### Over SSH
### Over HTTPS
## Listing matching branches
See merge request !4665
New diff notes
Fixes#12732, #14731, #19375, #14783
Builds on https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4110
To do:
- [x] Get it mostly working
- [x] Validate position validity
- [x] Fix: Don’t link to `#`
- [x] Fix: Base ref can be `nil`, potentially, when the MR has an oprhan source branch => Yep, doesn’t work. We need to store a `start_id`
- [x] Optimize: Fewer duplicate `git diff` compares
- [x] Optimize: Pass paths to `PositionTracer#diff` for faster diffs
- [x] Refactor: Use `head_id` in `MergeRequest`/`MergeRequestDiff` instead of `source_sha`
- [x] Refactor: Convert existing array-based diff refs to the DiffRefs model
- [x] Tweak: Use `note_type` in `Autosave` key
- [x] Tweak: Remove `line_code: note.line_code` from `link_to_reply_discussion`
- [x] Update: `SentNotifications` and reply-by-email receiver
- [x] Update: MR diff notification email
- [x] Update: API (MR, Commit note creation and entity)
- [x] Update: GitHub importer
- [x] Address any other TODO comments
- [x] Fix: Suppress "edited 4 minutes ago"
- [x] Write tests
- [x] `LineMapper`
- [x] `PositionTracer`
- [x] `Position`
- [x] `DiffPositionUpdateService`
- [x] `DiffNote`
- [x] `MergeRequests::RefreshService` / `MergeRequest#update_diff_notes_positions`
- [x] Make sure commits with diff notes don't get cleaned up, since this would prevent the diff notes from being rendered (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5062)
Future improvements:
- Display unresolved comments on files outside the diff, if the comment was added when that file _was_ part of the diff
- Allow commenting on sections between hunks, when expanding the diff using `...`
- (We'd need to generate line code based on Position if we have it, even if it falls outside bounds of diff)
- `diff_hunk` on diff note API entity
- Show diff hunk in notification email
- Resolved line notes would have a boolean, and be inactive through `notes.any? { !active? || resolved? }`
- Multi line notes would store a number of positions, and do the right thing (™) in grouping and then rendering if the first item is multiline? => true
- Image diff notes could store x,y,width,height instead of old_line,new_line for similar grouping. Does it need a reference to say if it's on old or new? These can't have line_codes, clearly. Rendering would be interesting.
- Show commit line comments in the MR diff
- Comment on specific selected words
- Comment on file header
- Unfold top of discussion diff note
- New diff notes API for commits and MRs
/cc @rspeicher
See merge request !4101
If the Sidekiq job fails for some reason, a project can be 'stuck'
pending deletion. The project can't be viewed, so it shouldn't be
available through the notification settings association as this will
throw an exception when we try to show the link.
As requested by the issue #14508 this adds an option in the application
settings to set newly registered users by default as external. The
default setting is set to false to stay backward compatible.
1. The `open_branches` method is used to provide a list of branches
while creating a protected branch.
2. It makes sense to include branches which are matched by one or more
wildcard protected branches, since the user might want to make exact
protected branches from these as well.
3. This also provides a large performance improvement. On my machine, in
a project with 5000 branches and 2000 protected branches, the
`ProtectedBranches#index` page went from a 40 seconds load time to 4
seconds (10x speedup).
Fix diff comments not showing up in activity feed
## What does this MR do?
It fixes the detection of note events to check for `Note` and `LegacyDiffNote`.
## Are there points in the code the reviewer needs to double check?
No? /cc @DouweM (since I believe you introduced `LegacyDiffNote`
## Why was this MR needed?
To fix#19092.
## What are the relevant issue numbers?
Fixes#19092.
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
- [x] Added for this feature/bug
- [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !5069
Fixing URL validation for import_url on projects
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/17536
This MR fixes problems related to bypassing `import_url` validation on projects. This makes sure the URL is properly validated so we don't enter crap and fail while running workers that handle this URL.
It also adds a migration to fix current invalid `import_url`s
See merge request !4753
1. The main implementation is in the `ProtectedBranch` model. The
wildcard is converted to a Regex and compared. This has been tested
thoroughly.
- While `Project#protected_branch?` is the main entry point,
`project#open_branches` and
`project#developers_can_push_to_protected_branch?`
have also been modified to work with wildcard protected branches.
- The regex is memoized (within the `ProtectedBranch` instance)
2. Improve the performance of `Project#protected_branch?`
- This method is called from `Project#open_branches` once _per branch_
in the project, to check if that branch is protected or not.
- Before, `#protected_branch?` was making a database call every
time it was invoked (in the above case, that amounts to once
per branch), which is expensive.
- This commit caches the list of protected branches in memory, which
reduces the number of database calls down to 1.
- A downside to this approach is that `#protected_branch?` _could_
return a stale value (due to the caching), but this is
an acceptable tradeoff.
3. Remove the (now) unused `Project#protected_branch_names` method.
- This was previously used to check for protected branch status.
* master: (98 commits)
Enable Style/EmptyLines cop, remove redundant ones
Update CHANGELOG
Cache results from jQuery selectors to retrieve namespace name
Fix import button when import fail due the namespace already been taken
Fix snippets comments not displayed
Fix emoji paths in relative root configurations
Exclude requesters from Project#members, Group#members and User#members
Upgrade Thin from 1.6.1 to 1.7.0.
Many squashed commits
Cache autocomplete results
Upgrade Sidekiq from 4.1.2 to 4.1.4.
Upgrade seed-fu from 2.3.5 to 2.3.6
use has_many relationship with events
Support creating a todo on issuables via API
Expose target, filter by state as string
Add todos API documentation and changelog
Improve the request / withdraw access button
Metrics for Rouge::Plugins::Redcarpet and Rouge::Formatters::HTMLGitlab
Groundwork for Kerberos SPNEGO (EE feature)
Update CHANGELOG 8.9.5 for runners related fixes
...
Exclude requesters from Project#members, Group#members and User#members
## What does this MR do?
It excludes requesters from the `Project#members`, `Group#members` and `User#members` associations, and adds new `Project#requesters` and `Group#requesters` associations.
## Are there points in the code the reviewer needs to double check?
No.
## Why was this MR needed?
Without this, if you call `project.members`, requesters are included in the results! This is at best misleading, and at worst can lead to security issues. By excluding requesters from the `#members` associations, we avoid introducing security inadvertently since you have to call the `#requesters` association explicitly to get requesters.
## What are the relevant issue numbers?
This is something I realized while fixing the security issue #19102.
## Does this MR meet the acceptance criteria?
- [x] I don't think this needs a CHANGELOG since this is an internal change
- Tests
- [x] Added for this feature/bug
- [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !4946
When pushing commits to existing branches we don’t
need to flush branch git data (branch names / counts)
When flushes the cache when pushing commits skip to
flush branch and tag git data (names / counts) because
those operations are managed explicitly in each case
Repopulated expired cache as soon as possible
This method now should return false if build is not completed. If build
is running then it is not retryable, therefore `Ci::Build#retryable?`
returned wrong value. This commit changes this behavior
The issue was with the `User#groups` and `User#projects` associations
which goes through the `User#group_members` and `User#project_members`.
Initially I chose to use a secure approach by storing the requester's
user ID in `Member#created_by_id` instead of `Member#user_id` because I
was aware that there was a security risk since I didn't know the
codebase well enough.
Then during the review, we decided to change that and directly store the
requester's user ID into `Member#user_id` (for the sake of simplifying
the code I believe), meaning that every `group_members` / `project_members`
association would include the requesters by default...
My bad for not checking that all the `group_members` / `project_members`
associations and the ones that go through them (e.g. `Group#users` and
`Project#users`) were made safe with the `where(requested_at: nil)` /
`where(members: { requested_at: nil })` scopes.
Now they are all secure.
Signed-off-by: Rémy Coutable <remy@rymai.me>
Postgres and MySQL don't guarantee that pagination with `LIMIT` and
`OFFSET` will work if the ordering isn't unique. From the Postgres docs:
> When using `LIMIT`, it is important to use an `ORDER BY` clause that
> constrains the result rows into a unique order. Otherwise you will get
> an unpredictable subset of the query's rows
Before:
[1] pry(main)> issues = 1.upto(Issue.count).map { |i| Issue.sort('priority').page(i).per(1).map(&:id) }.flatten
[2] pry(main)> issues.count
=> 81
[3] pry(main)> issues.uniq.count
=> 42
After:
[1] pry(main)> issues = 1.upto(Issue.count).map { |i| Issue.sort('priority').page(i).per(1).map(&:id) }.flatten
[2] pry(main)> issues.count
=> 81
[3] pry(main)> issues.uniq.count
=> 81
Resolve "Destroying a project causes post_decline_request to be executed"
## What does this MR do?
Ensure we don't send "access request declined" to access requesters when a project is deleted.
## Are there points in the code the reviewer needs to double check?
I've created a service to decouple the notification sending from the AR model.
## Why was this MR needed?
Because there was an issue.
## What are the relevant issue numbers?
Fixes#18755, #18750.
## Does this MR meet the acceptance criteria?
- [x] No CHANGELOG needed.
- [x] Tests
- [x] Added for this feature/bug
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !4744
A lot of git operations were being repeated, for example, to build a url
you would ask if the path was a Tree, which would call a recursive routine
in Gitlab::Git::Tree#where, then ask if the path was a Blob, which would
call a recursive routine at Gitlab::Git::Blob#find, making reference to
the same git objects several times. Now we call Rugged::Tree#path, which
allows us to determine the type of the path in one pass.
Some other minor improvement added, like saving commonly used references
instead of calculating them each time.
Rémy Coutable
Douwe Maan
Douwe Maan
Sean McGivern
Dravere
Timothy Andrew
Robert Speicher
Douglas Barbosa Alexandre
Dmitriy Zaporozhets
Valery Sizov
Tomasz Maczukin
Grzegorz Bizon
Paco Guzman
James Lopez
Alejandro Rodríguez
Tim
Robert Speicher
Ruben Davila
Yorick Peterse