Fix emoji paths in relative root configurations
## What does this MR do?
If a site specifies a relative URL root, emoji files would omit the path from the URL, leading to lots of 404s.
## Are there points in the code the reviewer needs to double check?
At first, I tried to use `ActionView::Helpers::AssetUrlHelper.asset_url` since this is what it's intended to do. But this helper function is extremely slow, and it took minutes to generate the URLs for the hundreds of links needed for each emoji.
## Why was this MR needed?
Because emojis were broken in relative URL installations
## What are the relevant issue numbers?
#15642
## Does this MR meet the acceptance criteria?
- [X] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
- [X] Added for this feature/bug
- [x] All builds are passing
- [X] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [X] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [X] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !5027
(cherry picked from commit 88dbc4d110)
Fixing problems with events for import/export
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/19202
A couple of issues related to target being missing in exported `Events` (as being polymorphic and not have `ActiveRecord` relationships is a bit more tricky than normal models) plus as the export was in JSON, the import retrieves hashed fields as stringified hashes and not symbolized - so fixed that as well, which was the cause of https://gitlab.com/gitlab-org/gitlab-ce/issues/19202
Also fixed / refactored tests
:simpl
Import/Export Version has been bumped to 0.1.1 as theses changes to events won't work very well with old exports - forcing users to generate a new export in the new version.
See merge request !4987
(cherry picked from commit c368cb605b)
Handle external issues in IssueReferenceFilter
Rendering issue references such as `#1` was broken for projects using an external issues tracker.
See gitlab-org/gitlab-ce#19036
See merge request !4988
(cherry picked from commit 6e82c0e06e)
Fix restore warning message
## What does this MR do?
Fix the restore Rake task so it properly outputs the database warning. This is a pretty important warning and it was not even being output. After this fix, the output looks like the screenshot below.
See merge request !4980
(cherry picked from commit 0144dce7ee)
Fix privilege escalation issue with OAuth external users
Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/19312
This MR fixes a privilege escalation issue, where manually set external users would be reverted back to internal users if they logged in via OAuth and that provider was not in the `external_providers` list.
/cc @douwe
See merge request !1975
(cherry picked from commit 5e6342b7ac)
Better debugging for memory killer middleware
This adds more info to the warning messages output by `MemoryKiller`.
Previously only the PID was showed, making it difficult to debug issues like https://gitlab.com/gitlab-org/gitlab-ce/issues/19124
This adds the worker class and job ID to the log messages.
See merge request !4936
(cherry picked from commit 3659992cd5)
Use clock_gettime for all performance timestamps
This MR adjusts the performance monitoring code to use `Process.clock_gettime` (thus `clock_gettime(3)`) instead of `Time.now`.
Using `Time.now` / `Time.new` adds more overhead than `Process.clock_gettime`, it also doesn't provide a way of getting timestamps in nanoseconds (which `Process.clock_gettime` does allow).
See merge request !4899
(cherry picked from commit 53ad9522c4)
Fix temp file being deleted after the request while importing a GitLab project
Fixes https://gitlab.com/gitlab-com/infrastructure/issues/151
In production, the temporary uploaded file is getting deleted straight after the request so the Sidekiq worker is unable to find it in `/tmp`
Also, improved erroring/logging of this situation.
See merge request !4894
Handle external issues in IssueReferenceFilter
Handling of external issues was broken when I refactored
`IssueReferenceFilter` to use fewer SQL queries.
Fixes#18827
See merge request !4789
Fix subsequent SAML sign ins
Fixes a bug when `auto_link_ldap_user` is `true` that causes SAML users to be unable to sign in a second time.
Fix the problem for https://gitlab.zendesk.com/agent/tickets/22546
See merge request !4718
Resolve "Destroying a project causes post_decline_request to be executed"
## What does this MR do?
Ensure we don't send "access request declined" to access requesters when a project is deleted.
## Are there points in the code the reviewer needs to double check?
I've created a service to decouple the notification sending from the AR model.
## Why was this MR needed?
Because there was an issue.
## What are the relevant issue numbers?
Fixes#18755, #18750.
## Does this MR meet the acceptance criteria?
- [x] No CHANGELOG needed.
- [x] Tests
- [x] Added for this feature/bug
- [x] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [x] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !4744
Signed-off-by: Rémy Coutable <remy@rymai.me>
Move pre_process into render_result
This MR moves `Banzai::Renderer.pre_process` into `Banzai::Renderer.render_result`.
The `pre_process` method was called even when its output would be ignored. See 11a5a4f359ee57029dbfcc9185fc6b47243ea2aa for more details.
See merge request !4830
GitLab CI Yaml template dropdown
## What does this MR do?
Make it possible to select a dropdown for an easy start with GitLab CI.
## What are the relevant issue numbers?
Closes#17521
## TODO
- [ ] Backend
- [x] CHANGELOG item
- [x] Fix rubocop failure
- [x] API Support
- [x] New tests
- [x] Add disclaimer to the top of the gitlab-ci.yml
- [ ] Frontend
- [x] New tests
See merge request !4411
Make it possible to lock runner on a specific project
Make it possible to lock runner on a specific project.
----
----
Closes#3407
See merge request !4093
Show proper image ID on registry page
## What does this MR do?
Display the container registry image ID (from the config blob) instead of the first image layer ID
## Are there points in the code the reviewer needs to double check?
## Why was this MR needed?
To show proper image ID on container_registry page.
This only supports manifest V2, since the manifest V1 doesn't expose information about Image ID.
## What are the relevant issue numbers?
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/18159.
## Screenshots (if relevant)
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- Tests
- [x] Added for this feature/bug
- [ ] All builds are passing
- [ ] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !4821
Allow to fetch LFS from CI
## What does this MR do?
This adds support for fetching LFS object from CI jobs (mostly it's made for supporting GitLab CI).
## What is left?
- [x] Write tests covering a new authorization mechanism
cc @grzesiek @marin
See merge request !4465
Fix RangeError exceptions when referring to issues or merge requests outside of max database values
When using #XYZ in Markdown text, if XYZ exceeds the maximum value of a signed 32-bit integer, we get an exception when the Markdown render attempts to run `where(iids: XYZ)`. Introduce a method that will throw out out-of-bounds values.
Closes#18777
See merge request !4777
Track method call times/counts as a single metric
This changes method call tracking so only a single metric is emitted regardless of the number of calls. This allows us to more accurately measure the total execution time of a method as well as the number of times a method is called. See 851e3ff7578973c2206628424eac3b951a3c656d for more details.
Method call tracking tracked calls individually meaning the end statistics may not always be accurate enough to get a good understanding of where time is spent.
See merge request !4754
Allow creating Personal Access Tokens through the website
Related to #2979
- Allow a user to create personal access tokens, and use them to authenticate
- Refactor `API::Helpers` into `API::Helpers::Core` and `API::Helpers::Authentication`
# Tasks
- [ ] #2979 (!3749) - Personal Access Tokens
- [x] Basic Implementation
- [x] Add UI to add "Personal Access Tokens"
- [x] Reload `lib/api` on every request
- [x] Respect these tokens for API requests
- [x] Just a param or a header too?
- [x] Allow revoking tokens
- [x] Expire tokens
- [x] Left bar should have a "PAT" icon
- [x] Scopes?
- [x] Copy to Clipboard
- [x] Show active/inactive tokens separately
- [x] No need to check for expired/revoked in the appropriate places
- [x] Why does regular ApplicationController check for private token?
- [x] Support non-API requests
- [x] Revert (or work on) `lib/api` eager loading
- [x] Create MR
- [x] Refactoring
- [x] Fix tests
- [x] Write more tests
- [x] Add screenshots to MR
- [x] Add description of query performance to MR
- [x] Limit the number of queries in the `personal_access_tokens` page
- [x] Wait for CI to pass
- [x] Fix merge issues in schema.rb
- [x] Assign MR to endboss
- [x] Wait for feedback
- [x] Fix feedback
- [x] Wait for CI to pass
- [x] Assign to @rspeicher
- [x] Fix @rspeicher's comments
- [x] Wait for CI to pass
- [x] Assign back to @rspeicher
- [x] Write documentation and ping @axil
- [x] Wait for Axil to respond
- [x] Assign to endboss
- [x] Address Douwe's feedback
- [x] Use the `private_token` or `authentication_token` param instead of `personal_access_token`
- [x] Ditto for the header
- [x] Assign to endboss
- [x] Make sure CI is green
- [x] Address Douwe's feedback
- [x] Don't go through the `authenticate_user_from_private_token!` method, if a private token is supplied (or combine them)
- [x] In `authenticate_user_from_personal_access_token!` don't hit DB if `token_string` is `nil`
- [x] Use `current_user.personal_access_tokens.build` in the controller
- [x] Remove the "We aren't using `personal_access_token` as the root param" comment
- [x] `No need for = "...", we can just have the Inactive ... #{...} on the next line` in the view
- [x] Render dates in a (more) human format
- [x] CSS issue with table
- [x] Don't show the tokens in the UI indefinitely
- [x] How to implement scopes? Add-on to current impl? Doorkeeper?
- [x] Wait for @DouweM's comments about scopes
- [x] Address @DouweM's second review
- [x] Try not using `native['innerHTML']`
- [x] use contexts for all "when ..."
- [x] Ensure consistency (styling) with other pages for "You don't have any tokens" message
- [x] "Actions" table column doesn't need a label
- [x] %td can be moved outside of the if/else statement
- [x] The header title should be "Profile Settings"
- [x] Can this be a `before_create`, so we don't need to use `generate`?
- [x] If it couldn't be revoked, will we show an error?
- [x] If it couldn't be saved, will we show an error?
- [x] Merge master
- [x] Update CHANGELOG entry
- [x] Add tests for form errors?
- [x] Post screenshots
- [x] Tag @jschatz1 for review
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/0dff6fd/builds) to pass
- [x] Respond to @jschatz1's comments
- [x] Hardcoded colors should be variables
- [x] Should not be allowed to chose a date in the past
- [x] Use the same table as in the Applications tab
- [x] button should say "Create Personal Access Token"
- [x] Float the revoke to the right on the `a`
- [x] Change revocation message. "Are you sure you want to revoke this certificate? This action cannot be undone."
- [x] Date stays selected and looks selected even though date is set as "never".
- [x] ~~hover on the calendar button shifts~~ (not caused by this MR - happens on `milestones#new` as well)
- [x] Don't use the panel for the created token
- [x] Use a normal flash for "Your new personal access token has been created"
- [x] Show the input (with the token) below it full width.
- [x] Put the "Make sure you save it - you won't be able to access it again." message near the input
- [x] Have the input highlight all on single click
- [x] Update screenshots
- [x] Merge master in + conflicts
- [x] Assign to @jschatz1 again
- [x] Respond to @jschatz1's comments
- [x] No button for clipboard, only link
- [x] text-danger
- [x] highlight fade on that area where the token was created
- [x] Make sure [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/d754d99179f1ffe846fcc1d8e858163b39efc5dc/builds) is green
- [x] Assign to @jschatz1
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/faa0e3f7580bc38d4d12916b4589c64d6c2678a7/builds) to pass
- [x] Respond to @DouweM's feedback
- [x] move the redirect_to out of the if/else
- [x] certificate -> token
- [x] datepicker back to text field
- [x] combine the get_user_from_private_token and get_user_from_personal_access_token methods in ApplicationController
- [x] combine the get_user_from_private_token and get_user_from_personal_access_token methods in `lib/api/helpers`
- [x] don't need the new constants
- [x] Wait for [build](https://gitlab.com/gitlab-org/gitlab-ce/commit/9d7cda3ddce52baad9618466a5d00319b333be57/builds) to pass
- [ ] Wait for merge
# Screenshots
See merge request !3749
Allow customising of queries used for `update_column_in_batches`
This MR makes two changes to `add_column_with_default` and `update_column_in_batches`:
1. `add_column_with_default` no longer wraps the entire set of updates in a single transaction, preventing any locks from sticking around for the duration of the entire transaction
2. `update_column_in_batches` now takes a block which can be used to customise the queries. This uses Arel as messing with raw SQL strings is a total pain
In !4381 there's a need for updating existing rows/columns in a table in batches using a custom `WHERE` condition. Without the changes in this MR this would not be possible.
See merge request !4680
Added API endpoint for Sidekiq Metrics.
## What does this MR do?
It adds an API endpoint to gather metrics about Sidekiq, it's jobs, queues, and processes.
## Why was this MR needed?
There was no API endpoint for Sidekiq information.
## What are the relevant issue numbers?
Fixes#7171
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)
- [x] API support added
- [x] Tests
- [x] Added for this feature/bug
- [x] All builds are passing
See merge request !4653
Reduce SQL query counts in IssueReferenceFilter
## What does this MR do?
This MR adds a preparation phase for reference filters that allows them to prepare/create data structures used while iterating over HTML nodes. In this particular case the preparation phase is used for issue references to greatly cut down the amount of queries executed to get projects/issues for Markdown references.
## Are there points in the code the reviewer needs to double check?
No.
## Why was this MR needed?
Rendering Markdown containing issue references would run at most two queries for every issue reference: one to get the project and one to get the issue from said project. When rendering Markdown with lots of issue references this would result in _a lot_ of queries being executed.
## What are the relevant issue numbers?
#18042
See merge request !4410
Improve pipeline status in case that pipeline has no jobs
## What does this MR do?
This MR resolves problem with pipeline status when there are no build in pipeline.
This can happen when builds were skipped - for example - by using `only`/`except` keyword in `.gitlab-ci.yml`.
## What are the relevant issue numbers?
Closes#17977
See merge request !4403
Add support for Docker Registry manifest v1
## What does this MR do?
Adds support for Manifest V1 generated by older versions of Docker (before 1.10).
## What are the relevant issue numbers?
Fixes https://gitlab.com/gitlab-org/gitlab-ce/issues/18609
See merge request !4669
Eager load project relations in IssueParser
## What does this MR do?
This changes the ReferenceParser class to eager load various associations. This in turn results in the permissions checking code (e.g. the `Ability` model) to _not_ run dozens if not hundreds of extra SQL queries depending on the amount of references involved (in a single document).
## Are there points in the code the reviewer needs to double check?
No.
## Why was this MR needed?
In !4410 it was revealed a _lot_ of a queries came from the `Ability` model and the code it would call. In many cases this was because the code would simply get a project, then get the owners; or get a group, then get some association of that. Eager loading these associations is a fairly simple solution and greatly cuts down the number of queries.
## What are the relevant issue numbers?
None.
## Does this MR meet the acceptance criteria?
- [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added
- [x] ~~[Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)~~
- [x] ~~API support added~~
- [ ] Tests
- [x] ~~Added for this feature/bug~~
- [ ] All builds are passing
- [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides)
- [ ] Branch has no merge conflicts with `master` (if you do - rebase it please)
- [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits)
See merge request !4675
Add environments and deployments
This MR is a continuation of https://gitlab.com/gitlab-org/gitlab-ce/issues/17009.
The current implementation is as follow:
1. We have two new tables: `environments` and `deployments`.
2. We have a new tab: `Environments` under `Pipelines` where you can see all you environments and add a new one.
3. We add a new option to `.gitlab-ci.yml` to track where we should create a deployment for environment.
4. If environment in `.gitlab-ci.yml` is specified it will create a deployment. **If environment does not exist it will be created.** (this got changed)
5. The deployment is always successful and shows the time of the action, in that case a build that presumably should do deployment. In the future we could extend deployment with statuses: success, failure. We could extend deployments with information that this is partial or full deployment.
6. User have to create environments that he will track first.
7. User can remove environments.
8. User can retry/rollback past deployment (in that case we retry past build). The new build when succeeds it will create a new deployment.
9. Currently environment have only one parameter: `name`. In the future it should have: `variables`, `credentials` and possibly `runners` and maybe other resources.
10. Currently deployment have this parameters: `sha`, `ref`, `deployable (in this case a build)`, `user (who triggered a deployment)`, `created_at`.
The `.gitlab-ci.yml`:
```
deploy to production:
stage: deploy
script: dpl travis...
environment: production
```
What needs to be done:
- [x] Write initial implementation
- [x] Improve implementation (@ayufan)
- [x] Write tests (@ayufan)
- [x] Improve UX of the forms (cc @markpundsack) - reviewed by @markpundsack
- [x] Improve implementation of the views (cc @jschatz1) - done by @iamphill
- [x] Write .gitlab-ci.yml documentation for `environments` - done by @ayufan
- [ ] Write user documentation (@ayufan and @markpundsack)
See merge request !4605
By eager loading these associations we can greatly cut down the number
of SQL queries executed when processing documents with lots of
references, especially in cases where there are references belonging to
the same project.
Since these associations are so specific to the reference parsing
process and the permissions checking process that follows it I opted to
include them directly in IssueParser instead of using something like a
scope. Once we have a need for it we can move this code to a scope or
method.
