These changes were pulled from GitLab EE to support configuring
an alternative API URL than the default https://api.github.com.
In addition, the `verify_ssl` flag allows users to disable SSL cert
checking.
One modification: add a default `args` option if it does not exist
to avoid breaking existing configurations.
Allow Omniauth providers to be marked as external
Related to #4009
With this MR we will be able to allow the user to set which Omniauth Providers they would like to have as external. All users login in via these providers will be marked as external, even if they already had an account before. If the provider is removed form the list of external providers, the users will be marked as internal at their next login.
MR for Omnibus: gitlab-org/omnibus-gitlab!727
/cc @dblessing @DouweM
See merge request !3657
A few things to note:
- The IncomingEmail feature is now enabled even without a
correctly-formatted sub-address
- Message-ID for new thread mail are kept the same so that subsequent
notifications to this thread are grouped in the thread by the email
service that receives the notification
(i.e. In-Reply-To of the answer == Message-ID of the first thread message)
- To maximize our chance to be able to retrieve the reply key, we look
for it in the In-Reply-To header and the References header
- The pattern for the fallback reply message id is "reply-[key]@[gitlab_host]"
- Improve docs thanks to Axil
Storing of application metrics in InfluxDB
This adds support for tracking metrics in InfluxDB, which in turn can be visualized using Grafana. For more information see #2936.
See merge request !2042
Sidekiq-cron configuration moved to gitlab.yml
When `sidekiq-cron` was introduced, jobs configuration where placed
in `schedule.yml` file.
As discussed in #3928, this is not desirable.
We moved it to `gitlab.yml`, exposing only the "cron" part of the configuration.
See merge request !2087
This ensures we don't end up wasting resources by tracking method calls
that only take a few microseconds. By default the threshold is 10
milliseconds but this can be changed using the gitlab.yml configuration
file.
This adds the ability to write application metrics (e.g. SQL timings) to
InfluxDB. These metrics can in turn be visualized using Grafana, or
really anything else that can read from InfluxDB. These metrics can be
used to track application performance over time, between different Ruby
versions, different GitLab versions, etc.
== Transaction Metrics
Currently the following is tracked on a per transaction basis (a
transaction is a Rails request or a single Sidekiq job):
* Timings per query along with the raw (obfuscated) SQL and information
about what file the query originated from.
* Timings per view along with the path of the view and information about
what file triggered the rendering process.
* The duration of a request itself along with the controller/worker
class and method name.
* The duration of any instrumented method calls (more below).
== Sampled Metrics
Certain metrics can't be directly associated with a transaction. For
example, a process' total memory usage is unrelated to any running
transactions. While a transaction can result in the memory usage going
up there's no accurate way to determine what transaction is to blame,
this becomes especially problematic in multi-threaded environments.
To solve this problem there's a separate thread that takes samples at a
fixed interval. This thread (using the class Gitlab::Metrics::Sampler)
currently tracks the following:
* The process' total memory usage.
* The number of file descriptors opened by the process.
* The amount of Ruby objects (using ObjectSpace.count_objects).
* GC statistics such as timings, heap slots, etc.
The default/current interval is 15 seconds, any smaller interval might
put too much pressure on InfluxDB (especially when running dozens of
processes).
== Method Instrumentation
While currently not yet used methods can be instrumented to track how
long they take to run. Unlike the likes of New Relic this doesn't
require modifying the source code (e.g. including modules), it all
happens from the outside. For example, to track `User.by_login` we'd add
the following code somewhere in an initializer:
Gitlab::Metrics::Instrumentation.
instrument_method(User, :by_login)
to instead instrument an instance method:
Gitlab::Metrics::Instrumentation.
instrument_instance_method(User, :save)
Instrumentation for either all public model methods or a few crucial
ones will be added in the near future, I simply haven't gotten to doing
so just yet.
== Configuration
By default metrics are disabled. This means users don't have to bother
setting anything up if they don't want to. Metrics can be enabled by
editing one's gitlab.yml configuration file (see
config/gitlab.yml.example for example settings).
== Writing Data To InfluxDB
Because InfluxDB is still a fairly young product I expect the worse.
Data loss, unexpected reboots, the database not responding, you name it.
Because of this data is _not_ written to InfluxDB directly, instead it's
queued and processed by Sidekiq. This ensures that users won't notice
anything when InfluxDB is giving trouble.
The metrics worker can be started in a standalone manner as following:
bundle exec sidekiq -q metrics
The corresponding class is called MetricsWorker.
Alejandro Rodríguez
Kamil TrzcinskiandPhil Hughes
Achilleas Pipinellis
Artem Sidorenko
Jacob Vosmaer
Stan Hu
Marin Jankovski
Dmitriy Zaporozhets
DJ Mountney
Robert Speicher
Patricio Cano
Jacob Schatz
Rémy Coutable
Daniel CarraroandRémy Coutable
Douwe Maan
Douwe Maan
Drew Blessing
Gabriel Mazetto
Yorick Peterse
Sytse Sijbrandij
tduehr